Unresolved directive in index.adoc - include::submodules/photon/docs/version.adoc[leveloffset=+1]

This document is maintained at https://github.com/muoncore/photon. Please submit issues at that repository.

Photon

Muon is designed to enable the construction and expression of distributed programming models, especially those following Reactive principles.

Event Systems are a very important and flexible programming model, especially when paired with the Event Sourcing approach designed by Greg Young, which you can read about here

Photon provides stream persistence and data analysis functionality that, along with the Event Client protocol in the Muon libraries, enables you to implement Events, Event Sourcing and CQRS within a Muon system

Quickstart

Download the Muon Starter repository and run through the instructions and pre-requisites from there. The latest released version of photon will be started using Docker Compose, along with a contained RabbitMQ instance for communication and other support microservices.

Install the Muon CLI and set up with the URL amqp://muon:microservices@localhost

You can then see photon running

> muon d

┌──────────────────────────────┬──────────────────────────────┬──────────────────────────────┐
│ SERVICE NAME                 │ TAGS                         │ CONTENT/TYPE                 │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ molecule                     │                              │ application/json             │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ photon                       │ photon,eventstore            │ application/json             │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘

You can emit an event using the CLI

> muon emit '{ "event-type": "SomethingHappened", "schema": "1","stream-name": "something","payload": {"message": "Hi there!"}}'

This will persist the event in the given stream. It can be replayed at any point later on

You can replay the persisted events using the CLI

> muon replay something

All the events stored will then be replayed.

Muon Events

Muon Events is a set of protocols and services (most notably, Photon), that enable the easy use of Events, Event Sourcing and Event Projections/ Aggregations in creating distributed applications

This section describes how you can develop using these concepts.

See the Muon Guide for more information on using these patterns

Streams

The central concept in Muon Events, and so Photon, is the stream. This is a set of Events that have occurred, persisted in the order they happened.

Streams
Figure 1. Streams in Photon
Events

Events are things that have happened. They are the facts of your system, that you can use to derive the current state

Muon Events generally expects that events are durable, which is what Photon will do for you.

Every Event should contain the following information

  • What Happened - The event type.

  • When it Happened - the event time.

  • A description of the change, the event payload

Muon Events also gives the option to include

  • Why it happened - the causing event id

  • Who caused it - the service that emitted the event

Photon persists all of this information, and allows you to replay and as you see later, project that data into representations you can use in your applications.

Replay

Streams can be replayed using the Reactive Streams protocol.

> muon replay mystream

This connects to photon using the Reactive Streams Protocol, and creates a subscription to the logical stream you have requested.

The default replay mode is 'hot-cold', where all of the historical events will be replayed, then all new data will continue to be played, including everything in the future. This subscription will never complete by itself, it remains open until the client closes it.

Stream Replay
Figure 2. Stream Replay

Partial Replay

TODO, describe how to play from an order-id or timestamp

Internal Streams

Photon persists everything as streams, including its own data structures.

The streams internal to Photon have a __ prefix/ suffix.

You can replay data from these internal streams, and they are useful for creating backups of Photon state.

In a future version of Photon, these streams will form the basis for a feature called Federation, complex, multi node installations.

__config__

Photon internal configuration is stored in the __config__ stream. Internally, this stream is played into a data structure that represents the current Photon configuration.

The Photon API layer acts as a front on this stream.

__all__

The second special stream is __all__. This contains all events, on all streams.

You can play this to obtain all events ever recieved by photon, in the order they were added in, no matter which stream, using the global order_id.

Projections

A very common pattern when dealing with stream based systems is aggregating the stream into a state representation. This goes by many names, but is fundamentally the reduce from the the well known map/ reduce pattern.

In this pattern, using a stateless function, you incrementally build a data structure by running the elements of the stream through the function. The function receives the current state, the event to process, and returns the new state.

This is such a common pattern that Photon has support this feature built in, known as stream Projections.

Creating

Querying

Deleting

Operations

Photon is designed to be lightweight when it comes to operational maintenance.

It is, however, a stateful component, and so that state needs to be looked after.

Installation

Photon is distributed as an uberjar and as a Docker image.

Startup options

Usage: java -jar photon-x.x.x-standalone.jar [-h] [-option value] ... [-option value]
Options:
-microservice.name    : Service ID, especially important for Muon (default = photon)
-rest.host            : The IP or hostname of the web server for frontend and API. Change it for external access (default = localhost)
-rest.port            : The port for the UI frontend and the REST API
-rest.keystore        : If set, the web server will be started in SSL mode using the certificates identified by this path
-rest.keypass         : The password required to open the keystore set in rest.keystore. Not required in not-SSL mode
-admin.user           : The default username for logging in and requesting API tokens (default = admin)
-admin.pass           : The default password for logging in and requesting API tokens (default = p4010n)
-admin.secret         : A secret string that will be used to encode authentication tokens (default is random on launch)
-projections.port     : Port to stream projection updates to (default = 8375)
-events.port          : Port to stream incoming events to (default = 8376)
-muon.url             : AMQP endpoint for Muon-based transport and discovery (default = amqp://localhost)
-parallel.projections : Number of cores assigned for parallel stream processing (default = number of cores on your machine)
-projections.path     : Local folder with projections, in EDN format, to pre-load on start (default = /tmp/photon)
-db.backend           : DB backend plugin to use (default=h2). Depending on the build of photon, this can be one of:
                        h2, cassandra, redis, file, mongo, riak, dummy.
-h2.path              : If using H2, the file prefix for the database file, including path (default = /tmp/photon.h2)
-cassandra.ip         : If using Cassandra, the host of the cluster
-file.path            : If using files as backend, the absolute path to the file
-mongodb.host         : If using MongoDB, the host of the cluster
-riak.default_bucket  : If using Riak, the name of the bucket
-riak.node.X          : If using Riak, the nodes that form the cluster (riak.node.1, riak.node.2, etc.)

Setting up a file for static configuration

Photon can be configured either directly from the command line or from a file, and parameters can be combined from different sources. The order of priority in which the configuration is build is the following:

  1. Command-line arguments

  2. photon.properties in the working directory

  3. resources/photon.properties

  4. resources/config.properties

Example of property file:

# Microservice identifier (default = photon):
microservice.name=photon
# AMQP endpoint (default = amqp://localhost):
muon.url=amqp://username:password@localhost
# Number of cores assigned for parallel stream processing
# (default = number of cores on your machine):
parallel.projections=8
# Local folder with projections, in EDN format, to pre-load on start
# (default = /tmp/photon):
file.path=/path/to/edn-files/
# DB backend plugin to use, several options currently available:
db.backend={file,mongodb,riak,cassandra}
# Depending on the backend, you'll need to set up the DB plugin:
cassandra.ip=127.0.0.1
file.path=/path/to/file.json
mongodb.host=localhost
riak.default_bucket=photon-eventstore-v1
riak.node.1=riak1.yourdomain.com
riak.node.2=riak2.yourdomain.com
riak.node.3=riak3.yourdomain.com

Running in a Docker based system

This includes Docker itself and related projects (compose, machine etc), and also in systems that can use Docker containers, such as Kubernetes, Mesos and the like.

Photon is published to an Artifactory backed Docker registry.

The coordinates for the image is simplicityitself-muon-image.jfrog.io/photon:latest

You can set the configuration options by altering the command line to be used.

You set the variable MUON_URL to the correct location for your system to enable Photon to connect to the Muon network.

This Docker Compose snippet shows how this can be done.

    photon:
    image: simplicityitself-muon-image.jfrog.io/photon
    environment:
      - MUON_URL=amqp://muon:microservices@rabbitmq
    links:
      - rabbitmq
    ports:
      - "3000:3000"
    expose:
      - "3000"
    command: -jar target/photon.jar -rest.host 0.0.0.0

This updates photon to listen for HTTP traffic on all network interfaces.

Backup

Photon is internally fully event sourced. All changes in internal state are mediated via one of the internal streams (denoted with __XX__)

You can perform a backup in one of two ways.

Copy backing data store

The backing data store (H2, Cassandra etc) contains all information necessary to back up the system. The data store, such as the h2 db file, can be copied and archived to create a usable backup.

The contents of the backing data store are kept compatible over minor version updates only, and may not be compatible over major versions.

Stream based

The stream __all__ contains all of the configuration updates made over the lifetime of the system, along with all of the data that it has persisted.

You can instruct photon to replay this stream in its entirety and persist this to disk. Once it has done so, you can use that file to recreate a runtime clone of the original Photon.

This method is fully portable, and will be compatible over major version upgrades of Photon.

> muon replay __all__ '{"stream-type":"cold"}' >> /tmp/backup

This creates a log file that contains all events in the Photon.

To restore from this file

> cat /tmp/backup | muon event

Monitoring

Backing data stores

Photon persists streams into a variety of different data stores, provided by internal persistence plugins.

The default docker image provided uses the embedded H2 plugin, which persists to the container local disk.

Embedded H2

Cassandra

MongoDB