Guide: Building an API

This guide is maintained at https://github.com/muoncore/guide Please submit issues at that repository.
  • Finish the Menu API

  • Build an Order API

  • Have it interact with the Menu API

  • Create reactive data flows between services

This last item is important. Understanding how data should move in a Reactive system is the most important lesson when building Microservices that will be able to tolerate network distribution and the various fun failure modes that comes with that.

Unresolved directive in index.adoc - include::guide/2-api/../12daypromo.adoc[]

Implement the Menu Microservice

Imagine a Pie Shop (a nice one!). You have tables, people looking at the menu, nice plants in the corner, the works

This is a advanced shop, and so at each of the tables is a tablet showing the current menu.

Given the boutique nature of the shop, the Menu is constantly updated through the day as the kitchen makes Pies and they are eaten, so reducing stock again.

The Menu needs to be able to show the various items available, their prices and be able to recieve push updates whenever these change.

To implement these, you need to update the service you created in part one, replacing its endpoints:

  • Get the current state as a request/ response, aka RPC. In clients, you will use this to get the initial state of the menu, and also any time you need to ad hoc query it.

  • Have current state always be up to date. This endpoint will push updates from the Menu to subscribed clients using the Reactive Stream protocol. This will allow you to distribute low latency updates to all clients.

We will implement updating the Menu state in the next episode of the series.

Introspection

Muon is built to enable you to use multiple protocols. Given that, you are going to need a way to understand what a service is able to do for you.

Since Muon is focused on communication and data models, it’s approach is generally "the answer is a protocol, what’s the question?"

The answer in Muon is the Introspection Protocol. This is a simple interaction that allows you to interrogate a remote system to see what it is capable of doing for you.

To call it via the CLI, execute muon introspect [servicename]

This is also available in the Muon SDK apis.

Try that now on your running menu microservice to see the protocol endpoints you have already made.

Internal Data

We need a menu data structure to work with, we’ll use Java classes for these in Java land.

Create a MenuItem class, like so:

package pieshop;

import java.util.*;

public class MenuItem {

  private String id = UUID.randomUUID().toString();
  private String name;
  private double price;

  public MenuItem(String name, double price) {
    this.name = name;
    this.price = price;
  }

  public String getName() {
    return name;
  }

  public double getPrice() {
    return price;
  }
  public String getId() {
    return id;
  }
}

Then, create a CurrentMenu class, like so:

package pieshop;

import java.util.List;

public class CurrentMenu {

  private List<MenuItem> items;

  public CurrentMenu(List<MenuItem> items) {
    this.items = items;
  }

  public List<MenuItem> getItems() {
    return items;
  }
}

These will form the local state of the Menu.

Lastly, rework your Menu class a little to make it a little more manageable.

public class Menu {

  private CurrentMenu menu = standardMenu();

  public Menu() {
    AutoConfiguration config =
      MuonConfigBuilder.withServiceIdentifier(
        "menu").build();

    Muon muon = MuonBuilder.withConfig(config).build();


    // define your Muon endpoints here...
  }

  private CurrentMenu standardMenu() {
    return new CurrentMenu(Arrays.asList(
      new MenuItem("Pork Pie", 2.50),
      new MenuItem("Blueberry Pie", 2.50),
      new MenuItem("Radish Pie", 2.50),
      new MenuItem("Apple Pie", 2.50)
    ));
  }

  public static void main(String[] args) {
    new Menu();
  }
}

This creates a representation of the Menu containing some pies. You will use this while developing the API as a runtime stub.

RPC Endpoint - /

Now, you need to update the RPC endpoint to be able to access this data.

    muon.handleRequest(all(), request -> {
      request.ok(menu);
    });

This simply passes the current menu state into the handler, where it will be serialised and sent through.

Run the service and then call it via the cli

>muon rpc rpc://menu

┌────────────┬───────────────────┬────────────────────────────────────────────────────────────┐
│ STATUS     │ CONTENT/TYPE      │ BODY                                                       │
├────────────┼───────────────────┼────────────────────────────────────────────────────────────┤
│ 200        │ application/json  │ {"items":[{"name":"Pork Pie","price":2.5},{"name":"Bluebe… │
└────────────┴───────────────────┴────────────────────────────────────────────────────────────┘

========= RESPONSE FULL BODY: ==================================================================

{ items:
   [ { name: 'Pork Pie', price: 2.5 },
     { name: 'Blueberry Pie', price: 2.5 },
     { name: 'Radish Pie', price: 2.5 },
     { name: 'Apple Pie', price: 2.5 } ] }

Stream Endpoint - /

Next, you need to update the stream endpoint to send the state of the menu every time it is updated.

There are multiple methods for doing this, depending on which FRP library you use to implement your service internals (Muon takes no position on what’s best in that field, only caring that they implement Reactive Streams).

Assuming you stick with RxJava2, one way to implement this would be to push the current state into an RxJava Subject.

The following code will create a new streaming endpoint that, whenever a new item of data is pushed into the subject, distribute the Menu to all subscribers.

Since we don’t yet have a mechanism to update the menu, you can simulate it for now, using a thread that periodically updates the state and pushes the new data out. We will fix this in the next episode

    PublishSubject<CurrentMenu> publishSubject = PublishSubject.create();  (1)

    muon.publishSource(
      "/live",                                                             (2)
      PublisherLookup.PublisherType.HOT,
      publishSubject.toFlowable(BackpressureStrategy.BUFFER));

    //Simulate menu updates for this episode of the Guide!
    new Thread(() -> {
      while(true) {
        try {
          Thread.sleep(1000);
          publishSubject.onNext(menu);
        } catch (InterruptedException e) {
          e.printStackTrace();
        }
      }
    }).start();
1 Create an RxJava PublishSubject. This can give us a Reactive Streams Publisher, and lets you arbitrarily push data in. There are probably better choices for most other circumstances, but for this, this is appropriate.
2 Obtain a Flowable from RxJava and expose this on a Muon stream endpoint

Re run the service, and now you can subscribe to this stream using the cli

> muon stream stream://menu/live

{ items:
   [ { name: 'Pork Pie', price: 2.5 },
     { name: 'Blueberry Pie', price: 2.5 },
     { name: 'Radish Pie', price: 2.5 },
     { name: 'Apple Pie', price: 2.5 } ] }
{ items:
   [ { name: 'Pork Pie', price: 2.5 },
     { name: 'Blueberry Pie', price: 2.5 },
     { name: 'Radish Pie', price: 2.5 },
     { name: 'Apple Pie', price: 2.5 } ] }

... every second ...

Now we have a system inside the Menu service that allows us to keep a consistent version of the menu data, and for remote clients to both retrieve that, and also have the Menu service keep that up to date via push updates.

Next, we will build the Order service that will use this!

Create the Order Microservice

Next up, we want to be able to take orders. To build this, we will use Node.js (go Polyglot!)

The API for an Order is an RPC endpoint. Take a request, process, send response back when done.

The complexity comes from the interaction between the Order and Menu services. How should we have the communicate?

The easy way is to have Order make an RPC call to the Menu, getting the state each time. This would add latency and some fragility to the service though. What happens if the Menu service goes down?

Here you have CAP theorum choice to make. Do you want to build a consistent, yet fragile system, and so always check the Menu at source, or do you want to build an available system that could make the occasional inconsistent data decision?

In our case, we will choose AP (because I always choose AP where possible, otherwise why are you building Microservices?)

The impact on our system is that the Order service will be able to place orders while the Menu service is down. The cost of this is that the Order service may place orders against an out of date Menu, either stock or price. This would only be the case if the Menu service is completely unavailable.

To do this, you will subscribe to the streaming endpoint on the Menu service and effectively keep a representation of the Menu data within the Order service. This gives us some nice advantages. Scale will be much easier, all the data is now local, and in memory. We can rework the data structure to make it more amenable for Order processing. If the stream connection is lost, we still have that data in memory until the Menu service comes back up, at which point the service reconnects and gets back in sync

Create the Microservice

First, install NPM and node as at http://npmjs.org

Once you have a working installation, create a new directory called 'orders' and then create a new npm project in there using npm init

Once you have done that, add a dependency for muon

npm install muon-core --save

Create a new file as the entry point, index.js

var Muon = require("muon-core")
var muonurl = process.env.MUON_URL || "amqp://muon:microservices@rabbitmq"

logger.info("Muon is enabled, booting up using url " + muonurl)
var muon = Muon.create("orders", muonurl); (1)
1 Create a Muon instance using the given URL for the transport/ discovery connection. In this case, defaulting to AMQP.

You can run this up

> node index.js

It will appear in the Muon discovery information, however it has no functionality, so you can’t do anything with it yet.

Get the Menu Data

Before we give the Order its own API, we will get some data from the Menu and store it in the Order service. This shows Muon in use as a client.

Start your Menu service in a different process and leave it running.

What we want to do now is to get the Menu data, and pull it into the Order service.

As we discussed above, this will allow the order service to keep on processing even when the Menu service goes down, an AP design decision.

var currentMenu = {}

function connectMenu() {
  console.log("Attempting to connect to Menu live stream")
  muon.subscribe("stream://menu/live", {}, (menu) => {                         
    currentMenu = menu(1)
    console.log("Menu Updated")
  }, (error) => {
    console.log("Disconnected from Menu service, reconnecting ", error)
    setTimeout(connectMenu, 10000)                                             
  }, (complete) => {(2)
    console.log("Stream was closed by the remote service, reconnecting ... ")
    setTimeout(connectMenu, 10000)
  })
}
connectMenu()
1 Subscribe to the menu live data. Remember that this is pushing new updates every second
2 On an error, attempt to reconnect 10s later.
3 Trigger the connection/ reconnect

Now run the server again. You should see the log messages every second. Try shutting down the Menu service, you’ll see error messages and attempts to reconnect.

Start the Menu service up again and you’ll see the subscription be re-established.

You now have menu data, held locally within the Order service, and eventually consistent with the Menu service itself.

Create the Order API

Now we have the Menu data, we can look to process orders!

This naturally fits an RPC style interaction. We will provide a single endpoint /order that lets you place an order.

This will be done by submitting the following data

{
  "client": "David"                               (1)
  "items": [                                      (2)
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc7", "value":2},
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc8", "value":2},
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc9", "value":2},
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc1", "value":2}
  ]
}
1 The client who placed this order.
2 The items, in a list matching item id to the number required

The response will show the full order

{
  "client": "David"
  "items": [
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc7", "value":2},
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc8", "value":2},
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc9", "value":2},
    {"id":"b4690eb0-8b7f-4707-896b-e7772910fcc1", "value":2}
  ],
  "total": 15.99                               (1)
}
1 The total order value

Implementing this is fairly straight forward. Create a new RPC endpoint in index.js

muon.handle("/order", (data, response) => {                          
(1)
  var orderRequest = data.body                                       
(2)
  var orderResponse = JSON.parse(JSON.stringify(orderRequest))       
(3)
  orderResponse.total = orderResponse.items.reduce((accum, current) => {  
    var item = currentMenu.items.find((item) => item.id == current.id)(4)
    return accum + item.price * current.value;
  }, 0);

  response(orderResponse)
})
1 Create an RPC endpoint for /order
2 Obtain the RPC payload
3 Deep clone the order.
4 Calculate the total field and add it to the response.

At this point, you aren’t storing the Orders coming in, but you can see how we can perform transactional processing and remain resilient in the face of a network failure.

To run this, you can use the CLI. FIrst obtain the IDs from the Menu as above, then call Order

> muon rpc rpc://orders/order '{"items": [{"id": "33c919e2-9291-44c6-8195-c9bbfb4517b5", "value": 5}] }'

┌────────────┬──────────────────┬────────────────────────────────────────────────────────────┐
│ STATUS     │ CONTENT/TYPE     │ BODY                                                       │
├────────────┼──────────────────┼────────────────────────────────────────────────────────────┤
│ 200        │ application/json │ {"items":[{"id":"33c919e2-9291-44c6-8195-c9bbfb4517b5","v… │
└────────────┴──────────────────┴────────────────────────────────────────────────────────────┘

========= RESPONSE FULL BODY: ==========================================

{ items: [ { id: '33c919e2-9291-44c6-8195-c9bbfb4517b5', value: 5 } ],
  total: 12.5 }

Summary

You now have two Microservices that interact in a Reactive way, communicating via streaming data and maintaining local data where it is needed, rather than abusing RPC style interactions to do that.

You have seen eventual consistency, the benefits of streaming data and how to build an AP system.

In the next episode of the series, you will add data persistence and distribution to your services in the form of event streams using a dedicated Microservice, Photon.