Using JSON and WebSockets

How did we get here?

RESTful APIs, the big lie challenged the widely-held belief that RESTful APIs are a good idea for modern web applications. The inspiration to write the post came from two sources. First, we were writing an new API for an SPA, so the topic was certainly at the top of my mind. Second, we are recruiting development team members, and the candidates placed a surprising amount of importance on RESTful APIs.

And now the big sequel

The goal of the big lie therefore was to provide an objective, easy to read, and well-ordered list of issues with RESTful APIs. In other words, the goal was to clearly define the problem before suggesting a solution.

This “big sequel” continues where the big lie ended and looks at some best API practices when freed from the constraints of REST. However, before we do let’s clear up a big misconception many readers had about the original post.

REST is good

REST is going to be around for a long time. It works very well for content delivery and we have an excellent and expensive infrastructure built around it. I’ve never disputed this. The original post, for example, contains this sentence:

REST is a great mechanism for many things such as content delivery, and it has served us well for two decades.

But I had made a mistake in the original post: I had titled it REST, the big lie instead of RESTful APIs, the big lie. I recognized the original title was too broad, but by the time I had fixed it a few hours later the damage had been done: search engines, links, and the social networks kept the original name. And, as they say, you can’t uncook an egg. This title misled many people to think the article was condeming all of REST. So let me be clear: REST is essential and good. And RESTful APIs are much less so.

RESTful APIs, not so much

The big lie about RESTful APIs is the commonly held belief that they are almost always the best choice for web application APIs. In truth, they are particularly ill-suited for modern web applications like SPAs. Most of issues are detailed in the the big lie, and we are not going to rehash them here.

Now, let’s get to the good stuff.

JSON-Pure best practices

RESTful APIs, the big lie concluded with a list of best-practices that evolved over 8 years of developing non-RESTful APIs for commercial Single Page web Applications. JSON-Pure APIs - a name I made up simply for brevity and convenience - follow these conventions. There isn’t an official standard or an Internet Engineering Task Force project for JSON-Pure APIs. At least not yet.

Below are my recommendations JSON-Pure APIs. The primary goal is to provide the best possible user experience for modern SPAs. Certainly the practices shown below are nothing new or revolutionary. You will see echoes of SOAP, JSON RPC, JSON API, JSend, JSON LD and Hydra in the recommendations - and probably a few other dozen standards I somehow missed.

• Use WebSockets and JSON
• Use an existing vocabulary
• Use four message types
• Use asynchronous transactions
• Use a reliable request
• Separate the request from the transport
• Use a reliable response
• Make it easy to debug
• Make it portable

This list is just slightly expanded from the summary of the big lie. Let’s look at each in detail, starting from the top.

Use Websockets and JSON

We recommend using the WebSockets transport with the JSON data exchange format.

WebSockets

The WebSockets transport begins as an HTTP/S request which is then upgraded by the client and server to a WebSocket connection. This means it generally passes through firewalls as well as HTTP/S, yet it provides truly asynchronous, full-duplex, persistent messaging channel between client and server with just a few bytes of overhead per message. Compare this to HTTP/S, where each message can require multiple kilobytes of overhead thanks to verbose headers and cookies. As a result, WebSockets not only provides great new capabilities, it can also be 4-10x faster.

Prior to WebSockets, real-time messaging between a browser and a server was expensive and error-prone. I spent some quality time with XMPP, BOSH, and Strophe. Trust me, it’s extremely painful in comparison.

Are WebSockets always required? Of course not. But the transport is easy to implement on modern browsers and its capabilities are extremely compelling for SPAs. APIs transported over WebSockets are decidedly not RESTful as there is only a single send() method for transport, and there are no visible response codes in web server logs. And that’s just fine.

JSON

JSON is a compelling data exchange format for web applications because JavaScript provide very fast, safe methods to read and write the data. That means that at least on the browser client, we will be using JSON.parse() and JSON.stringify() to parse and send the data respectively. These very same methods are available on the server if we use Node.js; if not, there are robust JSON libraries for all other major web server platforms.

JSON is also the native format of many minimal-structure databases such as MongoDB and CouchDB. Even when it’s not the native format, Node.js interfaces to other databases such as PostgreSQL return query responses in JSON. This means marshalling data from a database to a client can have substantially lower overhead compared ORMs such as Active Record (Ruby) or Hibernate (Java).

JSON does have its weaknesses compared other formats. Most notably a JSON document does not typically include a Document Type Declaration (DTD) which is useful to ensure the message contents are valid. We have found that that JSON-Schema addresses much of this deficit, and is discussed further below.

Now let’s look at using a clean, familiar vocabulary.

Use an existing vocabulary

The best APIs use a vocabulary that is widely understood with little nuance. This greatly reduces the cognitive overhead required to read or write messages.

Action verbs

The CRUD vocabulary is very widely understood. We use a superset for our actions verbs. Here they are organized by message type:

Server response verbs are directly correlated with requests and are in UPPERCASE to differentiat While at first this may appear almost trite, it actually is quite important and useful, as we shall see when we discuss indirect responses below.

We added the flush verb so the client can tell the server when it has removed data from its storage. Thus when the server distributes notification of changed records, it doesn’t have to send messages to clients that have flushed the containing record or data set.

Application status messages

Place all application status messages in a in a log_table (Update 2018-12-29: this used to be called log_list but I have since adopted the convention of using table to indicate a list of objects or list of lists). The JSON below shows all available log levels. I have adjusted this to match the syslog standard.

{ log_table: [
    { level_int : 0, level_str : "emerg", /* detail */ },
    { level_int : 1, level_str : "alert", ... },
    { level_int : 2, level_str : "crit",  ... },
    { level_int : 3, level_str : "error", ... },
    { level_int : 4, level_str : "warn",  ... },
    { level_int : 5, level_str : "notice",... },
    { level_int : 6, level_str : "info",  ... },
    { level_int : 7, level_str : "debug", ... },
  ],
  /* ... */
}

/* detail: log_id, code_key, code_str, user_msg */

The level_int value is useful to implement log filtering, either for the recipient, or during a request.

Here is an example of a full error message:

{ log_table :[
    { code_key  : "400",
      code_str  : "Bad request",
      level_int : 3,
      level_str : "error",
      log_id    : "1096",
      user_msg  : "action_str required"
    }
  ],
  /* ... */
}

Notice how we mapped a REST code to the error. We do that for convenience and familiarity. However, we generally use the REST error codes as a subset of our full vocabulary, since we can extend it as the application requires.

Use four message types

SPA APIs generally require four message types:

1. Requests
2. Direct responses
3. Indirect responses
4. Confirmations

Requests

Requests are messages sent from clients requesting data manipulation or delivery. An example request to change a specific record would have the update action verb.

Direct responses

Direct responses are messages from the server to attached clients directly in response to a request. A direct response to the update request in the example would use the UPDATED action verb.

Indirect responses

Indirect responses are sent from the server to clients without a direct corresponding request. This happens when the server needs to notify a client of a data change, such as an updated, deleted, or new record. An indirect response for the example above would have an UPDATED action verb.

Confirmations

Confirmations are sent from a client to a server to confirm a response has been received and processed. The client may indicate success with done, cancel the transaction with abort, or re-request the message with retry. Generally it is a good idea to avoid the complexities of supporting the idea of “partial success”.

Use a standard message structure

API messages should have a standard, minimal structure that is consistent and easy to read.

Message formats

Here is a template of suggested JSON message formats for the request, response, and confirm phases of a transaction:

// request
{ action_str    : "retrieve",  /* action verb */
  data_type     : "person",    /* record data */
  log_table      : [ /* application messages   */ ],
  request_map   : { /* request parameters     */ },
  trans_map     : { /* transaction meta-data  */ }
}

// direct and indirect responses
{ action_str   : "RETRIEVED", /* action verb */
  data_type    : "person",    /* record data */
  log_table    : [ /* application messages   */ ],
  response_map : { /* returned data          */ },
  trans_map    : { /* transaction meta-data  */ }
}

// confirm
{ action_str  : "done",      /* action verb */
  data_type   : "person",    /* record data */
  log_table   : [ /* application messages   */ ],
  confirm_map : { /* confirmation details   */ },
  trans_map   : { /* transaction meta-data  */ }
}

We have added a suffix to JSON keys to indicate type. This greatly helps the API consumer comprehend the meaning of fields at little cost. Suggested suffixes are detailed in Appendix A of Single Page Web Applications and a cheat sheet is freely available on GitHub.

Field meanings

• The action verb, action_str, is required. It is discussed above.
• The data_type indicator is application dependent.
• The application messages list, log_table, is optional. It is discussed above.
• The request_map, response_map, and confirm_map are required for each message type.
• The trans_map is required and includes transaction specific parameters. It is discussed below.

Message aggregation

Multiple messages may be aggregated by placing them in a list, for example:

[ { /* message 1 */ },
  { /* message 2 */ },
  ....
]

Messages are processed in the order received. An API can get rather chatty at times, so it can be a good idea to employ a strategy to aggregate messages on both the client and the server. One approach is to employ a message queue that is sent with a limited frequency.

Schema validation

We generally use a JSON schema to validate messages. Validation isn’t hard to implement, and it helps avoid silly and gnarly errors that can occur with invalid messages.

Use asynchronous transactions

We suggest using request-response-confirm for API transactions instead of the REST request-response convention. First, let’s consider when request-response is more appropriate.

Use request-response when appropriate

We want the classic request-response approach when the web browser is used to display published content like this blog page. It is simple, clean, and well-understood. Any benefit of using a more complex transaction is almost certainly outweighed by the costs.

Use request-response-confirm for APIs

SPAs, however, are better served by request-response-confirm transactions. For example, if a client requests a resource, the server issues a response. Once the client has received and processed the response, it will send a confirm to the server.

The benefits of the request-response-confirm transaction include:

• The confirmation goes beyond the transport status – was the message received? – and reports to the server the application status of the message – was the message properly processed and consumed? This is very important for an SPA so the server may keep track of client data sets.

• The transaction is asynchronous. The client, for example, may take 20 seconds to consume a large data set; or perhaps the client may wait for user approval before processing or denying a provided data set. During this time, the server can continue to process other request while while awaiting the confirmation. Implementing such a delay over a typical RESTful request-response cycle using something like BOSH can be difficult and expensive.

The request-response-confirm cycle is therefore often the best option if we need to sync records between a rich client and server.

Use a reliable request

Don’t embed any application meaning into the transport request. Instead, use the single, most reliable request method available.

On the transport layer, we are simply concerned about whether an API message is successfully transferred or not. Think of the transport layer as the “delivery driver” of web traffic: He will make sure our new sweater gets delivered (transport issue), but he’s not going to sit around for hours while we decide if we like it or not (application issue) and want to send it back.

WebSockets provides only a single send() method for requests. If we use HTTP for our API, it is wise to use only POST and ignore the other verbs.

Separate the request from the transport

A typical RESTful API call splits the application-level meaning of a message between the message body and the headers. We wish to instead leave headers alone and place all application-level details in the message. We do this through the trans_map and using a simple end point.

The transaction map

The transaction map, or trans_map was introduced earlier and contains transaction specific fields. Here is an example:

trans_map  : {
  auth_key : "c76aa3577f8b5a60206f9d041c76034a3c1f",
  trans_id : "eb99ec08-7e90-400d-9585-62a1385ec158"
}

The trans_id should be generated by the originator and should be immutable throughout the transaction. The auth_key will vary by implementation and client. A transaction may require additional details not shown here such as a server time stamp, but this is a good start.

The end point

Use an API endpoint that fully describes the API source and version. If using HTTP, for example, and endpoint might look like the following:

https://api.mycompany.com/3-1-a/

Better yet, put the API version (api_version) in the trans_map as that make the API transport independent.

Send all directives through the action_str property. Certainly, this precludes any sort of caching that RESTful API fans will undoubtedly miss. But the purpose of an an API is to communicate dynamic messages, not static content. Transfer of bulk data and caching can be done at many levels, and we can leverage the good parts of HTTP and REST when we need to. For example, instead of delivering image data over our API, we can instead provide a URL to a cache-able PNG image.

Use a reliable response

Don’t embed any application meaning into the transport response. Instead, use the single, most reliable response method available. As with the request, we are simply concerned about whether an API response is successfully transferred or not.

A WebSockets transport doesn’t allow us to impart meaning to a response. Either the message is received or it fails. If we use HTTP, then we should we use a 200 OK response whenever the transport succeeds, and leave any application-level communication to the message content.

Make it easy to debug

Debugging JSON-Pure APIs is quite straight-forward. First, we can check if the message transport has succeeded. Then we can proceed to inspect the JSON for any and all application-level meaning. A command-line JSON-schema utility is useful for checking message validity during this process.

Make it portable

An application API should be easily portable over any transport method such as HTTP/S, WebSockets, XMPP, or SMTP. A JSON-Pure API should be transport neutral, and should require only minor adjustment to work over nearly any transport.

Closing thoughts

The move away from RESTful APIs has already begun. Both FaceBook’s Relay and Netflix’s Falcor dispense with any pretense of being RESTful (Update 2018-12-29: And the trend continues with GraphQL).

I expect that REST will continue to be used for the majority of web traffic, especially content distribution. However, the more sophisticated APIs used by SPAs will move to more appropriate, non-RESTful design patterns. And that’s a good thing.

I hope you found this useful, and as always, any constructive feedback is welcome!

Cheers, Mike