Setting up Phoenix Channels to use MessagePack for Serialization

In this article, I’ll show you how to setup your Phoenix 1.3 app to use the binary serialization format MessagePack for sending and receiving web socket channel messages.

What is MessagePack?

MessagePack is a binary serialization format. You know how JSON uses strings to represent different kinds of data and objects? Well MessagePack does the same thing, but with binary, so the encoded things are generally smaller than their JSON equivalents.

Same guy, just in a smaller package.

What is Serialization?

Serialization is a nice word for encoding and decoding messages. In this article, I’ll be talking about two serializers – one for the server and one for the client.

The server serializer encodes Elixir data into MessagePack binary and decodes the binary to Elixir. The client serializer does the same thing but with JavaScript instead of Elixir.

Why use MessagePack?

The default for passing data around is pretty much always JSON. It’s fast, and relatively compact. But what about a case where you’re passing potentially hundreds of messages per second back and forth between client(s) and server. Any reduction in message size can add up really quickly.

Such is the case with many .io games, and Breakfast of Champions is no different. Using Msgpax, a MessagePack serializer written in Elixir, I was able to reduce the size of each message passed by an order of magnitude.

 


Setting Up Phoenix to Use MessagePack

Add Msgpax to Deps

First we need to make the Msgpax lib available to our app. Follow the installation instructions on the repo page: https://github.com/lexmag/msgpax.

Tell Phoenix to Use a Custom Serializer

NOTE: If you’re reading this in the future, this part will have changed when Phoenix 1.4 came out!

Next we need to edit the channels/user_socket.ex file. A Socket is an object (or module in this case) that deals with actions relating to the connection between a web socket server and client. Both the Phoenix server and JavaScript client have Sockets.

Find the line in the UserSocket module that says:

transport :websocket, Phoenix.Transports.WebSocket

Here transport is a macro that sets up the UserSocket’s transport property. If Sockets are objects that deal with web socket connections, a Socket’s transport property tells the socket how it’s communicating – in other words, what method is the socket using to transport messages. The two built in options for Phoenix are websocket and longpoll. The two parameters in the line above are telling Phoenix we’re using websockets to pass channel messages, and that transport behavior is defined in thePhoenix.Transports.WebSocket module.

So how do we tell the transport module that we want to use a different serializer to encode and decode our channel messages? If you take look at the implementation of the transport macro, you can see that it also accepts an optional config property. Upon further inspection, we see that config is a list that can have the key-value pair serializer: [//list of serializers and transport versions]. Bingo! Transform the above line into:

transport :websocket, Phoenix.Transports.WebSocket, serializer: [{YourAppName.MsgpaxSerializer, "~> 2.0.0"}]

Create a Custom Serializer

Ok, so we’ve told Phoenix we want to use a custom serializer, called MsgpaxSerializer, to encode and decode our messages, but that module doesn’t exist yet, so we better create it.

For inspiration, take a look at Phoenix’s own implementation of the websocket serializer. One of the first lines is: @behaviour Phoenix.Transports.Serializer. The @behaviour directive in Elixir is like an interface in Java or a protocol in Swift/Objective C. It defines a set of functions that have to be implemented by a module, and it raises errors if those functions are missing. We want our custom serializer to conform to this behavior (or behaviour) too, so let’s check it out, pasted here in it’s entirety:

phoenix/lib/phoenix/transports/serializer.ex

defmodule Phoenix.Transports.Serializer do
  @moduledoc """
  Defines a behaviour for `Phoenix.Socket.Message` serialization.
  """

  @doc "Translates a `Phoenix.Socket.Broadcast` struct to fastlane format"
  @callback fastlane!(Phoenix.Socket.Broadcast.t) :: term

  @doc "Encodes `Phoenix.Socket.Message` struct to transport representation"
  @callback encode!(Phoenix.Socket.Message.t | Phoenix.Socket.Reply.t) :: term

  @doc "Decodes iodata into `Phoenix.Socket.Message` struct"
  @callback decode!(iodata, options :: Keyword.t) :: Phoenix.Socket.Message.t
end

As you can see, we need three methods in our serializer to conform to the Phoenix.Transports.Serializer behavior: fastlane!, encode!, and decode!. Now that we know what we need, and have an example, let’s make the thing.

Make the module file at your_app/lib/your_app/msgpax_serializer.ex and add the following contents:

defmodule YourApp.MsgpaxSerializer do
  @moduledoc false
  @behaviour Phoenix.Transports.Serializer

  alias Phoenix.Socket.Reply
  alias Phoenix.Socket.Message
  alias Phoenix.Socket.Broadcast

  @doc """
  Translates a `Phoenix.Socket.Broadcast` into a `Phoenix.Socket.Message`.
  """
  def fastlane!(%Broadcast{} = msg) do
    msg = %Message{topic: msg.topic, event: msg.event, payload: msg.payload}

    {:socket_push, :binary, encode_v1_fields_only(msg)}
  end

  @doc """
  Encodes a `Phoenix.Socket.Message` struct to MessagePack binary.
  """
  def encode!(%Reply{} = reply) do
    msg = %Message{
      topic: reply.topic,
      event: "phx_reply",
      ref: reply.ref,
      payload: %{status: reply.status, response: reply.payload}
    }

    {:socket_push, :binary, encode_v1_fields_only(msg)}
  end
  def encode!(%Message{} = msg) do
    {:socket_push, :binary, encode_v1_fields_only(msg)}
  end

  @doc """
  Decodes MessagePack binary into `Phoenix.Socket.Message` struct.
  """
  def decode!(message, _opts) do
    message
    |> Msgpax.unpack!()
    |> Phoenix.Socket.Message.from_map!()
  end

  defp encode_v1_fields_only(%Message{} = msg) do
    msg
    |> Map.take([:topic, :event, :payload, :ref])
    |> Msgpax.pack!()
  end
end

It’s nearly identical the Phoenix’s websocket_serializer.ex with a few key differences. First, in the tuples we returned from fastlane! and encode!, we replaced :text with :binary, and second, we replaced all Poison encoding/decoding with Msgpax packing and unpacking.

That’s it. Phoenix is now sending all its channel messages in the MessagePack binary format and expecting that format in return. However, our js client is still sending and expecting JSON messages. We’ll change that in the next post!

2 Replies to “Setting up Phoenix Channels to use MessagePack for Serialization”

  1. Great write up! When can we expect part 2, sending binary from the browser (client). In particular can we down sample audio data from the client’s microphone and send realtime audio data for a small chat room (without requiring complicated webrtc connections for each peer?)

Leave a Reply

Your email address will not be published. Required fields are marked *