Binary data transfer

Motivation

Often for visualizations in genomics, massive social networks, or sensor data visualizations, it helps to be able to plot millions rather than simply hundreds of thousands of points.

By default, pydeck sends data from Jupyter to the frontend by serializing data to JSON. However, for massive data sets, the costs to serialize and deserialize this JSON can prevent a visualization from rendering.

In order to get around this, pydeck supports binary data transfer, which significantly reduces data size. Binary transfer relies on NumPy and its typed arrays, which are converted to JavaScript typed arrays and passed to deck.gl using precalculated binary attributes.

Usage

Binary transport will only work if the following requirements are met:

x y r g b
0 1 0 0 0
0 5 255 0 0
5 1 255 255 0

should be converted to a nested format like this–

position color
[0, 1] [0, 0, 0]
[0, 5] [255, 0, 0]
[5, 1] [255, 255, 0]

Example

To demonstrate this, we'll plot a series of nodes generated at random from the networkx library:

We will use deck.gl's OrbitView (example here) which can help us plot these points in 3D:

The data we'll render is a 3D position with some categorical data (group):

We need this flattened data in a nested format, so we transform it:

We'll also color the points by group and then normalize the colors since they are provided as floats–see the deck.gl documentation.

Confirming that data is being transferred over web sockets

You can open up your brower's developer console to confirm the data is being sent via binary transfer. In the console you should see a message like:

>> transport.js:68 Delivering transport message 
>> binary: {"binary-points": {…}}
>> transport: n {name: "Jupyter Transport (JavaScript <=> Jupyter Kernel)", _messageQueue: Array(0), userData: {…}, jupyterModel: n, jupyterView: n}
type: "json-with-binary"

You should also be able to see the binary payload in your developer console's Network tab.