Skip to content

NetStream Manual

Jump to bottom Edit New page
Yoann Pigné edited this page Feb 27, 2015 · 9 revisions

The NetStream Manual

The NetStream framework allows to export the idea of "streams of graph events" to other languages than Java, through a network interface. The aim is mainly to allow the use of GraphStream with other projects written in other languages. However, since it is a network interface it also allows the use of several machines. The protocol is optimized to be have as low overhead as possible.

If you are looking for a Java-to-Java network link between GraphStream and some other project, you may prefer GraphStream's RMI facilities.

This document is organized in 3 sections. The first one details the Receiver's mechanisms. The second section describes the Sender. The last section details the NetStream Protocol.

Receiver

This one is responsible for receiving graph events from the network following the "NetStream" protocol. Events are then dispatched to pipes according to a given names. Here we consider that several stream of events (independent one another) can be handled by the receiver. We thus introduce the idea of stream ID where a stream is identified by an ID.

The Receiver is composed of:

  • A socket server that handles multiples connections directed to multiple streams (pipes). That part is mostly a copy/past from Antoine's "MBox Receiver" code.
  • An implementation of the NetStream Protocol (see below) that parses the received byte arrays and creates/sends graph events to specified pipes.
  • a set of streams (ThreadProxyPipes) identified by an ID. From GraphStream's point of view, the NetStreamReceriver provides sources (actually pipes) on which sinks (or other pipes) can connect to, to receive graph events.

The Receiver's general behavior is:

  • Wait for messages from any sender received data is stored separately for each sender until a message is completely received. The receiver knows about a complete message because the first 4 bytes of the messages are an integer that gives the size of the message.
  • A complete message is decoded (according to the NetStream Protocol), an event is created and sent to the specified stream (pipe)

The graph event receiver listens at a given address and port. It runs on its own thread. Several senders can connect to it, the receiver will demultiplex the data flow and dispatch incoming events to specified pipes. No extra thread are created when client connect.

From the graph event stream point of view, the NetStream receiver can be seen as a set of pipes identified by an id. When an event is received is is directed to one specific stream. By default, senders not willing to handle different streams may send to the stream called "default".

The only way to receive events from the network is to ask for a stream by means of a ThreadProxyPipe to the Receiver. The getStream() and getDefaultStream() give access to such pipe. Asking a non-existing stream (with an unknown id) will create it, so those functions always return a pipe. On the opposite, any new stream introduced by a sender will be created by the receiver.

The following figure illustrates a receiver handling several streams with several senders in their own process
(optionally, on their own host). The two first senders both submit events to the same stream (the "default" one):
senders-receiver.png

Example

import java.io.IOException;
import java.net.UnknownHostException;

import org.graphstream.graph.Graph;
import org.graphstream.graph.implementations.MultiGraph;
import org.graphstream.stream.thread.ThreadProxyPipe;

/**
 *  A simple example of use of the NetStream receiver.
 */
public class ReceiverExample {

       public static void main(String[] args) throws UnknownHostException, IOException, InterruptedException {
               // ----- On the receiver side -----
               //
               // - a graph that will display the received events
               Graph g = new MultiGraph("G");
               g.display();
               // - the receiver that waits for events
               NetStreamReceiver net = new NetStreamReceiver(2001);
               // - received events end up in the "default" pipe
               ThreadProxyPipe pipe = net.getDefaultStream();
               // - plug the pipe to the sink of the graph
               pipe.addSink(g);
               // -The receiver pro-actively checks for events on the ThreadProxyPipe
               while (true) {
                       pipe.pump();
                       Thread.sleep(100);
               }
       }
}

Sender

A sender, from the GraphStream API, is first of all a sink where one can plug sources so that it can receive events. Receiving these events the sender will pack them into messages according to the NetStream Protocol and then send those messages to a defined receiver through a given port, host and stream ID.

Example

import java.io.IOException;
import java.net.UnknownHostException;

import org.graphstream.graph.Graph;
import org.graphstream.graph.implementations.MultiGraph;

/**
 *  A simple example of use of the NetStream sender.
 */
public class SenderExample {

       public static void main(String[] args) {
               Graph g = new MultiGraph("G");
               // - the sender
               NetStreamSender nsc = null;
               try {
                       nsc = new NetStreamSender(2001);
               } catch (UnknownHostException e) {
                       e.printStackTrace();
               } catch (IOException e) {
                       e.printStackTrace();
               }
               // - plug the graph to the sender so that graph events can be
               // sent automatically
               g.addSink(nsc);
               // - generate some events on the client side
               String style = "node{fill-mode:plain;fill-color:#567;size:6px;}";
               g.addAttribute("stylesheet", style);
               g.addAttribute("ui.antialias", true);
               g.addAttribute("layout.stabilization-limit", 0);
               for (int i = 0; i < 500; i++) {
                       g.addNode(i + "");
                       if (i > 0) {
                               g.addEdge(i + "-" + (i - 1), i + "", (i - 1) + "");
                               g.addEdge(i + "--" + (i / 2), i + "", (i / 2) + "");
                       }
               }
       }

}

The NetStream Protocol

Messages in the NetStream protocol are specified a the byte level. It is different than an XML-based protocols like client/server REST approaches. Here the content and different formats constituting a message are optimize as much as possible, so as to reduce the network payload.

A message, as it is created by a sender, is composed of three main parts, like shown of the figure :

message.png
  1. A 4 bytes integer that indicates the size (in bytes) of the remaining of this message (not including those 4 bytes).
  2. A string, encoded using the NetStream protocol (see TYPE_STRING below), that identifies the stream targeted by this event.
  3. The event itself, that can be decoded, according to the NetStream protocol.

Data Types

Before sending a value whose type is unknown (integer, double, string, array...) one have to specify its type (and if applicable, its length) to the server. Value types are defined to allow the server to recognize the type of a value. When applicable (strings, tables, raw data) types are followed by a length. This length is always coded with a 32-bits signed integer and usually represents the number of elements (for arrays). Apart from array length (but we should update) integer values tend to be encoded with a more space-efficient format, varints. Those varint can be signed or unsigned.

  • TYPE_BOOLEAN [0x50]

    Announces a boolean value. Followed by a byte whose value is 0 (false) or 1 (true).

  • TYPE_BOOLEAN_ARRAY [0X51]

    Announces an array of boolean values. Followed by first, a 32-bit integer that indicates the length of this array, and then, by the actual sequence of booleans.

  • TYPE_BYTE [0x52]

    Announces a byte. Followed by a 8-bit signed byte.

  • TYPE_INT_ARRAY [0x53]

    Announces an array of bytes. Followed by first, a 32-bit integer that indicates the length in number of elements of this array, and then, by the actual sequence of 8-bit signed bytes.

  • TYPE_SHORT [0x54]

    Announces a short. Followed by a signed varint.

  • TYPE_SHORT_ARRAY [0x55]

    Announces an array of shorts. Followed by first, a 32-bit integer that indicates the length in number of elements of this array, and then, by the actual sequence of signed varints.

  • TYPE_INT [0x56]

    Announces an integer. Followed by a signed varint.

  • TYPE_INT_ARRAY [0x57]

    Announces an array of integers. Followed by first, a 32-bit integer that indicates the length in number of elements of this array, and then, the actual sequence of signed varints.

  • TYPE_LONG [0x58]

    Announces a long. Followed by a signed varint.

  • TYPE_LONG_ARRAY [0x59]

    Announces an array of longs. Followed by first, a 32-bit integer that indicates the length in number of elements of this array, and then, by the actual sequence of signed varints.

  • TYPE_FLOAT [0x5A]

    Announces a float. Followed by a 32-bit single precision signed floating point number.

  • TYPE_FLOAT_ARRAY [0x5B]

    Announces an array of floats. Followed by first, a 32-bit integer that indicates the length in number of elements of this array, and then, by the actual sequence of 32-bit double precision signed floating point numbers.

  • TYPE_DOUBLE [0x5C]

    Announces a double. Followed by a 64-bit double precision signed floating point number.

  • TYPE_DOUBLE_ARRAY [0x5D]

    Announces an array of doubles. Followed by first, a 32-bit integer that indicates the length in number of elements of this array, and then, by the actual sequence of 64-bit double precision signed floating point numbers.

  • TYPE_STRING [0x5E]

    Announces an array of characters. Followed by first, a 32-bits integer for the size in bytes (not in number of characters) of the string, then by the unicode string itself.

  • TYPE_RAW [0x5F]

    Announces raw data, good for serialization or to exchange data the will then be understood in any language (an image, for instance). Followed by first, a 32-bits integer indicating the length in bytes of the dataset, and then by the data itself, as unsigned bytes.

  • TYPE_ARRAY [0x60]

    Announces an undefined-type array. Followed by first, a 32-bits integer indicating the number of elements, and then, the elements themselves. The elements themselves have to give their types. It may contain data of different types or even other arrays.

Graph Events

the graph event, as created by a sender, is the third part of the whole sent message. It is made of several parts that differ according the event. The common information is the first byte of the event, that identifies the event. Then, other data depending on the event follow up. Those event identifiers are one byte long. To avoid problems between languages (mainly because of java) those bytes are unsigned and only positive values are used. So, any event identifier will take a value between 0 and 127.

After the event type byte come two information: the StreamID and the TimeID. Those two are use by GraphStream in ordrer to keep the event stream consistant. The StreamID is always a string while the TimeID is an unsigned long (internaly encoded with an unsigned varint).

Here is a list of graph event identifiers followed by the expected information to fulfill these events:

  • EVENT_ADD_NODE [0x10]

    Add a node. Followed by a node id (TYPE_STRING format).

    EVENT_ADD_NODE.png

  • EVENT_DEL_NODE [0x11]

    Remove a node. Followed by a node id (TYPE_STRING format)

    EVENT_DEL_NODE.png

  • EVENT_ADD_EDGE [0x12]

    Add an edge. Followed by:

    • the edge id (TYPE_STRING format),
    • the source node id (TYPE_STRING format),
    • the target node id (TYPE_STRING format
    • a boolean indicating if the edge is directed (is it an arc?) (TYPE_BOOLEAN format)
    EVENT_ADD_EDGE.png

  • EVENT_DEL_NODE [0x13]

    Remove an edge. Followed by the string id of this edge.

    EVENT_DEL_EDGE.png

  • EVENT_STEP [0x14]

    Time step. Followed by a 64-bit double indicating the timestamp.

    EVENT_STEP.png

  • EVENT_CLEARED [0x15]

    Clear the graph. This event will remove any attribute or element in the graph.

    EVENT_CLEARED.png

  • EVENT_ADD_GRAPH_ATTR [0x16]

    Add an attribute to the graph. Followed by:

    • the attribute name (TYPE_STRING format)
    • the attribute value type (one of the bytes shown in the "Data Types" section)
    • the attribute value, encoded according to its value type (see the "Data Types" section)
    EVENT_ADD_GRAPH_ATTR.png

  • EVENT_CHG_GRAPH_ATTR [0x17]

    Change an existing attribute on the graph. Followed by:

    • the attribute name (TYPE_STRING format)
    • the attribute's old value type (one of the bytes shown in the "Data Types" section)
    • the old attribute value, encoded according to its value type (see the "Data Types" section)
    • the attribute's new value type (one of the bytes shown in the "Data Types" section)
    • the new attribute value, encoded according to its value type (see the "Data Types" section)
    EVENT_CHG_GRAPH_ATTR.png

  • EVENT_DEL_GRAPH_ATTR [0x18]

    Remove an attribute from the graph. Followed by the attribute name (encoded with the TYPE_STRING format).

    EVENT_DEL_GRAPH_ATTR.png

  • EVENT_ADD_NODE_ATTR [0x19]

    Add an attribute to a node. Followed by:

    • the ID of the considered node (TYPE_STRING format)
    • the attribute name (TYPE_STRING format)
    • the attribute value type (one of the bytes shown in the "Data Types" section)
    • the attribute value, encoded according to its value type (see the "Data Types" section)
    EVENT_ADD_NODE_ATTR.png

  • EVENT_CHG_NODE_ATTR [0x1A]

    Change an existing attribute on a given node. Followed by:

    • the ID of the considered node (TYPE_STRING format)
    • the attribute name (TYPE_STRING format)
    • the attribute's old value type (one of the bytes shown in the "Data Types" section)
    • the old attribute value, encoded according to its value type (see the "Data Types" section)
    • the attribute's new value type (one of the bytes shown in the "Data Types" section)
    • the new attribute value, encoded according to its value type (see the "Data Types" section)
    EVENT_CHG_NODE_ATTR.png

  • EVENT_DEL_NODE_ATTR [0x1B]

    Remove an attribute from a given node. Followed by:

    • the ID of the considered node (TYPE_STRING format)
    • the attribute name (encoded with the TYPE_STRING format).
    EVENT_DEL_NODE_ATTR.png

  • EVENT_ADD_EDGE_ATTR [0x1C]

    Add an attribute to an edge. Followed by:

    • the ID of the considered edge (TYPE_STRING format)
    • the attribute name (TYPE_STRING format)
    • the attribute value type (one of the bytes shown in the "Data Types" section)
    • the attribute value, encoded according to its value type (see the "Data Types" section)
    EVENT_ADD_NODE_ATTR.png

  • EVENT_CHG_EDGE_ATTR [0x1D]

    Change an existing attribute on a given edge. Followed by:

    • the ID of the considered edge (TYPE_STRING format)
    • the attribute name (TYPE_STRING format)
    • the attribute's old value type (one of the bytes shown in the "Data Types" section)
    • the old attribute value, encoded according to its value type (see the "Data Types" section)
    • the attribute's new value type (one of the bytes shown in the "Data Types" section)
    • the new attribute value, encoded according to its value type (see the "Data Types" section)
    EVENT_CHG_NODE_ATTR.png

  • EVENT_DEL_EDGE_ATTR [0x1E]

    Remove an attribute from a given edge. Followed by:

    • the ID of the considered edge (TYPE_STRING format)
    • the attribute name (encoded with the TYPE_STRING format).
    EVENT_DEL_NODE_ATTR.png
Add a custom footer
Add a custom sidebar
Clone this wiki locally