lsp fork stonework

paper：

"A Hybrid Data Plane SDN Architecture based on Cloud-native for Segment Routing over IPv6"

Donglei Yuan, Chi Jiang, Yuli Zhao, Hai Yu, Zhiliang Zhu

SRv6 (Segment Routing over IPv6) serves as the cornerstone for next-generation internet solutions. The current SDN architectures are incapable of providing concurrent control plane configuration for hybrid data planes that incorporate both VPP and Linux, thereby curtailing the cross-plane technological benefits inherent to SRv6. Furthermore, the necessity to maintain synchronization among disparate controllers escalates opera- tional costs and impedes the unified southbound management of SRv6 on the control plane for both Linux and VPP nodes. To address these challenges, we have reengineered the SDN architecture for cloud-native networking, introducing a south- bound interface LSP (Linux SR Plugin) component designed to facilitate SRv6 support. Additionally, we have integrated the LSP with a cloud platform through a key-value store, enhancing its northbound API server capabilities. This integration culminates in the deployment of the LSP as a plugin within the cloud envi- ronment, enabling users to focus solely on their business require- ments and manage the hybrid data plane seamlessly through the API interface. Subsequently, we constructed a simulation testing environment for the expanded plugin to validate its functionality and performance. The findings demonstrate that the LSP is capable of verifying all SRv6 configurations supported by both Linux and VPP data planes, while also meeting the performance benchmark of processing at least 400 configurations per second. This achievement underscores the LSP’s ability to provide a unified southbound control and management Architecture for SRv6 across both VPP and Linux data planes.

A high-performance data plane, modular control plane solution.

cncp-sdn-lsp supported linux and vpp data plane

This network appliance, however, is not a step back from distributed chained/meshed microservices, to monolithic architecture.

Instead, the integration is:

• Dynamic
• Based on container orchestration
• CNF discovery
• Sharing of network namespaces and
• Re-use of data paths for packet punting between CNFs

CHANGELOG:

• June 16, 2025:
• We supplemented the SRv6 testing implementation through netconf in different device types at this stage.
• These include:
  • H3C equipment
  • SONiC equipment
  • Linux Network stack

The main supplement is the experimental analysis to verify the advantages of LSP.

In the next stage, we will open source the implementation of the UI interface of the SRv6 controller part of the CNCP platform.

Release Schedule:

• Version policy:Follow semantic versioning: major version numbers.

• Major update: Breaking feature changes or API incompatibilities.

• Second version update: New features backward compatibility.

• Revision number Update: Bug fix or documentation improvement.

• Release Cycle: Released quarterly (with full testing and documentation).

• Hot fix: Release revisions as soon as urgent issues are fixed.

• CHANGELOG: Update the CHANGELOG.md with each release to record new features, fixes, and known issues.

Features

• High-performance VPP-based data plane
• Linux-based data plane
• Management agent build on top of Ligato VPP-Agent
• Suitable for both cloud & bare-metal deployments
• Can be deployed as either multiple interconnected instances (service function chaining), or a set of control/management plane microservices that use a single VPP instance for data plane (this is a trade-off between flexibility and resource utilization)
• Northbound APIs are modeled with protobuf and accessible over gRPC, REST, K8s CRDor through a key-value DB (etcd, redis, ...)
• Wide-range of networking features, natively implemented in VPP, e.g.:
  • High-performance device drivers (DPDK, RDMA, virtio)
  • Routing, switching
  • Tunneling (VXLAN, GRE, IP-IP)
  • ACL-based filtering and routing
  • NAT44, NAT64
  • Segment routing
  • VPN (Wireguard, IPSec)
  • Bridge domains, VRFs (multi-tenancy)
• Management features provided by the Linux network stack:
  • Routes, ARPs
  • iptables
  • Namespaces, VRFs (multi-tenancy)
• Dynamically (at run-time) extensible with additional features provided by CNFs from PANTHEON.tech

Examples

Before using StoneWork, we recommend reading this README and related documentation in the StoneWork [distribution folder][docs].

If you are new to StoneWork, it may be easier to first explore and run the provided examples, rather than trying to create deployment manifests from scratch.

Examples of deployment manifests and configurations for various use-cases can be found under the examples sub-directory.

The Getting Started example will guide you through your first StoneWork deployment.

Configuration

Configuration for StoneWork consists of two tasks:

1. VPP Startup Configuration

The VPP Startup Configuration comprises configuration options, which are set before VPP is started. They cannot be changed at the run-time, either by a management plane API or the VPP CLI). For StoneWork, the default VPP startup configuration file is packaged in the image, under /etc/vpp/vpp.conf.

Some of the examples override the default configuration with a customized version of vpp.confmounted into the container using volumes. Typically, the only configuration section that may require customization is the dpdk stanza, where PCI addresses of NICs, used by VPP, should be listed.

Run the lshw-class network -businfo command to view the available network devices and their respective PCI addresses. For example, if the PCI addresses of interfaces were 0000:00:08.0 and 0000:00:09.0 (e.g. inside and outside network), then the dpdk configuration would be:

dpdk {
    dev 0000:00:08.0 {
        name eth0
    }
    dev 0000:00:09.0 {
        name eth1
    }
}

Interface names can be selected arbitrarily, for example eth0 and eth1' in the above example.

More information about attaching physical interfaces into VPP can be found here.

2. Protobuf-modeled Network Configuration

StoneWork's network configuration (VPP, Linux, CNFs) is modeled using Google Protocol Buffers.

A summary of all configuration items and their attributes, with descriptions, can be found here (in markdown; also available as a single PDF document).

A JSON Schema is provided as well, and can be used to validate input configuration before it is submitted.

Some text editors, for example VS Code, can even load the Schema and provide autocomplete suggestions based on it, thus making the process of preparing input configuration a lot easier.

The original protobuf files, from which the documentation and schema were generated, can be found in the /api folder inside the StoneWork distribution. There is also the /api/models.spec.yaml file, which contains one YAML document with metadata for every configuration model.

These metadata are used to associate a configuration model with the corresponding protobuf definitions.

Network configuration is submitted into the control-plane agent either via:

• a CLI (YAML formatted), written into a key-value datastore (e.g. etcd; JSON-formatted)

• or applied programmatically over gRPC (serialized by protobuf) or REST (JSON) APIs. The initial configuration that should be applied immediately after StoneWork starts up can be mounted into the container under /etc/stonework/config/day0-config.yaml (YAML formatted).

Each of the attached examples has a sub-directory named config, where you can find configuration stanzas to learn from. Each example contains the startup configuration day0-config.yaml.

Additional *.yaml files are used to show how run-time configuration can be modified over CLI. Please refer to each examples README.md file for more information.

Installation

The following steps will guide you through the StoneWork installation process. The distribution package contains the StoneWork Docker image (stonework.image), documentation (*.md) and some examples to get you started.

Requirements

1. StoneWork requires an Ubuntu VM or a bare-metal server running Ubuntu, preferably version 18.04 (Bionic Beaver).

2. Next, Docker and docker-compose must be installed.

   Install with:

   $ apt-get install docker.io docker-compose
   

3. (DPDK Only) Install/Enable Drivers

   Depending on the type of NICs that VPP of StoneWork should bind to, you may have to install/enable the corresponding drivers.

   For example, in a VM environment, the Virtual Function I/O (VFIO) is preferred over the UIO framework for better performance and more security. In order to load a VFIO driver, run:

   $ modprobe vfio-pci
   $ echo "vfio-pci" > /etc/modules-load.d/vfio.conf
   

   Check with:

   $ lsmod | grep vfio_pci
   vfio_pci               45056  0
   

   More information about Linux network I/O drivers that are compatible with DPDK (used by VPP), can be found here.

4. (DPDK Only) Check Network Interfaces

   Make sure that the network interfaces are not already used by the Linux kernel, or else VPP/DPDK will not be able to grab them. Run ip link set dev {device} down for each device to un-configure it from Linux. Preferably disable the interfaces using configuration files to make the changes persistent (e.g. inside /etc/network/interfaces).

5. (DPDK Only) Huge Pages

   In order to optimize memory access, VPP/DPDK uses Huge Pages, which have to be allocated before deploying StoneWork. For example, to allocate 512 Huge Pages (1024MiB memory for default 2M hugepage size), run:

   $ echo "vm.nr_hugepages=512" >> /etc/sysctl.conf
   $ sysctl -p
   

   Detailed recommendations on allocations of Huge Pages for VPP can be found here.

6. Finally, the StoneWork image has to be loaded so that Docker/docker-compose/K8s is able to provision a container instance. Run:

   $ docker load <./stonework.image
   

Deployment

StoneWork is deployed using Docker Compose version 3.3 or newer. StoneWork itself is only a single container (with VPP and StoneWork agent inside), but every CNF that is deployed alongside it runs in a separate container, hence the use of Compose.

The following is a template for the docker-compose.yaml file, used to describe deployment in the language of Docker Compose. The template contains detailed comments, that explain the meaning of attributes contained in the template and how they work in StoneWork.

Angle brackets are used to mark placeholders that have to be replaced with appropriate actual values in the target deployment.

version: '3.3'

# Volume shared between StoneWork and every CNF deployed alongside it.
# CNFs and StoneWork use it to discover each other.
volumes:
  runtime_data: {}

services:
  stonework:
    container_name: stonework
    image: "ghcr.io/pantheontech/stonework:22.10"
    # StoneWork runs in the privileged mode to be able to perform administrative network operations.
    privileged: true
    # StoneWork runs in the PID namespace of the host so that it can read PIDs of CNF processes.
    pid: "host"
    environment:
      # Set log level (i.e. only log entries with that severity or anything above it will be printed).
      # Supported values: Trace, Debug, Info, Warning, Error, Fatal and Panic.
      INITIAL_LOGLVL: "debug"
      # MICROSERVICE_LABEL is used to mark container with StoneWork.
      MICROSERVICE_LABEL: "stonework"
      # By default etcd datastore is used as the source of the configuration.
      # Env. variable ETCD_CONFIG with empty value is used to disable etcd
      # and use CLI (agentctl) or gRPC as the primary source of the configuration.
      ETCD_CONFIG: ""
    ports:
      # Expose HTTP and gRPC APIs.
      - "9111:9111"
      - "9191:9191"
    volumes:
      # /run/stonework must be shared between StoneWork and every CNF.
      - runtime_data:/run/stonework
      # /sys/bus/pci and /dev are mounted for StoneWork to be able to access PCI devices over DPDK.
      - /sys/bus/pci:/sys/bus/pci
      - /dev:/dev
      # Docker socket is mounted so that StoneWork can obtain container metadata for every CNF.
      - /run/docker.sock:/run/docker.sock
      # To customize vpp startup configuration, create your own version of vpp.conf (here called vpp-startup.conf),
      # put it next to this docker-compose.yaml and mount it under /etc/vpp/vpp.conf.
      # Otherwise remove this mount.
      - ./vpp-startup.conf:/etc/vpp/vpp.conf
      # To start StoneWork with some initial configuration, create day0-config.yaml under the config
      # sub-directory, placed next to this docker-compose.yaml and mount it under /etc/stonework/config
      # Otherwise remove this mount.
      - ./config:/etc/stonework/config

  # Multiple CNFs may share the same Linux network namespace. This is in some case needed
  # if CNFs are to work together (e.g. BGP peering established over OSPF-learned routes).
  # The common network namespace is represented by a separate container (similar to the
  # sandbox container of a K8s Pod).
  router-ns:
    container_name: router-ns
    image: "busybox:1.29.3"
    command: tail -f /dev/null

  # CNF running alongside StoneWork (i.e. using the VPP of StoneWork as data-plane).
  # Name the container such that it is clear what services CNF provides (e.g. "cnf-dhcp").
  <cnf-name>:
    container_name: <cnf-name>
    image: "<cnf-image-name>"
    depends_on:
      - stonework
    # CNFs typically require privileges to perform administrative network operations.
    privileged: true
    # <cnf-name>-license.env is file that is obtained when the license of CNF is purchased.
    # Put <cnf-name>-license.env into the same directory as docker-compose.yaml.
    # It contains single line:
    # LICENSE=<signed license content>
    env_file:
      - <cnf-name>-license.env
    volumes:
      # /run/stonework must be shared between StoneWork and every CNF.
      - runtime_data:/run/stonework
    environment:
      INITIAL_LOGLVL: "debug"
      # MICROSERVICE_LABEL is effectively used to mark the container with CNF name.
      # StoneWork is then able to identify the CNF container among all containers.
      MICROSERVICE_LABEL: "<cnf-name>"
      ETCD_CONFIG: ""
      # If CNF runs alongside StoneWork (and not standalone), env. variable "CNF_MODE"
      # must be defined with value "STONEWORK_MODULE".
      CNF_MODE: "STONEWORK_MODULE"
    # Multiple CNFs may share the same Linux network namespace.
    # Use network_mode and point a group of CNFs to the same container (acting just like sandbox
    # container of a K8s Pod).
    network_mode: "service:router-ns"

  # here list other CNFs...

Development

• Build: Build instruction for StoneWork can be found here.
• Architecture: StoneWork architecture is described in detail here.
• CNF Compatibility: A guide on how to make CNF compatible with StoneWork can be found here.
• GNS3 & StoneWork: StoneWork GNS3 VM development documentation is here.