The following concept paper proposes an alternative implementation of farmOS, purposely built for distributed computing scenarios and in accordance with Runrig's design principles.
Runrig farmOS is a proposed suite of software packages for replicating and interoperating with the original client-server implementation of farmOS, which is written as a Drupal distribution in PHP. (To avoid confusion, I'll refer here to the former as Runrig farmOS or simply Runrig and to the latter as Standard farmOS or Standard.) Runrig farmOS packages are intended to run on servers, mobile devices, desktops, sensors, and embedded systems. They may be deployed as a substitute for a Standard farmOS instance, as a supplement to an existing one, or merely as one among many services bundled together as a municipal platform.1
Runrig farmOS is intended to enable local-first, peer-to-peer, and federated architectures. The aim is to bring these and other features to Standard farmOS, too, via Drupal modules that can be installed on any Standard farmOS instance. This should allow users to exchange data and functionality freely between Runrig and Standard and other free software services with ease. That way, as Runrig farmOS matures and Standard farmOS continues to evolve, their combined feature set can accommodate a wider range of requirements and use cases, complementing each other's strengths and weaknesses rather than competing for users and resources.
In pursuit of these aims, all packages in the Runrig farmOS suite must support (or at least not prohibit) the following core requirements, wherever applicable:
libfarmOS, a library implementation of the farmOS Data Model, written in Rust or C/C++ and compiled to binary for optimal portability & performance; it should be capable of serving as a sort of driver underlying servers, mobile devices, IoT sensors, even embedded systems, without reference to higher level abstractions, such as the network, storage, or presentation layers.- Multi-farm data representations and behavior, either as the proposed farmOS Organization entity or other compatible extension of the farmOS Data Model, in order to support multi-tenancy and easier aggregation.
- A shared, deterministic consensus algorithm, providing eventual consistency and predictable conflict resolution across concurrent devices and storage instances, but again, without reference to any specific network or storage protocols.
- Interoperation with Standard farmOS (v 3.0 or higher), if not in the form of realtime networked communication then at least by maintaining compatibility with data migrated to or from Standard servers.
The hope is that federated, P2P, and local-first architectures will emerge as convenient byproducts of these four requirements alone, so support for distributed architectures is not listed here specifically; however, that is still considered an essential feature in its own right, so the core requirements may be appended or amended if it does not emerge automatically as expected.
Another critical aspect of these packages is that they should adhere to a principle we might call schema-first design. As much as possible, all libraries and applications should derive their logic and structure from schema definitions, ontologies, RPCs, code generation, and various types of static configuration files that can be serialized, transmitted, and interpreted at runtime – not the other way around.
For instance, if a plugin introduces a new type of farmOS asset or log, rather than implement the entity in code and generate a schema and other static representations from that, instead the plugin should provide the necessary schema and behavior rules in a purely static form first – e.g., as a JSON Schema document, RDF ontology, or Protocol Buffer services – allowing the common engine or middleware to generate code based on that schema; or if code generation or RPC is impractical, at least infer common procedures like writing to the database or generating a route for API requests.
This approach should allow for greater portability across diverse languages and runtime environments, but without limiting the modularity of specific applications or the system as a whole. That is the ultimate object of schema-first design, because while portability and modularity may be two equally desirable characteristics of a given software system, they can also be the most difficult qualities to reconcile.
To sum up the main intent,
Schema-first Design ⇒ Portability + Modularity
where schema-first design enables portability and modularity to coincide within the same system, like an emulsifier encourages oil and water to mix.
These libraries and utilities should perform the lion's share of processing, storing, and transmitting farmOS-compliant data. Downstream packages in the Runrig farmOS suite, such as the Runrig Server and Runrig Client, can then act as thin wrappers around the Common Engine, while applications further downstream from the server and client themselves can elect to install additional middlewares or adapters, providing whatever encodings or platform support the application requires. Initially, these packages may be implemented in TypeScript, as an early expedient that at least meets the criteria for portability; eventually, however, most of packages would be ported to a compiled language, ideally Rust, for greater portability and improved performance.
The Common Engine will be the primary locus of development of Runrig farmOS's Core Requirements, particularly implementation of the farmOS Data Model (Reqs. 1 & 2) and the consensus algorithm (Req. 3). Interoperation with Standard farmOS (Req. 4), as well as other implementations, will mostly be the task of middlewares & adapters, though of course those utilities will rely heavily on the Common Engine.
As a sort of standard library, it might make sense to adopt the naming
convention common to low-level system libraries, especially those written in
C/C++ and Rust, and package this engine under the name libfarmOS or something
similar; this is akin to a range of widely used system libraries like
libcurl, libsodium, libpng, libsignal, libVLC, and
libsql. These examples variously serve a number of roles and functions:
dependencies for other system libraries and programs; drivers for higher-level
languages through foreign function interface (FFI), WebAssembly (WASM), or
running as an independent daemon; plug-n-play implementations of standards and
protocols that can be easily embedded into end-user applications to extend their
compatibility; or even, in the case of libVLC, a little bit of everything
above plus a headless CLI utility for processing large binary datasets. A
library like libfarmOS could go a tremendous ways to encouraging adoption of
the farmOS Data Model into existing projects, even if just as an afterthought.
Support for other platforms and runtimes could be provided more readily as
language bindings, similar to the roles currently fulfilled by the farmOS.py
and farmOS.js client libraries, but with far greater capabilities and far less
effort to maintain. In essence, they would function as fully fledged
implementations of the data model, which could then be coupled with the
middlewares & adapters or third-party networking utilities and storage drivers
to perform the full range of backend capabilities, as performed by Standard
farmOS servers today.
The Core Logic in particular will require special care to implement without
violating the principle of schema-first design. A degree of schema enforcement
might be achieved with remote procedure calls (RPC), but that won't necessarily
enforce the particulars of a given algorithm; mainly RPC would just guarantee a
consistent type signature for each procedure – e.g., getGeometry :: Asset -> Geometry. Some amount of code-sharing will likely prove unavoidable in the
first implementations of Runrig farmOS. Thankfully, the occasions of specific
logic in the farmOS Data Model are fairly limited, at the time of this draft,
currently just the three procedures listed in the requirements below. Changes
are unlikely to be introduced to Standard between major version releases. Also,
the three current logic procedures are standard to all relevant entities,
regardless of the bundle, so there appears to be no risk that farmOS modules
could add new bundles that would possibly alter that logic. For now then, the
logic will be implemented mostly in code, but the portability of the core
packages should suffice to ensure consistent behavior across systems.
As necessity dictates, greater algorithmic consistency might be achieved with the adoption of more rigorous type systems, such as Algebraic Data Types (ADTs). Though they don't add much in the way of schema enforcement, semantic ontologies like RDF may offer greater transparency of system configurations and capabilities. Even more comprehensive, in-band specification and enforcement mechanisms might be possible via more innovative solutions, such as Algebraic Property Graphs (essentially a marriage of IDL/RPC and RDF but reinforced by using ADTs as its core primitives) or the distributed protocols in development by the Spritely Institute, like OCapN, which expands upon Mark S. Miller's CapTP with a particular eye towards improving federated protocols like ActivityPub. As much as possible, OCapN aims to be backwards compatible with certain existing implementations of CapTP, like Cap'n Proto, which could make Cap'n Proto's RPC protocol and its schema language an especially attractive target for early implementations. Of course, we should remain careful not to impose specific transport protocols or communication models on the Common Engine itself – the actual implementation of Cap'n Proto or any RPC framework should be the purview of middlewares & adapters below. The Common Engine, meanwhile, should maintain its own internal representation of farmOS data structures and procedures that can interface with those protocols and frameworks. Ultimately, the Common Engine must be capable of ingesting schemas and logic extensions that can be interpreted consistently, regardless of their external representation.
Regarding the syncing protocol, this will strictly perform the role of a shared, deterministic consensus algorithm to handle concurrency across multiple systems. That introduces another element of logic that must be shared, but fortunately there exist a number of ready-made solutions for this, particularly Conflict-free Replicated Data Types (CRDTs), as well as established libraries like Automerge for implementing those data types consistently across disparate environments. An advantage of CRDTs (and other solutions similarly based upon data structures and mathematically rigorous transformations; ADTs excel here too) is that they also remain network-agnostic, so that behavior should remain predictable no matter the form of communication between nodes. The Automerge library is actually quite exemplary in this regard, insofar as it restricts itself to the processing, merging, and transformations of the data structures alone, in accordance to clearly specified behaviors and mathematical laws (e.g., commutativity, associativity, etc). It nevertheless remains thoroughly unaware of the network, a key element of Automerge's design principles, which are a primary source of inspiration for much of this proposal.
- Implementation of the farmOS Data Model
- Core Entities & Bundles
- Core Logic (preferably as RPC or IDL)
- Syncing Protocol
Optional libraries for hosting the Common Engine on specific platforms and environments, or for adapting the farmOS Data Model to other more generic protocols and standards. Unlike the common engine and application packages below, these will not be distributed as a single package but many smaller ones, preferably as granular as possible, though some may also be bundled together for convenience.
- Encoders/decoders, validators, etc. for Distributed Protocols
- Cap'n Proto for schema + RPC (or similar CapTP implementation)
- Protocol Buffers, Apache Thrift, or other IDL formats
- Linked Data: Atomic Data, Solid, IPLD, JSON-LD or other RDF
- GraphQL
- JSON Schema & JSON:API
- farmOS Conventions
- Language bindings & Foreign Function Interfaces (FFI)
- PHP
- Rust
- C/C++
- TypeScript/JavaScript (likely superseding farmOS.js)
- Python
- Ruby
- Elixir
- Framework middlewares, mixins & plugins
- Database adapters
- farmOS on SQLite
- 1 DB/tenant à la Turso/libsql
- or similar: SqlSync, Electric, VLCN/cr-sqlite
- Graph databases and triple stores
- Neo4j
- Apache Jena
- General SPARQL support
- farmOS on SQLite
A fully headless, minimalistic server implementation, mainly for the purpose of
providing remote storage and a syncing node for local-first applications,
federated servers, and standard farmOS instances that have the farm_runrig
module installed, all with very little overhead. For more comprehensive server
functionality, the server can be extended or wrapped by downstream applications,
or integrated into a microservice architecture.
- Integrates the Common Engine
- Extensible via shared middlewares & adapters, for a diversity of
- Serialization formats
- Network protocols
- Framework middlewares
- Database connections
- Connection to the Plugin Store
This will be a fairly lightweight client application, practically just a daemon
running from the system tray on desktop or as a background notifier on mobile,
but it will also serve as the host or kernel for the client plugin applications
that can be installed from the Plugin Store. In this respect, it will
supersede much of farmOS Field Kit's intended functionality, like the proposed
Field Module system, app shell, and the Vue composable useEntities for
CRUD & syncing operations. Much of this functionality will be replaced by the
Common Engine and Plugin Store, but wherever feasible, we'll try to recycle
any relevant portions of Field Kit's codebase. Depending on the platform target
(eg, desktop, mobile, IoT, etc.), the client may come stocked with a "Data
Viewer" or "Tasks List" plugin by default, similar to how Field Kit was planned
to come with the "Tasks" Field Module, so users have at least some kind of
minimal interface for interacting with data, without requiring any additional
Runrig Server plugins or Standard farmOS modules as dependencies, at least not
beyond the Common Engine and Standard farmOS's core modules, respectively.
- Integrates the Common Engine
- Runrig Server Connection
- farmOS JSON:API Connection
- Connection to Plugin Store
Plugins can wrap various middlewares & adapters, along with associated schemas, vocabularies, UI widgets, and other dependencies, to extend the functionality of servers and clients. By making these different components available from a known secure provider, federated services and devices can maintain compatibility with trusted peers within the network more easily and without sacrificing extensibility. The store can also make useful extensions more discoverable to end users and admins, with appropriate permissioning where necessary or desirable, through a GUI web application and CLI package manager.
For relevant open source examples, see:
- Secure signing of authorized plugins
- Runrig Server plugins
- Runrig Client plugins
- Schema & protocol publication
- Versioning & dependency management
- Package manager CLI (probably separate pkg)
Drupal modules for installation on Standard farmOS.
This module will allow any Standard farmOS server to act as a client for any Runrig Server instance, as well as a server for any Runrig Client, although it might be best to separate that functionality into two or more packages.
- Wraps the Common Engine
- Runrig Server Support
- Runrig Client Support
This is essentially a pre-configured Runrig Client, packaged up as a Drupal module for Standard farmOS and capable of running in the browser as a PWA.
Footnotes
-
A full exposition on municipal platforms, which might also be called Federated Autonomous Municipal Platforms, or FAMPs, will have to wait for now because they go far beyond the scope of Runrig farmOS alone. For the time being, however, municipal platforms can be understood as roughly equivalent to the concept I started out to describing as "service platforms" in The Runrig Plan from 2023. They are collectively owned and controlled digital platforms, similar to platform cooperatives, but each comprising multiple services or applications, in turn representing multiple businesses and consumer groups, rather than a monolithic single-service platform for a cooperative. A close approximation of this kind of multi-service, democratically governed and user-owned platform would be May First Movement Tech Co-op, though it is far more general in purpose and geographically dispersed, whereas the municipal platforms I intend to build with Runrig would focus more on the productive forces of a single foodshed or bioregion, with an emphasis on land management and food systems. As a final point of reference, they would also embody much of the traits of the "civic platforms" laid out in James Muldoon's Platform Socialism (Pluto Press, 2022). ↩