Composable Distributed OS

Spitballing a distributed OS design following the Perlis-Thompson Principle.

• Use Cases span the gamut. It scales down as well as scales up.

  • Hosting Static Files. You need something like nginx and a file system.
  • Hosting simple database-backed web apps (like Heroku does, e.g. "12 factor apps")
  • Hosting off-the-shelf distributed databases. It needs to be flexible enough for this.
  • Hosting a search engine: batch jobs for indexing, a tree of servers for serving posting lists, etc.
  • Hosting its own binaries: metrics, monitoring, alerting
  • Hosting another copy of itself: It needs to compose.
    • There is a "virtualization theorem" for hardware; likewise we want some kind of "virtualization" for a distributed. Not because this is necessarily a great idea, but because it's a test of the power of the abstractions developed. Arguably, the fact that VMs and containers exist is a sign of a deficit in the design of a Unix process. A process was a virtual machine, except where it leaked and caused problems.

• Design ideas

  • It's a source-based OS. You push source code (like Heroku) and build configuration, and the system spins up processes to build it into a binary / OCI image that can be deployed to many machines. Both the source code and binary image are identified with KIDs.
  • Being source-based means that distributed debugging and tracing tools can always refer back to source code. The system knows more about what's running on it than just opaque containers.

• The Inner Platform effect and the Bootstrapping Problem

  • We want to solve this.

• There is a single storage abstraction and single namespace. Let's call it a "Keg" to be concrete.

  • Think of Keg as a wrapper over git. It supports versioning and differential compression. It can efficiently hold 1,000 copies of a 1 GiB container (possibly by using "pointers". Programs can read and write its content with the regular POSIX file system API, not a RPC wrapper.

• Everything is a KID. A KID can be thought of as a value (in the Rich Hickey sense), and also a distributed pointer. It's a handle to versioned data.

  • To be concrete, let's write it as K-0123abcd. This is a hex number.

• Examples

  • Every user has a KID (could be a hash of an e-mail address)
  • Every machine has a KID. (It could be a hash of a human-readable name)
  • Every version of source code is represented by a KID (can be derived from git hash)
  • There is a single executable format like an OCI container, and each one has a unique KID.
  • Every running process had a KID. The Mayor maintains a map from PID to KID.

• Operations:

  • wait() on a set of process KIDs to exit.

• Components

  • Mayor: a distributed init. Maintains more of its own integrity than Borg.
    • Machine state: A table of (KID for binary, KID for user, )