Composable Distributed OS

Spitballing a distributed OS design following the Perlis-Thompson Principle.

• Use Cases span the gamut. It scales down as well as scales up.

  • Hosting Static Files. You need something like nginx and a file system.
  • Hosting simple database-backed web apps (like Heroku does, e.g. "12 factor apps")
  • Hosting off-the-shelf distributed databases. It needs to be flexible enough for this.
  • Hosting a search engine: batch jobs for indexing, a tree of servers for serving posting lists, etc.
  • Hosting its own binaries: metrics, monitoring, alerting
  • Hosting another copy of itself: It needs to compose.
    • There is a "virtualization theorem" for hardware; likewise we want some kind of "virtualization" for a distributed. Not because this is necessarily a great idea, but because it's a test of the power of the abstractions developed. Arguably, the fact that VMs and containers exist is a sign of a deficit in the design of a Unix process. A process was a virtual machine, except where it leaked and caused problems.

• The Inner Platform effect and the Bootstrapping Problem

  • We want to solve this.

• There is a single storage abstraction. Let's call it a "Keg" to be concrete.

  • Think of Keg as a wrapper over git. It supports versioning and differential compression. It can efficiently hold 1,000 copies of a 1 GiB container (possibly by using "pointers". Programs can read and write its content with the regular POSIX file system API, not a RPC wrapper.

• Everything is a KID. A KID can be thought of as a value (in the Rich Hickey sense), and also a distributed pointer. It's a handle to versioned data.

  • To be concrete, let's write it as K-0123abcd. This is a hex number.
  • Every machine has a KID
  • There is a single executable format like an OCI container, and each one has a unique KID.
  • Every user has a KID

• Components

  • Mayor: a distributed init. Maintains more of its own integrity than Borg.
    • Machine state: A table of (KID for binary, KID for user, )