Add methods in sandbox and endpoint to support container checkpoint and restore

[huikang]

This proposal is to explore adding libnetwork API to handle network connectivity during container checkpoint and restore. In general, there are two approaches.

• CRIU-based

  This approach relies on CRIU to restore the veth pair for the Linux namespace and attach to the docker bridge. An example implemeantion can be found in Allow restore network to have network connectivity boucher/docker#17

  This implementation requires libnetwork exposing the detailed interface information to the client (i.e., docker). Specifically, docker needs to passes (1) the interface name in the container namespcae (e.g., eth0), (2) the other end of the veth pair in the host (e.g., vethxxxxxxx), (3)the network name that includes the endpoint (e.g., docker0).

  Therefore, the criu-based approach requires new APIs from the endpoint and sandbox of libnetwork to expose the above interface and bridge information.

• libnetwork-based

  The second approach relies on libnetwork to create a new set of endpoint and sandbox for restoring the checkpointed container. However, this approach may requires modifications to the CRIU. This is because when criu restores the container process, CRIU creates the other end of the pair in the net namespace you launch criu from [1]. This veth end in the host is not known by libnetwork. Therefore, we may need to add some post-processing after criu restore the container. These post-processing should be libnetwork API calls.

[1] http://criu.org/Advanced_usage

---

Deprecated

Placeholder for designing new methods to support checkpoint and restore.

Considering adding the following methods

type Network interface {
    CheckpointEndpoint( )
}

type Controller interface {
    CheckpointSandbox()
}

These two methods will collaboratively reserve the endpoint (e.g., veth name) and the network namespace for the container when it is being checkpointed.

[mavenugo]

@huikang apologies on the delay and thanks for starting the design discussion on C/R. I was talking to Seid during the hackathon and we decided to approach it properly with proper end-to-end design discussion instead of retrofitting it with APIs. So, it would be really good if you can share the existing design that you have in mind and how you think libnetwork can help.
We can have a constructive discussion in this proposal and then come up with API requirements.
Does that work for you ?

[huikang]

@mavenugo Thanks for your response. I am very glad to join the discussion. I will update my thought at this thread (if you have any other preference way let me know). Thanks.

[huikang]

Updated my thought on networking C/R. Ping @mavenugo and @boucher

[boucher]

Sorry for the delay in responding. I personally believe its worth implementing option 1, since it seems to require very few modifications, while we figure out the best way to implement option 2. It seems to me like the libnetwork team strongly prefers eventually ending up at option 2.

[xemul]

What are the libnetwork team arguments for the option 2? I'd also like to note, that going this way would require recreating the net namespace in exact same state as it was on dump -- device names and indices, route tables, addresses -- al should remain intact as restoring sockets, especially TCP, may break if anything in the environment (netns) doesn't match the expectations.

[mavenugo]

Updated my thought on networking C/R. Ping @mavenugo and @boucher

@huikang Can you pls point me to that ?

@xemul @boucher pls refer to all the comments in moby/moby#15747 and that gives the reason on why option #1 is incorrect. Docker's networking model is based on CNM which enables a lot of good networking functionality. Hence we really like C/R functionality to work with it so that the end-user gets all the goodness.

BTW, i would love to get C/R support in docker for 1.10 if we all work together towards getting option#2 working end-to-end.

[huikang]

@mavenugo We all refer to the text on the top of this link. You must have seen it.

[SaiedKazemi]

@xemul @huikang @boucher

@mavenugo thanks for helping to move this forward. Per our discussion at Global Hack Day last week, below is how I did network C/R in Docker 1.5 before the introduction of libnetwork. I hope it will help in closing design discussions on C/R support in Docker/libnetwork.

1. Docker Daemon Starting

When Docker is started ($ sudo docker -d), it scans its repository directory (/var/lib/docker/containers) to load previously created containers. If we find a checkpointed container, we reserve its IP address because we anticipate that the container will be restored.

File: daemon/networkdriver/bridge/driver.go
Function: ReserveIP()

File: daemon/daemon.go
Function: restore()

2. Checkpointing a Container

Although a container exits after it's checkpointed ($ docker checkpoint container-id), we do not release its network resources (IP and MAC addresses) because we anticipate that it will be restored again and will want to continue using its previous network resources.

When calling CRIU from libcontainer to checkpoint a container, we specify the --tcp-established option to save information about its network connection in the checkpoint image files.

File: daemon/container.go
Function: cleanup()

File: vendor/src/github.com/docker/libcontainer/namespaces/checkpoint.go
Function: Checkpoint()

3. Restoring a Container

Setting up a container's network during restore ($ docker restore container-id) is more involved than the previous steps mentioned above.

We want to use the same IP and MAC addresses that the container was using when it was checkpointed. Therefore, for a checkpointed container, we do not error due to "using already allocated ip" when reusing these resources.

When CRIU is called with the --tcp-established option, it will create a new veth pair to restore the container's veth device (eth0) that was dismantled after checkpoint. We generate a unique name for the peer veth device in the host and tell CRIU to map the inside veth device (eth0) to the outside device and move it to Docker's bridge (--veth-pair eth0=vethXXXXXX@docker0). This is essentialy equivalent to:

    $ brctl addif docker0 vethXXXXXX
    $ ip link set dev vethXXXXXX up

This assumes that the contianer has not changed its network interface name from eth0. By examining /proc//net/dev, we can handle the case where the name has actually changed.

File: vendor/src/github.com/docker/libcontainer/namespaces/restore.go
Function: buildCriuArgs()

File: daemon/container.go
Function: AllocateNetwork()

File: daemon/networkdriver/ipallocator/allocator.go
Function: checkIP()

[xemul]

@SaiedKazemi let me correct you a little bit. The --tcp-established doesn't result in doing anything with veth pairs, it's only about repairing the TCP connection. The veth workflow is fully tied with netns restore -- if we have netns, we restore veth-s, if we don't -- we don't :)

@mavenugo , OK, option 1 can be achieved if we teach criu not to mess with net namespaces at all. Instead, docker would prepare the netns, put criu into it and ask for restore. But as I said, that's risky.

[SaiedKazemi]

@xemul thanks for your note. Reading the paragraph again, I see that it's misleading. Should have written "When CRIU is called to restore, we need to specify the --tcp-established option."

[mavenugo]

@xemul i think we are in sync on the expected behavior. I also understand your comment on why this can be risky due to

would require recreating the net namespace in exact same state as it was on dump -- device names and indices, route tables, addresses -- al should remain intact as restoring sockets, especially TCP, may break if anything in the environment (netns) doesn't match the expectations.

cc @mrjana as well and we can discuss further on this to see if we can make libnetwork honor this expectation.

thanks @SaiedKazemi for sharing the existing design. I will go over them and ping you with any questions that we might have. I also commented on the docker/docker project in support of adding it to 1.10 milestone and officially support it in libnetwork as well. I would like to get the feature consistency across these projects so that there is no misalignment in the priorities.

[huikang]

@SaiedKazemi Thanks for sharing your design. From my understanding, your approach in 1.5 aligns with option 1. Correct?