Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use docker containers as hosts for testing netplugin #37

Merged
merged 11 commits into from
Apr 7, 2015
66 changes: 66 additions & 0 deletions dockerhost/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
The multi node container networking can now be tested on a single VM by running docker inside docker.
https://blog.docker.com/2013/09/docker-can-now-run-within-docker/

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good edition... it will allow MAC users to natively test/experiment this. Did you try these on your laptop/mac's VM?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this on a ubuntu VM on my laptop. Not sure if we can run on native mac.


The outside docker containers act like physical hosts in our test and are connected using a standard linux bridge. Inside each "host container" we run a namespaced instance of docker, OVS , etcd and netplugin instance. One can now launch containers from within each "host containers" and use netplugin networking to connect them.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc deserves a reference from the main README. If this becomes the simplest way to try it out, we should replace this with Vagrant based configuration mentioned in the top level README.md

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree with @jainvipin

Actually can we move this README.md to docs/dockerhost.md and link it from the main README.md. That's how most of our docs are arranged.

Also the scripts can be moved to scripts/dockerhost/ directory, as all the scripts etc stay there. I am thinking we can use/extend most of these scripts to support docker based test and demo environments (eventually getting rid of the heavier vagrant environment).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving this to docs/Dockerhost.md and scripts to scripts/dockerhost. Added reference from main README.md . Working on adding a mode in system test to use dockerhost. Once this is stable we can decide if this should be default.

The steps to launch docker hosts are :

1. Compile the netplugin code
2. Change user to root
3. Set GOPATH to root of the netplugin code
4. Add $GOPATH/dockerhost to PATH

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is $GOPATH/dockerhost created? Did you mean $GOPATH/src/github.com/netplugin/dockerhost?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed dependency on GOPATH. Will start the script using make from the root directory the same way vagrant demo is started.

5. Set CONTIV_NODES to required number of nodes
6. Run start-dockerhosts

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we skip steps 4 above and import the path in start-dockerhosts script?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to specify, either here or in prerequisites, that nsenter must be installed for this to work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another pre-requisite is having docker installed on host as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

This will start CONTIV_NODES number of containers with docker image called ubuntu_netplugin which is just ubuntu image with docker and ovs installed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ubuntu image also needs to have etcd installed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added


Now start a shell within any of the "host containers" using following convenient wrapper around nsenter :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convenient -> convenience

```
docker-sh netplugin-node<x>
```

Start netplugin, post netplugin config and launch containers inside the "host containers" the same way you do on VMs.
Note : currently the demo is working only if config is posted before containers are started .. need to debug why the reverse is not working.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, I think one of our tests already tests this, may be something got broken. Or is it working for VMs? Can you file a issue to track this, if it's ok?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will debug and file an issue if its not dockerhost specific.


To cleanup all the docker hosts and the virtual interfaces created do
```
cleanup-dockerhosts
```

Example for testing TwoHostMultiVlan you can do :

1. Launch the two host containers

```
export CONTIV_NODES=2
start-dockerhosts
```

2. Load the netplugin configuration
```
docker-sh netplugin-node1
/netplugin/bin/netdcli -cfg /netplugin/examples/two_hosts_multiple_vlans_nets.json
```

3. Launch container1 on host1

```
docker-sh netplugin-node1
docker run -it --name=myContainer1 --hostname=myContainer1 ubuntu /bin/bash
```

4. Launch container3 on host2

```
docker-sh netplugin-node2
docker run -it --name=myContainer3 --hostname=myContainer1 ubuntu /bin/bash
```

5. Test connectivity between the containers using ping. Go to the shell for container1
```
root@myContainer1:/# ping -c3 11.1.2.2
PING 11.1.2.2 (11.1.2.2) 56(84) bytes of data.
64 bytes from 11.1.2.2: icmp_seq=1 ttl=64 time=3.15 ms
64 bytes from 11.1.2.2: icmp_seq=2 ttl=64 time=1.36 ms
```

20 changes: 20 additions & 0 deletions dockerhost/cleanup-dockerhosts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@

#!/bin/bash

num_nodes=1
if [ -z "$CONTIV_NODES" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this just be:

if [ -n $CONTIV_NODES ]; then
    num_nodes=$CONTIV_NODES
fi

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

then
num_nodes=1
else
num_nodes=$CONTIV_NODES
fi
echo $num_nodes

for i in `seq 1 $num_nodes`;
do
hostname="netplugin-node$i"
echo "Cleaning $hostname"
docker exec $hostname service docker stop
docker rm -f $hostname
ip link delete $i-ext
done
6 changes: 6 additions & 0 deletions dockerhost/create_loop_devices.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this script being used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need it if running many hosts as each need 2 loop devices

for i in `seq 9 20`;
do
sudo /bin/mknod -m640 /dev/loop$i b 7 $i
sudo /bin/chown root:disk /dev/loop$i
done
3 changes: 3 additions & 0 deletions dockerhost/docker-ip
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh

exec docker inspect --format '{{ .NetworkSettings.IPAddress }}' "$@"
3 changes: 3 additions & 0 deletions dockerhost/docker-pid
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh

exec docker inspect --format '{{ .State.Pid }}' "$@"
2 changes: 2 additions & 0 deletions dockerhost/docker-sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/bash
nsenter --target $(docker-pid $1) --mount --uts --ipc --net --pid
60 changes: 60 additions & 0 deletions dockerhost/start-dockerhosts
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/bin/bash

# Create a linux bridge between containers
brctl addbr br-em1

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious: we could use OVS for the outer connectivity too. Any specific reason to use a linux-bridge?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reason I can think of is that this relieves the host of one prerequisite of having ovs installed, as I think linux-bridge is part of all major Linux distros by default.

ip link set br-em1 up

num_nodes=1
if [ -z "$CONTIV_NODES" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this just be:

if [ -n $CONTIV_NODES ]; then
    num_nodes=$CONTIV_NODES
fi

then
num_nodes=1
else
num_nodes=$CONTIV_NODES
fi
echo $num_nodes

if [ -z "$GOPATH" ];
then
echo "GOPATH needs to be set"
exit
else
netplugin_path=$GOPATH
fi
cluster=""
first="true"
for i in `seq 1 $num_nodes`;
do
hostname="netplugin-node$i"
echo "Starting $hostname"
docker run -d -i -t --name $hostname --privileged -v /var/lib/docker -v $netplugin_path:/netplugin sachja/ubuntu_netplugin bash -c "/netplugin/dockerhost/start-service.sh & bash"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like sachja/ubuntu_netplugin is a pre-built container with many things in it. Would it be possible to add the Dockerfile for building this container in the checkin as well? It would be even better if the container image is built in this script if it doesn't exist.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to mount host's go-environment as well, like we do for vagrant-vms (checkout the Vagrantfile), this will come handy as we integrate these scripts with out test-environments. I think if you create a Dockerfile for the host, you can have all this set in there as well than having to pass through docker run

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a dockerfile and changed script to build the image at the beginning. So no reference to prebuilt container image now.
I will add the host's go enviroment and other mounts through dockerfile in next change when i try to run the system tests and unit tests in this environment.

nsenter -t $(docker-pid $hostname) -n hostname $hostname
ip link add $i-int type veth peer name $i-ext
brctl addif br-em1 $i-ext
ip link set netns $(docker-pid $hostname) dev $i-int
nsenter -t $(docker-pid $hostname) -n ip link set dev $i-int name eth2
nsenter -t $(docker-pid $hostname) -n ip link set eth2 up
if [ $i -gt "1" ]; then
cluster=$cluster",";
fi
docker exec $hostname service docker start
addr=$(docker-ip $hostname)
cluster=$cluster$hostname"=http://"$addr":2380"
first="false"
done

echo $cluster

for i in `seq 1 $num_nodes`;
do
hostname="netplugin-node$i"
echo "Starting etcd on $hostname"
addr=$(docker-ip $hostname)
docker exec $hostname etcd -name $hostname -data-dir /opt/etcd -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 -advertise-client-urls http://$addr:2379,http://$addr:4001 -initial-advertise-peer-urls http://$addr:2380 -listen-peer-urls http://$addr:2380 -initial-cluster $cluster -initial-cluster-state new &
done

for i in `seq 1 $num_nodes`;
do
hostname="netplugin-node$i"
echo "Starting netplugin on $hostname"
docker exec $hostname /netplugin/bin/netplugin -host-label host$i &
done
7 changes: 7 additions & 0 deletions dockerhost/start-service.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this file being used?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mapuri - this one seems used when container is run (this is the entry point) in dockerhosts.sh above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok...didn't notice that...

/etc/init.d/openvswitch-switch restart
while true
do
echo service is running >> service.log
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid a busy wait and continuous logging, I have seen folks do something like below. We also use it in our systemtests to start certain long lived containers. May be we can use it here as well:

/bin/bash -c 'mkfifo foo && < foo'

Also if we endup adding a Dockerfile, we can fold all this in as it's default entry-point

sleep 10
done