-
Notifications
You must be signed in to change notification settings - Fork 181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use docker containers as hosts for testing netplugin #37
Changes from 6 commits
ab59f37
db35897
11a861c
de6aea7
1fcea28
352cd59
e1548b5
53a1652
922bbdc
02d07fc
976b00c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
The multi node container networking can now be tested on a single VM by running docker inside docker. | ||
https://blog.docker.com/2013/09/docker-can-now-run-within-docker/ | ||
|
||
The outside docker containers act like physical hosts in our test and are connected using a standard linux bridge. Inside each "host container" we run a namespaced instance of docker, OVS , etcd and netplugin instance. One can now launch containers from within each "host containers" and use netplugin networking to connect them. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This doc deserves a reference from the main README. If this becomes the simplest way to try it out, we should replace this with Vagrant based configuration mentioned in the top level README.md There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, I agree with @jainvipin Actually can we move this README.md to docs/dockerhost.md and link it from the main README.md. That's how most of our docs are arranged. Also the scripts can be moved to scripts/dockerhost/ directory, as all the scripts etc stay there. I am thinking we can use/extend most of these scripts to support docker based test and demo environments (eventually getting rid of the heavier vagrant environment). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moving this to docs/Dockerhost.md and scripts to scripts/dockerhost. Added reference from main README.md . Working on adding a mode in system test to use dockerhost. Once this is stable we can decide if this should be default. |
||
The steps to launch docker hosts are : | ||
|
||
1. Compile the netplugin code | ||
2. Change user to root | ||
3. Set GOPATH to root of the netplugin code | ||
4. Add $GOPATH/dockerhost to PATH | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where is $GOPATH/dockerhost created? Did you mean $GOPATH/src/github.com/netplugin/dockerhost? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed dependency on GOPATH. Will start the script using make from the root directory the same way vagrant demo is started. |
||
5. Set CONTIV_NODES to required number of nodes | ||
6. Run start-dockerhosts | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we skip steps 4 above and import the path in start-dockerhosts script? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Need to specify, either here or in prerequisites, that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another pre-requisite is having docker installed on host as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added |
||
This will start CONTIV_NODES number of containers with docker image called ubuntu_netplugin which is just ubuntu image with docker and ovs installed. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think ubuntu image also needs to have etcd installed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added |
||
|
||
Now start a shell within any of the "host containers" using following convenient wrapper around nsenter : | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. convenient -> convenience |
||
``` | ||
docker-sh netplugin-node<x> | ||
``` | ||
|
||
Start netplugin, post netplugin config and launch containers inside the "host containers" the same way you do on VMs. | ||
Note : currently the demo is working only if config is posted before containers are started .. need to debug why the reverse is not working. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. interesting, I think one of our tests already tests this, may be something got broken. Or is it working for VMs? Can you file a issue to track this, if it's ok? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will debug and file an issue if its not dockerhost specific. |
||
|
||
To cleanup all the docker hosts and the virtual interfaces created do | ||
``` | ||
cleanup-dockerhosts | ||
``` | ||
|
||
Example for testing TwoHostMultiVlan you can do : | ||
|
||
1. Launch the two host containers | ||
|
||
``` | ||
export CONTIV_NODES=2 | ||
start-dockerhosts | ||
``` | ||
|
||
2. Load the netplugin configuration | ||
``` | ||
docker-sh netplugin-node1 | ||
/netplugin/bin/netdcli -cfg /netplugin/examples/two_hosts_multiple_vlans_nets.json | ||
``` | ||
|
||
3. Launch container1 on host1 | ||
|
||
``` | ||
docker-sh netplugin-node1 | ||
docker run -it --name=myContainer1 --hostname=myContainer1 ubuntu /bin/bash | ||
``` | ||
|
||
4. Launch container3 on host2 | ||
|
||
``` | ||
docker-sh netplugin-node2 | ||
docker run -it --name=myContainer3 --hostname=myContainer1 ubuntu /bin/bash | ||
``` | ||
|
||
5. Test connectivity between the containers using ping. Go to the shell for container1 | ||
``` | ||
root@myContainer1:/# ping -c3 11.1.2.2 | ||
PING 11.1.2.2 (11.1.2.2) 56(84) bytes of data. | ||
64 bytes from 11.1.2.2: icmp_seq=1 ttl=64 time=3.15 ms | ||
64 bytes from 11.1.2.2: icmp_seq=2 ttl=64 time=1.36 ms | ||
``` | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
|
||
#!/bin/bash | ||
|
||
num_nodes=1 | ||
if [ -z "$CONTIV_NODES" ]; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. could this just be:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
then | ||
num_nodes=1 | ||
else | ||
num_nodes=$CONTIV_NODES | ||
fi | ||
echo $num_nodes | ||
|
||
for i in `seq 1 $num_nodes`; | ||
do | ||
hostname="netplugin-node$i" | ||
echo "Cleaning $hostname" | ||
docker exec $hostname service docker stop | ||
docker rm -f $hostname | ||
ip link delete $i-ext | ||
done |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
#!/bin/bash | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this script being used? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We may need it if running many hosts as each need 2 loop devices |
||
for i in `seq 9 20`; | ||
do | ||
sudo /bin/mknod -m640 /dev/loop$i b 7 $i | ||
sudo /bin/chown root:disk /dev/loop$i | ||
done |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
#!/bin/sh | ||
|
||
exec docker inspect --format '{{ .NetworkSettings.IPAddress }}' "$@" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
#!/bin/sh | ||
|
||
exec docker inspect --format '{{ .State.Pid }}' "$@" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
#!/bin/bash | ||
nsenter --target $(docker-pid $1) --mount --uts --ipc --net --pid |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
#!/bin/bash | ||
|
||
# Create a linux bridge between containers | ||
brctl addbr br-em1 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Curious: we could use OVS for the outer connectivity too. Any specific reason to use a linux-bridge? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One reason I can think of is that this relieves the host of one prerequisite of having ovs installed, as I think linux-bridge is part of all major Linux distros by default. |
||
ip link set br-em1 up | ||
|
||
num_nodes=1 | ||
if [ -z "$CONTIV_NODES" ]; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. could this just be:
|
||
then | ||
num_nodes=1 | ||
else | ||
num_nodes=$CONTIV_NODES | ||
fi | ||
echo $num_nodes | ||
|
||
if [ -z "$GOPATH" ]; | ||
then | ||
echo "GOPATH needs to be set" | ||
exit | ||
else | ||
netplugin_path=$GOPATH | ||
fi | ||
cluster="" | ||
first="true" | ||
for i in `seq 1 $num_nodes`; | ||
do | ||
hostname="netplugin-node$i" | ||
echo "Starting $hostname" | ||
docker run -d -i -t --name $hostname --privileged -v /var/lib/docker -v $netplugin_path:/netplugin sachja/ubuntu_netplugin bash -c "/netplugin/dockerhost/start-service.sh & bash" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You might want to mount host's go-environment as well, like we do for vagrant-vms (checkout the Vagrantfile), this will come handy as we integrate these scripts with out test-environments. I think if you create a Dockerfile for the host, you can have all this set in there as well than having to pass through There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added a dockerfile and changed script to build the image at the beginning. So no reference to prebuilt container image now. |
||
nsenter -t $(docker-pid $hostname) -n hostname $hostname | ||
ip link add $i-int type veth peer name $i-ext | ||
brctl addif br-em1 $i-ext | ||
ip link set netns $(docker-pid $hostname) dev $i-int | ||
nsenter -t $(docker-pid $hostname) -n ip link set dev $i-int name eth2 | ||
nsenter -t $(docker-pid $hostname) -n ip link set eth2 up | ||
if [ $i -gt "1" ]; then | ||
cluster=$cluster","; | ||
fi | ||
docker exec $hostname service docker start | ||
addr=$(docker-ip $hostname) | ||
cluster=$cluster$hostname"=http://"$addr":2380" | ||
first="false" | ||
done | ||
|
||
echo $cluster | ||
|
||
for i in `seq 1 $num_nodes`; | ||
do | ||
hostname="netplugin-node$i" | ||
echo "Starting etcd on $hostname" | ||
addr=$(docker-ip $hostname) | ||
docker exec $hostname etcd -name $hostname -data-dir /opt/etcd -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 -advertise-client-urls http://$addr:2379,http://$addr:4001 -initial-advertise-peer-urls http://$addr:2380 -listen-peer-urls http://$addr:2380 -initial-cluster $cluster -initial-cluster-state new & | ||
done | ||
|
||
for i in `seq 1 $num_nodes`; | ||
do | ||
hostname="netplugin-node$i" | ||
echo "Starting netplugin on $hostname" | ||
docker exec $hostname /netplugin/bin/netplugin -host-label host$i & | ||
done |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
#!/bin/bash | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is this file being used? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @mapuri - this one seems used when container is run (this is the entry point) in dockerhosts.sh above. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ah, ok...didn't notice that... |
||
/etc/init.d/openvswitch-switch restart | ||
while true | ||
do | ||
echo service is running >> service.log | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To avoid a busy wait and continuous logging, I have seen folks do something like below. We also use it in our systemtests to start certain long lived containers. May be we can use it here as well: /bin/bash -c 'mkfifo foo && < foo' Also if we endup adding a Dockerfile, we can fold all this in as it's default entry-point |
||
sleep 10 | ||
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good edition... it will allow MAC users to natively test/experiment this. Did you try these on your laptop/mac's VM?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this on a ubuntu VM on my laptop. Not sure if we can run on native mac.