Severe Docker 1.12.1 performance regression with DB2 images (~10x slower)

[malduarte]

Expected behavior

Performance equal or better than 1.12.0

Actual behavior

Docker 1.12.1 is much slower on pretty much all DB2 activity. Not sure if it is DB2 only, but I'm only experiencing it on DB2. For example empty database creation takes ~24 seconds on 1.12.0 but on 1.12.1 it takes 10 times more, around ~253 seconds

docker stats seems to indicate lower cpu usage and lower I/O activity on 1.12.1

Information

• Di276C02C2-7AEC-4A1B-9F2C-9E6DAD3241CB
• Create a Dockerfile with the following content

FROM ibmcom/db2express-c:latest

ENV DB2INST1_PASSWORD=password
ENV LICENSE=accept
USER root
RUN /bin/bash -c 'su - db2inst1 -c "db2start && db2 create database test"'

Steps to reproduce the behavior

1. Install 1.12.0
2. Pull official db2 image docker pull ibmcom/db2express-c
3. Make sure you're running 1.12.0 docker --version should output Docker version 1.12.0, build 8eab29e
4. Create an image with the docker file above with the time command time docker build -t test . Consider running docker stats while it is running
5. Record the output
6. Remove the recently created image docker rmi <imageid>
7. Install 1.12.1
8. Make sure you're running 1.12.1 docker --version should output Docker version 1.12.1, build 6f9534c
9. Create an image with the attached docker file with the time command time docker build -t test .. Consider running docker stats while it is running
10. Compare the times.

12:05 $ docker --version
Docker version 1.12.0, build 8eab29e
✔ ~/docker_bench
12:05 $ time docker build -t test .
Sending build context to Docker daemon 66.56 kB
Step 1 : FROM ibmcom/db2express-c:latest
 ---> 7aa154d9b73c
Step 2 : ENV DB2INST1_PASSWORD password
 ---> Using cache
 ---> 8abd69a10768
Step 3 : ENV LICENSE accept
 ---> Using cache
 ---> 271672f5dc75
Step 4 : USER root
 ---> Using cache
 ---> 3f3ab0f72686
Step 5 : RUN /bin/bash -c 'su - db2inst1 -c "db2start && db2 create database test"'
 ---> Running in bd1c14701b68
libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory
SQL1063N  DB2START processing was successful.
DB20000I  The CREATE DATABASE command completed successfully.
 ---> 6bb79e3c1592
Removing intermediate container bd1c14701b68
Successfully built 6bb79e3c1592

real    0m23.646s
user    0m0.011s
sys 0m0.014s


12:10 $ docker --version
Docker version 1.12.1, build 6f9534c
✔ ~/docker_bench
12:11 $ time docker build -t test .
Sending build context to Docker daemon 66.56 kB
Step 1 : FROM ibmcom/db2express-c:latest
 ---> 7aa154d9b73c
Step 2 : ENV DB2INST1_PASSWORD password
 ---> Using cache
 ---> 8abd69a10768
Step 3 : ENV LICENSE accept
 ---> Using cache
 ---> 271672f5dc75
Step 4 : USER root
 ---> Using cache
 ---> 3f3ab0f72686
Step 5 : RUN /bin/bash -c 'su - db2inst1 -c "db2start && db2 create database test"'
 ---> Running in ebbf3069fb53
libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory
SQL1063N  DB2START processing was successful.
DB20000I  The CREATE DATABASE command completed successfully.
 ---> 032da4506157
Removing intermediate container ebbf3069fb53
Successfully built 032da4506157

real    4m23.451s
user    0m0.010s
sys 0m0.015s

[dave-tucker]

ping @dsheets

[justincormack]

I am going to test if this is an issue just on Linux, as there is not much visible Docker for Mac specific stuff here (eg no volumes or network)

[malduarte]

@justincormack I didn't try Linux, but I did try with docker 1.12.1 on docker-machine (virtualbox driver). Works fine there for this particular test, virtually the same time as docker 1.12.0 for mac

[justincormack]

@malduarte yes it may be Docker for Mac specific, but it is not obvious why, it could be for example related to the Linux runtime or something...

[justincormack]

Sounds like @djs55 has an idea of what may have caused this

[justincormack]

So it appears this may be because in previous versions it was not fully flushing blocks on fsync (and nor was Docker toolbox), so when we fixed this to add the full flush through to the OSX filesystem it slowed down. This would probably only be visible in database workloads where fsync is being used a lot.

[JanBednarik]

I have similar issue with PostgreSQL. It's approximately 10x slower after Docker for Mac upgrade to 1.12.1

[malduarte]

@justincormack : That makes sense. I understand current sync behaviour should be the default (event with the slow performance). Correctness should trump fast.

That being said Is there some kind of configuration option to disable fsync? It would be a boom for ephemeral container use cases. Example: container is spun up for a CI build and then thrown away.

[djs55]

Thanks very much for the easy to follow repro instructions. I ran 2 experiments:

• unmodified 1.12.1: time reports 5 minutes
• remove the call to fsync: time reports 30 seconds

It seems the VM calls the virtio-blk implementation of flush in hyperkit approximately 25000 times. Each of these is implemented currently by an fsync(F_FULLFSYNC) to avoid writes being partially written or re-ordered over a power loss (as recommended by the Apple docs). Unfortunately each fsync(F_FULLFSYNC) seems to take about 10ms which accounts for the slowdown.

I agree that there are important use-cases where data persistence is less important than throughput, especially on developer setups or CI builds where containers are ephemeral. I'll investigate the possibility of a configuration option.