[Help] Unable to get coverage for bash script

[Lilja]

Much like #45, i'm using shunit2 to test my main file called timelog. My unittests reside in test/unittest.sh, the unittests looks for a timelog command.

I'm getting output(short excerpt)

money_per_hour=140
currency=kr'
+++ grep -o 'project_id\ *\=\ *.*'
+++ cut -d = -f 2-
+++ awk '{$1=$1};1'
++ id=ts
++ printf '1: Test [ts]\n'
++ it=2
+ all_projects='1: Test [ts]'
++ echo '1: Test [ts]'
++ grep -o '\[.*\]'
+ project_ids_only='[ts]'

But the index.html does not report anything.

I'm using kcov with docker. The Dockerfile

FROM ubuntu:16.04
USER root
RUN apt-get update
RUN apt-get -y install git jq cmake g++ pkg-config zlib1g zlib1g-dev libcurl3-dev libelf-dev elfutils libdw-dev python libstdc++6-4.7-dev binutils-dev ruby curl
WORKDIR /tmp
RUN git clone https://github.com/SimonKagstrom/kcov
RUN cd kcov && mkdir build && cd build && cmake .. && make && make install && cd ..

RUN mkdir /tmp/timelog
RUN git clone https://github.com/lilja/timelog
RUN cd timelog/test && ./test_dep.sh && cd ..

RUN chmod +x /tmp/timelog/test/unittest.sh
RUN chmod +x timelog/bin/timelog

ENV PATH="$PATH:/tmp/timelog/bin"

WORKDIR /tmp/timelog/test

RUN mkdir /tmp/cov1

RUN kcov /tmp/cov1 ./unittest.sh

And finally, to run it.

    docker build . -t kcov_img && docker rm kcov_container && docker run --name kcov_container kcov_img ls && rm -r foo ; mkdir foo && docker cp kcov_container:/tmp/cov1 foo &&  firefox --new-tab foo/cov1/index.html

Which results in the screenshot taken above.

[SimonKagstrom]

Do you have a set +x somewhere in your scripts? Or perhaps setting of the PS4 environment variable? From the output, it looks like someone is changing PS4, which kcov uses to collect data.

[Lilja]

I haven't set set +x, but i think that maybe shunit2 might set something: set -x

Step 18/18 : RUN kcov /tmp/cov unittest.sh
 ---> Running in af0125df1dcd
ESC[91mrm: cannot remove 'dev/': No such file or directory
+ set -x
+ rm -r dev/
+ [[ '' = \-\v ]]
+ debug=
+ '[' -d dev ']'
+ mkdir dev/
ESC[0mESC[91m+ dir=/tmp/timelog/test/dev
ESC[0mESC[91m+ rm -r /tmp/timelog/test/dev/
ESC[0mESC[91m+ . shunit2-2.1.6/src/shunit2
ESC[0mESC[91m++ '[' -n '' ']'
++ SHUNIT_VERSION=2.1.6
++ SHUNIT_TRUE=0
++ SHUNIT_FALSE=1
++ SHUNIT_ERROR=2
++ SHUNIT_STRICT=0
ESC[0mESC[91m++ '[' -n '' ']'
++ __SHUNIT_ASSERT_MSG_PREFIX=ASSERT:
++ __SHUNIT_MODE_SOURCED=sourced

[SimonKagstrom]

Yes, that looks like it. You could try to use the --bash-method=DEBUG option to kcov, although I'm not sure if it will help in this case.

[SimonKagstrom]

The set +xstuff is #47 by the way.

[Lilja]

Hmm, I modified the tests so that shunit2 is never sourced so it will never run or configure any set mode. But it appears that set -x still appears. Is that kcov-magic?

Step 19/19 : RUN kcov /tmp/cov ./unittest.sh kcov
 ---> Running in 900eba8a49ee
+ set -x
+ [[ kcov = \-\v ]]
+ debug=
+ '[' -d dev ']'
+ rm -r dev/
+ mkdir dev/
++ echo /tmp/timelog/test/dev
+ dir=/tmp/timelog/test/dev
+ rm -r /tmp/timelog/test/dev/
+ '[' '' == kcov ']'
+ testFileSystem
+ touch foo
+ assertTrue 'Can not create files on filesystem' '[ -f foo ]'
+ :
+ rm foo
+ testCreateAndDeleteProject
+ createProjectTest 5
+ [[ 5 -eq 5 ]]
+ timelog --dev /tmp/timelog/test/dev create project
+ log_path=/root/.config/timelogs
+ log_level=0
+ VERSION=0.2.0
+ specify_project=n
+ get_all_projects
++ list_projects
+++ ls /root/.config/timelogs/def/
++ files=
++ it=1
+ all_projects=
+ [[ 4 -ge 1 ]]
+ logger_debug 'Processing '\''--dev'\'''
+ '[' 0 -ge 1 ']'

This still reports 0.0%

[Lilja]

I made the error even more trivial.

# trivial.sh
#!/bin/bash
echo "foo"

someFoo() {
  if [ "$1" = "hello" ]; then
    echo "Greetings"
  elif [ "$1" = "bye" ]; then
    echo "See you"
  else
    echo "Untested"
  fi
}
someFoo "hello"
someFoo "bye"

Step 20/20 : RUN kcov /tmp/cov ./trivial.sh
 ---> Running in 2e88293818c8
+ set -x
+ echo foo
+ someFoo hello
+ '[' hello = hello ']'
+ echo Greetings
+ someFoo bye
+ '[' bye = hello ']'
+ '[' bye = bye ']'
+ echo 'See you'
foo
Greetings
See you
 ---> 9236089beae1

Output:

[SimonKagstrom]

Then there is something in the docker environment I would guess - your trivial.sh example works fine when running from the command line here. Could you try it on your host system?

[Lilja]

I might be onto something. Here is the compilation of kcov
How important it DYNINST? Could NOT find DYNINST (missing: DYNINST_LIBRARIES DYNINST_INCLUDE_DIR)

Step 8/21 : RUN cd kcov && mkdir build && cd build && cmake .. && make && make install && cd ..
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found LIBBFD: /usr/lib/x86_64-linux-gnu/libbfd.so  
-- Could NOT find DYNINST (missing:  DYNINST_LIBRARIES DYNINST_INCLUDE_DIR)
-- Target architectures: x86_64
-- Checking for module 'zlib'
--   Found zlib, version 1.2.8
-- Checking for module 'libcurl'
--   Found libcurl, version 7.47.0
-- Found LibElf: /usr/lib/x86_64-linux-gnu/libelf.so  
-- Found ElfUtils: /usr/lib/x86_64-linux-gnu/libdw.so  
-- Configuring done
-- Generating done
-- Build files have been written to: /kcov/build
Scanning dependencies of target bash_execve_redirector
[  2%] Building C object src/CMakeFiles/bash_execve_redirector.dir/engines/bash-execve-redirector.c.o
[  4%] Linking C shared library libbash_execve_redirector.so
[  4%] Built target bash_execve_redirector
Scanning dependencies of target kcov_sowrapper
[  6%] Building C object src/CMakeFiles/kcov_sowrapper.dir/solib-parser/phdr_data.c.o
[  8%] Building C object src/CMakeFiles/kcov_sowrapper.dir/solib-parser/lib.c.o
[ 10%] Linking C shared library libkcov_sowrapper.so
[ 10%] Built target kcov_sowrapper
[ 12%] Generating version.c
[ 14%] Generating library.cc
[ 16%] Generating bash-redirector-library.cc
[ 18%] Generating python-helper.cc
[ 20%] Generating bash-helper.cc
[ 22%] Generating html-data-files.cc
Scanning dependencies of target kcov
[ 24%] Building CXX object src/CMakeFiles/kcov.dir/capabilities.cc.o
[ 26%] Building CXX object src/CMakeFiles/kcov.dir/collector.cc.o
[ 28%] Building CXX object src/CMakeFiles/kcov.dir/configuration.cc.o
[ 30%] Building CXX object src/CMakeFiles/kcov.dir/engine-factory.cc.o
[ 32%] Building CXX object src/CMakeFiles/kcov.dir/engines/bash-engine.cc.o
[ 34%] Building CXX object src/CMakeFiles/kcov.dir/engines/dyninst-file-format.cc.o
[ 36%] Building CXX object src/CMakeFiles/kcov.dir/engines/gcov-engine.cc.o
[ 38%] Building CXX object src/CMakeFiles/kcov.dir/engines/python-engine.cc.o
[ 40%] Building CXX object src/CMakeFiles/kcov.dir/filter.cc.o
[ 42%] Building CXX object src/CMakeFiles/kcov.dir/gcov.cc.o
[ 44%] Building CXX object src/CMakeFiles/kcov.dir/main.cc.o
[ 46%] Building CXX object src/CMakeFiles/kcov.dir/merge-file-parser.cc.o
[ 48%] Building CXX object src/CMakeFiles/kcov.dir/output-handler.cc.o
[ 50%] Building CXX object src/CMakeFiles/kcov.dir/parsers/bfd-disassembler.cc.o
[ 52%] Building CXX object src/CMakeFiles/kcov.dir/parser-manager.cc.o
[ 54%] Building CXX object src/CMakeFiles/kcov.dir/reporter.cc.o
[ 56%] Building CXX object src/CMakeFiles/kcov.dir/source-file-cache.cc.o
[ 58%] Building CXX object src/CMakeFiles/kcov.dir/utils.cc.o
[ 60%] Building CXX object src/CMakeFiles/kcov.dir/writers/cobertura-writer.cc.o
[ 62%] Building CXX object src/CMakeFiles/kcov.dir/writers/json-writer.cc.o
[ 64%] Building CXX object src/CMakeFiles/kcov.dir/writers/coveralls-writer.cc.o
[ 66%] Building CXX object src/CMakeFiles/kcov.dir/writers/html-writer.cc.o
[ 68%] Building CXX object src/CMakeFiles/kcov.dir/writers/sonarqube-xml-writer.cc.o
[ 70%] Building CXX object src/CMakeFiles/kcov.dir/writers/writer-base.cc.o
[ 72%] Building CXX object src/CMakeFiles/kcov.dir/engines/clang-coverage-engine.cc.o
[ 74%] Building CXX object src/CMakeFiles/kcov.dir/engines/ptrace.cc.o
[ 76%] Building CXX object src/CMakeFiles/kcov.dir/engines/kernel-engine.cc.o
[ 78%] Building CXX object src/CMakeFiles/kcov.dir/parsers/elf.cc.o
[ 80%] Building CXX object src/CMakeFiles/kcov.dir/parsers/elf-parser.cc.o
[ 82%] Building CXX object src/CMakeFiles/kcov.dir/parsers/dwarf.cc.o
[ 84%] Building CXX object src/CMakeFiles/kcov.dir/solib-handler.cc.o
[ 86%] Building C object src/CMakeFiles/kcov.dir/solib-parser/phdr_data.c.o
[ 88%] Building CXX object src/CMakeFiles/kcov.dir/library.cc.o
[ 90%] Building CXX object src/CMakeFiles/kcov.dir/bash-redirector-library.cc.o
[ 92%] Building CXX object src/CMakeFiles/kcov.dir/python-helper.cc.o
[ 94%] Building CXX object src/CMakeFiles/kcov.dir/bash-helper.cc.o
[ 96%] Building CXX object src/CMakeFiles/kcov.dir/html-data-files.cc.o
[ 98%] Building C object src/CMakeFiles/kcov.dir/version.c.o
[100%] Linking CXX executable kcov
[100%] Built target kcov
[  4%] Built target bash_execve_redirector
[ 10%] Built target kcov_sowrapper
[100%] Built target kcov
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/bin/kcov
-- Set runtime path of "/usr/local/bin/kcov" to ""
-- Installing: /usr/local/share/man/man1/kcov.1

[SimonKagstrom]

No, dyninst is only important for compiled code, and optional there as well. Are you able to try it locally, and does it behave the same way then?

[Lilja]

I got it to work locally, trivial.sh get's some coverage.

It sucks because I added every package i manually installed to the Dockerfile but it still gives me the 0.0% error. I'm currently diffing the installed packages between the container and my machine but it feels like finding a needle in a haystack.

[SimonKagstrom]

You could try running a printenv before the script to compare the environments. Maybe the docker environment has something which causes this?

[Lilja]

printenv had nothing interesting. Kcov doesn't have some verbose mode that would show verbose information etc?

[SimonKagstrom]

Well, it has --debug=15, but it shouldn't make much difference for bash scripts.

I'm not really sure how to tackle this. Perhaps you could try to manually run the script (without kcov) with PS4=kcov@${BASH_SOURCE}@${LINENO}@ set in the environment. You should then get lines like these instead of the "+" lines you have above.

[Lilja]

Closing, i'll run this on ubuntu 12 for now. Even if it's end of life'd.

These packages compile kcov on ubuntu 16 but produces the 0.0% bug.

RUN apt-get install -y pkg-config                                                                                                                                   
RUN apt-get install -y binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev cmake git python
RUN apt-get install -y curl

[SimonKagstrom]

Well Ubuntu 16.04 isn't the problem as such: I run it here without problems on the trivial.sh script, so there has to be something with the docker setup I think.

Anyway, if you're OK with Ubuntu 12, I guess that qualifies as a valid workaround!

[congma]

Is there any update on this problem other than the downgrade workround?

I'm having the same problem with bash. Outside the Docker container, everything works fine with the default PS4 method, on Linux and MacOS. Inside the container however, the coverage is always zero, and the output begins with the + character.

I'm running kcov on GitLab runner, by the way.

[SimonKagstrom]

@congma: I think this might be related to Issue #234. Bash upstream (or perhaps Ubuntu) has fixed a security issue which disallows setting PS4 when running as root. See the diff in the Ubuntu package: https://launchpad.net/ubuntu/+source/bash/4.3-7ubuntu1.7

And I guess in the docker container, you will run as root and then you probably get this behavior. A possible workaround is to use --bash-method=DEBUG, although I'm not sure.

[congma]

AFAIK, that upstream (CVE-2016-7543) bug is only triggered when Bash runs a setuid-root executable. However, I observe the strange behaviour reported by the OP when running kcov on Bash scripts with normal user/group/permissions and not as root, so perhaps the problem lies elsewhere. It's very perplexing and for now I'm also resorting to an old Docker image, ragnaroek/kcov built from Debian Jessie, just to run the coverage report.

[SimonKagstrom]

Well, the bug might be for that, but I believe the Ubuntu fix just checks for the root user:

+===================================================================
+--- bash-4.3.orig/variables.c	2017-05-16 07:42:30.000000000 -0400
++++ bash-4.3/variables.c	2017-05-16 07:43:09.292966687 -0400
+@@ -495,7 +495,11 @@ initialize_shell_variables (env, privmod
+ #endif
+       set_if_not ("PS2", secondary_prompt);
+     }
+-  set_if_not ("PS4", "+ ");
++
++  if (current_user.euid == 0)
++    bind_variable ("PS4", "+ ", 0);
++  else
++    set_if_not ("PS4", "+ ");
+ 
+   /* Don't allow IFS to be imported from the environment. */
+   temp_var = bind_variable ("IFS", " \t\n", 0);```

Anyway, if you see this for normal programs as non-root, it might be some entirely other issue.