Sporadic CI failures due to tools/tracker/tests/python/BUILD:57 null error

[billonahill]

These appear in CI often:

____(06-05 23:51:26.457) [298 / 427] Still waiting for 25 jobs to complete:
      Running (standalone):
        PexPython heron/tools/explorer/tests/python/explorer_unittest.pex, 18 s
        PexPython heron/tools/tracker/src/python/heron-tracker.pex, 11 s
      Scheduling:
        PexPython heron/tools/tracker/tests/python/query_operator_unittest.pex, 57 s
        PexPython heron/tools/tracker/tests/python/query_unittest.pex, 57 s
        Extracting interface //third_party/java:aws-java-sdk, 55 s
        PexPython heron/tools/tracker/tests/python/topology_helpers_unittest.pex, 55 s
        PexPython heron/tools/tracker/tests/python/topology_unittest.pex, 55 s
        PexPython heron/tools/tracker/tests/python/tracker_unittest.pex, 55 s
        Compiling heron/tmaster/src/cpp/manager/stats-interface.cpp, 53 s
        Compiling third_party/zookeeper/empty.cc, 53 s
        Compiling heron/common/src/cpp/zookeeper/zkclient.cpp, 51 s
        ... 14 more jobs
____(06-05 23:51:41.436) [300 / 427] Still waiting for 25 jobs to complete:
      Running (standalone):
        PexPython heron/tools/tracker/tests/python/query_operator_unittest.pex, 14 s
        PexPython heron/tools/tracker/tests/python/query_unittest.pex, 11 s
      Scheduling:
        Extracting interface //third_party/java:aws-java-sdk, 70 s
        PexPython heron/tools/tracker/tests/python/topology_helpers_unittest.pex, 70 s
        PexPython heron/tools/tracker/tests/python/topology_unittest.pex, 70 s
        PexPython heron/tools/tracker/tests/python/tracker_unittest.pex, 70 s
        Compiling heron/tmaster/src/cpp/manager/stats-interface.cpp, 68 s
        Compiling third_party/zookeeper/empty.cc, 68 s
        Compiling heron/common/src/cpp/zookeeper/zkclient.cpp, 66 s
        Building tools/java/src/com/twitter/bazel/checkstyle/libutil.jar (1 source file) [for host], 66 s
        Compiling heron/statemgrs/src/cpp/statemgr/heron-zkstatemgr.cpp, 62 s
        ... 14 more jobs
ERROR: (06-05 23:51:45.168) /home/travis/build/twitter/heron/heron/tools/tracker/tests/python/BUILD:57:1: null failed: _pex failed: error executing command 
  (cd /home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron && \
  exec env - \
  bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex --disable-cache --entry-point pytest bazel-out/local_linux-fastbuild/bin/heron/tools/tracker/tests/python/query_unittest.pex bazel-out/local_linux-fastbuild/bin/heron/tools/tracker/tests/python/query_unittest.pex.manifest): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Traceback (most recent call last):
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/_pex.py", line 211, in <module>
    sys.exit(main())
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/_pex.py", line 162, in main
    resolver_options_builder, interpreter=interpreter)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/bin/pex.py", line 490, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 162, in build
    local_package = Package.from_href(context.fetch(package))
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/http.py", line 109, in fetch
    with contextlib.closing(self.open(link)) as in_fp:
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/execroot/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/http.py", line 121, in open
    return urllib_request.urlopen(link.url)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1076, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
socket.error: [Errno 104] Connection reset by peer
____(06-05 23:51:45.202) Building complete.
____(06-05 23:51:45.220) Elapsed time: 572.801s, Critical Path: 555.29s

travis_time:end:007c5441:start=1496706130304249888,finish=1496706705251469619,duration=574947219731
�[0K
�[31;1mThe command "scripts/travis/ci.sh" exited with 1.�[0m

Done. Your build exited with 1.

[nlu90]

+1

[maosongfu]

+1. It is happening now on master branch: https://travis-ci.org/twitter/heron/builds/243842085?utm_source=email&utm_medium=notification

[objmagic]

@maosongfu I think this is one is slightly different.

[kramasamy]

@nlu90 @maosongfu @billonahill - there could be a couple of issues.

• Internet could be spotty during the travis-ci run
• pyyaml is actually referred as PyYAML, which might not be issue

However, I have fixed pyyaml to be PyYAML in the build files (just to be sure). Ran it a few times and so far it is not able to reproduce.