jtop docker container doesn't start after reboot

[hpetty-itscg]

Hello all,

I'm not sure how to classify my issue so forgive me if I didn't tag it properly.

I have a docker container that utilizes the jtop (4.2.12) python library to collect data about the hardware while inference is running in other containers. I have /run/jtop.sock mounted and the container is set to restart always. The machine is also configured to start up all current running containers after reboot. When I start it up everything works as expected. Here is an example of what my docker compose file looks like.

services:
  jtop:
    container_name: jtop
    image: <name of my image>
    network_mode: host
    restart: always
    environment:
      PYTHONUNBUFFERED: 1
    volumes:
      - /run/jtop.sock:/run/jtop.sock

However, when the machine reboots, all of the containers startup fine except for this jtop container. When trying to manually restart the container I will get this message.

Error response from daemon: Cannot restart container jtop: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/jtop.sock" to rootfs at "/run/jtop.sock" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

After about 30 seconds or so if I restart the container again manually it will restart and everything will work perfectly again. My guess is that docker starts up well before jtop has a chance to fully initialize.

So my question is what options do I have to ensure this container will start up automatically after reboot? I attempted to set the service to start before docker per google's recommendation using:

sudo systemctl enable --before jtop.service docker.service

Which did not work and I can't event find documentation that this is an actual command but it doesn't throw an error or show anything in the console so I don't know what to believe. I also tried to set up health checks in the compose for this container like so:

services:
    jtop:
      container_name: jtop
      image: <name of my image>
      network_mode: host
      restart: always
      environment:
        PYTHONUNBUFFERED: 1
      volumes:
        - /run/jtop.sock:/run/jtop.sock
      healthcheck:
        test: test -S /run/jtop.sock
        interval: 10s
        timeout: 30s
        retries: 3
        start_period: 60s

This also did not work as it seems like it only works on running containers. So I'm at a loss here. What can I do to ensure docker waits to attempt starting this container until after jtop is fully initialized? Any input is appreciated.

Thanks!

[LucasJSch]

In my case, I was having the same issue, but because of the error jtop.core.exceptions.JtopException: Lost connection with jtop server.

I realized that start_http_server() calls threading.Thread(target=httpd.serve_forever), which I'm guessing might not end the server upon an exception. And on top of that, the server runs in a different thread than the main. So the main thread is not seeing the jtop-related exceptions, and the server is continuously serving despite the exceptions. So the main process is not being terminated on these kind of exceptions.

My solution was to read in my main program the collector's output. This way, the main program ends on an exception when having issues related to jtop:

if __name__ == '__main__':

    start_http_server(8000, addr='::')
    collector = CustomCollector()
    REGISTRY.register(collector)
    while True:
        time.sleep(1)
        # This generates an exception if jtop connection is lost
        next(collector.collect())

After the exception being thrown, my container restarted as expected.