Skip to content

Commit 4e865e9

Browse files
vadymhlushko-mlnxmssonicbld
authored andcommitted
[memory_checker] Add a specific log message in a case when the docker service is not running. (#16018)
#### Why I did it To fix the logic introduced by [[memory_checker] Do not check memory usage of containers which are not created #11129](#11129). There could be a scenario before the reboot, where 1. The `docker service` has stopped 2. In a very short period of time, the monit service performs the `root@sonic:/home/admin# monit status container_memory_telemetry` In such scenario, the `memory_checker` script will throw an error to the syslog: ``` ERR memory_checker: Failed to retrieve the running container list from docker daemon! Error message is: 'Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))' ``` But, actually, this scenario is a correct behavior, because when the docker service is stopped, the Unix socket is destroyed and that is why we could see the `FileNotFoundError(2, 'No such file or directory'` exception in the syslog. #### How I did it Change the log severity to the warning and changed the return value. #### How to verify it It is really hard to catch the exact moment described in the `Why I did it` section. In order to check the logic: 1. Change the Unix socket path to non-existing in [/usr/bin/memory_checker](https://github.com/sonic-net/sonic-buildimage/blob/47742dfc2c0d1fa27198d69c9183ddc044e11b22/files/image_config/monit/memory_checker#L139) file on the switch. 2. Execute the `root@sonic:/home/admin# monit restart container_memory_telemetry` 3. Check the syslog for such messages: ``` WARNING memory_checker: Failed to retrieve the running container list from docker daemon! Error message is: 'Error while fetching server API version: ('Connection aborte d.', FileNotFoundError(2, 'No such file or directory'))' INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running! ```
1 parent 9a63b9d commit 4e865e9

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

files/image_config/monit/memory_checker

+5
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,11 @@ def get_running_container_names():
140140
running_container_list = docker_client.containers.list(filters={"status": "running"})
141141
running_container_names = [ container.name for container in running_container_list ]
142142
except (docker.errors.APIError, docker.errors.DockerException) as err:
143+
if not is_service_active("docker"):
144+
syslog.syslog(syslog.LOG_INFO,
145+
"[memory_checker] Docker service is not running. Error message is: '{}'".format(err))
146+
return []
147+
143148
syslog.syslog(syslog.LOG_ERR,
144149
"Failed to retrieve the running container list from docker daemon! Error message is: '{}'"
145150
.format(err))

0 commit comments

Comments
 (0)