[Services] Restart BGP service upon unexpected critical process exit

[yozhao101]

• What I did
  Restart BGP service if one of critical processes running in BGP container exited or crashed abnormally.

• How I did it
  Generally I follow the framework created by Joe to implement this feature in BGP container.
  First, add supervisor-proc-exit-listener event listener option in Supervisord configuration file in BGP Docker container. Supervisord will read a list of critical processes for which to monitor the unexpected crashed and exited.
  Second, configure bgp.service to always auto-restart the service if it stops, with a delay of 30 seconds. Also set a rate limit of 3 restarts within 20 minutes (1200 seconds).

• How to verify it
  On your switch device, please use docker ps command to list all running docker containers.
  Then use docker exec -it container_id /bin/bash to login your container. Typing top command
  on the shell will display all the processes dynamically and you will spot the process id of one
  of the critical processes. Finally type the command kill -9 process_id to kill one process.
  After exiting the container, you can use watch -n 1 docker ps to dynamically see the restart
  of BGP container.

[jleveque]

Retest this please

[nikos-github]

@jleveque Until the diffs in #3486 are merged, these changes shouldn't be pushed.

[lguohan]

@nikos-github , I have concerns on PR #3486 exactly because this reason. We do want to have a unified mechanism to manage process restart. I do not agree that PR 3486 should be merged first. I suggest to find different approach to address watchfrr issue or integrated mode issue. It is frr issue, not sonic issue.

[nikos-github]

@lguohan We need to have a technical reason for requesting a unified approach. You haven't presented one and there isn't one. #3486 will address what this PR is trying to do for the frr services. Your concern is unfounded and isn't backed up/driven by data. More importantly, routing application should be driven independently and not have tight coupling to how sonic is managed - today it's frr, tomorrow it may be some other routing stack. Your proposal doesn't allow alternative routing stacks to easily come in as plug-and-play into sonic. I am also concerned that we are so easily dismissing the issues we have reported and overall we are refusing to address them for the community while at the same time so casually removing without proper review, functionality that existed in sonic.

[hasan-brcm]

Instead of restarting the whole docker, shouldn't zebra be restarted alone? Fpmsyncd can start the reconciliation with app-db when it reconnects to zebra.