Skip to content

[master] Errors in docker database supervisord init - flush_unused_database failed #11597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vaibhavhd opened this issue Aug 1, 2022 · 5 comments
Assignees
Labels
MSFT Triaged this issue has been triaged

Comments

@vaibhavhd
Copy link
Contributor

vaibhavhd commented Aug 1, 2022

Description

Database container init has errors in latest master image. Likely due to this error other database dependent services also fail to start.

I think the issue is because of this change: #11215

The change replaced swsssdk.SonicDBConfig with swsscommon.SonicDBConfig, but the function get_instancelist is not in swsscommon. Should that change to getInstanceList?

Steps to reproduce the issue:

  1. Install latest master image (tested so far on 2700 t0)
  2. Cold reboot to it.
  3. Check syslog for errors (attached here is an example).

Describe the results you received:

Errors seen in flush_unused_database script start. This is part of database container's supervisord conf:

https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-database/supervisord.conf.j2#L39

I think this prevent other database-dependent services from starting, as swss, syncd, bgp did not start after this error in database init.

Aug  1 18:37:15.087226 sonic INFO database#supervisord: flushdb Traceback (most recent call last):
Aug  1 18:37:15.087468 sonic INFO database#supervisord: flushdb   File "/usr/local/bin/flush_unused_database", line 14, in <module>
Aug  1 18:37:15.087598 sonic INFO database#supervisord: flushdb     instlists = swsscommon.SonicDBConfig.get_instancelist()
Aug  1 18:37:15.087703 sonic INFO database#supervisord: flushdb AttributeError: type object 'SonicDBConfig' has no attribute 'get_instancelist'
Aug  1 18:37:15.105005 sonic INFO database#supervisord 2022-08-01 18:37:15,104 INFO exited: flushdb (exit status 1; not expected)
Aug  1 18:37:22.607330 str2-msn2700-spy-1 NOTICE systemd[1]: switch state service is not active.

Describe the results you expected:

Database container init should not throw errors.
Critical services and containers should start.

Output of show version:


SONiC Software Version: SONiC.master.129002-cdd278611
Distribution: Debian 11.4
Kernel: 5.10.0-12-2-amd64
Build commit: cdd278611
Build date: Sun Jul 31 18:51:20 UTC 2022
Built by: AzDevOps@sonic-build-workers-001V01

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700
ASIC: mellanox
ASIC Count: 1
Serial Number: N/A
Model Number: N/A
Hardware Revision: A1
Uptime: 20:31:14 up  1:59,  1 user,  load average: 0.18, 0.17, 0.15
Date: Mon 01 Aug 2022 20:31:14

Output of show techsupport:

[syslog.log](https://github.com/Azure/sonic-buildimage/files/9237381/syslog.log)

Additional information you deem important (e.g. issue happens only occasionally):

@liuh-80
Copy link
Contributor

liuh-80 commented Aug 2, 2022

ACK, will create a fix for this issue soon

@liuh-80
Copy link
Contributor

liuh-80 commented Aug 2, 2022

Draft PR created, will request review after all validation passed: #11600

@liuh-80
Copy link
Contributor

liuh-80 commented Aug 2, 2022

PR ready for review: #11600
Manually verifyed with following command:

admin@vlab-01:~$ sudo docker exec -it database bash
root@vlab-01:/# docker-database-init.sh
/usr/local/lib/python3.9/dist-packages/supervisor/options.py:473: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
self.warnings.warn(
2022-08-02 09:11:55,805 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2022-08-02 09:11:55,806 INFO Set uid to user 0 succeeded
Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
For help, use /usr/local/bin/supervisord -h
root@vlab-01:/# /usr/local/bin/flush_unused_database
root@vlab-01:/#

liuh-80 added a commit that referenced this issue Aug 3, 2022
#### Why I did it
Fix docker-database flush_unused_database failed issue: #11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

#### How I did it
Change flush_unused_database code to use swsscommon API:
Change get_instancelist to getInstanceList.
Change get_dblist to getDbList.

#### How to verify it
Pass all E2E test.
Manually check syslog make sure error log not exist and swss, syncd, bgp service started.
Search code in Azure make sure there all similer case are fixed in this PR.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
Fix docker-database flush_unused_database failed issue: #11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)



Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>
@liuh-80
Copy link
Contributor

liuh-80 commented Aug 3, 2022

PR merged to master branch.

@zhangyanzhao zhangyanzhao added Triaged this issue has been triaged MSFT labels Aug 3, 2022
@zhangyanzhao
Copy link
Collaborator

@vaibhavhd please close this issue if the fix is verified. Thanks.

liuh-80 added a commit that referenced this issue Aug 10, 2022
Fix docker-database flush_unused_database failed issue: #11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

Change flush_unused_database code to use swsscommon API:
Change get_instancelist to getInstanceList.
Change get_dblist to getDbList.

Pass all E2E test.
Manually check syslog make sure error log not exist and swss, syncd, bgp service started.
Search code in Azure make sure there all similer case are fixed in this PR.

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

Fix docker-database flush_unused_database failed issue: #11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>
liuh-80 added a commit that referenced this issue Aug 12, 2022
Fix docker-database flush_unused_database failed issue: #11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

Change flush_unused_database code to use swsscommon API:
Change get_instancelist to getInstanceList.
Change get_dblist to getDbList.

Pass all E2E test.
Manually check syslog make sure error log not exist and swss, syncd, bgp service started.
Search code in Azure make sure there all similer case are fixed in this PR.

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

Fix docker-database flush_unused_database failed issue: #11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it

#### How I did it

#### How to verify it

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
skbarista pushed a commit to skbarista/sonic-buildimage that referenced this issue Aug 17, 2022
…) (sonic-net#11677)

Fix docker-database flush_unused_database failed issue: sonic-net#11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

Change flush_unused_database code to use swsscommon API:
Change get_instancelist to getInstanceList.
Change get_dblist to getDbList.

Pass all E2E test.
Manually check syslog make sure error log not exist and swss, syncd, bgp service started.
Search code in Azure make sure there all similer case are fixed in this PR.

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

Fix docker-database flush_unused_database failed issue: sonic-net#11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

Co-authored-by: liuh-80 <azureuser@liuh-dev-vm-02.5fg3zjdzj2xezlx1yazx5oxkzd.hx.internal.cloudapp.net>

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it

#### How I did it

#### How to verify it

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
liushilongbuaa pushed a commit to liushilongbuaa/sonic-buildimage that referenced this issue Sep 23, 2022
…se failed issue

#### Why I did it
Fix docker-database flush_unused_database failed issue: sonic-net#11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

#### How I did it
Change flush_unused_database code to use swsscommon API:
Change get_instancelist to getInstanceList.
Change get_dblist to getDbList.

#### How to verify it
Pass all E2E test.
Manually check syslog make sure error log not exist and swss, syncd, bgp service started.
Search code in Azure make sure there all similer case are fixed in this PR.

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205

#### Description for the changelog
Fix docker-database flush_unused_database failed issue: sonic-net#11597
When change flush_unused_database from use swsssdk to use swsscommon, get_instancelist() and get_dblist() name changed but not update.

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MSFT Triaged this issue has been triaged
Projects
None yet
Development

No branches or pull requests

3 participants