hostcfgd: Handle missed tacacs updates between load & listen

[renukamanavalan]

Why I did it

The time gap between last config load & db-listen seem to have increased.
Any config updates that occurred in this gap gets missed by db-listen.
This could miss updating /etc/pam.d/common-auth-sonic

How I did it

Add a one shot timer, just before db-listen. The timer will fire after the subscribe is done
When the timer fires, reload tacacs & aaa

How to verify it

A tough repro. So you may explicitly add some sleep just before listen.
Make sure the sleep is at least 10 seconds less than timer.
Write a log just before sleep.
restart hostcfgd and watch syslog for the added log line
Upon this log, make some tacacs updates that would have some change in
/etc/pam.d/common-auth-sonic.
Watch for changes in that file

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

[arlakshm]

Do we need this change in master as well ?

[arlakshm]

iso of global_lock, can we move this lock to the AaaCfg() ? To prevent another classes in this file from using this lock

[renukamanavalan]

If anyone else (another subscriber) gets a need, they could use this.

[abdosi]

@renukamanavalan and @yxieca : when the issue is seen the steps are:-

a) load minigraph
b) config tacacs * commands

Basically delay bewteen restart hostcfgd and config command is 10-15 seconds. I think if we can randomize delay we might be able to repro this

Restart of hostcfgd service as part of load minigraph

Config save after load minigrpah

Tacacs configuration

[renukamanavalan]

@renukamanavalan and @yxieca : when the issue is seen the steps are:-

a) load minigraph
b) config tacacs * commands

Basically delay bewteen restart hostcfgd and config command is 10-15 seconds. I think if we can randomize delay we might be able to repro this

Restart of hostcfgd service as part of load minigraph
Config save after load minigrpah

Tacacs configuration

Managed a repro with following script. I could repro reliably in two systems. In one case, the pause time of 5 seconds worked, but in another it took 11 seconds. Again this is pretty dependent on many other factors. With help of load_minigraph, we could put control plane on high load, which helped get a distinct pause between load & listen.

for ((;;))
do
    tsleep=1
    tinterval=1

    for ((i=0; i<16; i++))
    do
        for ((j=0; j<5; j++))
        do
            config load_minigraph -y
            sleep 30s
            systemctl restart hostcfgd

            # variable sleep since hostcfgd restart
            sleep ${tsleep}

            # Update config
            config tacacs passkey xxxx
            config tacacs  authtype login
            config aaa authentication failthrough enable
            config aaa authentication login tacacs+
            echo "tacacs configured; sleep 30s before testing pam file"

            # Pause to allow hostcfgd to react
            sleep 30s

            # Check pam.d conf file
            n=$(grep testing123 /etc/pam.d/common-auth-sonic | wc -l)
            if [ $n -ne 1 ]; then
                echo "$(date): sleep little longer"
                sleep 30s
                n=$(grep testing123 /etc/pam.d/common-auth-sonic | wc -l)
                if [ $n -ne 1 ]; then
                    echo "$(date): Failure encountered"
                    exit -2
                fi
            fi
            echo "$(date): Run i=$i j=$j tsleep=${tsleep} complete"
        done
        tsleep=$((tsleep + tinterval))
    done
done

[abdosi]

do we don't need this code anymore ? all these changes looks unrelated to this fix.
Can we have separate PR for reason of these changes or at least can we add to PR description

[renukamanavalan]

We don't need this code anymore. The whole purpose of this code is to keep a local cache of tacacs & aaa. Maintain the cache on updates and use the cache. We don't do caching anymore.

We get it when we need it. This handles the missed-updates in simple manner and btw, we don't gain any by keeping duplicate cache in mem.