Skip to content

Commit e63f47e

Browse files
author
maksymbelei95
authored
[warm-reboot] Fix failures of warm reboot on disconnect of ssh session (sonic-net#1529)
Starting the script in background mode and detaching the process from terminal session to prevent failures, caused by closing or sudden disconnecting of terminal session. Redirecting output of the script to the file to prevent failures, when the script tries to write an output to file descriptor of the nonexistent terminal session. Adding new parameter to script to be able to explicitly run the script in foreground mode with output to the terminal. Updating the command reference doc according to the changes. - What I did Resolves sonic-net#7127 Fixed failures of warm reboot, when the SSH session is being disconnected. As the script will now be executed in background mode by default, added parameter to explicitly run it as usual, in foreground mode. Updated command reference according to the changes. - How I did it By restarting the script in background mode with detaching it from the terminal session. All the output has redirected to file /var/log/warm-reboot.txt for warm-reboot case, or /var/log/fast-reboot.txt for fast-reboot, depends on REBOOT_TYPE. This will prevent crashes of the script in case, when it will try to write some data to the file descriptor of the disconnected terminal session. - How to verify it 1. Connect to the switch with SSH; 2. Execute sudo warm-reboot -v; 3. See the current progress of warm reboot with cat /var/log/warm-reboot.txt; 4. Close SSH connection before warm reboot finish; Warm reboot should finish successfully, in spite of status of the SSH session. - New command output (if the output of a command-line utility has changed) The script will be running in background detached mode with output to the file. The related log will be shown in terminal before restarting in background mode: admin@sonic:~$ sudo warm-reboot Detaching the process from the terminal session. Redirecting output to /var/log/warm-reboot.txt. All the usual logs will be written to warm-reboot.txt.
1 parent c05845d commit e63f47e

File tree

2 files changed

+19
-2
lines changed

2 files changed

+19
-2
lines changed

doc/Command-Reference.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -8944,7 +8944,7 @@ This command requires root privilege.
89448944
89458945
- Usage:
89468946
```
8947-
warm-reboot [-h|-?|-v|-f|-r|-k|-x|-c <control plane assistant IP list>|-s]
8947+
warm-reboot [-h|-?|-v|-f|-r|-k|-x|-c <control plane assistant IP list>|-s|-D]
89488948
```
89498949
89508950
- Parameters:
@@ -8958,6 +8958,7 @@ This command requires root privilege.
89588958
-c : specify control plane assistant IP list
89598959
-s : strict mode: do not proceed without:
89608960
- control plane assistant IP list.
8961+
-D : detached mode - closing terminal will not cause stopping reboot
89618962
```
89628963
89638964
- Example:

scripts/fast-reboot

+17-1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ LOG_SSD_HEALTH="/usr/local/bin/log_ssd_health"
2525
PLATFORM_FWUTIL_AU_REBOOT_HANDLE="platform_fw_au_reboot_handle"
2626
SSD_FW_UPDATE="ssd-fw-upgrade"
2727
TAG_LATEST=yes
28+
DETACH=no
29+
LOG_PATH="/var/log/${REBOOT_TYPE}.txt"
2830

2931
# Require 100M available on the hard drive for warm reboot temp files,
3032
# Size is in 1K blocks:
@@ -73,13 +75,14 @@ function showHelpAndExit()
7375
echo " -s : strict mode: do not proceed without:"
7476
echo " - control plane assistant IP list."
7577
echo " -t : Don't tag the current kube images as latest"
78+
echo " -D : detached mode - closing terminal will not cause stopping reboot"
7679

7780
exit "${EXIT_SUCCESS}"
7881
}
7982

8083
function parseOptions()
8184
{
82-
while getopts "vfidh?rkxc:s" opt; do
85+
while getopts "vfidh?rkxc:sD" opt; do #TODO "t" is missing
8386
case ${opt} in
8487
h|\? )
8588
showHelpAndExit
@@ -114,6 +117,9 @@ function parseOptions()
114117
t )
115118
TAG_LATEST=no
116119
;;
120+
D )
121+
DETACH=yes
122+
;;
117123
esac
118124
done
119125
}
@@ -487,8 +493,18 @@ then
487493
exit "${EXIT_FAILURE}"
488494
fi
489495
496+
497+
# re-run the script in background mode with detaching from the terminal session
498+
if [[ x"${DETACH}" == x"yes" && x"${ALREADY_DETACHED}" == x"" ]]; then
499+
echo "Detaching the process from the terminal session. Redirecting output to ${LOG_PATH}."
500+
ALREADY_DETACHED=yes $0 "$@" &>$LOG_PATH & disown %%
501+
exit $?
502+
fi
503+
504+
490505
check_conflict_boot_in_fw_update
491506
507+
492508
sonic_asic_type=$(sonic-cfggen -y /etc/sonic/sonic_version.yml -v asic_type)
493509
494510
# Check reboot type supported

0 commit comments

Comments
 (0)