fix(v3.8.x): rpi_boot.update_firmware: ensure /tmp folder exists when doing flash-kernel

[Bodong-Yang]

Introduction

Starting from otaclient v3.8.x, a new mechanism is used to replace the previous two-stage reboot mechanism for executing flash-kernel. This new mechanism will chroot into the standby slot, and call flash-kernel on standby slot directly to deploy new dtbs, kernel and initrd img.

However, current OTA image is configured to ignore with pattern /tmp, which means even the top-level /tmp folder will be ignored. When OTA just updates the standby slot, there will be no /tmp folder existed on it, causing flash-kernel failed due to flash-kernel needs /tmp folder during execution.

This PR fixes this problem by ensuring /tmp folder on standby slot.

Other changes:

1. now rpi_boot will wait for 6 seconds, and issue a logging before rebooting.

Tests

• test passes on directly executing the rpi_boot control flow.

https://tier4.atlassian.net/browse/RT4-15340

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/ota_metadata/legacy
__init__.py	11	0	100%
parser.py	341	40	88%	103, 162, 167, 203–204, 214–215, 218, 230, 288, 298–301, 340–343, 423, 426, 434–436, 449, 458–459, 462–463, 675–676, 686, 688, 691, 718–720, 770, 773–775
types.py	84	13	84%	37, 40–42, 112–116, 122–125
src/ota_proxy
__init__.py	36	10	72%	59, 61, 63, 72, 81–82, 102, 104–106
__main__.py	7	7	0%	16–18, 20, 22–23, 25
_consts.py	17	0	100%
cache_control_header.py	68	4	94%	71, 91, 113, 121
cache_streaming.py	144	13	90%	211, 225, 229–230, 265–266, 268, 280, 349, 367–370
config.py	17	0	100%
db.py	74	18	75%	110, 116, 154, 160–161, 164, 170, 172, 193–200, 202–203
errors.py	5	0	100%
lru_cache_helper.py	47	2	95%	84–85
ota_cache.py	228	61	73%	70–71, 140, 151–152, 184–185, 202, 239–243, 247–249, 251, 253–260, 262–264, 267–268, 272–273, 277, 324, 332–334, 407, 434, 437–438, 460–462, 466–468, 474, 476–478, 483, 509–511, 546–548, 575, 581, 596
server_app.py	141	39	72%	79, 82, 88, 107, 111, 170, 179, 221–222, 224–226, 229, 234–235, 238, 241–242, 245, 248, 251, 254, 267–268, 271–272, 274, 277, 303–306, 309, 323–325, 331–333
utils.py	14	0	100%
src/otaclient
__init__.py	5	2	60%	17, 19
__main__.py	1	1	0%	16
log_setting.py	52	5	90%	53, 55, 64–66
src/otaclient/app
__main__.py	1	1	0%	16
configs.py	76	0	100%
errors.py	120	0	100%
interface.py	3	0	100%
main.py	46	5	89%	52–53, 75–77
ota_client.py	381	115	69%	80, 88, 109, 136, 138–139, 142–143, 145–146, 150, 154–155, 160–161, 167, 169, 207–210, 216, 220, 226, 345, 357–358, 360, 369, 372, 377–378, 381, 387, 389–393, 412–415, 418–429, 457–460, 506–507, 511, 513–514, 544–545, 554–561, 568, 571–577, 622–625, 633, 669–671, 676–678, 681–682, 684–685, 687, 745–746, 749, 757–758, 761, 772–773, 776, 784–785, 788, 799, 818, 845, 864, 882
ota_client_stub.py	393	109	72%	75–77, 79–80, 88–91, 94–96, 100, 105–106, 108–109, 112, 114–115, 118–120, 123–124, 127–129, 134–139, 143, 146–150, 152–153, 161–163, 166, 203–205, 210, 246, 271, 274, 277, 381, 405, 407, 431, 477, 534, 604–605, 644, 663–665, 671–674, 678–680, 687–689, 692, 696–699, 752, 841–843, 850, 880–881, 884–888, 897–906, 913, 919, 922–923, 927, 930
update_stats.py	104	9	91%	57, 103, 105, 114, 116, 125, 127, 148, 179
src/otaclient/boot_control
__init__.py	4	0	100%
_common.py	248	112	54%	74–75, 96–98, 114–115, 135–136, 155–156, 175–176, 195–196, 218–220, 235–236, 260–266, 287, 295, 313, 321, 340–341, 344–345, 368, 370–379, 381–390, 392–394, 413, 416, 424, 432, 448–450, 452–457, 550, 555, 560, 673, 677–678, 681, 689, 691–692, 718–719, 721–724, 729, 735–736, 739–740, 742, 749–750, 761–767, 777–779, 783–784, 787–788, 791, 797
_firmware_package.py	94	22	76%	83, 87, 137, 181, 187, 210–211, 214–219, 221–222, 225–230, 232
_grub.py	417	128	69%	217, 265–268, 274–278, 315–316, 323–328, 331–337, 340, 343–344, 349, 351–353, 362–368, 370–371, 373–375, 384–386, 388–390, 469–470, 474–475, 527, 533, 559, 581, 585–586, 601–603, 627–630, 642, 646–648, 650–652, 711–714, 739–742, 765–768, 780–781, 784–785, 820, 826, 846–847, 849, 861, 864, 867, 870, 874–876, 894–897, 925–928, 933–941, 946–954
_jetson_cboot.py	262	262	0%	20, 22–25, 27–29, 35–38, 40–41, 57–58, 60, 62–63, 69, 73, 132, 135, 137–138, 141, 148–149, 157–158, 161, 167–168, 176, 185–189, 191, 197, 200–201, 207, 210–211, 216–217, 219, 225–226, 229–230, 233–235, 237, 243, 248–250, 252–254, 259, 261–264, 266–267, 276–277, 280–281, 286–287, 290–294, 297–298, 303–304, 307, 310–314, 319–322, 325, 328–329, 332, 335–336, 339, 343–348, 352–353, 357, 360–361, 364, 367–370, 372, 375–376, 380, 383, 386–389, 391, 398, 402–403, 406–407, 413–414, 420, 422–423, 427, 429, 431–433, 436, 440, 443, 446–447, 449, 452, 460–461, 468, 478, 481, 489–490, 495–498, 500, 507, 509–511, 517–518, 522–523, 526, 530, 533, 535, 542–546, 548, 560–563, 566, 569, 571, 578, 582–583, 585–586, 588–590, 592, 594, 597, 600, 603, 605–606, 609–613, 617–619, 621, 629–633, 635, 638, 642, 645, 656–657, 662, 672, 675–683, 687–696, 700–709, 713, 715–717, 719–720, 722–723
_jetson_common.py	172	45	73%	132, 140, 288–291, 294, 311, 319, 354, 359–364, 382, 408–409, 411–413, 417–420, 422–423, 425–429, 431, 438–439, 442–443, 453, 456–457, 460, 462, 506–507
_jetson_uefi.py	397	271	31%	127–129, 134–135, 154–156, 161–164, 331, 449, 451–454, 458, 462–463, 465–473, 475, 487–488, 491–492, 495–496, 499–501, 505–506, 511–513, 517, 521–522, 525–526, 529–530, 534, 537–538, 540, 545–546, 550, 553–554, 559, 563–565, 569–571, 573, 577–580, 582–583, 605–606, 610–611, 613, 617, 621–622, 625–626, 633, 636–638, 641, 643–644, 649–650, 653–656, 658–659, 664, 666–667, 675, 678–681, 683–684, 686, 690–691, 695, 703–707, 710–711, 713, 716–720, 723, 726–730, 734–735, 738–743, 746–747, 750–753, 755–756, 763–764, 774–777, 780, 783–786, 789–793, 796–797, 800, 803–806, 809, 811, 816–817, 820, 823–826, 828, 834, 839–840, 859–860, 863, 871–872, 879, 889, 892, 899–900, 905–908, 916–919, 927–928, 940–943, 945, 948, 951, 959, 970–972, 974–976, 978–982, 987–988, 990, 1003, 1007, 1010, 1020, 1025, 1033–1034, 1037, 1041, 1043–1045, 1051–1052, 1057, 1065–1072, 1077–1085, 1090–1098, 1104–1106
_rpi_boot.py	295	141	52%	56, 59, 123–124, 128, 136–139, 153–156, 163–164, 166–167, 172–173, 176–177, 186–187, 225, 231–235, 238, 256–258, 262–264, 269–271, 275–277, 287–288, 291, 294, 296–297, 299–300, 303, 305–307, 309–311, 317, 320–321, 331–334, 342–347, 349, 351–352, 357–358, 365–371, 402, 404–407, 417–420, 424, 427–428, 430–435, 463–466, 485–488, 493, 496, 514–517, 522–530, 535–543, 560–563, 569–571, 574, 577
configs.py	55	0	100%
protocol.py	4	0	100%
selecter.py	41	29	29%	45–47, 50–51, 55–56, 59–61, 64, 66, 70, 78–80, 82–83, 85–86, 90, 92, 94–95, 97, 99–100, 102, 104
src/otaclient/configs
_common.py	8	0	100%
ecu_info.py	58	1	98%	108
proxy_info.py	52	2	96%	88, 90
src/otaclient/create_standby
__init__.py	12	5	58%	29–31, 33, 35
common.py	224	44	80%	62, 65–66, 70–72, 74, 78–79, 81, 127, 175–177, 179–181, 183, 186–189, 193, 204, 278–279, 281–286, 298, 335, 363, 366–368, 384–385, 399, 403, 425–426
interface.py	5	0	100%
rebuild_mode.py	97	9	90%	93–95, 107–112
src/otaclient_api/v2
api_caller.py	39	6	84%	45–47, 83–85
api_stub.py	17	0	100%
types.py	256	23	91%	86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 586
src/otaclient_common
__init__.py	34	8	76%	42–44, 61, 63, 69, 76–77
common.py	156	18	88%	47, 202, 205–207, 222, 229–231, 297–299, 309, 318–320, 366, 370
downloader.py	199	10	94%	107–108, 126, 153, 369, 424, 428, 516–517, 526
linux.py	61	15	75%	51–53, 59, 69, 74, 76, 108–109, 133–134, 190, 195–196, 198
logging.py	29	1	96%	55
persist_file_handling.py	118	18	84%	113, 118, 150–152, 163, 192–193, 228–232, 242–244, 246–247
proto_streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
proto_wrapper.py	398	48	87%	87, 165, 172, 184–186, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 707, 713, 760–763, 765, 803–805
retry_task_map.py	105	5	95%	158–159, 161, 181–182
typing.py	25	3	88%	69–70, 72
TOTAL	6340	1688	73%

Tests	Skipped	Failures	Errors	Time
217	0 💤	0 ❌	0 🔥	13m 23s ⏱️

[airkei]

LGTM, thank you 🎉 I understood that flash-kernel is used only in rasp pi.

[airkei]

(nits) I wonder, if we can use flush instead of waiting for constant time like the following code.

for handler in logger.handlers:
    handler.flush()

[Bodong-Yang]

Emmm currently our _LogTeeHandler at otaclient._logging in otaclient v3.8.x doesn't implement the flush API.

[Bodong-Yang]

So for v3.8.x, let's make the thing easier, just wait for 6 seconds.

And in the main branch, let's implement the flush API in the otaclient LogHandler 👍

[airkei]

Yes, this is nits comment. Please feel free to ignore as necessary. And I agree with waiting for constant value for this hot-fix.

Emmm currently our _LogTeeHandler at otaclient._logging in otaclient v3.8.x doesn't implement the flush API.

Oh..., I understood we need to implement it by ourselves. the original logging.Handler.flush() just pass the function...

[Bodong-Yang]

To be honest, I think OTA image should preserve the top-level folder that being defined in the Linux file system layout.

Currently the default ignore setting not only ignoring everything under /tmp, but also ignore the /tmp itself, see https://github.com/tier4/webauto-ci-buildspecs/blob/trunk/builder/ota/ignore_user_default.txt for more details.
(To preserve the /tmp itself, the entry should be /tmp/**/*).

But because we are in a hurry now, let's discuss this topic later.

[Bodong-Yang]

@airkei What do you think above?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
20.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[airkei]

To be honest, I think OTA image should preserve the top-level folder that being defined in the Linux file system layout.
Currently the default ignore setting not only ignoring everything under /tmp, but also ignore the /tmp itself, see https://github.com/tier4/webauto-ci-buildspecs/blob/trunk/builder/ota/ignore_user_default.txt for more details.
(To preserve the /tmp itself, the entry should be /tmp/**/*).
But because we are in a hurry now, let's discuss this topic later.

@Bodong-Yang Good points. I also consider if it should be fixed in image side. But this is from flash-kernel spec, thus I think this approach is also reasonable.
As you say, I think preserving top directory in OTA image (or by otaclient) is also reasonable approach for current sytem. But depending on package like ubuntu/raspbian(ubuntu-mate?), sometimes these packages might not follow the FHS spec. So we might need to be care what top directory should be preserved rather than preserving all of the top directories.

[Bodong-Yang]

@airkei Yes I agree 👍 , so I just do this hotfix in rpi_boot, which we know that /tmp should exist, and don't apply this fix to other boot control implementations. (And I also don't have plan to widely release otaclient v3.8.5, but only release it to rpi based ECU).

I think fixing the problem at OTA image side is more reasonable. Let's later talk to CI/CD team about this issue, and let them adjust the default ignore setting.