Add support for upgrade with custom https repos #1106

dkubek · 2023-08-02T09:35:56Z

Currently, upgrades with custom repositories hosted on private https servers are not functional. The reason is that the server certificates are unavailable in the upgrade container as they are not copied from the source system; therefore, the upgrade fails when trying to access the custom repository.

This change fixes the issue by copying all files from the /etc/pki folder of the source system, while respecting any new files installed by packages into the container. If a conflict arises a file installed by a package into a container is preferred. Any broken symlinks are skipped and ignored.

Jira (internal): OAMG-6388
BZ (internal): RHBZ: 2040706

github-actions · 2023-08-02T09:36:07Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please @oamg/developers to notify leapp developers of the review request
/packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp from PR#42, use /packit test oamg/leapp#42

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and latest upstream leapp build as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-sst to schedule sst tests using this pr build and latest upstream leapp build as artifacts
/rerun-sst 42 to schedule sst tests using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

fernflower · 2023-08-09T10:02:23Z

/rerun

fernflower · 2023-08-09T10:05:50Z

@dkubek if you rebase against master you should get meaningful packit test run (without Pipeline timeout exceeded)

github-actions · 2023-08-09T10:07:09Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/6259949

github-actions · 2023-08-09T10:17:51Z

Testing Farm request for RHEL-7.9-rhui/6259949 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-08-09T10:28:16Z

Testing Farm request for RHEL-8.6-rhui/6259949 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

matejmatuska · 2023-08-17T11:20:12Z

/rerun

github-actions · 2023-08-17T11:24:25Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/6311591

matejmatuska · 2023-08-17T11:33:28Z

Tests

RHUI

Since the patch touches RHUI code, I reproduced the issue when the _copy_certificates function is unconditional and I verified that with this patch on RHUI on Azure 8->9 IPU it's working as expected, the RHUI certs are no longer rewritten.

General

I also did a test in the internal vagrant box with IPU 7->8 and verified that:

new content from target userspace packages, such as /etc/pki/product-default/479.pem is there.
if there is a file that is provided by both source and target system packages, such as RPM GPG keys in /etc/pki/rpm-gpg/, the file from target system package is prefered.
the files from source system that aren't in the target system packages, such as custom certificates, are also present in the target userspace.

Integration

The patch also passes all the testcases in the yet unmerged integration tests (see Jira task for those).

github-actions · 2023-08-17T11:45:34Z

Testing Farm request for RHEL-8.6-rhui/6311591 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-08-17T11:51:44Z

Testing Farm request for RHEL-7.9-rhui/6311591 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

matejmatuska · 2023-08-21T09:50:13Z

Tests

RHUI

Since the patch touches RHUI code, I reproduced the issue when the _copy_certificates function is unconditional and I verified that with this patch on RHUI on Azure 8->9 IPU it's working as expected, the RHUI certs are no longer rewritten.

General

I also did a test in the internal vagrant box with IPU 7->8 and verified that:

new content from target userspace packages, such as /etc/pki/product-default/479.pem is there.

if there is a file that is provided by both source and target system packages, such as RPM GPG keys in /etc/pki/rpm-gpg/, the file from target system package is prefered.

the files from source system that aren't in the target system packages, such as custom certificates, are also present in the target userspace.

Integration

The patch also passes all the testcases in the yet unmerged integration tests (see Jira task for those).

I reran the prepared test with the lastest patch and all cases passed.

Currently, upgrades with custom repositories hosted on private https servers are not functional. The reason is that the server certificates are unavailable in the upgrade container as they are not copied from the source system; therefore, the upgrade fails when trying to access the custom repository. This change fixes the issue by copying all files from the ``/etc/pki`` folder of the source system, while respecting any new files installed by packages into the container. If a conflict arises a file installed by a package into a container is preferred. Any broken symlinks are skipped and ignored.

matejmatuska

LGTM, for completeness I will wait for the checks to complete and merge

## Packaging - Requires leapp-framework 5.0 ## Upgrade handling ### Fixes - Add el8toel9 actor to handle directory -> symlink with ruby IRB. (oamg#1076) - Do not try to update GRUB core on IBM Z systems (oamg#1117) - Fix failing upgrades with devtmpfs file systems specified in FSTAB (oamg#1090) - Fix the calculation of the required free space on each partitions/volume for the upgrade transactions (oamg#1097) - Fix the generation of the report about hybrid images (oamg#1064) - Handle correctly the installed certificates to allow upgrades with custom repositories using HTTPs with enabled SSL verification (oamg#1106) - Minor improvements and fixes of various reports (oamg#1066, oamg#1067, oamg#1085) - Update error messages about leapp data files to inform user how to obtain valid data files (oamg#1121) - Update links in various reports (oamg#1062, oamg#1086) - Update the repomap data to cover changed repoids in RHUI Azure (oamg#1087) - [IPU 7 -> 8] Fix false positive report about invalid symlinks on RHEL 7 (oamg#1052) - [IPU 8 -> 9] Inhibit the upgrade when unsupported x86-64 microarchitecture is detected (oamg#1059) ### Enhancements - Include updated leapp data files in the RPM (oamg#1046, oamg#1092, oamg#1119) - Update the set of supported upgrade paths (oamg#1077): - RHEL with SAP HANA 7.9 -> 8.6, 8.8 (default: 8.6) - RHEL with SAP HANA 8.8 -> 9.2 - Introduce new upgrade paths: - RHEL 7.9 -> 8.9 (default) - RHEL 8.9 -> 9.3 - Correctly update grub2 when /boot resides on multiple devices aggregated in RAID (oamg#1093, oamg#1115) - Enable upgrades for machines using RHUI on AlibabaCloud (oamg#1088) - Introduce possibility to add kernel drivers to initramfs (oamg#1081) - Redesign handling of information about kernel (booted and target) in preparation for new changes in RHEL 9 (oamg#1107) - Redesign source system overlay to use disk images backed by sparse files to optimize disk space consumption (oamg#1097, oamg#1103) - Requires leapp-framework 5.0 (oamg#1061, oamg#1116) - Use new leapp CLI API which provides better report summary output (oamg#1061, oamg#1116) - [IPU 8 -> 9] Detect and report use of deprecated Xorg drivers (oamg#1078) - [IPU 8 -> 9] Introduce IPU for systems with FIPS enabled (oamg#1053) ## Additional changes interesting for devels - Deprecated `GrubInfo.orig_device_name` field in the `GrubInfo` model (replaced by `GrubInfo.orig_devices`) (oamg#1093) - Deprecated `InstalledTargetKernelVersion` model (replaced by `InstalledTargetKernelInfo`) (oamg#1107) - Deprecated `leapp.libraries.common.config.version.is_rhel_realtime` (check the type in msg `KernelInfo`, field `type`) (oamg#1107) - Deprecated `leapp.libraries.common.grub.get_grub_device()` (replaced by `leapp.libraries.common.grub.get_grub_devices()`) (oamg#1093) - Introduced new devel envar LEAPP_DEVEL_KEEP_DISK_IMGS=1 to skip the removal of the created disk images for OVL. That's sometimes handy for the debugging. (oamg#1097)

## Packaging - Requires leapp-framework 5.0 ## Upgrade handling ### Fixes - Add el8toel9 actor to handle directory -> symlink with ruby IRB. (#1076) - Do not try to update GRUB core on IBM Z systems (#1117) - Fix failing upgrades with devtmpfs file systems specified in FSTAB (#1090) - Fix the calculation of the required free space on each partitions/volume for the upgrade transactions (#1097) - Fix the generation of the report about hybrid images (#1064) - Handle correctly the installed certificates to allow upgrades with custom repositories using HTTPs with enabled SSL verification (#1106) - Minor improvements and fixes of various reports (#1066, #1067, #1085) - Update error messages about leapp data files to inform user how to obtain valid data files (#1121) - Update links in various reports (#1062, #1086) - Update the repomap data to cover changed repoids in RHUI Azure (#1087) - [IPU 7 -> 8] Fix false positive report about invalid symlinks on RHEL 7 (#1052) - [IPU 8 -> 9] Inhibit the upgrade when unsupported x86-64 microarchitecture is detected (#1059) ### Enhancements - Include updated leapp data files in the RPM (#1046, #1092, #1119) - Update the set of supported upgrade paths (#1077): - RHEL with SAP HANA 7.9 -> 8.6, 8.8 (default: 8.6) - RHEL with SAP HANA 8.8 -> 9.2 - Introduce new upgrade paths: - RHEL 7.9 -> 8.9 (default) - RHEL 8.9 -> 9.3 - Correctly update grub2 when /boot resides on multiple devices aggregated in RAID (#1093, #1115) - Enable upgrades for machines using RHUI on AlibabaCloud (#1088) - Introduce possibility to add kernel drivers to initramfs (#1081) - Redesign handling of information about kernel (booted and target) in preparation for new changes in RHEL 9 (#1107) - Redesign source system overlay to use disk images backed by sparse files to optimize disk space consumption (#1097, #1103) - Requires leapp-framework 5.0 (#1061, #1116) - Use new leapp CLI API which provides better report summary output (#1061, #1116) - [IPU 8 -> 9] Detect and report use of deprecated Xorg drivers (#1078) - [IPU 8 -> 9] Introduce IPU for systems with FIPS enabled (#1053) ## Additional changes interesting for devels - Deprecated `GrubInfo.orig_device_name` field in the `GrubInfo` model (replaced by `GrubInfo.orig_devices`) (#1093) - Deprecated `InstalledTargetKernelVersion` model (replaced by `InstalledTargetKernelInfo`) (#1107) - Deprecated `leapp.libraries.common.config.version.is_rhel_realtime` (check the type in msg `KernelInfo`, field `type`) (#1107) - Deprecated `leapp.libraries.common.grub.get_grub_device()` (replaced by `leapp.libraries.common.grub.get_grub_devices()`) (#1093) - Introduced new devel envar LEAPP_DEVEL_KEEP_DISK_IMGS=1 to skip the removal of the created disk images for OVL. That's sometimes handy for the debugging. (#1097)

dkubek force-pushed the https_repos branch from 66995c1 to 51155a0 Compare August 9, 2023 10:03

dkubek force-pushed the https_repos branch from 51155a0 to d552289 Compare August 9, 2023 10:18

dkubek force-pushed the https_repos branch from d552289 to 3f02380 Compare August 13, 2023 17:53

dkubek changed the title ~~Draft: Add support for upgrade with custom https repos~~ Add support for upgrade with custom https repos Aug 13, 2023

pirat89 added this to the 8.9/9.3 milestone Aug 15, 2023

pirat89 self-assigned this Aug 17, 2023

matejmatuska self-assigned this Aug 17, 2023

pirat89 requested a review from matejmatuska August 21, 2023 09:16

dkubek force-pushed the https_repos branch 2 times, most recently from 5d0ece8 to aefbe25 Compare August 21, 2023 10:51

dkubek force-pushed the https_repos branch from aefbe25 to e054f0f Compare August 21, 2023 11:43

matejmatuska approved these changes Aug 21, 2023

View reviewed changes

matejmatuska merged commit 6fdb5b0 into oamg:master Aug 22, 2023

matejmatuska added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Aug 22, 2023

pirat89 mentioned this pull request Aug 23, 2023

Release 0.19.0 #1122

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for upgrade with custom https repos #1106

Add support for upgrade with custom https repos #1106

Uh oh!

dkubek commented Aug 2, 2023

Uh oh!

github-actions bot commented Aug 2, 2023

Uh oh!

fernflower commented Aug 9, 2023

Uh oh!

fernflower commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

matejmatuska commented Aug 17, 2023

Uh oh!

github-actions bot commented Aug 17, 2023

Uh oh!

matejmatuska commented Aug 17, 2023

Uh oh!

github-actions bot commented Aug 17, 2023

Uh oh!

github-actions bot commented Aug 17, 2023

Uh oh!

matejmatuska commented Aug 21, 2023

Tests

RHUI

General

Integration

Uh oh!

matejmatuska left a comment •

edited

Loading

Uh oh!

Uh oh!

Add support for upgrade with custom https repos #1106

Add support for upgrade with custom https repos #1106

Uh oh!

Conversation

dkubek commented Aug 2, 2023

Uh oh!

github-actions bot commented Aug 2, 2023

Thank you for contributing to the Leapp project!

Uh oh!

fernflower commented Aug 9, 2023

Uh oh!

fernflower commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

matejmatuska commented Aug 17, 2023

Uh oh!

github-actions bot commented Aug 17, 2023

Uh oh!

matejmatuska commented Aug 17, 2023

Tests

RHUI

General

Integration

Uh oh!

github-actions bot commented Aug 17, 2023

Uh oh!

github-actions bot commented Aug 17, 2023

Uh oh!

matejmatuska commented Aug 21, 2023

Tests

RHUI

General

Integration

Uh oh!

matejmatuska left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matejmatuska left a comment •

edited

Loading