Include leapp data files in the RPM & repository #1046

pirat89 · 2023-02-17T12:00:37Z

In the past it was needed to obtain the data by

automatic download of data files from RH Insights
(cloud.redhat.com)
- which required access to the server & have the system registered
manual download from the article:
https://access.redhat.com/articles/3664871
which required to login to the portal, download the archive
and install its content manually on the system

Additional problem was with the syncing of data, as because of the separation of data from the code, people had ensure they are using the right combination of data and SW manually - in case the data has not been downloaded automatically from RH Insights.

Having the data in the RPM makes our lives easier so let's install them to /etc/leapp/files/ via the package directly.
Set /etc/leapp/files/* as configuration files to ensure that modified files will be backed up with *.rpmsave suffix as
expected for configuration files. This could be important in case users manually updated e.g. repomap.json file to reflect their internal settings of the satellite server.

The complete table how the configuration files are handled during
rpm installation/upgrade/... is here:

https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html

Keeping original functionality still available, so if people remove the files from the system, data can be still downloaded from the server automatically. It seems it does not make sense, but we could still use the possibility to download more up-to-date data from RH Insights.

Configure codespell to ignore .etc/leapp/files

github-actions · 2023-02-17T12:00:48Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please to notify leapp developers of review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-sst to schedule sst tests using this pr build and leapp*master* as artifacts
/rerun-sst 42 to schedule sst tests using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

pirat89 · 2023-02-17T12:44:20Z

@vinzenz surprise! :) I will later consider actually use of LFS. keeping in draft for the weekend

pirat89 · 2023-02-17T14:17:43Z

/rerun

github-actions · 2023-02-17T14:27:32Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5537550

github-actions · 2023-02-17T14:48:39Z

Testing Farm request for RHEL-8.6-rhui/5537550 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-17T14:55:33Z

Testing Farm request for RHEL-7.9-rhui/5537550 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-17T17:47:29Z

Testing Farm request for RHEL-8.6.0-Nightly/5537550 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-17T18:03:08Z

Testing Farm request for RHEL-8.7.0-Nightly/5537550 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-17T18:44:04Z

Testing Farm request for RHEL-7.9-ZStream/5537550 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-17T18:51:02Z

Testing Farm request for RHEL-7.9-ZStream/5537550 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

pirat89 · 2023-03-10T10:08:19Z

@oamg/developers The failed codespell test is false positive. The issue has been reported in upstream: codespell-project/codespell#2786

pirat89 · 2023-05-03T08:42:30Z

/packit build

pirat89 · 2023-05-03T09:22:07Z

@oamg/developers @oamg/qe Tested manually on el7 & el8.

install new packages, no config files present on the system before

[root@localhost ~]# find /etc/leapp/files
find: ‘/etc/leapp/files’: No such file or directory
[root@localhost ~]# dnf install "leapp-upgrade-*pr1046*" -y
....
Complete!
[root@localhost ~]# find /etc/leapp/files
/etc/leapp/files
/etc/leapp/files/device_driver_deprecation_data.json
/etc/leapp/files/pes-events.json
/etc/leapp/files/repomap.json

install new packages, config files already present on the system

[root@localhost files]# mkdir -p /etc/leapp/files
[root@localhost files]# get_data_files prod
...
[root@localhost files]# dnf install "leapp-upgrade-*pr1046*" -y
...
Complete!
[root@localhost files]# find /etc/leapp/files
/etc/leapp/files
/etc/leapp/files/device_driver_deprecation_data.json.rpmorig
/etc/leapp/files/device_driver_deprecation_data.json
/etc/leapp/files/pes-events.json.rpmorig
/etc/leapp/files/pes-events.json
/etc/leapp/files/repomap.json.rpmorig
/etc/leapp/files/repomap.json

I do not have now tests to show how the rpmsave files are created as it would require additional rpm with new data (tested before when I updated the data files, but as we want to proceed with these files anyway, I am not creating an extra build for it to show the behaviour in the comment. just believe the rpm functionality)

pirat89 · 2023-05-03T09:26:08Z

the changelog updated in advance

In the past it was needed to obtain the data by * automatic download of data files from RH Insights (cloud.redhat.com) - which required access to the server & have the system registered * manual download from the article: https://access.redhat.com/articles/3664871 which required to login to the portal, download the archive and install its content manually on the system Additional problem was with the syncing of data, as because of the separation of data from the code, people had ensure they are using the right combination of data and SW manually - in case the data has not been downloaded automatically from RH Insights. Having the data in the RPM makes our lives easier so let's install them to `/etc/leapp/files/` via the package directly. Set /etc/leapp/files/* as configuration files to ensure that modified files will be backed up with *.rpmsave suffix as expected for configuration files. This could be important in case users manually updated e.g. repomap.json file to reflect their internal settings of the satellite server. The complete table how the configuration files are handled during rpm installation/upgrade/... is here: https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html Keeping original functionality still available, so if people remove the files from the system, data can be still downloaded from the server automatically. It seems it does not make sense, but we could still use the possibility to download more up-to-date data from RH Insights. Configure codespell to ignore .etc/leapp/files

pirat89 · 2023-05-04T10:42:30Z

rebased against up-to-date master to see whether it resolves some of failed tests

Rezney

The failing tests are known.

## Packaging - Requires leapp-framework 5.0 ## Upgrade handling ### Fixes - Add el8toel9 actor to handle directory -> symlink with ruby IRB. (oamg#1076) - Do not try to update GRUB core on IBM Z systems (oamg#1117) - Fix failing upgrades with devtmpfs file systems specified in FSTAB (oamg#1090) - Fix the calculation of the required free space on each partitions/volume for the upgrade transactions (oamg#1097) - Fix the generation of the report about hybrid images (oamg#1064) - Handle correctly the installed certificates to allow upgrades with custom repositories using HTTPs with enabled SSL verification (oamg#1106) - Minor improvements and fixes of various reports (oamg#1066, oamg#1067, oamg#1085) - Update error messages about leapp data files to inform user how to obtain valid data files (oamg#1121) - Update links in various reports (oamg#1062, oamg#1086) - Update the repomap data to cover changed repoids in RHUI Azure (oamg#1087) - [IPU 7 -> 8] Fix false positive report about invalid symlinks on RHEL 7 (oamg#1052) - [IPU 8 -> 9] Inhibit the upgrade when unsupported x86-64 microarchitecture is detected (oamg#1059) ### Enhancements - Include updated leapp data files in the RPM (oamg#1046, oamg#1092, oamg#1119) - Update the set of supported upgrade paths (oamg#1077): - RHEL with SAP HANA 7.9 -> 8.6, 8.8 (default: 8.6) - RHEL with SAP HANA 8.8 -> 9.2 - Introduce new upgrade paths: - RHEL 7.9 -> 8.9 (default) - RHEL 8.9 -> 9.3 - Correctly update grub2 when /boot resides on multiple devices aggregated in RAID (oamg#1093, oamg#1115) - Enable upgrades for machines using RHUI on AlibabaCloud (oamg#1088) - Introduce possibility to add kernel drivers to initramfs (oamg#1081) - Redesign handling of information about kernel (booted and target) in preparation for new changes in RHEL 9 (oamg#1107) - Redesign source system overlay to use disk images backed by sparse files to optimize disk space consumption (oamg#1097, oamg#1103) - Requires leapp-framework 5.0 (oamg#1061, oamg#1116) - Use new leapp CLI API which provides better report summary output (oamg#1061, oamg#1116) - [IPU 8 -> 9] Detect and report use of deprecated Xorg drivers (oamg#1078) - [IPU 8 -> 9] Introduce IPU for systems with FIPS enabled (oamg#1053) ## Additional changes interesting for devels - Deprecated `GrubInfo.orig_device_name` field in the `GrubInfo` model (replaced by `GrubInfo.orig_devices`) (oamg#1093) - Deprecated `InstalledTargetKernelVersion` model (replaced by `InstalledTargetKernelInfo`) (oamg#1107) - Deprecated `leapp.libraries.common.config.version.is_rhel_realtime` (check the type in msg `KernelInfo`, field `type`) (oamg#1107) - Deprecated `leapp.libraries.common.grub.get_grub_device()` (replaced by `leapp.libraries.common.grub.get_grub_devices()`) (oamg#1093) - Introduced new devel envar LEAPP_DEVEL_KEEP_DISK_IMGS=1 to skip the removal of the created disk images for OVL. That's sometimes handy for the debugging. (oamg#1097)

## Packaging - Requires leapp-framework 5.0 ## Upgrade handling ### Fixes - Add el8toel9 actor to handle directory -> symlink with ruby IRB. (#1076) - Do not try to update GRUB core on IBM Z systems (#1117) - Fix failing upgrades with devtmpfs file systems specified in FSTAB (#1090) - Fix the calculation of the required free space on each partitions/volume for the upgrade transactions (#1097) - Fix the generation of the report about hybrid images (#1064) - Handle correctly the installed certificates to allow upgrades with custom repositories using HTTPs with enabled SSL verification (#1106) - Minor improvements and fixes of various reports (#1066, #1067, #1085) - Update error messages about leapp data files to inform user how to obtain valid data files (#1121) - Update links in various reports (#1062, #1086) - Update the repomap data to cover changed repoids in RHUI Azure (#1087) - [IPU 7 -> 8] Fix false positive report about invalid symlinks on RHEL 7 (#1052) - [IPU 8 -> 9] Inhibit the upgrade when unsupported x86-64 microarchitecture is detected (#1059) ### Enhancements - Include updated leapp data files in the RPM (#1046, #1092, #1119) - Update the set of supported upgrade paths (#1077): - RHEL with SAP HANA 7.9 -> 8.6, 8.8 (default: 8.6) - RHEL with SAP HANA 8.8 -> 9.2 - Introduce new upgrade paths: - RHEL 7.9 -> 8.9 (default) - RHEL 8.9 -> 9.3 - Correctly update grub2 when /boot resides on multiple devices aggregated in RAID (#1093, #1115) - Enable upgrades for machines using RHUI on AlibabaCloud (#1088) - Introduce possibility to add kernel drivers to initramfs (#1081) - Redesign handling of information about kernel (booted and target) in preparation for new changes in RHEL 9 (#1107) - Redesign source system overlay to use disk images backed by sparse files to optimize disk space consumption (#1097, #1103) - Requires leapp-framework 5.0 (#1061, #1116) - Use new leapp CLI API which provides better report summary output (#1061, #1116) - [IPU 8 -> 9] Detect and report use of deprecated Xorg drivers (#1078) - [IPU 8 -> 9] Introduce IPU for systems with FIPS enabled (#1053) ## Additional changes interesting for devels - Deprecated `GrubInfo.orig_device_name` field in the `GrubInfo` model (replaced by `GrubInfo.orig_devices`) (#1093) - Deprecated `InstalledTargetKernelVersion` model (replaced by `InstalledTargetKernelInfo`) (#1107) - Deprecated `leapp.libraries.common.config.version.is_rhel_realtime` (check the type in msg `KernelInfo`, field `type`) (#1107) - Deprecated `leapp.libraries.common.grub.get_grub_device()` (replaced by `leapp.libraries.common.grub.get_grub_devices()`) (#1093) - Introduced new devel envar LEAPP_DEVEL_KEEP_DISK_IMGS=1 to skip the removal of the created disk images for OVL. That's sometimes handy for the debugging. (#1097)

pirat89 force-pushed the introduce-leapp-data-in-rpm branch from bce781f to e3f9cec Compare February 17, 2023 12:19

pirat89 force-pushed the introduce-leapp-data-in-rpm branch from ae1c459 to 30a953f Compare February 17, 2023 12:53

pirat89 changed the title ~~Introduce leapp data in the RPM & repository~~ [do not merge] Introduce leapp data in the RPM & repository Feb 17, 2023

pirat89 force-pushed the introduce-leapp-data-in-rpm branch from 071fea0 to 32dc6b0 Compare March 10, 2023 09:44

pirat89 added the enhancement New feature or request label Mar 10, 2023

pirat89 force-pushed the introduce-leapp-data-in-rpm branch from 32dc6b0 to 4aa6554 Compare April 21, 2023 12:55

pirat89 added this to the 8.9/9.3 milestone Apr 25, 2023

pirat89 changed the title ~~[do not merge] Introduce leapp data in the RPM & repository~~ Introduce leapp data in the RPM & repository May 3, 2023

pirat89 marked this pull request as ready for review May 3, 2023 08:29

pirat89 force-pushed the introduce-leapp-data-in-rpm branch from 4aa6554 to ed9f161 Compare May 3, 2023 08:56

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label May 3, 2023

pirat89 changed the title ~~Introduce leapp data in the RPM & repository~~ Introduce leapp data files in the RPM & repository May 3, 2023

pirat89 changed the title ~~Introduce leapp data files in the RPM & repository~~ Include leapp data files in the RPM & repository May 3, 2023

pirat89 force-pushed the introduce-leapp-data-in-rpm branch from ed9f161 to 1e052c8 Compare May 4, 2023 10:42

Rezney approved these changes May 5, 2023

View reviewed changes

Rezney merged commit 818945d into oamg:master May 5, 2023

pirat89 mentioned this pull request Aug 23, 2023

Release 0.19.0 #1122

Merged

pirat89 deleted the introduce-leapp-data-in-rpm branch October 11, 2023 10:14

Include leapp data files in the RPM & repository #1046

Include leapp data files in the RPM & repository #1046

Uh oh!

Conversation

pirat89 commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 17, 2023

Thank you for contributing to the Leapp project!

Uh oh!

pirat89 commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pirat89 commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

github-actions bot commented Feb 17, 2023

Uh oh!

pirat89 commented Mar 10, 2023

Uh oh!

pirat89 commented May 3, 2023

Uh oh!

pirat89 commented May 3, 2023

Uh oh!

pirat89 commented May 3, 2023

Uh oh!

pirat89 commented May 4, 2023

Uh oh!

Rezney left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pirat89 commented Feb 17, 2023 •

edited

Loading

pirat89 commented Feb 17, 2023 •

edited

Loading