ThirdPartyContainerManagement(TPCM)_in_SonicPackageManager (#2815)

ThirdPartyContainerManagement(TPCM) support in SonicPackageManager allows third party dockers to be installed on the sonic system. The Manifest file is generated from a custom local default file. The Manifest file could be updated through "sonic-package-manager manifests update" command and later the running package could be updated with the new manifest file through "sonic-package-manager update"

#### What I did
There are many Third Party application dockers, that can be used in SONiC to provision, manage and monitor SONiC devices. The dockers need not be compatible with SONiC, but can almost work independently with minimal SONiC interfaces. These are extensions to SONiC and require additional capabilities to seamlessly integrate with SONiC. These are related to installation, upgrade, and configuration. This change is an enhancement to the SONiC Application Extension Infrastructure to enable integrating a Third Party Application in the form of dockers with SONiC. 
Moreover, the process of downloading image tarballs for the dockers (packages) supports SCP, SFTP, and URL before installing them.

#### How I did it
The Sonic-package-manager framework has been enhanced to support ThirdPartyContainerManagement (TPCM). In case no manifest is found in the image labels, the framework treats it as a TPCM package and creates a default manifest for it. During installation, a new manifest file is created with a specified name using the --name option. Users can use the "sonic-package-manager manifests create/update/delete" commands to modify or delete the manifest file. The location for custom local package manifest files is set to "/var/lib/sonic-package-manager/manifests/". Finally, the "sonic-package-manager update" command can be used to apply the updated manifest file to the running TPCM docker.


#### How to verify it

sonic-package-manager install --from-repository <package without manifest, say httpd> --name mytpcm 

sonic-package manager install --from-tarball <local tar/scp tar/sftp tar/http tar> --name <> 												--use-local-manifest
											

Manifests Commands(tpcm):
sonic-package-manager manifests create <> --from-json <>
sonic-package-manager manifests update <>  --from-json <>	
sonic-package-manager manifests list
sonic-package-manager manifests show <>
sonic-package-manager manifests delete <>


sonic-package manager update <package>

Author: sg893052

setup.py

@@ -257,6 +257,7 @@
         'xmltodict==0.12.0',
         'lazy-object-proxy',
         'six==1.16.0',
+        'scp==0.14.5',
     ] + sonic_dependencies,
     setup_requires= [
         'pytest-runner',

sonic_installer/main.py

@@ -337,6 +337,8 @@ def migrate_sonic_packages(bootloader, binary_image_version):
     new_image_docker_mount = os.path.join(new_image_mount, "var", "lib", "docker")
     docker_default_config = os.path.join(new_image_mount, "etc", "default", "docker")
     docker_default_config_backup = os.path.join(new_image_mount, TMP_DIR, "docker_config_backup")
+    custom_manifests_path = os.path.join(PACKAGE_MANAGER_DIR, "manifests")
+    new_image_package_directory_path = os.path.join(new_image_mount, "var", "lib", "sonic-package-manager")
 
     if not os.path.isdir(new_image_docker_dir):
         # NOTE: This codepath can be reached if the installation process did not
@@ -372,6 +374,8 @@ def migrate_sonic_packages(bootloader, binary_image_version):
             run_command_or_raise(["chroot", new_image_mount, DOCKER_CTL_SCRIPT, "start"])
             docker_started = True
             run_command_or_raise(["cp", packages_path, os.path.join(new_image_mount, TMP_DIR, packages_file)])
+            run_command_or_raise(["mkdir", "-p", custom_manifests_path])
+            run_command_or_raise(["cp", "-arf", custom_manifests_path, new_image_package_directory_path])
             run_command_or_raise(["touch", os.path.join(new_image_mount, "tmp", DOCKERD_SOCK)])
             run_command_or_raise(["mount", "--bind",
                                 os.path.join(VAR_RUN_PATH, DOCKERD_SOCK),

sonic_package_manager/main.py

@@ -15,6 +15,7 @@
 from sonic_package_manager.errors import PackageManagerError
 from sonic_package_manager.logger import log
 from sonic_package_manager.manager import PackageManager
+from sonic_package_manager.manifest import MANIFESTS_LOCATION
 
 BULLET_UC = '\u2022'
 
@@ -157,6 +158,13 @@ def repository(ctx):
     pass
 
 
+@cli.group()
+@click.pass_context
+def manifests(ctx):
+    """ Custom local Manifest management commands. """
+
+    pass
+
 @cli.group()
 @click.pass_context
 def show(ctx):
@@ -280,6 +288,73 @@ def changelog(ctx,
         exit_cli(f'Failed to print package changelog: {err}', fg='red')
 
 
+@manifests.command('create')
+@click.pass_context
+@click.argument('name', type=click.Path())
+@click.option('--from-json', type=str, help='specify manifest json file')
+@root_privileges_required
+def create_manifest(ctx, name, from_json):
+    """Create a new custom local manifest file."""
+
+    manager: PackageManager = ctx.obj
+    try:
+        manager.create_package_manifest(name, from_json)
+    except Exception as e:
+        click.echo("Error: Manifest {} creation failed - {}".format(name, str(e)))
+        return
+
+
+@manifests.command('update')
+@click.pass_context
+@click.argument('name', type=click.Path())
+@click.option('--from-json', type=str, required=True)
+@root_privileges_required
+def update_manifest(ctx, name, from_json):
+    """Update an existing custom local manifest file with new one."""
+
+    manager: PackageManager = ctx.obj
+    try:
+        manager.update_package_manifest(name, from_json)
+    except Exception as e:
+        click.echo(f"Error occurred while updating manifest '{name}': {e}")
+        return
+
+
+@manifests.command('delete')
+@click.pass_context
+@click.argument('name', type=click.Path())
+@root_privileges_required
+def delete_manifest(ctx, name):
+    """Delete a custom local manifest file."""
+    manager: PackageManager = ctx.obj
+    try:
+        manager.delete_package_manifest(name)
+    except Exception as e:
+        click.echo("Error: Failed to delete manifest file '{}'. {}".format(name, e))
+
+
+@manifests.command('show')
+@click.pass_context
+@click.argument('name', type=click.Path())
+@root_privileges_required
+def show_manifest(ctx, name):
+    """Show the contents of custom local manifest file."""
+    manager: PackageManager = ctx.obj
+    try:
+        manager.show_package_manifest(name)
+    except FileNotFoundError:
+        click.echo("Manifest file '{}' not found.".format(name))
+
+
+@manifests.command('list')
+@click.pass_context
+@root_privileges_required
+def list_manifests(ctx):
+    """List all custom local manifest files."""
+    manager: PackageManager = ctx.obj
+    manager.list_package_manifest()
+
+
 @repository.command()
 @click.argument('name', type=str)
 @click.argument('repository', type=str)
@@ -334,6 +409,14 @@ def remove(ctx, name):
               help='Allow package downgrade. By default an attempt to downgrade the package '
               'will result in a failure since downgrade might not be supported by the package, '
               'thus requires explicit request from the user.')
+@click.option('--use-local-manifest',
+              is_flag=True,
+              default=None,
+              help='Use locally created custom manifest file. ',
+              hidden=True)
+@click.option('--name',
+              type=str,
+              help='custom name for the package')
 @add_options(PACKAGE_SOURCE_OPTIONS)
 @add_options(PACKAGE_COMMON_OPERATION_OPTIONS)
 @add_options(PACKAGE_COMMON_INSTALL_OPTIONS)
@@ -348,7 +431,9 @@ def install(ctx,
             enable,
             set_owner,
             skip_host_plugins,
-            allow_downgrade):
+            allow_downgrade,
+            use_local_manifest,
+            name):
     """ Install/Upgrade package using [PACKAGE_EXPR] in format "<name>[=<version>|@<reference>]".
 
     The repository to pull the package from is resolved by lookup in package database,
@@ -378,16 +463,58 @@ def install(ctx,
     if allow_downgrade is not None:
         install_opts['allow_downgrade'] = allow_downgrade
 
+    if use_local_manifest:
+        if not name:
+            click.echo('name argument is not provided to use local manifest')
+            return
+        original_file = os.path.join(MANIFESTS_LOCATION, name)
+        if not os.path.exists(original_file):
+            click.echo(f'Local Manifest file for {name} does not exists to install')
+            return
+
     try:
         manager.install(package_expr,
                         from_repository,
                         from_tarball,
+                        use_local_manifest,
+                        name,
                         **install_opts)
     except Exception as err:
         exit_cli(f'Failed to install {package_source}: {err}', fg='red')
     except KeyboardInterrupt:
         exit_cli('Operation canceled by user', fg='red')
 
+# At the end of sonic-package-manager install, a new manifest file is created with the name.
+# At the end of sonic-package-manager uninstall name,
+# this manifest file name and name.edit will be deleted.
+# At the end of sonic-package-manager update,
+# we need to mv maniests name.edit to name in case of success, else keep it as such.
+# So during sonic-package-manager update,
+# we could take old package from name and new package from edit and at the end, follow 3rd point
+
+
+@cli.command()
+@add_options(PACKAGE_COMMON_OPERATION_OPTIONS)
+@add_options(PACKAGE_COMMON_INSTALL_OPTIONS)
+@click.argument('name')
+@click.pass_context
+@root_privileges_required
+def update(ctx, name, force, yes, skip_host_plugins):
+    """ Update package to the updated manifest file. """
+
+    manager: PackageManager = ctx.obj
+
+    update_opts = {
+        'force': force,
+        'skip_host_plugins': skip_host_plugins,
+        'update_only': True,
+    }
+    try:
+        manager.update(name, **update_opts)
+    except Exception as err:
+        exit_cli(f'Failed to update package {name}: {err}', fg='red')
+    except KeyboardInterrupt:
+        exit_cli('Operation canceled by user', fg='red')
 
 @cli.command()
 @add_options(PACKAGE_COMMON_OPERATION_OPTIONS)