Implement build-time package dependencies #274

matteodelabre · 2021-02-04T23:35:56Z

(Fixes #85.)

The existing depends field only addressed install-time package dependencies. This PR makes it so that dependencies of packages are also installed at build time, before each package is built. Both dependencies on Debian packages (“build” packages) and other Entware or Toltec packages (“host” packages) are supported. When running make repo, recipes are built in topological order following the declared dependencies.

To illustrate and test this new feature, the following packages are also changed by this PR:

calculator, evtest, plato, wireguard, zoneinfo-utils: turn manual invocations of apt-get into makedepends entries
all packages using the existing depends field: upgrade to the v1.3.2 toolchain version that ships with the opkg binary
vnsee: create a separate libvncserver recipe that contains the libvncserver.so and libvncclient.so shared libraries, make vnsee link to that library instead of bundling it in the vnsee binary
recrossable: create a separate libdlib recipe that contains the libdlib.so shared library, and link recrossable to that library

Test plan:

Checked that all packages build without error and in the correct order (libvncserver before vnsee, libdlib before recrossable) when running make repo-local from an empty repo.
Checked that when a recipe-level dependency cycle is created or a recipe depends on itself, a CycleError is raised.
Checked that vnsee and recrossable install their required libraries and that they run properly (no dynamic linker error).

The existing `depends` field only addressed install-time package dependencies. This PR makes it so that dependencies of packages are also installed at build time, before each package is built. Both dependencies on Debian packages (“build” packages) and other Entware or Toltec packages (“host” packages) are supported. When running `make repo`, recipes are built in topological order following the declared dependencies. To illustrate and test this new feature, the following packages are also changed by this PR: * _calculator, evtest, plato, wireguard, zoneinfo-utils:_ turn manual invocations of `apt-get` into `makedepends` entries * _all packages using the existing `depends` field:_ upgrade to the v1.3.2 toolchain version that ships with the `opkg` binary * _vnsee:_ create a separate _libvncserver_ recipe that contains the _libvncserver.so_ and _libvncclient.so_ shared libraries, make _vnsee_ link to that library instead of bundling it in the _vnsee_ binary * _recrossable:_ create a separate _libdlib_ recipe that contains the _libdlib.so_ shared library, and link _recrossable_ to that library Test plan: * Checked that all packages build without error and in the correct order (libvncserver before vnsee, libdlib before recrossable) when running `make repo-local` from an empty repo. * Checked that when a recipe-level dependency cycle is created or a recipe depends on itself, a CycleError is raised. * Checked that _vnsee_ and _recrossable_ install their required libraries and that they run properly (no dynamic linker error)

Eeems · 2021-02-04T23:40:22Z

What is the purpose of host: and build: for the depends array? I understand how it's used in makedepends, but not depends.

matteodelabre · 2021-02-04T23:50:17Z

There can only be host-type dependencies in the depends field. Since it's using the same parser as makedepends, it is valid (but redundant, since it's the default anyway) to add the host: prefix to its entries. I should probably not have added the prefix in the depends arrays, I'll revert that.

raisjn

is the plan to keep the changes in vnsee, recrossable, etc?

raisjn · 2021-02-13T14:25:19Z

scripts/repo_build.py

-    "-n",
-    "--no-fetch",
+    "-d",
+    "--diff",
    action="store_true",
-    help="do not fetch missing packages from the remote repository",
+    help="only keep new packages that do not exist on the remote repository",


can you describe the difference in behavior, if any? (i think the PR artifact will still only retain the important packages - will downloads increase with this change?)

The downloads do increase with this change. With this flag, instead of ignoring locally missing packages that already exist on the remote, the script now fetches them for the time of the build, then discards them. The rationale is that we may have some packages that depend on other existing packages at build time, so even if we do not want them in the final PR artifacts, we still need them to build the updated packages.

Do you think the flag’s description is clear enough as it stands or do you think it should be improved?

the flag's description is fine

matteodelabre · 2021-02-13T17:03:17Z

is the plan to keep the changes in vnsee, recrossable, etc?

Yes. I plan to take advantage of the separation of libvncserver out of VNSee later on, to make packaging rM-vnc-server easier, as they share the same dependency. As for recrossable, moving the dlib dependency out of the toolchain and into a separate package is a step towards lighter Docker images (I removed dlib from the toolchain in version 1.3).

raisjn

this seems good to me. i built an assortment of packages (evtest, calculator, rmkit, vnsee) but not all. i understand the changes and motivation, all makes sense

for vnsee, i get the following:

okay@chalk:~/tonka/src/remarkable/toltec$ ./scripts/package_build.py -v vnsee
[   DEBUG] docker.utils.config: Trying paths: ['/home/okay/.docker/config.json', '/home/okay/.dockercfg']
[   DEBUG] docker.utils.config: No config file found
[   DEBUG] docker.utils.config: Trying paths: ['/home/okay/.docker/config.json', '/home/okay/.dockercfg']
[   DEBUG] docker.utils.config: No config file found
The build directory 'build/package/vnsee' for recipe 'vnsee' already exists.
Would you like to [c]ancel, [r]emove that directory, or [k]eep it (not recommended)? [C/r/k] r
[    INFO] toltec.builder: vnsee: Fetching source files
[   DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): github.com:443
[   DEBUG] urllib3.connectionpool: https://github.com:443 "GET /matteodelabre/vnsee/archive/3f07c04e1991c1c5ff3b5a4758aa42301bf86fae.zip HTTP/1.1" 302 158
[   DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): codeload.github.com:443
[   DEBUG] urllib3.connectionpool: https://codeload.github.com:443 "GET /matteodelabre/vnsee/zip/3f07c04e1991c1c5ff3b5a4758aa42301bf86fae HTTP/1.1" 200 207831
[   DEBUG] toltec.builder: vnsee: Skipping prepare (nothing to do)
[    INFO] toltec.builder: vnsee: Building artifacts
[   DEBUG] urllib3.connectionpool: http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 88
[   DEBUG] urllib3.connectionpool: http://localhost:None "GET /v1.35/containers/a6caa5880a0161dfbe50d4b0b1caa0c8fef61cce1868338f2aa4419b9936b9d2/json HTTP/1.1" 200 None
[   DEBUG] urllib3.connectionpool: http://localhost:None "POST /v1.35/containers/a6caa5880a0161dfbe50d4b0b1caa0c8fef61cce1868338f2aa4419b9936b9d2/start HTTP/1.1" 204 0
[   DEBUG] urllib3.connectionpool: http://localhost:None "GET /v1.35/containers/a6caa5880a0161dfbe50d4b0b1caa0c8fef61cce1868338f2aa4419b9936b9d2/logs?stderr=1&stdout=1&timestamps=0&follow=1&tail=all HTTP/1.1" 200 None
[   DEBUG] urllib3.connectionpool: http://localhost:None "GET /v1.35/containers/a6caa5880a0161dfbe50d4b0b1caa0c8fef61cce1868338f2aa4419b9936b9d2/json HTTP/1.1" 200 None
[   DEBUG] toltec.builder: vnsee: * opkg_prepare_url_for_install: Couldn't find anything to satisfy 'libvncclient'.
[   DEBUG] urllib3.connectionpool: http://localhost:None "POST /v1.35/containers/a6caa5880a0161dfbe50d4b0b1caa0c8fef61cce1868338f2aa4419b9936b9d2/wait HTTP/1.1" 200 None
[   DEBUG] urllib3.connectionpool: http://localhost:None "DELETE /v1.35/containers/a6caa5880a0161dfbe50d4b0b1caa0c8fef61cce1868338f2aa4419b9936b9d2?v=False&link=False&force=False HTTP/1.1" 204 0
[   ERROR] toltec.builder: vnsee: build() failed
Traceback (most recent call last):
  File "./scripts/package_build.py", line 44, in <module>
    if not builder.make(recipe, packages):
  File "/home/okay/tonka/src/remarkable/toltec/scripts/toltec/builder.py", line 129, in make
    self._build(recipe, src_dir)
  File "/home/okay/tonka/src/remarkable/toltec/scripts/toltec/builder.py", line 295, in _build
    self._print_logs(logs, "build()")
  File "/home/okay/tonka/src/remarkable/toltec/scripts/toltec/builder.py", line 502, in _print_logs
    raise err
  File "/home/okay/tonka/src/remarkable/toltec/scripts/toltec/builder.py", line 483, in _print_logs
    for line in logs:
  File "/home/okay/tonka/src/remarkable/toltec/scripts/toltec/bash.py", line 413, in run_script_in_container
    raise ScriptError(f"Script exited with code {result['StatusCode']}")
toltec.bash.ScriptError: Script exited with code 255

this makes me think i should make sure the whole repo builds, so i will run make repo now while i do some other tasks

raisjn · 2021-02-19T14:05:59Z

scripts/repo_build.py

-    "-n",
-    "--no-fetch",
+    "-d",
+    "--diff",
    action="store_true",
-    help="do not fetch missing packages from the remote repository",
+    help="only keep new packages that do not exist on the remote repository",


the flag's description is fine

raisjn · 2021-02-19T14:43:38Z

also, i think that might be expected behavior - do i need to run make libvncclient and then make vnsee after?

raisjn · 2021-02-19T14:54:15Z

make repo works (yay!), so its just behavior when building single package that has dependencies

okay@chalk:~/tonka/src/remarkable/toltec$ make repo
./scripts/repo_build.py
[    INFO] toltec.repo: Scanning for missing packages
[    INFO] toltec.repo: Package vnsee_0.3.1-2_armv7-3.2 (vnsee) is missing
[    INFO] toltec.repo: Package libdlib_19.21-1_armv7-3.2 (libdlib) is missing
[    INFO] toltec.repo: Package libdlib-dev_19.21-1_armv7-3.2 (libdlib) is missing
[    INFO] toltec.repo: Package recrossable_0.0.0-4_armv7-3.2 (recrossable) is missing
[    INFO] toltec.repo: Package libvncserver_0.9.13-1_armv7-3.2 (libvncserver) is missing
[    INFO] toltec.repo: Package libvncclient_0.9.13-1_armv7-3.2 (libvncserver) is missing
[    INFO] toltec.repo: Package libvncserver-dev_0.9.13-1_armv7-3.2 (libvncserver) is missing
[    INFO] toltec.builder: libvncserver: Fetching source files
[    INFO] toltec.builder: libvncserver: Building artifacts
[    INFO] toltec.builder: libvncserver: Stripping binaries
[    INFO] toltec.builder: libvncserver (libvncserver): Packaging build artifacts
[    INFO] toltec.builder: libvncserver (libvncserver): Creating archive
[    INFO] toltec.builder: libvncserver (libvncclient): Packaging build artifacts
[    INFO] toltec.builder: libvncserver (libvncclient): Creating archive
[    INFO] toltec.builder: libvncserver (libvncserver-dev): Packaging build artifacts
[    INFO] toltec.builder: libvncserver (libvncserver-dev): Creating archive
[    INFO] toltec.repo: Generating package index
[    INFO] toltec.builder: libdlib: Fetching source files
[    INFO] toltec.builder: libdlib: Building artifacts
[    INFO] toltec.builder: libdlib: Stripping binaries
[    INFO] toltec.builder: libdlib (libdlib): Packaging build artifacts
[    INFO] toltec.builder: libdlib (libdlib): Creating archive
[    INFO] toltec.builder: libdlib (libdlib-dev): Packaging build artifacts
[    INFO] toltec.builder: libdlib (libdlib-dev): Creating archive
[    INFO] toltec.repo: Generating package index
The build directory 'build/package/vnsee' for recipe 'vnsee' already exists.
Would you like to [c]ancel, [r]emove that directory, or [k]eep it (not recommended)? [C/r/k] r
[    INFO] toltec.builder: vnsee: Fetching source files
[    INFO] toltec.builder: vnsee: Building artifacts
[    INFO] toltec.builder: vnsee: Stripping binaries
[    INFO] toltec.builder: vnsee (vnsee): Packaging build artifacts
[    INFO] toltec.builder: vnsee (vnsee): Creating archive
[    INFO] toltec.repo: Generating package index
[    INFO] toltec.builder: recrossable: Fetching source files
[    INFO] toltec.builder: recrossable: Building artifacts
[    INFO] toltec.builder: recrossable: Stripping binaries
[    INFO] toltec.builder: recrossable (recrossable): Packaging build artifacts
[    INFO] toltec.builder: recrossable (recrossable): Creating archive
[    INFO] toltec.repo: Generating package index
[    INFO] toltec.repo: Generating package index
[    INFO] toltec.repo: Generating web listing

matteodelabre · 2021-02-19T20:14:13Z

Thanks for testing! As you’ve noticed, I did not change the make RECIPE command to automatically build the dependencies of the recipe. I did this to keep things simple since it’s not strictly needed and the diff is already substantial. I can implement it I you think it’s worth it, though.

raisjn

i don't mind when it's fixed, but i think that it will come up eventually (when someone tries to build something) and we'll get support requests then.

i've read through the code so many times, but i think i should just be taking time to come up with scenarios that might break. either way, it looks good to me

matteodelabre · 2021-03-01T21:27:25Z

Thanks for reviewing! I’ll merge this so that we can move forward with packages that need build dependencies. We’ll revisit dependency handling in make RECIPE when it comes up.

new packages: [ddvk-hacks] Add ddvk-hacks (toltec-dev#247) updated packages: [wireguard][1.0.20210219] - Updated package (and include wireguard-tools) (toltec-dev#285) [rm2fb] update rm2fb to work with xochitl 2.6 (v1.0.1) (toltec-dev#301) [recrossable] Update recrossable (toltec-dev#312) [wikipedia] Initial wikipedia package. [appmarkable] Update appmarkable to 0.0.0-9 and rmservewacominput to 0.3.0-1 (toltec-dev#308) with rm2 support [rmkit] patch genie to fix crash in testing (toltec-dev#304) [oxide] Update Oxide to v2.1.2 (toltec-dev#241) [rm2fb] update rm2fb with wait ioctl and no-op on rM1 (toltec-dev#298) [rmkit] add bufshot app, add lamp, add iago, add changelog (toltec-dev#276) [rmkit] update rmkit to latest (2021-02-17) (toltec-dev#286) [zshelf][0.3.1] - Updated Package (toltec-dev#287) tooling: Pin the Ubuntu version used in workflows to 20.04 (toltec-dev#316) Provide better version number error messages (toltec-dev#314) util.auto_extract: Extract broken symlinks and missing directories (toltec-dev#302) change web background color to #fcfaf8 (toltec-dev#280) Implement build-time package dependencies (toltec-dev#274) Rewrite repo-build-web in Python (toltec-dev#266) Print last 50 lines of output on build error (toltec-dev#263) Hardcode REMOTE_HTTP secret in PR workflows (toltec-dev#262) Rewrite repo-build and package-build in Python (toltec-dev#218) Make bootstrap execution conditional on hash verification (toltec-dev#257) Add Toltec web home page (toltec-dev#193)

New packages: * bufshot - 0.1.0-2 (#276) * ddvk-hacks - 17.04-1 (#247) * iago - 0.1.0-1 (#276) * lamp - 0.1.0-1 (#276) * libdlib, libdlib-dev - 19.21-1 (#274) * libvncserver, libvncserver-dev - 0.9.13-1 (#274) * wikipedia - 0.1.0-2 (#311) Updated packages: * appmarkable - 0.0.0-9 (#308) * decay - 2.1.2~1 (#241) * erode - 2.1.2~1 (#241) * fret - 2.1.2~1 (#241) * genie - 0.1.4-2 (#304) * harmony - 0.1.2-1 (#286) * mines - 0.1.2-1 (#286) * nao - 0.1.2-1 (#286) * oxide - 2.1.2~1 (#241) * recrossable - 0.0.0-5 (#274, #312) * remux - 0.1.8-1 (#286) * rm2fb - 1.0.1-1 (#298, #301) * rmservewacominput - 0.3.0-1 (#308) * rot - 2.1.2~1 (#241) * simple - 0.1.3-1 (#286) * tarnish - 2.1.2~1 (#241) * vnsee - 0.3.1-2 (#274) Tooling: * Pin the Ubuntu version used in workflows to 20.04 (#316) * Provide better version number error messages (#314) * util.auto_extract: Extract broken symlinks and missing directories (#302) * change web background color to #fcfaf8 (#280) * Implement build-time package dependencies (#274) * Rewrite repo-build-web in Python (#266) * Print last 50 lines of output on build error (#263) * Hardcode REMOTE_HTTP secret in PR workflows (#262) * Rewrite repo-build and package-build in Python (#218) * Make bootstrap execution conditional on hash verification (#257) * Add Toltec web home page (#193) Co-authored-by: okay <okay@arkose> Co-authored-by: Mattéo Delabre <[email protected]>

The existing `depends` field only addressed install-time package dependencies. This PR makes it so that dependencies of packages are also installed at build time, before each package is built. Both dependencies on Debian packages (“build” packages) and other Entware or Toltec packages (“host” packages) are supported. When running `make repo`, recipes are built in topological order following the declared dependencies. To illustrate and test this new feature, the following packages are also changed by this PR: * _calculator, evtest, plato, wireguard, zoneinfo-utils:_ turn manual invocations of `apt-get` into `makedepends` entries * _all packages using the existing `depends` field:_ upgrade to the v1.3.2 toolchain version that ships with the `opkg` binary * _vnsee:_ create a separate _libvncserver_ recipe that contains the _libvncserver.so_ and _libvncclient.so_ shared libraries, make _vnsee_ link to that library instead of bundling it in the _vnsee_ binary * _recrossable:_ create a separate _libdlib_ recipe that contains the _libdlib.so_ shared library, and link _recrossable_ to that library Test plan: * Checked that all packages build without error and in the correct order (libvncserver before vnsee, libdlib before recrossable) when running `make repo-local` from an empty repo. * Checked that when a recipe-level dependency cycle is created or a recipe depends on itself, a CycleError is raised. * Checked that _vnsee_ and _recrossable_ install their required libraries and that they run properly (no dynamic linker error)

This reverts commit c2791a1.

matteodelabre added the tooling Set of scripts and configuration files for building the packages label Feb 4, 2021

This comment has been minimized.

Sign in to view

matteodelabre marked this pull request as draft February 4, 2021 23:40

matteodelabre added 4 commits February 5, 2021 11:21

Remove host: prefix in depends arrays

32ffb16

Copy Python 3.9’s graphlib module for compatibility

6ac6262

Fix formatting

14050a4

Add documentation

679a141

matteodelabre marked this pull request as ready for review February 5, 2021 20:36

matteodelabre and others added 2 commits February 5, 2021 21:48

Use a more reliable server for fetching dlib’s source

8be33e6

Merge branch 'testing' into tooling/implement-build-depends

c66846a

raisjn reviewed Feb 13, 2021

View reviewed changes

raisjn reviewed Feb 19, 2021

View reviewed changes

Merge branch 'testing' into tooling/implement-build-depends

501a6d6

matteodelabre mentioned this pull request Feb 19, 2021

Add Qt packages #212

Open

raisjn approved these changes Feb 20, 2021

View reviewed changes

Merge branch 'testing' into tooling/implement-build-depends

94ba777

matteodelabre linked an issue Mar 1, 2021 that may be closed by this pull request

Add builddepends metadata field to recipes #85

Closed

matteodelabre merged commit 256bcac into testing Mar 1, 2021

matteodelabre deleted the tooling/implement-build-depends branch March 1, 2021 21:38

matteodelabre mentioned this pull request Mar 25, 2021

Stable merge for week 13 of 2021 #320

Merged

danshick added a commit to danshick/toltec that referenced this pull request May 5, 2021

Revert "Implement build-time package dependencies (toltec-dev#274)"

0d7c91b

This reverts commit c2791a1.

matteodelabre mentioned this pull request Jul 31, 2021

Make package_build build-dependency-aware toltec-dev/build#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement build-time package dependencies #274

Implement build-time package dependencies #274

Uh oh!

matteodelabre commented Feb 4, 2021

Uh oh!

This comment has been minimized.

This comment has been minimized.

Eeems commented Feb 4, 2021

Uh oh!

matteodelabre commented Feb 4, 2021

Uh oh!

raisjn left a comment

Uh oh!

raisjn Feb 13, 2021

Uh oh!

matteodelabre Feb 13, 2021

Uh oh!

raisjn Feb 19, 2021

Uh oh!

matteodelabre commented Feb 13, 2021

Uh oh!

raisjn left a comment

Uh oh!

raisjn Feb 19, 2021

Uh oh!

raisjn commented Feb 19, 2021

Uh oh!

raisjn commented Feb 19, 2021

Uh oh!

matteodelabre commented Feb 19, 2021

Uh oh!

raisjn left a comment

Uh oh!

matteodelabre commented Mar 1, 2021

Uh oh!

Uh oh!

Implement build-time package dependencies #274

Implement build-time package dependencies #274

Uh oh!

Conversation

matteodelabre commented Feb 4, 2021

Uh oh!

This comment has been minimized.

This comment has been minimized.

Eeems commented Feb 4, 2021

Uh oh!

matteodelabre commented Feb 4, 2021

Uh oh!

raisjn left a comment

Choose a reason for hiding this comment

Uh oh!

raisjn Feb 13, 2021

Choose a reason for hiding this comment

Uh oh!

matteodelabre Feb 13, 2021

Choose a reason for hiding this comment

Uh oh!

raisjn Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

matteodelabre commented Feb 13, 2021

Uh oh!

raisjn left a comment

Choose a reason for hiding this comment

Uh oh!

raisjn Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

raisjn commented Feb 19, 2021

Uh oh!

raisjn commented Feb 19, 2021

Uh oh!

matteodelabre commented Feb 19, 2021

Uh oh!

raisjn left a comment

Choose a reason for hiding this comment

Uh oh!

matteodelabre commented Mar 1, 2021

Uh oh!

Uh oh!