Size of the typst/packages repository

[gasche]

The typst/packages repository takes 1.9Gio on my machine with a current clone. As Typst gets more popular, its size will increase at a higher-than-linear speed, and there is a risk that it becomes painful in practice to operate with the package repository: at some point, people with low bandwidth will have trouble cloning the repository to contribute their own package.

The size of the repository is currently roughly:

• 500MiB of git metadata
• 1.4GiB of package data

(in particular, doing a shallow clone will not help much)

On my current checkout of the repository, there are

• 585 packages in total
• 343 packages which take less than 1Mio of disk space, they consume 117Mio in total
• 219 that take between 1Mio and 10Mio, they consume 724Mio in total
• 23 which take more than 10Mio, they consume 567Mio in total

In the short term, the following could work:

• in packaging guidelines, encourage people to stick to small packages below 1Mio (maybe templates need different recommendations)
• replace identical asset files by symbolic links, to avoid duplication of assets across different versions

Replacing identical asset files by symbolic links can be done by package authors if they are told how to do it, or by repository maintainers after the fact. (git already deduplicates its internal data, so it is not strictly necessary to do it at package-submission time.) A quick experiment suggests that doing this with the current repository should shrink its size from 1.4Gio to 947Mio, which is a sizeable win.

In the long term, I think that repository maintainers should maybe consider git-lfs or other options. The end goal would be that package authors do not need to download all other packages to submit theirs.

[elegaanz]

We are well aware of this issue, and even if a shallow clone wouldn't help, a sparse checkout should normally keep the repository size smaller on your disk. As I said in #2007 I plan to rework the packaging guidelines at some point, and explanations on how to do that (as well as general tips on how to keep a package small) will definitely be part of it.

On the long term, we will probably move away from Git and GitHub to store and review packages to use a custom solution instead.

[gasche]

I plan to rework the packaging guidelines at some point, and explanations on how to do that (as well as general tips on how to keep a package small) will definitely be part of it.

Would you be able to suggest a way to do this incrementally, so that we can get this useful documentation out right now, without being in conflict with your medium-plans for the README? It's a shame to find out how to generate thumbnails or cloning the repository later on, after having already done both in a non-optimal way, because the information is hard to find.

For example, an immediate plan would be to create a doc/ repository, and move the content of the current README in subfiles there (with links from the main README). I would think of the following separate topics:

• submitting-packages.md: the process of how to submit a new package
• packaging-guidelines.md: general packaging rules and recommendations
• package-content.md: recommended content of a package (README, thumbnails... including the recommended filesystem layout)
• typst.toml.md: how to write typst.toml
• local-packages.md: local packages / how to test packages locally

This is mostly for package authors. The main README should work as a landing page for package users by briefly mentioning how to use packages, and how to browse the set of existing packages. (The documentation on local packages can also be useful there.)

Would you be willing to review a PR that splits up the existing guidelines in this way?

[elegaanz]

Yes, that would be extremely helpful. There are few other things I have in mind when it comes to improving the docs, but this can be an incremental process, splitting everything as you suggested would be a good first step.