Build system for CJK fonts

[NightFurySL2001]

Currently Google Fonts seem to slowly start to accept CJK fonts after a long period without CJK fonts except Noto CJK. That is a good thing, except when it comes to the build system everything went haywire.

In the current documentation, Google Fonts allowed any build system as long as it uses only open-source tools and is repeatable with one command (or at lease one build.sh file).

https://github.com/googlefonts/googlefonts.github.io/blob/27a81526ff40693e533cb25be39fb0284c7d0dc0/gf-guide/production.md?plain=1#L50C1-L50C136

• Fonts should be built using open-source tools.** This ensures that they can be built under the same conditions on any platform.

• Fonts should be built in one step. All GF font production tools can be run from the command line. This allows to use them to generate font families by running a single command.
  If the build process necessitates more than one step / one command, then every step needed to build the fonts should be included in a single build script. See the chapter about building fonts for more information.

• Font builds must be repeatable. Given all of GF tools are written in Python and distributed using pypi/pip, this allows to use specific versions of each package, ensuring the same conditions for each build with the same quality of results.

And a list of build tools are provided in https://googlefonts.github.io/gf-guide/build.html:

Fontmake
gftools
gen-stat
gftools builder
Useful tools

However, after observing (and working with a few) the onboarding process, it seems that the GF team currently only accepts the makefile build system for CJK fonts and reject any currently-using build system. This has caused delays and frustrations among font designers, where many of them do not have any knowledge of programming or software development.

For example, the currently live LXGW Wenkai TC had undergone an overhaul of its original build system by @\aaronbell (observe tagged v1.330 before the overhaul and v1.500 after the overhaul). I had personally worked with the font designer LXGW in lxgw/LxgwWenkaiTC#16 , where the designer had many questions on why the Latin parts were separated and the original source files are abandoned. Although the final build system is better in terms of modularity, it had made the font designer's job harder as his original source files cannot be used anymore, and instead be forced to use the new UFO files generated by @\aaronbell in the build system, which is originally made with Glyphs where both the font designer and me don't have. I had to research on how to convert the UFO files back to workable source files for the font designer, and helped him to migrate the other Wenkai repo to use the same build system locally.

The same actually happened in lxgw/LxgwMarkerGothic#7, where this time it is not even obvious why the build system has to be changed when the original fontmake build system works perfectly. The original repo uses fontmake because that was what GF was suggesting at the time of release, and gftools weren't stable to work with.

Another is the currently live Cactus Serif, where the build system is replaced from a build.sh file to makefile in MoonlitOwen/CactusSerif#2 . This is quite unnecessary and the reason given is only "Updated the build system to use GF's standard builder tool.", which they were already using (in build.sh it was calling gftools builder). The source files are also regenrated with Glyphs, which as usual, the original font designer doesn't have. I do not know the font author personally so I couldn't provide help for him, and the build system change ended up not merging into the upstream which breaks the version control flow in the future (e.g. google/fonts#8694 this bug was introduced in downstream by @\aaronbell, so downstream has to bump one version without upstream bumping the version).

There are also reasons to not use the gftools builder, like when I was trying it out and could not choose the overlapping backend as gftools builder did not provide the correct option. This is reported in googlefonts/gftools#654 and only fixed after 1 year, where I had to let https://github.com/maoken-fonts/unbounded-sans stay with fontmake because gftools builder couldn't do what I need it to do. Other build systems that aren't using the fontmake/gftools buildchain exist too, like Adobe's Source Han / Google Noto which uses the AFDKO system.

These issues have raised a concern that if the GF team had any compulsary requirements that all new onboarding font must use the new makefile build system, which takes a long time from @\aaronbell and break the font designer's workflow altogether. I hope the GF team can provide some accurate answer/stance on this issue.

[aaronbell]

Thanks for the feedback. Before diving into it, I wanted to mention that this repository issue tracker is not as actively monitored as the main one so I'd suggest posting issues and questions over there for more rapid attention! :)

A few points in response to your comments:

1. There is no explicit requirement by Google Fonts to use the gftools builder.

However, the builder makes many things convenient for build system management by:

• (a) using a standardized config file that is easier to understand than commandline,
• (b) provide more accessible commands to extract, compress, build, and manage the sources in a single location (the 'makefile') which is more straightforward than having to run a bunch of python files directly, and
• (c) automatically implement things that Fontmake doesn't do (like generating a STAT table) which are required by Google Fonts.

I understand your frustration with the overlaps problem which prevented your use of the builder previously, but hopefully now that it has been resolved, you can appreciate the improvements offered by the new system.

2. Google Fonts does require a true source to be present in the repositories of the fonts it onboards.

In the case of LXGW (and MoonlitOwen), it sounds like the designer maintains a local source file where the work is done, and then output UFOs to the server. Unfortunately, that presents a problem when I, or anyone else, needs to make changes to the source files. I don't have access to your local files, so all I can do is update the UFOs, which is an accepted interchange format (not Glyphs-specific) which is readable / writable by all professional type design software.

3. Splitting of source material in LXGW WenKai TC

The font uses significant duplicative content across the two large families. Part of my job as onboarder is to think about future maintenance and making sure we limit the potential for human error. In the case of such families, I split apart the files to ensure that there is 1 copy of each glyph that needs to be updated at any given time and that change is automatically implemented in every relevant font file. Otherwise you'd need to go through and make the same change multiple times to keep consistency, and potentially make a mistake in the process. It adds some complexity, admittedly, but I've found it a very useful approach in the expansive projects that I've had to manage.

To summarize:
(1) No, the builder is not a requirement, and if you don't want to use it, you don't have to, but the fonts must still pass validation. I find the builder easier to work with, and hope that you will too now that the pathops problem is solved.
(2) The source on the repository must be a true source, which means anyone can edit it without it needing to be ported to a local source. I realize that has caused some consternation in this case, but it is a requirement for it to be the true source.
(3) Splitting of files, such as for WenKai TC is a standard software development approach intended to reduce the chance for human error. This isn't to say it won't still happen, but there are significant benefits to this approach that I think it worth the trouble.

All of this aside, I'm always open to discussion (or explanation!) on any PR that I make, and am happy to work with you to find a common solution that works for both of us.

[NightFurySL2001]

To start off, here is two core fact that is the basis for the rest of discussion:

1. Font designers are not programmers, and many of them are using Windows.
2. All font editors have features that are not supported by UFO and may not support everything in UFO.

---

Discussion 1: Remaking the build system

Firstly, I do understand the easiness of using gftools builder. However, the gftools builder is harder to understand than the command line, where "give a command and the program build" (eg fontmake) makes more intuitive sense than "write a config file that the program reads and then build a file" (eg builder) for non-programmers.

It doesn't help the fact that the call method to the build program in the new system PR'd by @\aaronbell is wrapped in a Makefile, which is harder to work with on Windows (at least personally I never successfully ran anything with a Makefile at all on Windows); it also makes it harder to explain to non-programmers ("this file runs a command that calls a program that reads a config file before building the font").

Sometimes what the font designer end up understanding from the PR is the build system just turns from a readable shell script to a mess of files with unknown usages. When they don't know what the PR is doing, they can't even start to ask the "why" in doing such things and might have just abandon the project due to the high maintenance that might be required in the future.

So there are two issues in this case: the calling method (Makefile) and the actual font builder system (gftools). They are just packaged in one PR which the PR author expects the font designers with no programming experience to just understand. What could've been done to ease the onboarding step is enforcing only one update to the build system at a time. If that helps, I would suggest Google Fonts to consider requiring a Makefile for consistent entry point for the build system (but note that currently there is no suitable alternative to make on Windows yet). If the font authors opted to use fontmake, then they can put the fontmake command in the Makefile, otherwise they can follow a tutorial and put the preferred system (gftools builder) in the Makefile instead.

For the builder system, the issue is that currently there is only the tutorial on https://googlefonts.github.io/gf-guide/build.html#gftools-builder and no detailed documentations on the available parameters for the config file, which makes it hard to reference and understand the config file. The documentation site linked above isn't only for gftools builder, and is mixed with other tools in gftools. It would be better if Google Fonts can provide a more detailed documentation on gftools builder specifically for font designers.

---

Discussion 2: UFO source files

Firstly, I understand that UFO is the current preferred text-based source files for OpenType fonts, agnostic of font editor. However, it is unrealistic to expect all font designers to only work in publicly-available UFOs. There exist a critical difference of UFOs and the normal source code though: UFOs are almost always generated by authoring tools instead of written by humans.

I am ok with contributors editing the UFO and sending in a PR, however in this case there should be one expectation that are hidden in the normal OSS PR flow: the changes from the PR should be reproducible and validatabale. This is because any changes to the font should be understood by the font designers and validated such that it won't break compatibility or introduced unintended side effects (what if a PR changed the metrics? eg huertatipografica/Caladea#4); it should be reproducible in the sense that if I open the same UFO in any font editors, I should be able to edit the same items that were edited in the PR.

However, in the PRs that @\aaronbell submitted, the first edit to the UFO files is just opening it in Glyphs and then re-exporting to UFO. This introduced tons of useless updates to the UFO files and make the commit unnecessarily large (e.g. in MoonlitOwen/CactusSerif#2 the first commit is so large it crashed GitHub). Some examples of the updates include: reordering of plistlib keys, missing keys, dumped unknown keys, adding new Glyphs-specific items, etc. These updates are not reproducible in many cases as most hobbyist font designers I know (and me personally) do not have a Mac and Glyphs to validate the updates.

There also is the problem of font editors having poor support of UFOs. As @\aaronbell has mentioned in this comment:

I suspect that this will be an issue with most applications that translate UFO into app-specific data structures and then back to UFO. I'd suggest using Robofont as it is a native UFO reader / writer, but it appears to have non-existent CJK / vertical typesetting support. Otherwise, it is probably best to store in a particular app's format, then use a reliable conversion tool (like glyphs2ufo) to obtain the desired output. It makes it harder for others to work on your sources, but at least you can avoid things being broken unexpectedly like we've observed here.

Sadly, font editors on Windows do not have reliable conversion tools, and are lacking some features too (FontLab doesn't support vhea/vmtx and IVD yet, FontCreator's UFO is just a jumbled mess of bugs and errors, Glyphs' IVD is still lacking). There is also the problem of unsupported features (eg smart components in Glyphs/FontLab 8) that could not be stored in UFO yet, forcing designers to must have a local source files in the editor's native format. This is outside of our controls though so not much can be done here, except that it would be really helpful if only the required changes are submitted in each PR.

I would kindly remind of the discussions that happened during the onboarding process at lxgw/LxgwWenkaiTC#16 would be a helpful records on improving the whole process.

[For LXGW Wenkai, there is the issue that there are also other versions that the font author is maintaining, which makes it even more so harder that only TC is updated, forcing different maintenance procedures to different fonts]

[aaronbell]

Thank you for your comments and insight.

1. Build system

I can see your point that adding a config file into the build system can lead to some confusion wherein it isn't a 1:1 mapping of input (command line command) to output (program builds). In a way, though, using build.sh is essentially the same thing wherein you've listed all of the settings in a line of code in a file which you then have to run to get an output. So one has to be adept at reading the code in the build.sh commands to understand what the build system is doing. Whereas with the builder, the config instructions are fairly human-readable and can be understood.

It is useful to know that the Makefile approach is not viable for your cases. If I'm honest, in all of the onboarding work that I've done, no one has ever complained about the Makefile, or appeared to have difficulty running it, so I apologize for making it more difficult for you to manage. Happy to re-adjust the build system to provide a more universal method.

and no detailed documentations on the available parameters for the config file, which makes it hard to reference and understand the config file.

Totally agree about the lack of documentation regarding the builder config. I've generally found the parameters self-explanatory, but the full list of them is not readily available. I end up always having to go to https://github.com/googlefonts/gftools/blob/main/Lib/gftools/builder/schema.py to find the name of the config entry that I want to use. It might be useful to you as well if you haven't seen it.

2. UFOs

This does raise an interesting challenge. The GF toolchain is built to support UFOs and (relatively simple) Glyphs source files. I think that there's a Fontlab conversion tool out there too, but not sure as to its capabilities / status.

And even with support for Glyphs sources, not all of the features are included (which forced me to write this build script that loads Glyphs up in the background to export the font before building for a project some years back due to a Round Filter). And it also isn't always clear why something gets exported sometimes and not other times (like that issue with the vmtx table going missing on Cactus—I added the vertical metric data into the source, but for whatever reason that wasn't sufficient when it had been previously).

I am ok with contributors editing the UFO and sending in a PR, however in this case there should be one expectation that are hidden in the normal OSS PR flow: the changes from the PR should be reproducible and validatabale. This is because any changes to the font should be understood by the font designers and validated such that it won't break compatibility or introduced unintended side effects (what if a PR changed the metrics? eg huertatipografica/Caladea#4); it should be reproducible in the sense that if I open the same UFO in any font editors, I should be able to edit the same items that were edited in the PR.

The problem is that pretty much every editor has its own slightly different take on how to order the data, what data to include, and even the types of tabs / spaces to use. Like you said, it makes it unnecessarily harder to parse through.

And I totally understand that regenerating the UFO with Glyphs was probably very frustrating. At the time, I was brought in to do the onboarding with a very, very tight timeline, so was trying to complete the work well, and as quickly as possible. For this round, when I've had more time to consider how to approach these projects, I realized it made more sense to use a different tool (Robofont) which still makes (inconsequential) changes to the UFO source but would make fewer ones than Glyphs did.

In any case, all of this is leading me to think that there needs to be a rethink in how to conduct repro management for large projects such as these where maintaining the source in the original format (like .fcp) is valuable, even if the fontmake tool chain cannot process it at this time.

I'll discuss with the onboarding team and report back.

[aaronbell]

Hi @NightFurySL2001

I've had a chance to discuss with the onboarder team and wanted to follow up.

As you stated, a build.sh is completely acceptable per the GF guide content on Scalable font production. There's a lot of benefits to the gftools builder system (documented here) but no specific requirement to use it, nor a makefile per the spec.

As I mentioned earlier, the makefile system is more convenient for me to rapidly prep repositories for onboarding, but I recognize that it can make things more difficult for maintainers who prefer a different method.

If there are projects that you're involved that you'd rather switch back to a build.sh file, please let me know and we can look into doing that.

It is useful to know that FontCreator is actively-used among the TC (and SC?) Open Source font community for development. It does pose a challenge as the .fcp format does not have a script-based conversion method (which would be a requirement, per Fonts should be built in one step), but for the time being we are OK with using the output UFO as an interchange format until such a script to directly convert .fcp becomes available.

However, GF does require that the "true source" be made available on Github, so I request that the FontCreator source files are uploaded to the repositories that you manage. Especially since FontCreator now has a mac version, it becomes easier for an onboarder to make a modification on the source directly, rather than needing to either (a) file a bug on the repository, or (b) submit a modification on the UFO which you then have to parse and figure out how to implement it from there.

To this end, I will be posting an issue on each of the repositories that I worked on either in the last cycle, or this cycle to (a) ensure that the build system is acceptable to the maintainers, and (b) confirm that the true source is uploaded to the repro. I've also submitted a PR #155 (which needs review) to clarify some of the language around inclusion of sources. Please keep an eye out! :)

[davelab6]

I request that the FontCreator source files are uploaded to the repositories

Yes, I'd like 'real' source files to be published, even if we do not currently have libre software to process them, since that can change over time. @simoncozens has this very nice https://github.com/simoncozens/babelfont project that I hope will learn to speak FCP :)

[NightFurySL2001]

That seems to be an acceptable path, however it should be noted some preprocessing might be required for some of the source formats (e.g. adding vertical metrics for CJK in FontLab, decomposing smart components in Glyphs) and version control might be harder (FontCreator prevent files edited in newer versions from opening in older version, i.e. FC 14 cannot open FC 15 files, neither does 15.0.0.2900 opening 15.0.0.3000).

[rsheeter]

Naively, would it make sense to eventually support FCP in fontc?

[aaronbell]

That seems to be an acceptable path, however it should be noted some preprocessing might be required for some of the source formats (e.g. adding vertical metrics for CJK in FontLab, decomposing smart components in Glyphs)

In cases where there is processing required for the source to become usable in fonttools, the original source should be provided as well as the modified source. That's the route we had to take in Iansui because the cap components were not compatible. (I should note that glyphsLib does support smart components, at least to some extent!)

and version control might be harder (FontCreator prevent files edited in newer versions from opening in older version, i.e. FC 14 cannot open FC 15 files, neither does 15.0.0.2900 opening 15.0.0.3000).

That's fine, IMO. Opening files in older versions could result in problematic results.

[simoncozens]

I'm working with Erwin to develop file format specs and a libre reader library for FCP. If we add it to babelfont-rs, we'll get fontir generation for free.

[rsheeter]

I'm working with Erwin to develop file format specs and a libre reader library for FCP

Great news!

Do we have many fcp families? - if we do, and fcp is the preferred form for modification, it would be nice to have fcp2fontir directly in fontc.

[aaronbell]

@rsheeter FC appears to be common for TC designers. We don’t have many prod families based on it presently but there are several prospective ones we want to onboard.

[NightFurySL2001]

FC is common in SC community more than TC, especially those that do not have a Mac and Glyphs.

Note that major FC release license key are not version-compatible, meaning font authors might be locked out of their own source files after editing. FC 10-14 are still in common use and those do not have a corresponding Mac release.

[NightFurySL2001]

As I mentioned earlier, the makefile system is more convenient for me to rapidly prep repositories for onboarding, but I recognize that it can make things more difficult for maintainers who prefer a different method.

Given the fact that it is unrealistic for Google Fonts to continue maintain the build system in the future, it would be better if the build system remains one that is maintainable by other people. A makefile + gftools builder config + fcp + ufo combination does not sound maintainable in the future.

The issue with the first few TC font onboarding (LXGW WenKai TC, Cactus Serif, Chocolate Sans) is that it was rushed and the original build system provided in the repo was totally discarded in favour of Glyphs-exported UFO + makefile + gftools builder. The repo owner for Cactus Serif and Chocolate Sans could not understand what had happened to the repo, which ended up with two downstream repo under @\aaronbell, as well as the vertical typeset metrics issue in google/fonts#8694 which is still not deployed to live yet. The updates from the repo owner also did not go into the downstream repo causing some version lagging on the GF version and upstream version.

Chinese font requires thousands of hours to make and refine, with nearly 10k+ glyphs being just the bare minimum for a usable Chinese font. The sheer scale of designing Chinese font means that font authors will not be learning the proper techniques of open-source, especially those that involves programming. Commercial Chinese font foundries have a separate position of "font programmer" from "font designer" FYI, which handles the packaging and building of final font products.

The main point I want to talk across to those in Google Fonts is that it is simply unreasonable for GF team to expect Chinese font designers to learn programming and setup the required open-source build toolchain, which is just another burden to the font desingers. Given it is also unreasonable for Google Fonts to keep maintaining their Chinese fonts forks (see the various ZCOOL SC fonts on GF now that doesn't receive any updates anymore), the path forward for GF team is to allow direct contributions from Chinese font designers to Google Fonts from their own repos (without GF forks).

Sure, the "true source" way @\aaronbell mentioned above will help a lot, but it is unfeasible currently and comes with major problems for versioning. It would be better if GF team only review -- instead of fully rewrite -- any build systems currently in use by Chinese font designers which will save much of the time onboarding (unless it is a critical oversight like LXGW WenKai TC), and also give some clearer technical guidance on what to expect for Chinese fonts to be onboarded onto Google Fonts. Please, Chinese font designers aren't programmers, they don't need another lecture from programmers about font engineering when they already have the 10k+ glyphs per font to worry about.