Output normalization hook

[muzimuzhi]

To apply additional normalization, after built-in output normalization is done. Could be named normalize_hook().

Use case: to simplify specific test output, for example the log-based m3plain002.lvt test in latex3, added in latex3/latex3#1467

latex3/latex3#1467 (comment):

Commit 6f301f1 adds a plainTeX test that loads expl3-generic.tex in a group with \tracingrestores turned on. I think it dumps too many lines hence am considering dropping it in final state.

Is there a way to track lines containing =undefined} only? If so, I'm willing to keep m3plain002.lvt.

[cfr42]

This would be useful for testing font packages because it would let me remove glue/box lines. (I could post-process the output with something like sed or perl, much as @davidcarlisle suggested, but that means abandoning cross-platform/cross-distribution compatibility.) As far as I know there's no tracing option for pdfTeX/TeX which will give me (all) the font lines I want without the lines I don't. Right now the only cross-platform options appear to be to rewriting the normalisation function or tolerating the 'noise'.

[josephwright]

I think before doing this we maybe should refactor the normalisation code anyway. We used to worry about long lines, but the 'plan' has changes so we run with a very large line length and avoid that. So perhaps we should make the latter a requirement, simplify the normalisation, then look at this?

[muzimuzhi]

We used to worry about long lines, but the 'plan' has changes so we run with a very large line length and avoid that.

But that's configurable and I've seen cases where package maintainers chose to remain the old value of maxprintline, for example tuna/thuthesis@45c0b37. So maybe existing normalisation code for wrapped long lines should be kept. Or it can be turned off if maxprintline ~= 9999.

[josephwright]

We used to worry about long lines, but the 'plan' has changes so we run with a very large line length and avoid that.

But that's configurable and I've seen cases where package maintainers chose to set the old value of maxprintline, for example tuna/thuthesis@45c0b37. So maybe existing normalisation code for wrapped long lines should be kept.

Indeed - that's why I said about making it a requirement - we can't simplify the normalisation unless we also force the line length. That's a breaking change but over time we have made those where it makes sense - it's not hard to update .tlg files when one knows the only change is in l3build. But that is likely a separate discussion to the question of a normalisation hook.

[josephwright]

OK, do we want to provide extra normalization on a per-line basis or for the whole of the 'content'? I guess I favour for a single line: so we'd have normalize_addn() or something that takes and returns a string?

[josephwright]

I'm not sure about hook here as I think we want to take the content (or line) as an argument.

[cfr42]

for versioning specifically, could you generalise to remove more version details? otherwise, I think you need an option to add something before the standard normalising of versions. currently, half of the version information is normalised and half isn't. e.g. most pgf files, as well as anything which uses svn-prov \revinfo.

[josephwright]

@cfr42 Why would it need to come before? In the other issue, you give the example of the string (SVN Rev: 10946), which we could search and replace with e.g.

s = string.gsub(s, "%(SVN Rev: %d+%)", "(SVN Rev: ...)")

without needing to know the rest of the line.

[cfr42]

@cfr42 Why would it need to come before? In the other issue, you give the example of the string (SVN Rev: 10946), which we could search and replace with e.g.

s = string.gsub(s, "%(SVN Rev: %d+%)", "(SVN Rev: ...)")
without needing to know the rest of the line.

because pgf just puts e.g. (3.8.1) and that seems dangerously generic ...

[josephwright]

because pgf just puts e.g. (3.8.1) and that seems dangerously generic ...

OK, pgf has something like

Package: pgfrcs 2023-01-15 v3.1.10 (3.1.10)

in the raw log, which we normalise to

Package: pgfrcs ....-..-.. v... (3.1.10)

so one could go with something like

s = string.gsub(s, "v%.%.%. %(%d+%.%d+%.%d+[%d%a.+%-]*%)", "v..., (...)")

or even include the pgf part

local p = "%(%d+%.%d+%.%d+[%d%a.+%-]*%)$"
if string.match(s, "pgf") and string.match(s, p) then
  s = string.gsub(s, p, "(...)")
end

(The pattern here matches semantic-like version strings.)

In the end, there's always a worry about over-normalising - we try in the core to be relatively conservative within what we need ourselves, but for an individual author a bit more risk is probably acceptable - the only issue is missing real failures.

[josephwright]

I'm not seeing really how normalising before the standard run would help here - you still need to look for more-or-less the same stuff, and done at the end, you are modifying what you can already see in the .tlg. If we add an custom function at the start, you have to work out what might mess up the standard code.

[cfr42]

I'm not seeing really how normalising before the standard run would help here - you still need to look for more-or-less the same stuff, and done at the end, you are modifying what you can already see in the .tlg. If we add an custom function at the start, you have to work out what might mess up the standard code.

It just seems safer to substitute for the full version string than only part of it. If I do the substitution afterwards, I have to figure out what you've normalised to and use something like gsub(s, "(v%.%.%. %()%d+%.%d[%d%.]*(%))", "%1...%2").

But if it is lines rather than the whole content, I don't think it really matters.

I think it is better for authors to add further normalisation than l3build because then you can do it for particular tests, which seems preferable since in many cases this information is omitted anyway.

[I hope it is lines. I don't like pattern matching whole content with Lua ;).]