Reimplement and simplify Hugo's template system

[bep]

Some key differences from v0.145.0:

• We now consider the complete Page.Path, not just the top level (section/type) when resolving templates.
• All template lookup follows the same code path. This means that you can use the same set of identifiers in the filenames for page layouts (including base, e.g. baseof.list.html, templates), shortcode templates, and render hook templates.
• _shortcodes and _markup directories can be added to any level in the file tree.
• The directories shortcodes and partials are renamed _shortcodes and _partials.
• The (confusing) concept of _internal/ templates are removed (but will still work, will be deprecated a later Hugo version), so e.g. the Twitter card template needs to be loaded via {{ partial "twitter_cards.html" . }}.
• Add a new catch-all layout named all, see Add new standard layout 'all' (in addition to single,list) #13545.

Note

I have gone to great lengths to make this as backwards compatible as possible. All existing test passes and I have added lots more. But with the amount of Hugo sites in the wild, I would surprised if we end up with no "my site is broken" reports after releasing this, but we really need to do this.

Identifiers and their weights

Name	Description	W1	W2	W3
Distance	To find the best matching template, we walk from the root down to the page's Path. The distance is measured in directory levels from the page Path up to the template path.
Kind	The Page Kind, one of home, page, section, term or taxonomy.	3	1
Layout Standard	all, list or single.	2	1
Layout Custom	Custom layout set in the layout front matter key, e.g. mylayout.	4	2
Type	Set in type front matter key. For template lookups, this effectively changes the root directory of Page.Path.
OutputFormat	Output format name, e.g. html.	2		1
MediaType	Media type, e.g. text/html.	1		1
Lang	Language, e.g. en.	1		1
Variant1	Contextual variant, currently used to identify a render hook, e.g. link.	4
Variant2	Contextual variant, currently used to identify e.g. the code language in codeblock render hooks.	2

Template Resolving

This section covers template resolving for page layouts, shortcodes and render hook templates. Partials has currently a much simpler logic.

To resolve a template, we

1. Create a Page descriptor with the identifiers listed above.
2. Walk the template tree down to the Page path and compare the Page descriptor with the current template descriptor.
   1. If the template descriptor is a subset of the Page descriptor, we calculate the weights w1, w2 and w3.
   2. If this is the first match, we store it away.
   3. Else we compare it to the previous best match:
      1. If the new match's path is closer to the Page path than the previous best match's path, we compare their values for w2 and w3, in that order. For example, given a Page path /foo/bar, a layout in /layouts/foo/single.html would win over /layouts/page.html even if the latter has a higher w1 weight.
      2. Else we compare their values for w1.
      3. All things equal, these are the tie breakers:
         1. Shortest distance.
         2. If layout is set, we prefer any value before all.
         3. The lesser layout Path (lexicographically).

There are some special cases to the above:

• We prefer user provided templates over embedded templates.
• TODO(bep) complete this.

Example layouts directory

layouts
├── baseof.html
├── home.html
├── list.html
├── single.html
├── taxonomy.html
├── term.html
├── term.mylayout.section.en.rss.xml
├── _markup
│   ├── render-codeblock-go.term.mylayout.no.rss.xml
│   └── render-link.html
├── _partials
│   └── mypartial.html
├── _shortcodes
│   ├── myshortcode.html
│   └── myshortcode.section.mylayout.en.rss.xml
├── docs
│   ├── baseof.html
│   ├── _shortcodes
│   │   └── myshortcode.html
│   └── api
│       ├── mylayout.html
│       ├── single.html
│       └── _markup
│           └── render-link.html
└── tags
    ├── taxonomy.html
    ├── term.html
    └── blue
        └── list.html

Fixes #13515
Updates #7964
Updates #13365
Updates #12988
Updates #4891

Fix/adjusment tasks

• Fix output format resolve issue in head/css.html and similar partials.
• --printWarnings vs All values after the first dot in content/layout/etc. base filenames as identifiers (e.g. language, ext) #13515.
• TestLayoutAllRevised

[jmooring]

This is really, really good.

[bep]

This is really, really good.

Yea, I'm pretty happy with the end result, but I have changed my mind a few times. Note that this is still a draft; it's mostly working, but there's some TODOs sprinkled around, some commented out tests, and everything I say about shortcodes above isn't yet entirely true, but it will be.

[jmooring]

still a draft
it's mostly working
isn't yet entirely true

I've gone-to-market with less, but in most cases under protest.

[jmooring]

I am not trying to increase the scope of this PR, but am curious if either of these are affected:

• Reverse the lookup order of partials #8998
• Not possible to "define" two inline templates with the same name #3996

[bep]

@jmooring none of those issues is somehow fixed by this (whatever that would mean). But the order of things is certainly easier to reason about in the new setup:

if err := s.insertTemplates(nil, false); err != nil {
  return nil, err
}
if err := s.insertEmbedded(); err != nil {
  return nil, err
}
if err := s.parseTemplates(); err != nil {
  return nil, err
}
if err := s.extractInlinePartials(); err != nil {
  return nil, err
}
if err := s.transformTemplates(); err != nil {
  return nil, err
}
if err := s.tns.createPrototypes(true); err != nil {
  return nil, err
}
if err := s.prepareTemplates(); err != nil {
  return nil, err
}

So, while we extractInlinePartials later, we don't overwrite partials with the same path¹. If we did, that would for one break how we use the file system overlay to override templates (project partial wins over theme partial). That could possibly be handled, but that is certainly not in scope here.

Footnotes

1. ... but if you should be able to override by using e.g. mypartial.html.html (e.g. both output format and media type) ↩

[Copilot]

Pull Request Overview

This PR reimplements and simplifies Hugo’s template system by unifying template lookup rules across page layouts, shortcodes, and render hook templates. It also refactors internal data structures (e.g. the radix tree and path parsing logic) and renames constants and methods for improved consistency.

• Unified template lookup using a single code path with a new TemplateStore.
• Refactored concurrency handling in the radix tree and improved path parsing to incorporate language, output format, and type.
• Updated tests and configuration to reflect these internal API and naming changes.

Reviewed Changes

Copilot reviewed 133 out of 133 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
hugolib/doctree/simpletree.go	Refactored tree interfaces and locking; clarified thread safety.
hugolib/content_render_hooks_test.go	Added tests for render hooks and shortcode reuse behavior.
hugolib/content_map_page.go	Updated references from old constants to new type names.
hugolib/content_map.go	Adjusted switch cases to newer type constants and template store ref.
hugolib/content_factory.go	Replaced old template handler calls with TemplateStore methods.
hugolib/alias_test.go	Updated test setup to use inline file definitions and new templates.
hugolib/alias.go	Switched alias handling to use the new TemplateStore API.
deps/deps.go	Removed TemplateProvider in favor of TemplateStore and added counters.
create/content.go	Changed key generation for archetypes to exclude language/format.
config/allconfig/allconfig.go	Extended ContentPathParser with output format detection.
common/types/types.go	Added Locker and RWLocker interfaces for consistency.
common/paths/*	Renamed and reworked path parsing and type handling logic.
common/maps/*	Minor enhancements to ordered map and cache functionality.
commands/server.go	Updated error template lookup to use the TemplateStore.

[bep]

@jmooring All tests are green and I have completed my TODO list. I will do some more manual testing of this myself, but I would appreciate if you could take it for a spin, and especially see if you agree with my choices in the "lookup department"; the most opinionated is probably the path/depth logic. I have tried to explain it above, I'm not sure how well I succeeded.

[jmooring]

@bep

git clone https://github.com/jmooring/test-site-new-template-system
cd test-site-new-template-system
hugo server

At the top of the page you'll see this:

[bep]

@jmooring good catch, odd that none of my tests caught this. Probably a confusion between text and html parsing for the partial func. If you do this, it "works":

{{ partialCached "head/css.html" . | safeHTML }}

Which is obviously not what we want. I will add a test for this and fix it.

[bep]

Here's what happens:

head/css.html => outputformat = css output format => plain text template.

The isn't entirely incorrect and comes from us now handling all templates more or less the same in this department, but I see that we need to add an additional check to make sure that the suffix matches the output format.

[bep]

@jmooring I have pushed a fix for the output format issue in fd073d2 -- I have also created a "task list" in the intro comment to indicate status of any similar future issue.

[jmooring]

Found while verifying behavior of #13515.

$ hugo new content content/v0.123.0.md
Content "/home/jmooring/temp/test-site-new-template-system/content/v0.123.0.md" created

$ hugo new content content/v0.123.1.md
panic: [BUG] no Page found for "/home/jmooring/temp/test-site-new-template-system/content/v0.123.1.md"

I don't think this is anything new. See:

• panic when trying to create a page resource with "hugo new" #12786
• Panic with a misleading error on new page creation. ?When similar page exists? #12602
• Panic when creating new content _index.md then index.md #12194

[jmooring]

Also found while verifying behavior of #13515.

content/
├── _index.md
├── v0.123.0.md
└── v0.123.1.md

As expected, both are published to public/v0/index.html. But --printPathWarnings doesn't report the collision. I think that the release notes should encourage users to run hugo --printPathWarnings while testing the upgrade to detect pages that are affected by #13515.

[jmooring]

This was unexpected, perhaps a specificity problem with the "all" layout.

./
├── content/
│   ├── s1/
│   │   └── p1.md
│   └── s2/
│       └── p2.md
├── layouts/
│   ├── s1/
│   │   └── all.html
│   ├── list.html
│   └── single.html
├── public/
│   ├── s1/
│   │   ├── p1/
│   │   │   └── index.html <-- rendered by layouts/single.html (unexpected)
│   │   └── index.html     <-- rendered by layouts/list.html (unexpected)
│   ├── s2/
│   │   ├── p2/
│   │   │   └── index.html <-- rendered by layouts/single.html 
│   │   └── index.html     <-- rendered by layouts/list.html
│   └── index.html         <-- rendered by layouts/list.html
└── hugo.toml

test case

func TestLayoutAllRevised(t *testing.T) {
	t.Parallel()

	files := `
-- hugo.toml --
disableKinds = ['rss','sitemap','taxonomy','term']
-- content/s1/p1.md --
---
title: p1
---
-- content/s2/p2.md --
---
title: p2
---
-- layouts/single.html --
layouts/single.html
-- layouts/list.html --
layouts/list.html
-- layouts/s1/all.html --
layouts/s1/all.html
`

	b := hugolib.Test(t, files)

	b.AssertFileContent("public/index.html", "layouts/list.html")
	b.AssertFileContent("public/s1/index.html", "layouts/s1/all.html")    // fails
	b.AssertFileContent("public/s1/p1/index.html", "layouts/s1/all.html") // fails
	b.AssertFileContent("public/s2/index.html", "layouts/list.html")
	b.AssertFileContent("public/s2/p2/index.html", "layouts/single.html")
}

[bep]

This was unexpected, perhaps a specificity problem with the "all" layout.

index is not a layout anymore (to my recollection, that was only used for the "home page" before, and that still works). I'm not saying that that is the "problem" in the above, but I find it curious that you use that as a use case.

[jmooring]

I am not using index as a layout; including the public directory in the tree above might have made that unclear. I just added a test case.

[bep]

OK, sorry, I read that a little fast. I have put it on my list.

[bep]

@jmooring I have pushed fixes to 2 issues reported above, the path warnings and the all layout behavior.

[jmooring]

This is also related to #13515. The existing behavior may be correct, but it caught my attention.

When given a content path such as p2.x, content adapters do not remove the extraneous portions of the file name.

integration test

func TestContentTemplateDotSegments(t *testing.T) {
	t.Parallel()

	files := `
-- hugo.toml --
disableKinds = ['home','rss','section','sitemap','taxonomy','term']
-- content/s2/_content.gotmpl --
{{ $page := dict "path" "p2.x" "title" "p2 x"}}
{{ .AddPage $page }}
-- content/s1/p1.x.md --
---
title: p1 x
---
-- layouts/single.html --
{{ .Title }}
`

	b := hugolib.Test(t, files)

	b.AssertFileExists("public/s1/p1/index.html", true)
	b.AssertFileExists("public/s2/p2/index.html", true) // fail, got public/s2/p2.x/index.html instead
}

[bep]

The existing behavior may be correct, but it caught my attention.

I see your point, but I think I'm going to leave this as-is. The "dot" issue is about delmiters in base filenames used to create a path. In your example, the path is pre-defined. There's no filename in play, no delimiters, no ambigouity,.