Tests: regression with Docutils HEAD (0.22-rc)

[jayaddison]

Describe the bug

The Sphinx unit tests when running with the latest HEAD commit of docutils are failing.

An illustrative case from 2025-05-08 can be found in the failed build at: https://github.com/sphinx-doc/sphinx/actions/runs/14915538556/job/41900098124

There appear to be at a few categories of error:

Child-node count mismatches (ref):

    def assert_node(node: Node, cls: Any = None, xpath: str = '', **kwargs: Any) -> None:
        if cls:
            if isinstance(cls, list):
                assert_node(node, cls[0], xpath=xpath, **kwargs)
                if cls[1:]:
                    if isinstance(cls[1], tuple):
                        assert_node(node, cls[1], xpath=xpath, **kwargs)
                    else:
                        assert (
                            isinstance(node, nodes.Element)
                        ), f'The node{xpath} does not have any children'  # fmt: skip
                        assert len(node) == 1, (
                            f'The node{xpath} has {len(node)} child nodes, not one'
                        )
                        assert_node(node[0], cls[1:], xpath=xpath + '[0]', **kwargs)
            elif isinstance(cls, tuple):
                assert (
                    isinstance(node, list | nodes.Element)
                ), f'The node{xpath} does not have any items'  # fmt: skip
                assert (
>                   len(node) == len(cls)
                ), f'The node{xpath} has {len(node)} child nodes, not {len(cls)!r}'  # fmt: skip
E               AssertionError: The node has 4 child nodes, not 3

An index range error (ref):

    def visit_section(self, node: Element) -> None:
>       self._title_char = self.sectionchars[self.sectionlevel]
E       IndexError: string index out of range

...and a missing-attribute error (ref):

    def check_subsection(self, source, style, lineno) -> bool:
        """
        Check for a valid subsection header.  Update section data in `memo`.
    
        When a new section is reached that isn't a subsection of the current
        section, set `self.parent` to the new section's parent section.
        """
        memo = self.memo
        title_styles = memo.title_styles
>       mylevel = len(memo.section_parents)
E       AttributeError: 'types.SimpleNamespace' object has no attribute 'section_parents'

How to Reproduce

$ git clone https://github.com/sphinx-doc/sphinx.git && cd sphinx
$ git checkout c4929d026c8d22ba229b39cfc2250a9eb1476282
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install '.[test]'
$ pip install git+https://repo.or.cz/docutils.git@2310dd9b468e9f86b00401b1aa60d47aabc199ac#subdirectory=docutils
$ pytest tests/test_directives/test_directive_only.py tests/test_environment/test_environment_toctree.py tests/test_util/test_util_docutils_sphinx_directive.py

Environment Information

Platform:              linux; (Linux-6.12.25-rt-amd64-x86_64-with-glibc2.41)
Python version:        3.13.3 (main, Apr 10 2025, 21:38:51) [GCC 14.2.0])
Python implementation: CPython
Sphinx version:        8.3.0+/c4929d026
Docutils version:      0.22rc2.dev
Jinja2 version:        3.1.6
Pygments version:      2.19.1

Sphinx extensions

Additional context

No response

Edit: fixup: include attribute-error and context instead of a hyperlink to it.

[gmilde]

This regression is a showstopper for the planned release of Docutils 0.23 :(

From the PR, it seems the reason is Docutils commit r10093 "Change section handling to not rely on exceptions and reparsing."

It seems the PR only changes an auxilliary function in the test code.
Are there other cases in Sphinx where docutils.parsers.rst.states.RSTState.memo is set "by hand" or where the code relies on the details of section handling?

Could you try, whether just adding section_parents=[], to the memo solves
the problem in a backwards-compatible manner?

@@ -39,14 +40,14 @@ def make_directive_and_state(
     inliner.init_customizations(state.document.settings)
     state.inliner = inliner
     state.parent = None
     state.memo = SimpleNamespace(
         document=state.document,
         language=english,
         inliner=state.inliner,
         reporter=state.document.reporter,
         section_level=0,
         title_styles=[],
+        section_parents=[],
     directive = SphinxDirective(
         name='test_directive',
         arguments=[],

If not, it may help to adapt the test code for different values of docutils.__version_info__.

[jayaddison]

It seems the PR only changes an auxilliary function in the test code.
Are there other cases in Sphinx where docutils.parsers.rst.states.RSTState.memo is set "by hand" or where the code relies on the details of section handling?

That's correct - the PR only modifies the test code so far. I may extend it with other fixes as we discover them, but I'm not familiar with the docutils codebase or how/where it interfaces with Sphinx.

As far as I am aware: no, the sole place where we assign to the RSTState.memo attribute in this repository/codebase is from the test suite (specifically from the make_directive_and_state helper method).

[jayaddison]

Could you try, whether just adding section_parents=[], to the memo solves
the problem in a backwards-compatible manner?

Yep, that sounds worth a try. I should be able to attempt that within the next few hours.

[jayaddison]

Could you try, whether just adding section_parents=[], to the memo solves
the problem in a backwards-compatible manner?

Yep, that sounds worth a try. I should be able to attempt that within the next few hours.

That does indeed work, and seems to be a lighter touch / simpler fix. I'll push a commit with that here in a few moments.

[AA-Turner]

From the PR, it seems the reason is Docutils commit r10093 "Change section handling to not rely on exceptions and reparsing."

From the issue, I am suspicious of "I was having problems with a custom directive's nested parse of its content when section headers were used". Directives generally should not contain section headers, and we have extensive work-arounds in Sphinx for this (see nested_parse_with_titles etc). Can we revert this change in Docutils @gmilde?

A

[gmilde]

From the issue, I am suspicious of "I was having problems with a custom directive's nested parse of its content when section headers were used".

I understand your suspicion. However, the patch published in "the issue" is rather a by-product of working on the custom directive. The orignal "section-in-directive parsing" problem was directly reported to the docutils-devel mailing list. It is not solved by the patch.

The reasons for accepting the patch are:

• It implements a cleaner and more straightforward approach to parsing section headings than the previous "try, fake-error, retry" relying on a section_bubble_up_kludge.
• It is more efficient (I see a small but consistent decrease in the runtime of the test suite).
• It solves bug #346 "reST parser warns twice for short underline" from 2018-07-29 (which is a side-effect of the "retry" in the old implementation).
• It fixes wrong line numbers in system messages.
• It allows us to remove states.RSTStateMachine.memo.section_bubble_up_kludge and states.Line.eofcheck.

Can we revert this change in Docutils?

I am very reluctant to revert/postpone this change (and re-open Bug#346) until it turns out to be an insurmountable obstacle to the compatibility of Doctils 0.23 and Sphinx.

[AA-Turner]

Thanks Günter (and also sorry I haven't had much time recently for Docutils work). Hopefully we can get to the bottom of the failures within Sphinx.

A

[gmilde]

AFAIK, the reason is that the "mock" memo in the test suite needs to set up the section_parents=[] stack introduced in Docutils commit r10093.

The first attempt of a fix failed, maybe, because it copied the initialization from the Docutils source but did not fully adapt it to the different context: The line inliner=state.inliner is replaced by inliner=inliner.

Regarding the failure of the re-occuring failure "8 minutes ago", I can only see Error: (in red) after opening the drop-down list of the failing "Test with pytest".

[AA-Turner]

You may need to refresh the page? I see the below, and then scrolling back shows the previous lines. This is on Firefox.

If that doesn't work at all, you can open the raw logs (gear symbol in the top-right -> 'View raw logs'), it should be a link like https://github.com/sphinx-doc/sphinx/commit/fa12cc6bac609c12afec755ec01d78ae2cce6f96/checks/42086233845/logs

[AA-Turner]

AFAIK, the reason is that the "mock" memo in the test suite needs to set up the section_parents=[] stack introduced in Docutils commit r10093.

See #13548 to introduce section_parents in the mocked state.memo. There are still four failing tests following that change (all marked with @xfail_du_22).

A

[jayaddison]

A note from a post I sent to the docutils-develop mailing list (pending moderation) - there could be a clue in the fact that Sphinx's parsing of only.rst emits some critical-annotated output:

$ sphinx-build -b html tests/roots/test-directive-only/ _build
... [ snip ] ...
 CRITICAL: Title level inconsistent:

1.5.2. Subsection
~~~~~~~~~~~~~~~~~ [docutils]
... [ snip ] ...

NB: this didn't occur with docutils 0.21 -- but that may not necessarily be useful information, because these critical log messages were also added by the same SVN revision r10093 already identified -- so they could theoretically be exposing an existing, albeit hidden, problem -- or they could be a bug related to the patch.

[AA-Turner]

The problem seems to be with the .. only:: directive. The following reproducer shows the error:

from __future__ import annotations

import shutil
from pathlib import Path

from sphinx.cmd.make_mode import run_make_mode

Path('index.rst').write_text("""\
reproducer
==========

1. Section
----------
Lorem ipsum.

1.1. Subsection
~~~~~~~~~~~~~~~
Lorem ipsum.

.. only:: not nonexisting_tag

   2. Section
   ----------
   Lorem ipsum.

3. Section
----------
Lorem ipsum.

.. only:: not nonexisting_tag

   4. Section
   ----------
   Lorem ipsum.

5. Section
----------
Lorem ipsum.

.. only:: not nonexisting_tag

   6. Section
   ----------
   Lorem ipsum.

7. Section
----------
Lorem ipsum.

7.1. Subsection
~~~~~~~~~~~~~~~
Lorem ipsum.

.. only:: not nonexisting_tag

   7.1.1. Subsubsection
   ####################
   Lorem ipsum.

8. Section
------------
Fails!
""", encoding='utf-8')

Path('conf.py').write_bytes(b'')
shutil.rmtree('_build', ignore_errors=True)
run_make_mode(['text', '.', '_build', '--show-traceback', '--fail-on-warning'])

[jayaddison]

Resolved by #13548 + #13549. Thanks @AA-Turner @gmilde!

[AA-Turner]

With #13549 and #13551, tests are now passing. Thanks to @gmilde and @jayaddison for the investigations and help. This is a hard breaking change from Docutils, though, as section_level now seems to have no effect and you must use the section_parents attribute.

This will break breathe, sphinx-needs (cc @chrisjsewell), sphinx.includechangelog, the linux kernel documentation, etc. We (in Docutils) should find a way to keep the new feature but revert the hard breakage, or work to communicate this much better, and ideally propose fixes for the breakage introduced by the changes in Docutils.

A

[AA-Turner]

Re-opening to discuss the ecosystem impact of this breaking change

[gmilde]

Please test, if Docutils commit r10129 fixes/alleviates the compatibility problems.

[gmilde]

@AA-Turner
With Docutils commit r10129, there is no longer a need to add section_parents to a "mock memo". (Section parents are now collected "on the fly".) I recommend reverting commits adding section_parents as to not confuse people looking for a clue how to set up a mock memo...

[gmilde]

The problem seems to be with the .. only:: directive. The following reproducer shows the error:

reproducer
==========
[...] 
8. Section
------------
Fails!

In my local environment, the reproducer no longer fails with sphinx-build 8.2.3 and Docutils HEAD.

[AA-Turner]

See #13560 to remove section_parents. Tests are failing, I will only be able to investigate later, though.

A

[gmilde]

Tests are failing, I will only be able to investigate later, though.

@AA-Turner Could you re-run the tests with current Docutils HEAD, please?
It would be good to have feedback before the release of Docutils 0.22.rc2 on thursday.

Commit r10131 (with sphinx-build 8.2.3) solves the problem in a local test project.

[gmilde]

According to the checks for #13560, tests ran OK 20h ago
(https://github.com/sphinx-doc/sphinx/actions/runs/15124762116/job/42514658174?pr=13560).
Is this a good sign or due to expected failures?

Let's see whether the projects mentioned in #13539 (comment) are fine, too.