gh-131421: fix ASDL grammar for `Dict` to have an `expr?*` keys field #131419

Demonstrandum · 2025-03-18T13:48:08Z

In the ast documentation for Python:

https://docs.python.org/3/library/ast.html#ast.Dict it is made clear that:

When doing dictionary unpacking using dictionary literals the expression to be expanded goes in the values list, with a None at the corresponding position in keys.

Hence, keys is really a expr?* and not a expr*.

Issue: Wrong Python.asdl type for keys in Dict literals #131421

In the `ast` documentation for Python: * https://docs.python.org/3/library/ast.html#ast.Dict it is made clear that: > When doing dictionary unpacking using dictionary literals the expression to be expanded goes in the values list, with a `None` at the corresponding position in `keys`. Hence, `keys` is really a `expr?*` and *not* a `expr*`.

ghost · 2025-03-18T13:48:11Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2025-03-18T13:48:12Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

picnixz · 2025-03-18T14:47:20Z

Parser/Python.asdl

@@ -63,7 +63,7 @@ module Python
         | UnaryOp(unaryop op, expr operand)
         | Lambda(arguments args, expr body)
         | IfExp(expr test, expr body, expr orelse)
-         | Dict(expr* keys, expr* values)
+         | Dict(expr?* keys, expr* values)


This kind of change makes me wonder whether we should also add '+' to indicate 1 or more items. For instance nonlocal requires at least one name, same for import, global and others.

Similarly, there is no explanation of what * and ? actually mean. While they can be inferred, I would expect a caption for them. WDYT @JelleZijlstra?

JelleZijlstra · 2025-03-18T14:53:58Z

Python.asdl is used to generate the AST code, and this change doesn't compile (to reproduce run make regen-ast locally; ideally we'd fix CI to do that).

I don't think it's really worth changing the grammar here.

picnixz · 2025-03-18T14:56:53Z

I don't think it's really worth changing the grammar here.

Can we perhaps add a comment instead? because now that I think about it, for static analysis tools, we could be surprised (like, if we only focus on the ASDL and not on the documented entry itself). [I could be surprised in Sphinx though I don't think I've encountered an issue with this]

Demonstrandum · 2025-03-18T16:38:01Z

It represents a point where it completely breaks expectation, where up until then, the ASDL (as far as I have observed) was being completely respected by the ast module. Since * and ? are modifiers on types, and expr? is still a type, morally a star should be able to follow any type, including expr? to get a expr?*, for example.

Would this be very difficult to parse / break a lot of handling?

JelleZijlstra · 2025-03-18T16:50:29Z

It's been like this for a long time and I don't recall any other reports of confusion about it, so there's a good argument to stick with the working code.

That said, if you can provide a patch that parses this new syntax correctly and that compiles correctly, we can consider it.

Demonstrandum · 2025-03-18T18:17:47Z

@JelleZijlstra I updated the Parser/asdl.py ASDL parser to support chained quantifiers. I made sure the opt and seq interface it exposes remains the same (based of the last quantifier supplied). This means the generated Python/Python-ast.c is unchanged.

Naturally, make regen-ast works now.

tomasr8 · 2025-03-18T18:23:33Z

Parser/asdl.py

+class Quantifier:
+    class _Optional: pass
+    class _Sequence: pass
+    OPTIONAL = _Optional()
+    SEQUENCE = _Sequence()


Could this just be an enum?

Sure, I just prefer these to have unique/singleton values (as opposed to ints or whatnot).

I can change it if its outside the expected coding style.

If you wanted singletons you could also just do OPTIONAL = object() but to be honest I think an Enum/StrEnum is a better choice here since it's more explicit.

(one reason to do the class trick is to have a bit of a nicer repr(), but OTOH, I don't think we need to worry about importing enum)

Is it worth importing enum? If you want to add your suggested change I can commit it.

Yes let's use an enum here, it's the right tool.

picnixz

I'm not sure if this is already the case or not, but can you quickly check that the tests for ast.parse('{}') handle this case?

Parser/asdl.py

picnixz · 2025-03-18T18:30:20Z

Parser/asdl.py

-        elif self.cur_token.kind == TokenKind.Question:
-            is_opt = True
+        quantifiers = []
+        while self.cur_token.kind in (TokenKind.Asterisk, TokenKind.Question):


Shouldn't we check that we don't have ** for instance or ???

Maybe. But both ?? and ** are legitimate types, ** perhaps more legitimate than ?? is.

Ah yes, actually the current parser doesn't care. I thought that it would have warned the user. And should T** actually mean a list of list of T? is the obtained C code designed to handle this kind of grammar?

I have no strong opinion now on whether we should warn ??.

There is essentially no change to the C code at all at this point. I don't know how this was handled before, since expr?* was always there implicitly, but this didn't seem to matter.

Yeah I think this is fine as is

picnixz · 2025-03-18T18:32:44Z

Parser/asdl.py

 class Field(AST):
-    def __init__(self, type, name=None, seq=False, opt=False):
+    def __init__(self, type, name=None, quantifiers=None):


Is there a need to support more than two qualifiers? and/or shouldn't we check the consistency of those qualifiers as well?

Currently there are no more than two quantifiers on any type, as per my change to Python.asdl.

I'm just implementing the EBNF grammar described at the top of the asdl.py file. And I don't think this files does any amount of extra validation so far anyway.

Co-authored-by: Bénédikt Tran <[email protected]>

JelleZijlstra

Looks good, I'll push the enum change

JelleZijlstra · 2025-05-04T22:55:33Z

Parser/asdl.py

+class Quantifier:
+    class _Optional: pass
+    class _Sequence: pass
+    OPTIONAL = _Optional()
+    SEQUENCE = _Sequence()


Yes let's use an enum here, it's the right tool.

bedevere-bot · 2025-05-04T23:08:57Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL 3.x (tier-1) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1609/builds/2443) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1609/builds/2443

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])

bedevere-bot · 2025-05-04T23:11:12Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x (tier-2) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1046/builds/8548) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1046/builds/8548

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])


Traceback (most recent call last):
  File "/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)

bedevere-bot · 2025-05-04T23:12:14Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 NoGIL 3.x (tier-2) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1270/builds/4691) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1270/builds/4691

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)


Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])

bedevere-bot · 2025-05-04T23:13:19Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian 3.x (tier-3) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/424/builds/10540) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/424/builds/10540

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes
test_async_remote_stack_trace - test.test_external_inspection.TestGetStackTrace.test_async_remote_stack_trace

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_external_inspection.py", line 271, in test_async_remote_stack_trace
    self.assertEqual(stack_trace, expected_stack_trace)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Lists differ: [[('c[62 chars]y', 10), ('c4', '/tmp/test_python_kwzu5r9s/tmp[1353 chars]]]]]] != [[('c[62 chars]y', 11), ('c4', '/tmp/test_python_kwzu5r9s/tmp[1368 chars]]]]]]


Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])


Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)

bedevere-bot · 2025-05-04T23:18:26Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86 Debian Non-Debug with X 3.x (no tier) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1245/builds/5302) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1245/builds/5302

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/test_asdl_parser.py"�[0m, line �[35m72�[0m, in �[35mtest_attributes�[0m
    �[31mself.assertEqual�[0m�[1;31m(repr(stmt.attributes[0]), 'Field(int, lineno)')�[0m
    �[31m~~~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
�[1;35mAssertionError�[0m: �[35m'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)
�[0m


Traceback (most recent call last):
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/test_asdl_parser.py"�[0m, line �[35m63�[0m, in �[35mtest_product�[0m
    �[31mself.assertEqual�[0m�[1;31m(�[0m
    �[31m~~~~~~~~~~~~~~~~�[0m�[1;31m^�[0m
        �[1;31mstr(alias),�[0m
        �[1;31m^^^^^^^^^^^�[0m
        �[1;31m'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '�[0m
        �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
        �[1;31m'[Field(int, lineno), Field(int, col_offset), '�[0m
        �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
        �[1;31m'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')�[0m
        �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
�[1;35mAssertionError�[0m: �[35m'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])
�[0m

bedevere-bot · 2025-05-04T23:18:31Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD14 3.x (tier-3) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1232/builds/5658) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1232/builds/5658

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)

PR python#131419 broke this, but we failed to run tests on the PR due to a bug in our script.

JelleZijlstra · 2025-05-04T23:27:36Z

Tests didn't run due to a bug in our logic and unfortunately this did break some tests, I put up another PR to fix.

bedevere-bot · 2025-05-04T23:34:44Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD 3.x (tier-3) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1223/builds/6120) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1223/builds/6120

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])


Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)

PR #131419 broke this, but we failed to run tests on the PR due to a bug in our script.

bedevere-bot · 2025-05-05T00:07:32Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x (no tier) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/469/builds/10996) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/469/builds/10996

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])

bedevere-bot · 2025-05-05T00:19:43Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 MacOS Intel NoGIL 3.x (tier-1) has failed when building commit 3084070.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1258/builds/4902) and take a look at the build logs.
Check if the failure is related to this commit (3084070) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1258/builds/4902

Failed tests:

test_asdl_parser

Failed subtests:

test_product - test.test_asdl_parser.TestAsdlParser.test_product
test_attributes - test.test_asdl_parser.TestAsdlParser.test_attributes

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/test/test_asdl_parser.py", line 72, in test_attributes
    self.assertEqual(repr(stmt.attributes[0]), 'Field(int, lineno)')
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Field(int, lineno, quantifiers=[])' != 'Field(int, lineno)'
- Field(int, lineno, quantifiers=[])
+ Field(int, lineno)


Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/traceback.py", line 14, in <module>
    import _colorize
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/_colorize.py", line 163, in <module>
    set_theme()
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/_colorize.py", line 148, in set_theme
    colors = get_colors()
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/_colorize.py", line 92, in get_colors
    if colorize or can_colorize(file=file):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/_colorize.py", line 136, in can_colorize
    return os.isatty(file.fileno())
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/resource_tracker.py", line 314, in main
    _CLEANUP_FUNCS[rtype](name)
FileNotFoundError: [Errno 2] No such file or directory


Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/test/_test_multiprocessing.py", line 513, in _sleep_some
    time.sleep(100)
    ~~~~~~~~~~^^^^^
KeyboardInterrupt
k


Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/util.py", line 216, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/synchronize.py", line 87, in _cleanup
    unregister(name, "semaphore")
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/resource_tracker.py", line 216, in unregister
    self._send('UNREGISTER', name, rtype)
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/resource_tracker.py", line 226, in _send
    warnings.warn(
UserWarning: ResourceTracker called reentrantly for resource cleanup, which is unsupported. The semaphore object '/mp-5xlj1a0h' might leak.
Warning -- Unraisable exception
Exception ignored while calling weakref callback <Finalize object, dead>:
Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/util.py", line 216, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/synchronize.py", line 87, in _cleanup
    unregister(name, "semaphore")
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/resource_tracker.py", line 216, in unregister
    self._send('UNREGISTER', name, rtype)
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/resource_tracker.py", line 226, in _send
    warnings.warn(
UserWarning: ResourceTracker called reentrantly for resource cleanup, which is unsupported. The semaphore object '/mp-s8g8kwq9' might leak.
k


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/multiprocessing/resource_tracker.py", line 317, in main
    warnings.warn('resource_tracker: %r: %s' % (name, e))
UserWarning: resource_tracker: '/mp-s8g8kwq9': [Errno 2] No such file or directory


Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-intel-aws.nogil/build/Lib/test/test_asdl_parser.py", line 63, in test_product
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        str(alias),
        ^^^^^^^^^^^
        'Product([Field(identifier, name), Field(identifier, asname, opt=True)], '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '[Field(int, lineno), Field(int, col_offset), '
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Prod[22 chars] name, quantifiers=[]), Field(identifier, asna[209 chars]])])' != 'Prod[22 chars] name), Field(identifier, asname, opt=True)], [113 chars]e)])'
- Product([Field(identifier, name, quantifiers=[]), Field(identifier, asname, quantifiers=[, OPTIONAL])], [Field(int, lineno, quantifiers=[]), Field(int, col_offset, quantifiers=[]), Field(int, end_lineno, quantifiers=[, OPTIONAL]), Field(int, end_col_offset, quantifiers=[, OPTIONAL])])
+ Product([Field(identifier, name), Field(identifier, asname, opt=True)], [Field(int, lineno), Field(int, col_offset), Field(int, end_lineno, opt=True), Field(int, end_col_offset, opt=True)])

Demonstrandum requested review from pablogsal and lysnikolaou as code owners March 18, 2025 13:48

bedevere-app bot added the awaiting review label Mar 18, 2025

Demonstrandum changed the title ~~Python.asdl: Fix Dict variant to have an expr?* keys field.~~ gh-131421: Python.asdl: Fix Dict variant to have an expr?* keys field. Mar 18, 2025

bedevere-app bot mentioned this pull request Mar 18, 2025

Wrong Python.asdl type for keys in Dict literals #131421

Closed

picnixz added the skip news label Mar 18, 2025

picnixz reviewed Mar 18, 2025

View reviewed changes

picnixz changed the title ~~gh-131421: Python.asdl: Fix Dict variant to have an expr?* keys field.~~ gh-131421: fix ASDL grammar for Dict to have an expr?* keys field Mar 18, 2025

Update Parser/asdl.py to parse multiple chained quantifiers.

2a19020

Demonstrandum requested review from isidentical, JelleZijlstra and Eclips4 as code owners March 18, 2025 18:14

tomasr8 reviewed Mar 18, 2025

View reviewed changes

picnixz reviewed Mar 18, 2025

View reviewed changes

Remove newline before return.

cbccd92

Co-authored-by: Bénédikt Tran <[email protected]>

JelleZijlstra reviewed May 4, 2025

View reviewed changes

JelleZijlstra added 2 commits May 4, 2025 15:58

enum

da764a3

Merge remote-tracking branch 'upstream/main' into Demonstrandum/main

9406ddb

JelleZijlstra merged commit 3084070 into python:main May 4, 2025
23 checks passed

bedevere-app bot removed the awaiting review label May 4, 2025

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this pull request May 4, 2025

pythongh-131421: Fix ASDL tests

d9916d1

PR python#131419 broke this, but we failed to run tests on the PR due to a bug in our script.

This was referenced May 4, 2025

gh-131421: Fix ASDL tests #133408

Merged

Tools/build/compute-changes.py looks at the wrong branches #133410

Open

ambv pushed a commit that referenced this pull request May 4, 2025

gh-131421: Fix ASDL tests (#133408)

483d130

PR #131419 broke this, but we failed to run tests on the PR due to a bug in our script.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-131421: fix ASDL grammar for `Dict` to have an `expr?*` keys field #131419

gh-131421: fix ASDL grammar for `Dict` to have an `expr?*` keys field #131419

Demonstrandum commented Mar 18, 2025 •

edited by bedevere-app bot

Loading

ghost commented Mar 18, 2025 •

edited by ghost

Loading

bedevere-app bot commented Mar 18, 2025

picnixz Mar 18, 2025

JelleZijlstra commented Mar 18, 2025

picnixz commented Mar 18, 2025 •

edited

Loading

Demonstrandum commented Mar 18, 2025

JelleZijlstra commented Mar 18, 2025

Demonstrandum commented Mar 18, 2025

tomasr8 Mar 18, 2025

Demonstrandum Mar 18, 2025

tomasr8 Mar 18, 2025

picnixz Mar 18, 2025

Demonstrandum Mar 19, 2025

JelleZijlstra May 4, 2025

picnixz left a comment

picnixz Mar 18, 2025

Demonstrandum Mar 18, 2025

picnixz Mar 18, 2025

Demonstrandum Mar 18, 2025

JelleZijlstra May 4, 2025

picnixz Mar 18, 2025

Demonstrandum Mar 18, 2025

JelleZijlstra left a comment

JelleZijlstra May 4, 2025

bedevere-bot commented May 4, 2025

bedevere-bot commented May 4, 2025

bedevere-bot commented May 4, 2025

bedevere-bot commented May 4, 2025

bedevere-bot commented May 4, 2025

bedevere-bot commented May 4, 2025

JelleZijlstra commented May 4, 2025

bedevere-bot commented May 4, 2025

bedevere-bot commented May 5, 2025

bedevere-bot commented May 5, 2025

gh-131421: fix ASDL grammar for Dict to have an expr?* keys field #131419

gh-131421: fix ASDL grammar for Dict to have an expr?* keys field #131419

Conversation

Demonstrandum commented Mar 18, 2025 • edited by bedevere-app bot Loading

ghost commented Mar 18, 2025 • edited by ghost Loading

bedevere-app bot commented Mar 18, 2025

Choose a reason for hiding this comment

JelleZijlstra commented Mar 18, 2025

picnixz commented Mar 18, 2025 • edited Loading

Demonstrandum commented Mar 18, 2025

JelleZijlstra commented Mar 18, 2025

Demonstrandum commented Mar 18, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

picnixz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelleZijlstra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

JelleZijlstra commented May 4, 2025

bedevere-bot commented May 4, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 5, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 5, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gh-131421: fix ASDL grammar for `Dict` to have an `expr?*` keys field #131419

gh-131421: fix ASDL grammar for `Dict` to have an `expr?*` keys field #131419

Demonstrandum commented Mar 18, 2025 •

edited by bedevere-app bot

Loading

ghost commented Mar 18, 2025 •

edited by ghost

Loading

picnixz commented Mar 18, 2025 •

edited

Loading