tokenize.py: NEWLINE/NL mixup after line continuation character in indented block

[lysnikolaou]

Bug report

There's the following difference in behavior between 3.11 and 3.12.

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ cat tmp/t.py         
def f():
  x = 0
  \
  %                                                                                                                                                                             

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ../cpython-versions/3.11.3/build/bin/python3.11 -m tokenize tmp/t.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,5:            NAME           'f'            
1,5-1,6:            OP             '('            
1,6-1,7:            OP             ')'            
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,2:            INDENT         '  '           
2,2-2,3:            NAME           'x'            
2,4-2,5:            OP             '='            
2,6-2,7:            NUMBER         '0'            
2,7-2,8:            NEWLINE        '\n'           
4,2-4,3:            NEWLINE        ''             
5,0-5,0:            DEDENT         ''             
5,0-5,0:            ENDMARKER      ''             

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe -m tokenize tmp/t.py                                   
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,5:            NAME           'f'            
1,5-1,6:            OP             '('            
1,6-1,7:            OP             ')'            
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,2:            INDENT         '  '           
2,2-2,3:            NAME           'x'            
2,4-2,5:            OP             '='            
2,6-2,7:            NUMBER         '0'            
2,7-2,8:            NEWLINE        '\n'           
4,2-4,3:            NL             ''             <-- This used to be a NEWLINE
5,0-5,0:            DEDENT         ''             
5,0-5,0:            ENDMARKER      ''             

I tend to believe that the previous NEWLINE was correct, but I'm not sure. Keep in mind, this breaks some tests on IPython. What do you think?

cc @pablogsal @mgmacias95

[pablogsal]

@lysnikolaou Hummmm, I cannot reproduce this on my side with the latest main:

❯ cat lel.py
def f():
  x = 0
  \
  %

main on  main [$?⇕]
❯ ./python.exe --version
Python 3.13.0a0

main on  main [$?⇕]
❯ ./python.exe -m tokenize lel.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,5:            NAME           'f'
1,5-1,6:            OP             '('
1,6-1,7:            OP             ')'
1,7-1,8:            OP             ':'
1,8-1,9:            NEWLINE        '\n'
2,0-2,2:            INDENT         '  '
2,2-2,3:            NAME           'x'
2,4-2,5:            OP             '='
2,6-2,7:            NUMBER         '0'
2,7-2,8:            NEWLINE        '\n'
4,2-4,3:            OP             '%'
4,3-4,4:            NEWLINE        '\n'
5,0-5,0:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

`

[pablogsal]

Ah, maybe that '%' is a EOF character of some sort?

[lysnikolaou]

@lysnikolaou Hummmm, I cannot reproduce this on my side with the latest main:

The % sign is not there. It's output cat adds to my file to signify that there's no newline. The file itself only contains the following (underscores signify spaces):

def f():
__x = 0
__\
__  

[pablogsal]

Ah, indeed, I can reproduce now:

❯ cat
def f():
  x = 0
  \


❯ python -m tokenize lel.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,5:            NAME           'f'
1,5-1,6:            OP             '('
1,6-1,7:            OP             ')'
1,7-1,8:            OP             ':'
1,8-1,9:            NEWLINE        '\n'
2,0-2,2:            INDENT         '  '
2,2-2,3:            NAME           'x'
2,4-2,5:            OP             '='
2,6-2,7:            NUMBER         '0'
2,7-2,8:            NEWLINE        '\n'
4,0-4,1:            NEWLINE        '\n'
5,0-5,0:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

main on  main [$?⇕]
❯ ./python.exe -m tokenize lel.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,5:            NAME           'f'
1,5-1,6:            OP             '('
1,6-1,7:            OP             ')'
1,7-1,8:            OP             ':'
1,8-1,9:            NEWLINE        '\n'
2,0-2,2:            INDENT         '  '
2,2-2,3:            NAME           'x'
2,4-2,5:            OP             '='
2,6-2,7:            NUMBER         '0'
2,7-2,8:            NEWLINE        '\n'
4,0-4,1:            NL             '\n'
5,0-5,0:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

[pablogsal]

Well, I am a bit torn, as this is technically invalid (incomplete) Python, so the output is technically not stable, no?

[lysnikolaou]

It's not limited to incomplete input though, this was just a minimal example. This valid file runs, but contains the difference as well.

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ cat tmp/t.py | tr " " "_"
def_f():
__print(0)
__\
__
f()

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe -m tokenize tmp/t.py                                   
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,5:            NAME           'f'            
1,5-1,6:            OP             '('            
1,6-1,7:            OP             ')'            
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,2:            INDENT         '  '           
2,2-2,7:            NAME           'print'        
2,7-2,8:            OP             '('            
2,8-2,9:            NUMBER         '0'            
2,9-2,10:           OP             ')'            
2,10-2,11:          NEWLINE        '\n'           
4,2-4,3:            NL             '\n'           
5,0-5,0:            DEDENT         ''             
5,0-5,1:            NAME           'f'            
5,1-5,2:            OP             '('            
5,2-5,3:            OP             ')'            
5,3-5,4:            NEWLINE        '\n'           
6,0-6,0:            ENDMARKER      ''             

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ../cpython-versions/3.11.3/build/bin/python3.11 -m tokenize tmp/t.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,5:            NAME           'f'            
1,5-1,6:            OP             '('            
1,6-1,7:            OP             ')'            
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,2:            INDENT         '  '           
2,2-2,7:            NAME           'print'        
2,7-2,8:            OP             '('            
2,8-2,9:            NUMBER         '0'            
2,9-2,10:           OP             ')'            
2,10-2,11:          NEWLINE        '\n'           
4,2-4,3:            NEWLINE        '\n'           
5,0-5,0:            DEDENT         ''             
5,0-5,1:            NAME           'f'            
5,1-5,2:            OP             '('            
5,2-5,3:            OP             ')'            
5,3-5,4:            NEWLINE        '\n'           
6,0-6,0:            ENDMARKER      ''             

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe tmp/t.py                                               
0

[pablogsal]

Ah, thanks for clarifying! I understand now the problem. This is how I see this: IIRC NL means that the new line character is not semantically relevant, which means that technically we can remove it and the code would do the same no? For example in this code:

def f():
  x = 0




f()

we can see (in both 3.11 and 3.12):

0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,5:            NAME           'f'
1,5-1,6:            OP             '('
1,6-1,7:            OP             ')'
1,7-1,8:            OP             ':'
1,8-1,9:            NEWLINE        '\n'
2,0-2,2:            INDENT         '  '
2,2-2,3:            NAME           'x'
2,4-2,5:            OP             '='
2,6-2,7:            NUMBER         '0'
2,7-2,8:            NEWLINE        '\n'
3,0-3,1:            NL             '\n'
4,0-4,1:            NL             '\n'
5,0-5,1:            NL             '\n'
6,0-6,1:            NL             '\n'
7,0-7,0:            DEDENT         ''
7,0-7,1:            NAME           'f'
7,1-7,2:            OP             '('
7,2-7,3:            OP             ')'
7,3-7,4:            NEWLINE        '\n'
8,0-8,0:            ENDMARKER      ''

Those NL mean that they can be removed and the function will tokenize the same semantically. In our case, it seems that is indeed the case as well, no? Because when we write:

def_f():
__print(0)
__\
__
f()

Technically this is the same as:

def_f():
__print(0)
____
f()

And that tokenizes as:

0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,5:            NAME           'f'
1,5-1,6:            OP             '('
1,6-1,7:            OP             ')'
1,7-1,8:            OP             ':'
1,8-1,9:            NEWLINE        '\n'
2,0-2,2:            INDENT         '  '
2,2-2,3:            NAME           'x'
2,4-2,5:            OP             '='
2,6-2,7:            NUMBER         '0'
2,7-2,8:            NEWLINE        '\n'
3,0-3,1:            NL             '\n'
4,0-4,0:            DEDENT         ''
4,0-4,1:            NAME           'f'
4,1-4,2:            OP             '('
4,2-4,3:            OP             ')'
4,3-4,4:            NEWLINE        '\n'
5,0-5,0:            ENDMARKER      ''

This is true for both tokenizers (notice the NEWLINE + NL). So i think the correct thing here is indeed NEWLINE + NL so the new code is right and the old one is wrong.

[lysnikolaou]

Agreed. Closing this then.