Since July update cause java highlight break

[ericsia]

I love to use VSC Java plugin but I had so much pain working with java now.
Now the highlight is glitchy, it don't highlight properly when Semantic Highlight is turn on.
Comment don't get highlighted :( I feel sad
Also I would appreciate it if it is easier to use junit assertEqual, when I try to open a single .java file, it cannot import org.junit

Environment

• Operating System: Windows 10
• JDK version: 14
• Visual Studio Code version: 1.48.2 before update also the same
• Java extension version: -

Steps To Reproduce

1. Turn on semantic highlight
2. start debugging

Current Result

Expected Result

Additional Informations

[fbricon]

Are you saying semantic highlighting is fine when not debugging?

[ericsia]

Are you saying semantic highlighting is fine when not debugging?

after I close and open it sometime it appears fine

[0dinD]

Does the highlighting also "fix" itself after you type a character?

Not sure if it's related, but I've seen something very similar happen when applying large changes to the document, such as undo/redo or using git reset --hard. I unfortunately could not find a way to consistently reproduce the issue, but the semantic highlighting broke in the same way as in your screenshot.

@ericsia if you could find a way to consistently reproduce the issue, that would help a lot in finding a fix. Does it always happen when debugging? Also, what version of the extension are you using? The latest?

[clankill3r]

I also notice that this happens a lot since July, I'm on a mac OSX 10.12.6.
Often typing and deleting something fixes it, but it stays annoying.

[0dinD]

@clankill3r can you provide some more info about what you're doing when the issue occurs? It would make the debugging process a lot easier if we had a way to reproduce the issue. And by "since July", do you mean the 0.65.0 update or do you not know which version you were using?

Personally I've seen this issue occur earlier than 0.65.0, but I couldn't find a way to consistently reproduce it. It seemed to occur somewhat randomly when applying large changes to the document. Spamming undo/redo sometimes reproduced it, but there was not pattern to it.

[ericsia]

@clankill3r can you provide some more info about what you're doing when the issue occurs? It would make the debugging process a lot easier if we had a way to reproduce the issue. And by "since July", do you mean the 0.65.0 update or do you not know which version you were using?

Personally I've seen this issue occur earlier than 0.65.0, but I couldn't find a way to consistently reproduce it. It seemed to occur somewhat randomly when applying large changes to the document. Spamming undo/redo sometimes reproduced it, but there was not pattern to it.

most accurately I didn't know since when do I turn it on. During last 2 update if I didn't rmb wrongly, I saw VS Code prompt me a msg to enable semantic highlight. So I go and turn it on. There is where I notice the highlights broken.
It is the best to try trace back from older build

[ericsia]

Does the highlighting also "fix" itself after you type a character?

Not sure if it's related, but I've seen something very similar happen when applying large changes to the document, such as undo/redo or using git reset --hard. I unfortunately could not find a way to consistently reproduce the issue, but the semantic highlighting broke in the same way as in your screenshot.

@ericsia if you could find a way to consistently reproduce the issue, that would help a lot in finding a fix. Does it always happen when debugging? Also, what version of the extension are you using? The latest?

During I start typing code from scratch I think. When I added comment, add more code then it might occurs

[0dinD]

During last 2 update if I didn't rmb wrongly

Ok, that strengthens my belief that this issue was not introduced via any of my changes in 0.65.0, and has been present since the initial implementation.

Again, if someone could provide a way to consistently reproduce this issue (via steps), that would make the debugging process a lot less time-consuming. (Meaning a fix can be released sooner.)

@ericsia how often do you estimate that this issue appears for you? And what's the connection between this issue and debug mode? Are you saying that the issue only appears when debugging, or that it appears more often when debugging?

[ericsia]

During last 2 update if I didn't rmb wrongly

Ok, that strengthens my belief that this issue was not introduced via any of my changes in 0.65.0, and has been present since the initial implementation.

Again, if someone could provide a way to consistently reproduce this issue (via steps), that would make the debugging process a lot less time-consuming. (Meaning a fix can be released sooner.)

@ericsia how often do you estimate that this issue appears for you? And what's the connection between this issue and debug mode? Are you saying that the issue only appears when debugging, or that it appears more often when debugging?

The color highlight when get messed like that doesn't go back to normal (when stop debug) until I reopen vscode again. I am also trying hard to find how to reproduce it but right now I'm not writing any code atm.
Now when I freshly open the java code everything look normal. But maybe tomorrow or 2 days later I will need to code again I will pay extra attention

[0dinD]

While I have not come up with a reliable way to reproduce this issue, it did occur again during development, so I thought I'd share another screenshot. For me it occurred quite randomly when typing code, possibly as an effect of pasting ~10 lines of code, but I'm not sure yet whether the two were related. The bug doesn't happen very often, but it's still annoying the few times where it does.

What's interesting to note is that just like the screenshot shared in the first comment of this issue, the tokens are not completely wrong, just offset by a few characters. Weirdly enough, the Developer: Inspect Editor Tokens and Scopes command reports that the semantic tokens are in the correct positions. So the offset either only appears visually, or the Developer: Inspect Editor Tokens and Scopes command is somehow also broken and doesn't show the correct information.

The issue goes away as soon as new semantic tokens are requested, so all you have to do is type a character and the highlighting is no longer broken.

[0dinD]

This issue has proven hard to debug, but I believe I've found a possible explanation, even if I don't know the exact cause or how to fix it. It seems to occur in some rare cases, when changes are made to the document during a request for semantic tokens.

The most reliable I've been able to reproduce it is to make multiple small changes to the document in rapid succession, but the timing required to reproduce the issue is very specific. I would also guess that the issue can occur when there is a change to the document during a particularly long running request for semantic tokens. Perhaps then, the issue will occur more frequently on machines with lower performance or when working with very large Java files.

All of the information that I've gathered seems to suggest that this is a bug in VS Code, rather than this extension. But again, I'm not entirely sure, as this issue is hard to reproduce. The code below this line looks like it could be part of the cause. It seems to shift the semantic tokens around and wait for new ones, if changes are made to the document during a request for semantic tokens.

I think I've noticed this behavior when trying to reproduce the issue, as the tokens do get shifted around (like in the previous screenshot!), only to get corrected a split second later. My best guess is that the request for new semantic tokens occasionally gets ignored on this line for some reason, in which case this issue would appear since the tokens have been shifted around but not fixed by requesting new ones. Again, as soon as you type a character, new semantic tokens are requested and the issue is gone.

If someone with more time on their hands or experience with the VS Code source-code could look into this, that would be greatly appreciated. I might submit an issue over at the VS Code repo, but right now I'm not sure if it's entirely the fault of VS Code, and I don't have a very reliable way to reproduce the issue.

[JoeyRxy]

yeah, I've encountered this problem since the beginning of semantic color rendering tech.
it's a severe problem I think.

[Eskibear]

@0dinD good findings. I also observe the same behavior, but it prevents me from digging deep as I cannot stably reproduce it. The root cause might be either of the below:
a) JDT.LS calculates semantic tokens based on an out-of-date state (corresponding CompilationUnit) of current document, and vscode highlights the file using wrong tokens.
b) JDT.LS does provide the correct result, but vscode somehow wrongly highlight the document.

If you have time, one way to validate might be, setting java.trace to "verbose" and to inspect the request and response. But It won't be easy as the responses are already encoded. If we do find clues/evidence to prove it's vscode's issue, we can request them for a fix.

[0dinD]

@Eskibear thanks for the tip, I'll try to figure out more details when I can. One thing that makes me think it's more of a VS Code-issue is that the Developer: Inspect Editor Tokens and Scopes command is reporting the correct positions for the tokens, while in reality, the text color has been offset by a few characters.

[ericsia]

@Eskibear thanks for the tip, I'll try to figure out more details when I can. One thing that makes me think it's more of a VS Code-issue is that the Developer: Inspect Editor Tokens and Scopes command is reporting the correct positions for the tokens, while in reality, the text color has been offset by a few characters.

here I produced it again, after copy paste. I am copy paste from a document to another java file.
Also I used find and replace a few times already.
Not just that, if you look carefully it said the syntax that is correct as wrong, in this case the right bracket

[0dinD]

@ericsia does the highlighted problem appear at the same time as the highlighting breaks? And does it also go away when the highlighting problems "fix" themselves? If the two are related, that's an interesting find.

I spent some more time debugging the semantic highlighting issue today, and come to the following conclusions:

• The issue is definitely caused by eclipse.jdt.ls, not VS Code.
  • I have verified that VS Code receives tokens with incorrect positions, by decoding the tokens on the client-side and printing the text at their position.
  • The reason why the Developer: Inspect Editor Tokens and Scopes command was reporting the correct positions is because it sends a separate request for semantic tokens, which means that it will always show the correct tokens while the editor could still have wrong ones.
• This probably means that whatever model of the document eclipse.jdt.ls is using to encode the tokens does not get updated when the document has changes during a request.

@Eskibear @fbricon do you have any idea why this might be happening? Are there any good ways to check if the document has had changes during the request? If so, maybe we can recompute the token positions when a change to the document is detected during the request. I'm just not that familiar with the Eclipse JDT APIs.

[ericsia]

@ericsia does the highlighted problem appear at the same time as the highlighting breaks? And does it also go away when the highlighting problems "fix" themselves? If the two are related, that's an interesting find.

I spent some more time debugging the semantic highlighting issue today, and come to the following conclusions:

• The issue is definitely caused by eclipse.jdt.ls, not VS Code.

  • I have verified that VS Code receives tokens with incorrect positions, by decoding the tokens on the client-side and printing the text at their position.
  • The reason why the Developer: Inspect Editor Tokens and Scopes command was reporting the correct positions is because it sends a separate request for semantic tokens, which means that it will always show the correct tokens while the editor could still have wrong ones.

• This probably means that whatever model of the document eclipse.jdt.ls is using to encode the tokens does not get updated when the document has changes during a request.

@Eskibear @fbricon do you have any idea why this might be happening? Are there any good ways to check if the document has had changes during the request? If so, maybe we can recompute the token positions when a change to the document is detected during the request. I'm just not that familiar with the Eclipse JDT APIs.

Yes. When I notice the color started to change, often it tells me some syntax error. But I can still compile it. The color won't fix automatically for me, I have to close vs code and reopen to appear normal again.

[0dinD]

Finally, after another debugging session, I think I've found the real issue, and a fix for it. The strange offset for semantic tokens seems to be introduced on this line, when the encoding algorithm needs to figure out the line and column number of a semantic token. The reason for the offset is that the document used to calculate the line and column number from the offset can sometimes get out-of-sync in the middle of a semantic tokens request, due to frequent updates of the document. This means that some token positions will be calculated based on an older version of the document, which causes them to be offset by the number of characters in the document edit.

@fbricon @Eskibear I've submitted a PR to fix this issue, could you please take a look at it?

[0dinD]

After playing around a little more with my changes in eclipse-jdtls/eclipse.jdt.ls#1552, I realized that there were two separate problems that can cause the tokens to become offset while editing.

1. The document used by SemanticTokensVisitor to calculate the semantic token positions can become out-of-sync with its corresponding compilation unit, causing the token position calculations to become offset. (fixed by Fix semantic tokens offset due to document updates eclipse-jdtls/eclipse.jdt.ls#1552)

2. The document used on the server side can become out-of-sync with the real document on the client side. This issue is a bit less noticeable since it won't cause the tokens to become offset unless text has changed which affects the line or column number of the semantic tokens. Additionally, this issue seems to always fix itself a split second after you see it, and never "gets stuck" like the first issue. This is probably because of the fact that VS Code seems to requests new tokens after it detects changes during a request. After looking at the implementation of the TypeScript semantic tokens provider, I believe I have found a fix for this issue as well, see Fix semantic tokens offset due to document updates #1632.

When using both eclipse-jdtls/eclipse.jdt.ls#1552 and #1632, I can no longer reproduce this issue, no matter how many changes I spam to the document.

@fbricon @Eskibear please have a look at both PRs.

[0dinD]

@ericsia the strange offset to warning underlines looks like it could be somewhat related, but I decided to create a new issue (#1633) for it since I do not have as much insight into the code behind that feature and how to fix it.

[ericsia]

Finally, after another debugging session, I think I've found the real issue, and a fix for it. The strange offset for semantic tokens seems to be introduced on this line, when the encoding algorithm needs to figure out the line and column number of a semantic token. The reason for the offset is that the document used to calculate the line and column number from the offset can sometimes get out-of-sync in the middle of a semantic tokens request, due to frequent updates of the document. This means that some token positions will be calculated based on an older version of the document, which causes them to be offset by the number of characters in the document edit.

@fbricon @Eskibear I've submitted a PR to fix this issue, could you please take a look at it?

omg you are amazing, you are the first one to discovered the fix. Owh that makes more sense now

[ericsia]

@ericsia the strange offset to warning underlines looks like it could be somewhat related, but I decided to create a new issue (#1633) for it since I do not have as much insight into the code behind that feature and how to fix it.

@0dinD it is getting worst after vscode since 2 updates before. Whenever I save the code, it auto-format and everything get messed up.

[0dinD]

@ericsia is this about #1597 or #1633 (or both)? Could you provide me with some details about which formatter and settings you're using? It would be really helpful if you've found a way to reliably reproduce the issue.

For the record, the issue with the semantic tokens doesn't seem to be entirely fixed, because I've still seen it happen on very rare occasions. Usually it happens when doing multiple edits in quick succession at the same time as I'm starting some CPU-intensive process (which will cause the language server to be slow at computing the semantic tokens). #1632 seems to have mitigated the issue, but not fixed the root cause.

Unfortunately I don't have much time to investigate this at the moment, so any help in reproducing the issue or finding the cause is greatly appreciated.

[ericsia]

@ericsia is this about #1597 or #1633 (or both)? Could you provide me with some details about which formatter and settings you're using? It would be really helpful if you've found a way to reliably reproduce the issue.

For the record, the issue with the semantic tokens doesn't seem to be entirely fixed, because I've still seen it happen on very rare occasions. Usually it happens when doing multiple edits in quick succession at the same time as I'm starting some CPU-intensive process (which will cause the language server to be slow at computing the semantic tokens). #1632 seems to have mitigated the issue, but not fixed the root cause.

Unfortunately I don't have much time to investigate this at the moment, so any help in reproducing the issue or finding the cause is greatly appreciated.

my bad, it is the bug after plugin update #1917

[ericsia]

@ericsia is this about #1597 or #1633 (or both)? Could you provide me with some details about which formatter and settings you're using? It would be really helpful if you've found a way to reliably reproduce the issue.

For the record, the issue with the semantic tokens doesn't seem to be entirely fixed, because I've still seen it happen on very rare occasions. Usually it happens when doing multiple edits in quick succession at the same time as I'm starting some CPU-intensive process (which will cause the language server to be slow at computing the semantic tokens). #1632 seems to have mitigated the issue, but not fixed the root cause.

Unfortunately I don't have much time to investigate this at the moment, so any help in reproducing the issue or finding the cause is greatly appreciated.

They broke the offset again like 3 updates ago. At first I thought it was the workspace request permission fault.
Anyway, because I got auto-indentation On when I click save, now each time when I hit debug my code got messed up. Worst is I can't CTRL + Z to revert it back, if I do the old code get pasted somewhere else. Omg this is a very serious issue @fbricon @0dinD

[0dinD]

@ericsia Right, so you have format-on-save, that does seem likely to offset the semantic tokens based on my findings. Starting the debugger is likely also a contributing factor, as that may increase response times from the language server due to CPU load, which increases the likelihood of the semantic tokens becoming offset.

Worst is I can't CTRL + Z to revert it back, if I do the old code get pasted somewhere else.

Not sure what you mean here, can't you just undo and then redo? (Ctrl + Z then Ctrl + Y)
The semantic tokens should "repair" themselves if you just change something in the code, even if you change it back. Either that or you've found another more serious bug.

Either way, the original "fix" for this issue was more of a workaround (#1632), but with my new findings in eclipse-jdtls/eclipse.jdt.ls#1806 I think I've found a more robust solution to fix this bug once and for all. Speaking of, do you have time to review eclipse-jdtls/eclipse.jdt.ls#1806 and the related #2000 @Eskibear?

See this comment, specifically the second section, for a more detailed explanation of how this offset bug happens with semantic tokens.

[ericsia]

@ericsia Right, so you have format-on-save, that does seem likely to offset the semantic tokens based on my findings. Starting the debugger is likely also a contributing factor, as that may increase response times from the language server due to CPU load, which increases the likelihood of the semantic tokens becoming offset.

Worst is I can't CTRL + Z to revert it back, if I do the old code get pasted somewhere else.

Not sure what you mean here, can't you just undo and then redo? (Ctrl + Z then Ctrl + Y)
The semantic tokens should "repair" themselves if you just change something in the code, even if you change it back. Either that or you've found another more serious bug.

Either way, the original "fix" for this issue was more of a workaround (#1632), but with my new findings in eclipse/eclipse.jdt.ls#1806 I think I've found a more robust solution to fix this bug once and for all. Speaking of, do you have time to review eclipse/eclipse.jdt.ls#1806 and the related #2000 @Eskibear?

See this comment, specifically the second section, for a more detailed explanation of how this offset bug happens with semantic tokens.

If I don't remember incorrectly it can't be REDO after UNDO too since the offset is changed.
Owh btw, I am using this profile due to school asked us to line up braces since this plugin doesn't have option to line up braces plus the default switch-case indentation is weird.
Not sure if this is the cause.
configuration.zip

[0dinD]

If I don't remember incorrectly it can't be REDO after UNDO too since the offset is changed.
Owh btw, I am using this profile due to school asked us to line up braces since this plugin doesn't have option to line up braces plus the default switch-case indentation is weird.
Not sure if this is the cause.

Is the issue you're talking about related to the code being formatted wrong, or the syntax highlighting (semantic tokens) being wrong? Could you please share some screenshots showing the current problem?

Also, you should always be able to redo a change that you have just undone, not sure what you mean by "it can't be REDO after UNDO too since the offset is changed". You may need to undo/redo more than once since the formatter can create multiple edits.

[ericsia]

If I don't remember incorrectly it can't be REDO after UNDO too since the offset is changed.
Owh btw, I am using this profile due to school asked us to line up braces since this plugin doesn't have option to line up braces plus the default switch-case indentation is weird.
Not sure if this is the cause.

Is the issue you're talking about related to the code being formatted wrong, or the syntax highlighting (semantic tokens) being wrong? Could you please share some screenshots showing the current problem?

Also, you should always be able to redo a change that you have just undone, not sure what you mean by "it can't be REDO after UNDO too since the offset is changed". You may need to undo/redo more than once since the formatter can create multiple edits.

code being formatted wrong
I can't reproduce it now atm, if I met it again I'll snapshot it

[0dinD]

If what you've been talking about is a bug with the formatter rather than syntax highlighting, please report that in a new issue (after searching for an existing one). This issue tracks the bug where semantic tokens sometimes got offset during multiple edits, causing strange syntax highlighting artifacts as shown in your initial screenshot.

[ericsia]

If what you've been talking about is a bug with the formatter rather than syntax highlighting, please report that in a new issue (after searching for an existing one). This issue tracks the bug where semantic tokens sometimes got offset during multiple edits, causing strange syntax highlighting artifacts as shown in your initial screenshot.

I just reproduced it. Apparently it can undo but now I can't save it.
This happens when I start typing code for long time.

[Eskibear]

@ericsia It looks more like a formating issue. Have you turned on "formatOnSave"? I suggest you to do following to get help:

• Create a new issue talking about this issue, including the GIF above. If possible, attach a sample project and describe how to reproduce it.
• Check if vscode setting "Format On Save" is turned on . If so, check if it works after turning it off.
  

[ericsia]

@ericsia It looks more like a formating issue. Have you turned on "formatOnSave"? I suggest you to do following to get help:

• Create a new issue talking about this issue, including the GIF above. If possible, attach a sample project and describe how to reproduce it.
• Check if vscode setting "Format On Save" is turned on . If so, check if it works after turning it off.
  

the thing is I need the format on save thought that is why I turn it on. I see, I thought this belong to same category of issue. If not mistaken this also cause semantic highlight changed too.