malformed semantic tokens in some case

[Eskibear]

Open this java file in an empty folder:

public class MatriksSatuClass {
  public static int[][] setBaris(int[][] m, int baris) {
    int[][] newNilai = new int[baris][m[]];
  }
}

The last semantic token is malformed as below:

// deltaLine, deltaCharacter, length, tokenTypeIndex, tokenModifierSet
0, -2, 5, 12, 0

We are producing unsorted tokens.

references:
microsoft/vscode#135079
microsoft/vscode#134973
microsoft/vscode-java-pack#764 (comment)

[0dinD]

The tokens aren't unsorted, the problem is that JDT Core seems to think that m[] is a type literal (and doesn't set the ASTNode.RECOVERED flag), which means this heuristic fails and produces a malformed token. This isn't always the case, I can't reproduce it if I have a similar expression with m[] as a field.

I thought we could skip resolving the type binding and just check the flags on the type literal node to find out if it's recovered, but it seems like that doesn't work in this specific case (I think that's a bug with JDT Core, right?). But the type binding does contain the correct information about it being recovered, so we can use that instead. I've submitted a PR, see #1924.

[0dinD]

Actually we still need to check if the TypeLiteral node itself has the recovered flag, otherwise we introduce a regression for redhat-developer/vscode-java#1921. I've added regression tests for both cases now in #1924.

[0dinD]

Hmm, actually relying on whether or not the binding is recovered only helps in this case, but it won't work if the type binding isn't recovered:

public class MatriksSatuClass {
  public static int[][] setBaris(int[][] m, int baris) {
    int[][] newNilai = new int[baris][int[]];
  }
}

Here you can see that int[] gets the keyword token, because it's a non-recovered type literal according to the AST (which I consider a bug). You can see this if you hover int[] in my example: the diagnostic error says "Type mismatch: cannot convert from Class<int[]> to int", which makes no sense since int[] shouldn't be considered a complete type literal.

I am planning to implement more robust semantic highlighting for keywords, which can cover this edge case correctly and highlight other keywords, but I don't have time to spend on that right now.

Edit: I came up with a workaround.

[Eskibear]

@0dinD Thank you for the investigation and quick response.

because it's a non-recovered type literal according to the AST (which I consider a bug).

Then probably we should fire a bug in upstream project. And the workaround looks fine to me before we get it fixed in upsteam.