Parser accepts control chars 0x00 and 0x1F

[torsten-schenk]

In the file jv_parse.c, I cam across the following line (currently line 497):

if (c >0 && c < 0x001f) // reject character

I also tested it with an actual .json-file containing strings with unescaped character 0x00 / 0x1f, in both cases the strings got accepted.

[wader]

Hi, i get error for 0x00 but 0x1f is accepted. What version are you using and can be elaborate a bit more what behaviour your expecting?

$ jq --version
jq-1.7

$ echo -ne '"\x00hello"' | jq .
jq: parse error: Unfinished string at EOF at line 1, column 1

$ echo -ne '"\x1fhello"' | jq .
"\u001fhello"

$ echo -ne '"\x10hello"' | jq .
jq: parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 1, column 8

https://datatracker.ietf.org/doc/html/rfc8259#section-7

All Unicode characters may be placed within the
quotation marks, except for the characters that MUST be escaped:
quotation mark, reverse solidus, and the control characters (U+0000
through U+001F).

So should the reject condition actually be if (c >= 0 && c =< 0x001f)?

[torsten-schenk]

No problem. I attached three .json-files, each containing a control character in the string.

If characters 00 of 1f (test_00.txt and test_1f.txt) are in the string, the parser accepts this string and escapes the characters when printing them.

If character 01 (test_01.txt) and other control characters are in the string, the parser rejects the string with the message "jq: parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 1, column 4"

The expected behaviour is, that all control characters are rejected by the parser, so that this error message is printed for all three attached files.

You actually replicated this behaviour in your second test, echo -ne '"\x1fhello"' | jq . should have been rejected in the same way as echo -ne '"\x10hello"' | jq .

So should the reject condition actually be if (c >= 0 && c =< 0x001f)?

Yes, that would solve the issue.

test_00.txt
test_01.txt
test_1f.txt

[wader]

@nicowilliams was there some reason 0x00 and 0x1f was excluded in the check? used to be handled somewhere else etc?

[nicowilliams]

@nicowilliams was there some reason 0x00 and 0x1f was excluded in the check? used to be handled somewhere else etc?

Nope, just an off-by-one bug.