Bug using +=

[pkoppstein]

CORRECT:

jq -c '.VALUE |= .+[length]' <<< '{"DATE":"September","VALUE":[5,17,8,9]}'
{"DATE":"September","VALUE":[5,17,8,9,4]}

INCORRECT:

jq -c '.VALUE += [length]' <<< '{"DATE":"September","VALUE":[5,17,8,9]}'
{"DATE":"September","VALUE":[5,17,8,9,2]}

This behavior can be observed with jq1.4, jq 1.5, and the master version
(jq-1.5rc2-245-g7b81a83).

[muhmuhten]

The documentation is inaccurate. The RHS of update-assignment doesn't receive the LHS as input. The actual behaviour is more like [length] as $_ | .VALUE |= .+$_.

It probably makes more sense to update the documentation than break compatibility, since this is arguably the more useful operation: you're more likely to be doing arithmetic on values at the same level than, say, merging a field of an object with the object itself. e.g. jq -nc '{a:3,b:4}|.a+=.b' And the alternatives to achieve this behaviour are all much longer, compared to just two extra characters to achieve the documented behaviour.

[nicowilliams]

Actually, I think this is a bug indeed!

$ jq -c '(.VALUE) += [debug|length]' <<< '{"DATE":"September","VALUE":[5,17,8,9]}'
["DEBUG:",{"DATE":"September","VALUE":[5,17,8,9]}]
$ jq -c '(.VALUE) |= .+[debug|length]' <<< '{"DATE":"September","VALUE":[5,17,8,9]}'
["DEBUG:",[5,17,8,9]]
{"DATE":"September","VALUE":[5,17,8,9,4]}

The problem is that gen_update() doesn't put the val block in the update lambda, but computes it first with the same input (.) as the LHS:

static block gen_update(block object, block val, int optype) {
  block tmp = gen_op_var_fresh(STOREV, "tmp");
  return BLOCK(gen_op_simple(DUP),
               val,
               tmp,
               gen_call("_modify", BLOCK(gen_lambda(object),
                                         gen_lambda(gen_binop(gen_noop(),
                                                              gen_op_bound(LOADV, tmp),
                                                              optype)))));
}

Compare to the |= operator:

Exp "|=" Exp {
  $$ = gen_call("_modify", BLOCK(gen_lambda($1), gen_lambda($3)));
} |

I don't know if we can fix this now. It's always been thus. @stedolan?

[emanuele6]

I think this should not be considered a bug.
I have always used non-|= assignment operators assuming the right hand side is evaluated once using . as input, just like for =.
Maybe this is not how it worked in jq 1.4 or earlier, but I don't think this behaviour should be changed now.

[nicowilliams]

Re-thinking this now, lhs += rhs is really taking whatever rhs is and adding it to whatever lhs is, but what should be the input to rhs? Well, "who cares" is a pretty good answer -- it could be anything at all and the += operator should still work as intended, as long as the rhs ignores its input. Typically the rhs will just be a literal, and literal values ignore their input. Thus, whatever that input is today, that is just fine.

I think perhaps it would have been best to pass the current value of the lhs to the rhs so that one could write expressions like .a += if iseven then 0 else 1 end.

On the other hand, we have documented what the input to the rhs:

I think most likely we could make the code match the docs and it should more than likely not break any existing jq code because most jq code using += and friends probably use RHS expressions that ignore their input.

OTOH, this has been like this for a long time, and we could just leave it as-is and instead fix the docs to say that the input to the RHS is unspecified, thus letting us wait until whatever release follows 1.7 to change this.

[wader]

For reference #2407 might similar or dup of this? also #2410 about //= might be interesting

[muhmuhten]

For what it's worth, I stand by the position that it's more useful to document the current behavior for the arith-assign ops so that e.g. {a:1,b:2}|.a+=.b yields {"a":3,"b":2}, which is an occasionally useful operation that's substantially less intuitive to express otherwise, especially if the lhs is a nontrivial expression. I'm having a lot more trouble coming up with a use for the documented behavior for the arith ops.

For the //= issue, my (not too considered) thought is that the best unifying semantic might be for lhs op= rhs to be the equivalent of . as $_|lhs |= . op ($_|rhs), which if I'm not misthinking something should provide identical results to current behavior for the ops that always evaluate the rhs, while having a more useful behavior for //=.

[nicowilliams]

For reference #2407 might similar or dup of this? also #2410 about //= might be interesting

Yes, //= in particular is really busted:

//= would be silly to send the current value to the rhs since that value would be null or the lhs is not even set.

[emanuele6]

@nicowilliams A common way I use += is .foo += (-1, 0, 1); example in my day 14 solution for advent of code 2022 to generate the possible directions in which the grain of sand can fall (this is the version that draws an animation in the terminal if used as ./foo.jq input | awk -v RS= '{ system("clear") } 1; { system("sleep .1") }')

#!/bin/sh --
# \
exec jq -nRrf "$0" -- "$@"

[
    inputs / " -> " |
    map(. / "," | map(tonumber)) |
    range(0; length - 1) as $i |
    .[$i:$i + 2]
] |

(map(.[][0]) | min) as $minx |
(map(.[][0]) | max) as $maxx |
(map(.[][1]) | max) as $maxy |
[ 500 - $minx, 0 ] as $sandIn |

.[][][0] -= $minx |
reduce .[] as $line ([];
    def xrange: $line | map(.[0]) | range(min; max + 1);
    def yrange: $line | map(.[1]) | range(min; max + 1);
    .[xrange][yrange] = "#"
) |

def gridToString:
    transpose |
    (.[][] | nulls) = "." |
    map(join("")) |
    join("\n");

{ grid: ., $sandIn, units: [] } |
while(.grid[.sandIn[0]][.sandIn[1]] == null;
    .units += [ .sandIn ] |
    def directions($grid):
        select(.[1] <= $maxy) |
        .[1] += 1 |
        .[0] += (0, -1, 1) |
        select(.[0] < 0 or $grid[.[0]][.[1]] == null);
    reduce (.units | keys_unsorted[]) as $k ({ offset: 0, v: . };
        def unit: .v.units[$k + .offset];
        .v.grid as $grid |
        [ unit | first(directions($grid)) ] as $dir |
        if $dir == [] then
            .v.grid[unit[0]][unit[1]] //= "o" |
            del(unit) |
            .offset -= 1
        else
            unit = $dir[0] |
            select($dir[0][0] < 0) |= (
                .v.grid |= [ [] ] + . |
                (.v | .sandIn, .units[])[0] += 1
            )
        end
    ) |
    .v
) |
reduce .units[] as $u (.grid; .[$u[0]][$u[1]] = "%") |
gridToString,
""

Some example inputs (describe the map in which the sand falls):

498,4 -> 498,6 -> 496,6
503,4 -> 502,4 -> 502,9 -> 494,9

480,10 -> 500,10
510,10 -> 520,10  
480,20 -> 490,20
500,20 -> 500,18 -> 520,18

But I have many other more pratical examples in which I use .foo += manyvalues to generate multiple values with a modified .foo. This is way more useful than having it work as .foo |= . + (-1, 0, 1) so as just .foo += -1.

In my scripts, I can also find many instances of me using .foo += [ if something then x else y end ] where something, uses . as input and x, y are either constants or expressions that use . as input, not .foo.

Also, if you need something like .a += if iseven then 0 else 1 end, you should just use (.a | select(iseven | not)) += 1 ;)

[nicowilliams]

Oh, hmm, no, //= is not particularly busted.

[nicowilliams]

@emanuele6 and @muhmuhten have the winning arguments. Let's fix the docs then.