|
| 1 | +# Processing JSON using jq |
| 2 | + |
| 3 | +jq is useful to slice, filter, map and transform structured json data. |
| 4 | + |
| 5 | +## Installing jq |
| 6 | + |
| 7 | +### On Mac OS |
| 8 | + |
| 9 | +`brew install jq` |
| 10 | + |
| 11 | +### On AWS Linux |
| 12 | + |
| 13 | +Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ |
| 14 | + |
| 15 | +Installing from the source proved to be tricky. |
| 16 | + |
| 17 | +## Useful arguments |
| 18 | + |
| 19 | +When running jq, the following arguments may become handy: |
| 20 | + |
| 21 | +| Argument | Description | |
| 22 | +| ------ | :--------- | |
| 23 | +| `--version`| Output the jq version and exit with zero. | |
| 24 | +| `--sort-keys` | Output the fields of each object with the keys in sorted order.| |
| 25 | + |
| 26 | +## Basic concepts |
| 27 | + |
| 28 | +The syntax for jq is pretty coherent: |
| 29 | + |
| 30 | +| Syntax | Description | |
| 31 | +| ------ | :--------- | |
| 32 | +| , | Filters separated by a comma will produce multiple independent outputs| |
| 33 | +| ? | Will ignores error if the type is unexpected | |
| 34 | +| [] | Array construction | |
| 35 | +| {} | Object construction | |
| 36 | +| + | Concatenate or Add | |
| 37 | +| - | Difference of sets or Substract | |
| 38 | +| length | Size of selected element | |
| 39 | +| | | Pipes are used to chain commands in a similar fashion than bash| |
| 40 | + |
| 41 | + |
| 42 | +## Dealing with json objects |
| 43 | + |
| 44 | +| Description | Command | |
| 45 | +| ------ | :--------- | |
| 46 | +| Display all keys | `jq 'keys'` | |
| 47 | +| Adds + 1 to all items | `jq 'map_values(.+1)'` | |
| 48 | +| Delete a key| `jq 'del(.foo)'` | |
| 49 | +| Convert an object to array | `to_entries | map([.key, .value])` | |
| 50 | + |
| 51 | +## Dealing with fields |
| 52 | + |
| 53 | +| Description | Command | |
| 54 | +| ------ | :--------- | |
| 55 | +| Concatenate two fields| `fieldNew=.field1+' '+.field2` | |
| 56 | + |
| 57 | + |
| 58 | +## Dealing with json arrays |
| 59 | + |
| 60 | +### Slicing and Filtering |
| 61 | + |
| 62 | +| Description | Command | |
| 63 | +| ------ | :--------- | |
| 64 | +| All | `jq .[]` | |
| 65 | +| First | `jq '.[0]'` | |
| 66 | +| Range | `jq '.[2:4]'` | |
| 67 | +| First 3 | `jq '.[:3]'` | |
| 68 | +| Last 2 | `jq '.[-2:]'` | |
| 69 | +| Before Last | `jq '.[-2]'`| |
| 70 | +| Select array of int by value | `jq 'map(select(. >= 2))'` | |
| 71 | +| Select array of objects by value| `jq '.[] | select(.id == "second")'` | |
| 72 | +| Select by type | `jq '.[] | numbers'` with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars | |
| 73 | + |
| 74 | +### Mapping and Transforming |
| 75 | + |
| 76 | +| Description | Command | |
| 77 | +| ------ | :--------- | |
| 78 | +| Add + 1 to all items | `jq 'map(.+1)'` | |
| 79 | +| Delete 2 items| `jq 'del(.[1, 2])'` | |
| 80 | +| Concatenate arrays | `jq 'add'` | |
| 81 | +| Flatten an array | `jq 'flatten'` | |
| 82 | +| Create a range of numbers | `jq '[range(2;4)]'` | |
| 83 | +| Display the type of each item| `jq 'map(type)'` | |
| 84 | +| Sort an array of basic type| `jq 'sort'` | |
| 85 | +| Sort an array of objects | `jq 'sort_by(.foo)'` | |
| 86 | +| Group by a key - opposite to flatten | `jq 'group_by(.foo)'` | |
| 87 | +| Minimun value of an array| `jq 'min'` .See also min, max, min_by(path_exp), max_by(path_exp) | |
| 88 | +| Remove duplicates| `jq 'unique'` or `jq 'unique_by(.foo)'` or `jq 'unique_by(length)'` | |
| 89 | +| Reverse an array | `jq 'reverse'` | |
0 commit comments