jq - just Bash to travel over JSON
When call jq and supply a valid JSON data with a JSON path we want, we will obtain the expected result at ease.
JSON is a very popular file format and I have a blog for this (here). I pretty sure you readers know it so well. We have many tools and methods to work with this. This time I would like to introduce a super basic yet effective tool we can use just our command line and get a job done for JSON files, “jq”.
In case you want to read more about Bash script on command line interface, you can visit this blog.
What is jq?
jq is a package for command line to access and process JSON contents. When call jq and supply a valid JSON data with a JSON path we want, we will obtain the expected result at ease.
We can install jq via homebrew which I wrote about recently.
Syntax
1
2
cat <json_file> | jq '<jq_statement>'
jq '<jq_statement>' <json_file>
First command means read a JSON file, then pipe (|) to parse contents of the file to jq command to do something. Second one is to execute jq directly on the file without reading beforehand.
In case of wanting to replace jq output back to the same file, we could try sponge. Sponge is a package to handle file writing in Linux. Also available on Homebrew here.
When sponge is ready in the machine, we can apply this command.
1
2
cat <json_file> | jq '<jq_statement>' | sponge <json_file>
jq '<jq_statement>' <json_file> | sponge <json_file>
Now the output is written to the output file described after sponge.
jq applications
Assume we have this JSON.
We have many ways to compute and generate outputs using jq functions.
Here are 9 examples for this blog.
1. direct access
1
2
3
cat contents.json | jq '.id'
cat contents.json | jq '.id, .name'
cat contents.json | jq '.id, .name, .items'
We can output a value of a key directly with a single dot in front. This implies we are accessing “inside” from root level.
2. inside an array
1
2
cat contents.json | jq '.classes[0]'
cat contents.json | jq '.classes[0].town'
If it’s an array, we can access by giving an index.
3. simple select
1
2
cat contents.json | jq '.classes[] | select(.level==50)'
cat contents.json | jq '.classes[] | select(.level==50) | .job'
With select(), we can filter an array by this function and supplying a boolean expression.
.classes[] means treating from a single array to a set of elements, then select (.level==50) in order to filter with the key “level” inside each element if that equals 50.
4. regex
1
2
cat contents.json | jq '.items[] | match("^st.*")'
cat contents.json | jq '.items[] | match("^st.*") | .string'
Regex is available as well. match() is one of functions using with Regex. The match() returns an object of matching products.
Example above is to find out words in “items” which starts (^) with st and followed by anything (.*).
Wanna read more about regex? I have my blog for this here.
5. regex with select
1
cat contents.json | jq '.classes[] | select(.job|match("ma"))'
We can combine select() and match() together to filter on a specific key in objects array.
First we extract an array using [] then select(). Inside select we will supply a boolean statement by choosing a key and do regex on the key with pipe |.
This example is to find elements inside classes, where the key job is matched with string ma.
6. basic altering
1
cat contents.json | jq '.cash | map(.amount |= .+200)'
map is also a useful function. This iterates through an iterator, you can see that “cash” is an array but we don’t add [] unlike above functions.
As the example, we choose the key “amount” of the array “cash” then apply |= .+200 which means update its value (|=) by adding itself (.) with 200.
map function returns an array.
7. alter and direct access
1
2
cat contents.json | jq '.cash | map(.amount |= .+200) | map(.)[]'
cat contents.json | jq '.cash | map(.amount |= .+200) | map(.)[] | .id,.amount'
As map returns an array, we are able to add next pipe and apply map again, supply themselves with a dot and [] to extract. Now it’s ready for accessing a key inside.
8. alter and make a new JSON
1
cat contents.json | jq '.cash | map(.amount |= .+200) | map(.)[] | {id:.id, amount:.amount}' | jq -s
Outputs are ready to be a new object using {...}.
After that, pack them all together into a new JSON using jq -s where -s is “slurp” flag for “read entire input into a large array”.
9. Replace file using Sponge
sponge package we discussed in the topic “Syntax” has a scene here.
1
cat contents.json | jq '...' | sponge contents.json
Read a file, process with jq, and sponge back to the same file. Easy.
