JQ and OpenAPI Series - Part 3

Modifying OpenAPI files with JQ

By Arnaud Lauret, June 28, 2020

Ever wanted to quickly find, extract or modify data coming from some JSON documents on the command line? JQ is the tool you’re looking for. Thanks to the two previous parts of this JQ and OpenAPI Series, we learned how to extract data from JSON (OpenAPI) files by discovering many filters, creating modules and using command line arguments. Now we will discover how to modify them; how to replace, add or delete elements in processed documents.

JQ and OpenAPI Series

JQ’s documentation is quite complete and there are many tutorials and Stackoverflow answers, so why bother writing this series? First reason, I regularly meet people working with APIs and/or JSON files who actually don’t know JQ exists and how it could save their life (or at least their time). Second reason, I often use it with OpenAPI specification files and I found that showing how JQ can be used on such a widely adopted and familiar JSON based format could help to learn how to use it (and also writing this post actually helped me to improve my JQ skills!).

1 - Using JQ to extract data from OpenAPI files
2 - Using JQ command line arguments, functions and modules
3 - Modifying OpenAPI files with JQ
4 - Bonus: Coloring JQ's raw output

Get post’s content

All examples shown in this post are based on JQ 1.6 and OpenAPI 3. All examples can be copied using the button and downloaded using the one on code snippets. All source code can be retrieved from the JQ and OpenAPI post series’ github repository.

git clone https://github.com/arno-di-loreto/jq-and-openapi/
cd jq-and-openapi
git checkout part-3

[apihandyman.io]$ git clone https://github.com/arno-di-loreto/jq-and-openapi/
[apihandyman.io]$ cd jq-and-openapi
[apihandyman.io]$ git checkout part-3

We will go on using the demo-api-openapi.json OpenAPI file in this post:

demo-api-openapi.json

{"openapi":"3.0.0","info":{"title":"Banking API","version":"1.0.0-snapshot","description":"The Banking API provides access to the [Banking Company](http://www.bankingcompany.com) services, which include bank account information, beneficiaries, and money transfer management.<!--more-->\n\n# Authentication\n\n## How to \n- Register\n- Create an APP\n- Request credentials\n\n# Use cases\n\n## Transferring money to an account or preexisting beneficiary\n\nThe _transfer money_ operation allows one to transfer an `amount` of money from a `source` account to a `destination` account or beneficiary.\nIn order to use an appropriate `source` and `destination`, we recommend to use _list sources_ and _list source's destinations_ as shown in the figure below (instead of using _list accounts_ and _list beneficiaries_).\n\n![Diagram](http://localhost:9090/12.2-operation-manual-diagram.svg)\n\n## Cancelling a delayed or recurring money transfer\n\n- List money transfers: To list existing money transfers and select the one to delete\n- Cancel a money transfer: To cancel the selected money transfer\n","contact":{"name":"The Banking API team","email":"[email protected]","url":"developer.bankingcompany.com"}},"tags":[{"name":"Transfers","description":"Everything you need to manage money transfers. A money transfer consists in transferring money from a source account to a destination account."},{"name":"Beneficiaries","description":"Everything you need to manage money transfer beneficiaries. Beneficiaries are pre-registred external accounts that can be used as destinations for money transfers."}],"paths":{"/accounts":{"get":{"tags":["Accounts"],"summary":"List accounts","responses":{"200":{"description":"User's accounts","content":{"application/json":{"schema":{"required":["properties"],"properties":{"items":{"type":"array","items":{"$ref":"#/components/schemas/Account"}}}}}}}}}},"/accounts/{id}":{"get":{"tags":["Accounts"],"summary":"Get an account","parameters":[{"name":"id","in":"path","description":"Account's id","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"The account","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Account"}}}},"401":{"description":"Unauthorized"}},"x-implementation":{"security":{"description":"Only accounts belonging to user referenced in security data;\nreturn a 404 if this is not the case\n","source":{"system":"security","location":"jwt.sub"},"fail":404}}}},"/beneficiaries":{"post":{"tags":["Beneficiaries"],"summary":"Register a beneficiary","security":[{"BankingAPIScopes":["beneficiary:create","beneficiary:admin"]}],"responses":{"201":{"description":"Beneficiary added"},"401":{"description":"Unauthorized"},"403":{"description":"Forbidden"}}},"get":{"tags":["Beneficiaries"],"summary":"List beneficiaries","security":[{"BankingAPIScopes":["beneficiary:read","beneficiary:admin"]}],"responses":{"200":{"description":"The beneficiaries list"}}}},"/beneficiaries/{id}":{"parameters":[{"name":"id","in":"path","description":"Beneficiary's id","required":true,"schema":{"type":"string"}}],"delete":{"deprecated":true,"tags":["Beneficiaries"],"summary":"Delete a beneficiary","security":[{"BankingAPIScopes":["beneficiary:delete","beneficiary:admin"]}],"responses":{"204":{"description":"Beneficiary deleted"},"401":{"description":"Unauthorized"}}},"patch":{"deprecated":true,"tags":["Beneficiaries"],"summary":"Updates a beneficiary","security":[{"BankingAPIScopes":["beneficiary:admin"]}],"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/BeneficiaryUpdate"}}}},"responses":{"200":{"description":"The updated beneficiary"},"401":{"description":"Unauthorized"}}},"get":{"tags":["Beneficiaries"],"summary":"Get a beneficiary","security":[{"BankingAPIScopes":["beneficiary:read","beneficiary:admin"]}],"responses":{"200":{"description":"The beneficiary"}}}},"/sources":{"get":{"summary":"List transfer sources","tags":["Transfers"],"description":"Not all bank accounts can be used as a source\nfor a money transfers. This operation returns\nonly the accounts elligible as a money transfer\nsource.\n","responses":{"200":{"description":"The transfer sources"}}}},"/sources/{id}/destinations":{"parameters":[{"name":"id","in":"path","description":"Source's id","required":true,"schema":{"type":"string"}}],"get":{"summary":"List transfer source's destinations","tags":["Transfers"],"description":"Depending on the source account, only specific\nbeneficiaries or accounts can be used as a money\ntransfer destination.\nThis operation returns them.\n","responses":{"200":{"description":"The transfer destination"}}}},"/transfers":{"post":{"summary":"Transfer money","security":[{"BankingAPIScopes":["transfer:create","transfer:admin"]}],"tags":["Transfers"],"description":"This operation allows one to transfer an `amount` of money from a `source` account to a `destination` account.\nThere are three different types of money transfer:\n  - Immediate -- these are executed as soon as the request is received \n  - Delayed -- these are executed upon a given future `date`\n  - Recurring -- these are executed a given `occurrences` number of times at a given `frequency` -- the first occurrence being executed immediately or at a given `date`\n","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/TransferRequest"},"examples":{"immediate":{"summary":"Immediate transfer","description":"The money transfer is executed immediately","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2}},"delayed":{"summary":"Delayed transfer","description":"The money transfer is executed at a given date","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2,"date":"2019-03-19"}},"recurring":{"summary":"Recurring transfer","description":"The money transfer is executed at a given date reurringly","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2,"date":"2019-03-19","occurrences":1,"frequency":"MONTHLY"}}}}}},"responses":{"201":{"description":"Immediate or recurring transfer executed","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TransferResponse"},"examples":{"immediate":{"summary":"Immediate transfer","description":"The money transfer is executed immediately","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2}},"recurring":{"summary":"Recurring transfer","description":"The first occurence is executed immediately","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2,"date":"2019-03-19","occurrences":1,"frequency":"MONTHLY"}}}}}},"202":{"description":"Delayed or recurring delayed transfer accepted","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TransferResponse"},"examples":{"delayed":{"summary":"Delayed transfer","description":"The money transfer is executed at a given date","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2,"date":"2019-03-19"}},"recurring":{"summary":"Recurring transfer","description":"The money transfer is executed at a given date reurringly","value":{"source":"000534115776675","destination":"000567689879878","amount":456.2,"date":"2019-03-19","occurrences":1,"frequency":"MONTHLY"}}}}}},"400":{"description":"The transfer is rejected due to an error in the request properties or an insufficient balance.\nEach error provides the property `source` of the error along with a human-readable `message` and its `type`:\n\n- MANDATORY_PROPERTY: The property indicated in `source` is missing\n- INVALID_FORMAT: The format of the property indicated in `source` is invalid\n- INVALID_VALUE: The value of the property indicated in `source` is invalid\n- INSUFFICIENT_BALANCE: The `amount` property is higher than the `source` account balance\n","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ConsumerError"}}}}}},"get":{"summary":"List money transfers","tags":["Transfers"],"security":[{"BankingAPIScopes":["transfer:read","transfer:admin"]}],"responses":{"200":{"description":"Transfers list","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TransferList"}}}}}}},"/transfers/{id}":{"parameters":[{"name":"id","in":"path","description":"Transfer's id","required":true,"schema":{"type":"string"}}],"get":{"summary":"Get a money transfer","tags":["Transfers"],"security":[{"BankingAPIScopes":["transfer:read","transfer:admin"]}],"responses":{"200":{"description":"The money transfer"},"404":{"description":"The money transfer does not exist"}}},"x-tension-example":{"some":"value"},"patch":{"tags":["Transfers"],"responses":{"200":{"description":"The money transfer has been update"}}},"delete":{"summary":"Cancel a money transfer","tags":["Transfers"],"security":[{"BankingAPIScopes":["transfer:delete","transfer:admin"]}],"description":"Only delayed or recurring money transfer can be canceled","responses":{"204":{"description":"The money transfer has been deleted"},"404":{"description":"The money transfer does not exist"}}}}},"components":{"securitySchemes":{"BankingAPIScopes":{"type":"oauth2","flows":{"implicit":{"authorizationUrl":"https://auth.bankingcompany.com/authorize","scopes":{"transfer:create":"Create transfers","transfer:read":"Read transfers","transfer:delete":"Delete transfers","transfer:admin":"Create, read, and delete transfers","beneficiary:create":"Create beneficiaries","beneficiary:read":"List beneficiaries","beneficiary:delete":"delete beneficiaries","beneficiary:admin":"Create, read, and delete beneficiaries","account:read":"Read accounts","account:admin":"Read accounts"}}}}},"schemas":{"BeneficiaryUpdate":{"description":"A beneficiary update parameter","properties":{"name":{"type":"string"}}},"UselessSchema":{"description":"An unused useless schema","type":"string"},"TransferRequest":{"description":"A money transfer request","required":["source","destination","amount"],"properties":{"deprecatedPropertyExample":{"deprecated":true,"type":"string","description":"An example of a deprecated property"},"source":{"type":"string","description":"Source account number","minLength":15,"maxLength":15,"pattern":"^\\d{15}$","example":"000534115776675"},"destination":{"type":"string","description":"Destination account number","minLength":15,"maxLength":15,"pattern":"^\\d{15}$","example":"000567689879878"},"amount":{"type":"number","example":456.2,"minimum":0,"exclusiveMinimum":true},"date":{"type":"string","format":"date","description":"Execution date for a delayed transfer\nor first execution date for a recurring one\n","example":"2019-03-19"},"occurrences":{"type":"integer","description":"Number of times a recurring transfer will be executed\n","example":2,"minimum":2,"maximum":100},"frequency":{"type":"string","description":"Frequency of recurring transfer's execution","example":"MONTHLY","enum":["WEEKLY","MONTHLY","QUARTERLY","YEARLY"]}}},"TransferResponse":{"allOf":[{"required":["id","type","status"],"properties":{"id":{"type":"string","example":"1611e71f-1bb2-412f-8c43-92b275a5c321"},"type":{"type":"string","enum":["IMMEDIATE","DELAYED","RECURRING"],"example":"RECURRING"},"status":{"type":"string","description":"An immediate transfer is always `EXECUTED`, a delayed transfer can be `EXECUTED` or `PENDING` and a recurring one is always `PENDING`\n","enum":["EXECUTED","PENDING"],"example":"PENDING"},"requestDate":{"type":"string","example":"2019-09-19"}}},{"$ref":"#/components/schemas/TransferRequest"}]},"TransferList":{"properties":{"items":{"type":"array","items":{"$ref":"#/components/schemas/TransferResponse"}}}},"ConsumerError":{"required":["errors"],"properties":{"errors":{"description":"A list of errors providing detailed information about the problem","type":"array","minItems":1,"items":{"required":["source","type","message"],"properties":{"source":{"description":"the property source of the error","type":"string","example":"amount","enum":["source","destination","amount","date","occurrences","frequency"]},"type":{"type":"string","example":"MANDATORY_PROPERTY","enum":["MANDATORY_PROPERTY","INVALID_FORMAT","INVALID_VALUE","INSUFFICIENT_BALANCE"]},"message":{"description":"a human-readable error message","type":"string","example":"The money transfer's amount must be provided"}}}}}},"ProviderError":{"properties":{"errors":{"type":"array","minItems":1,"maxItems":1,"items":{"properties":{"message":{"type":"string"}}}}}},"Account":{"properties":{"balance":{"description":"The balance in the account's default currency","type":"object","title":"Amount","required":["value","currency"],"properties":{"value":{"description":"Balance's value using the number of decimal places defined by ISO 4217","externalDocs":{"description":"Decimal places table","url":"https://www.currency-iso.org/en/home/tables/table-a1.html"},"type":"number","x-implementation":{"description":"The real time balance (not the daily one!)","source":{"system":"Core Banking","data":"ZBAL0.RTBAL"}}},"currency":{"description":"An ISO 4217 code","externalDocs":{"url":"https://www.iso.org/iso-4217-currency-codes.html"},"type":"string","example":"USD","x-implementation":{"source":{"system":"Core Banking","data":"ZBAL0.RTCUR"}}}}}}}}}}

Replacing elements

It’s fairly common to tweak OpenAPI files, especially before putting them in an API portal. You may have to replace some server URLs, update version number and replace some descriptions. That can be easily done with JQ. To learn how to replace values, we’ll work on the info section of demo-api-openapi.json with = and |= operators . And we will also see how to save the modified file because it wouldn’t make any sense to not be able to save our modifications.

JQ Operators
`.info.description = "New description."` `.info.description = .info.contact.name`	Replaces a value, the input to the right-hand-side (rhs) is the same as the input to the left-hand-side (lhs) rather than the value at the lhs path
`.info.contact \|= .name` `.info.version \|= sub("-snapshot";"")`	Replaces a value, it takes a filter on the right-hand side and works out the new value for the property of `.` being assigned to by running the old value through this expression

This section’s content is also available as an asciinema bash session:

Replacing elements

Replacing a value with =

The following command line shows how to print the .info.description property of demo-api-openapi.json file (as we have learned to do so in part 1 of this post series):

jq '.info.description' demo-api-openapi.json

Printing .info.description original value

[apihandyman.io] $jq '.info.description' demo-api-openapi.json 
"The Banking API provides access to the [Banking Company](http://www.bankingcompany.com) services, which include bank account information, beneficiaries, and money transfer management.<!--more-->\n\n# Authentication\n\n## How to \n- Register\n- Create an APP\n- Request credentials\n\n# Use cases\n\n## Transferring money to an account or preexisting beneficiary\n\nThe _transfer money_ operation allows one to transfer an `amount` of money from a `source` account to a `destination` account or beneficiary.\nIn order to use an appropriate `source` and `destination`, we recommend to use _list sources_ and _list source's destinations_ as shown in the figure below (instead of using _list accounts_ and _list beneficiaries_).\n\n![Diagram](http://localhost:9090/12.2-operation-manual-diagram.svg)\n\n## Cancelling a delayed or recurring money transfer\n\n- List money transfers: To list existing money transfers and select the one to delete\n- Cancel a money transfer: To cancel the selected money transfer\n"

The next listing shows how modifying this value is as simple, thanks to = operator. By adding ="New description" after the property’s path, its content is modified like in any programming language. And now instead of returning the .info.description value, jq returns the whole document in which this property’s value has been replaced by “New description”.

The listing shows also 2 different ways to only show what we need to see. The first one is to use head (which show only the nth first line) and the second one consists in piping the result of jq in another jq command to show only the value of .info.description.

jq '.info.description="New description"' demo-api-openapi.json
jq '.info.description="New description"' demo-api-openapi.json | head -n6
jq '.info.description="New description"' demo-api-openapi.json | jq '.info.description'

Modifying .info.description

[apihandyman.io] $jq '.info.description="New description"' demo-api-openapi.json
{
...full modified document ...
}
[apihandyman.io] $jq '.info.description="New description"' demo-api-openapi.json | head -n6 
{
  "openapi": "3.0.0",
  "info": {
    "title": "Banking API",
    "version": "1.0.0-snapshot",
    "description": "New description",
[apihandyman.io] $jq '.info.description="New description"' demo-api-openapi.json | jq '.info.description'
"New description"

Anything can be put on the right side of the = operator as shown in the following listing. The command line allows to replace the value of .info.contact by another JSON object.

jq '.info.contact' demo-api-openapi.json
jq '.info.contact = { name: "The Awesome Banking API Team", url: "www.bankingcompany.com" }' demo-api-openapi.json | jq '.info.contact'

Modifying an entire object value

[apihandyman.io] $jq '.info.contact' demo-api-openapi.json
{
  "name": "The Banking API team",
  "email": "[email protected]",
  "url": "developer.bankingcompany.com"
}
[apihandyman.io] $jq '.info.contact = { name: "The Awesome Banking API Team", url: "www.bankingcompany.com" }' demo-api-openapi.json | jq '.info.contact'
{
  "name": "The Awesome Banking API Team",
  "url": "www.bankingcompany.com"
}

Saving (a copy of) modified file

But those command lines did not actually saved the modified files; indeed the modified content is just printed in the terminal. Unfortunately jq does not come with in-place modification like sed. But that’s not really a problem, we can use a good old > to save result in another file as shown below (and once really sure of what we have done, we can replace the original one).

jq '.info.description="New description"' demo-api-openapi.json > demo-api-openapi-mod.json
jq '.info.description' demo-api-openapi-mod.json

Saving modified file

[apihandyman.io] $jq '.info.description="New description"' demo-api-openapi.json > demo-api-openapi-mod.json 
[apihandyman.io] $jq '.info.description' demo-api-openapi-mod.json
"New description"

Using filters when replacing a value

When I said that anything can be put on the right side, it’s virtually anything; including complex filters chains like the one we have learned to create previously in part 1 and 2. Let’s see that with a very simple example. The following listing shows how to delete the “-snapshot” suffix from the .info.version property using the sub filter (which replaces a string by another one inside a string) on the .info.version property.

jq '.info.version' demo-api-openapi.json 
jq '.info.version = (.info.version | sub("-snapshot";""))' demo-api-openapi.json
jq '.info.version = (.info.version | sub("-snapshot";""))' demo-api-openapi.json | jq '.info.version'

Using filters

[apihandyman.io]$ jq '.info.version' demo-api-openapi.json
"1.0.0-snapshot"
[apihandyman.io]$ jq '.info.version = (.info.version | sub("-snapshot";""))' demo-api-openapi.json
{
  ...full modified document
}
[apihandyman.io]$ jq '.info.version = (.info.version | sub("-snapshot";""))' demo-api-openapi.json | jq '.info.version'
"1.0.0"

Did you noticed the parenthesis around the right side of the = operator? They are very important. If you forget them, be ready to face more or less unexpected consequences depending on what you do. Here it hopefully break swith an error without silently doing nasty stuff. Indeed, here, without (), the result of .info.version = .info.version, which is the whole document (a JSON object), is piped into the sub filter which expects a string. So always put parenthesis around the right side if there are some piped filters.

jq '.info.version = (.info.version | sub("-snapshot";""))' demo-api-openapi.json
jq '.info.version = .info.version | sub("-snapshot";"")'  demo-api-openapi.json

Do not forget parenthesis

[apihandyman.io]$ jq '.info.version = (.info.version | sub("-snapshot";""))' demo-api-openapi.json | jq '.info.version'
"1.0.0"
[apihandyman.io]$ jq '.info.version = .info.version | sub("-snapshot";"")'  demo-api-openapi.json
jq: error (at demo-api-openapi.json:0): object ({"openapi":...) cannot be matched, as it is not a string

And again, when I said that anything could be put on the right side, it’s really anything. Even anything from from anywhere. For example, the following command line put the modified version number into the description.

jq '.info.description' demo-api-openapi.json
jq '.info.description = (.info.version | sub("-snapshot";""))' demo-api-openapi.json | jq '.info.description'

Using anything from anywhere

[apihandyman.io]$ jq '.info.description' demo-api-openapi.json
"The Banking API provides access to the [Banking Company](http://www.bankingcompany.com) services, which include bank account information, beneficiaries, and money transfer management.<!--more-->\n\n# Authentication\n\n## How to \n- Register\n- Create an APP\n- Request credentials\n\n# Use cases\n\n## Transferring money to an account or preexisting beneficiary\n\nThe _transfer money_ operation allows one to transfer an `amount` of money from a `source` account to a `destination` account or beneficiary.\nIn order to use an appropriate `source` and `destination`, we recommend to use _list sources_ and _list source's destinations_ as shown in the figure below (instead of using _list accounts_ and _list beneficiaries_).\n\n![Diagram](http://localhost:9090/12.2-operation-manual-diagram.svg)\n\n## Cancelling a delayed or recurring money transfer\n\n- List money transfers: To list existing money transfers and select the one to delete\n- Cancel a money transfer: To cancel the selected money transfer\n"
[apihandyman.io]$ jq '.info.description = (.info.version | sub("-snapshot";""))' demo-api-openapi.json | jq '.info.description'
"1.0.0"

Replacing a value with |=

deleting “-snapshot” from the version number can be done in a more elegant way using the |= operator, as shown in the following listing. Thanks to this operator, when using . on the right side, you only get what was passed on the left side and so you may not need to use | (reminder . | some_filter is equivalent to some_filter).

jq '.info.version' demo-api-openapi.json 
jq '.info.version |= sub("-snapshot";"")' demo-api-openapi.json
jq '.info.version |= sub("-snapshot";"")' demo-api-openapi.json | jq '.info.version'

Using |= instead of =

[apihandyman.io]$ jq '.info.version' demo-api-openapi.json
"1.0.0-snapshot"
[apihandyman.io]$ jq '.info.version |= sub("-snapshot";"")' demo-api-openapi.json
{
  ...full modified document
}
[apihandyman.io]$ jq '.info.version |= sub("-snapshot";"")' demo-api-openapi.json | jq '.info.version'
"1.0.0"

That works with objects, let’s invert email and url values in the contact object. As you can see, url value is accessed with .url as only the contact object value is available on the right side (same for email); if we had used = we should have used .info.contact.url. Note also how name is kept unmodified by just using name.

jq '.info.contact' demo-api-openapi.json 
jq '.info.contact |= { name, url: .email, email: .url }' demo-api-openapi.json
jq '.info.contact |= { name, url: .email, email: .url }' demo-api-openapi.json | jq '.info.contact'

|= operator works on object too

[apihandyman.io]$ jq '.info.contact' demo-api-openapi.json
{
  "name": "The Banking API team",
  "email": "[email protected]",
  "url": "developer.bankingcompany.com"
}
[apihandyman.io]$ jq '.info.contact |= { name, url: .email, email: .url }' demo-api-openapi.json
{
  ...full modified document
}
[apihandyman.io]$ jq '.info.contact |= { name, url: .email, email: .url }' demo-api-openapi.json | jq '.info.contact'
"1.0.0"

Chaining modifications

Obviously, jq allows to do more than one modification at a time. You probably already guessed how to do so, you just need to pipe all modifications one after another as shown below on line 12 and 16. Here, we replace .info.description value by “New description.” using = and then | the result to another modification consisting in deleting “-snapshot” from .info.version using |= and sub. The final result shows both modifications.

jq '.info' demo-api-openapi.json
jq '(.info.description = "New description.") | (.info.version |= sub("-snapshot";""))' demo-api-openapi.json
jq '(.info.description = "New description.") | (.info.version |= sub("-snapshot";""))' demo-api-openapi.json | jq '.info'

[apihandyman.io]$ jq '.info' demo-api-openapi.json
{
  "title": "Banking API",
  "version": "1.0.0-snapshot",
  "description": "The Banking API provides access to the [Banking Company](http://www.bankingcompany.com) services, which include bank account information, beneficiaries, and money transfer management.<!--more-->\n\n# Authentication\n\n## How to \n- Register\n- Create an APP\n- Request credentials\n\n# Use cases\n\n## Transferring money to an account or preexisting beneficiary\n\nThe _transfer money_ operation allows one to transfer an `amount` of money from a `source` account to a `destination` account or beneficiary.\nIn order to use an appropriate `source` and `destination`, we recommend to use _list sources_ and _list source's destinations_ as shown in the figure below (instead of using _list accounts_ and _list beneficiaries_).\n\n![Diagram](http://localhost:9090/12.2-operation-manual-diagram.svg)\n\n## Cancelling a delayed or recurring money transfer\n\n- List money transfers: To list existing money transfers and select the one to delete\n- Cancel a money transfer: To cancel the selected money transfer\n",
  "contact": {
    "name": "The Banking API team",
    "email": "[email protected]",
    "url": "developer.bankingcompany.com"
  }
}
[apihandyman.io]$ jq '(.info.description = "New description.") | (.info.version |= sub("-snapshot";""))' demo-api-openapi.json
{
  ...full modified document...
}
[apihandyman.io]$ jq '(.info.description = "New description.") | (.info.version |= sub("-snapshot";""))' demo-api-openapi.json | jq '.info'
{
  "title": "Banking API",
  "version": "1.0.0",
  "description": "New description.",
  "contact": {
    "name": "The Banking API team",
    "email": "[email protected]",
    "url": "developer.bankingcompany.com"
  }
}

JQ module

Here’s a JQ module demonstrating the various use of = and |= we have seen (use jq -r replace.jq demo-api-openapi.json to see it in action).

replace.jq

# Basic value replacement using =
(.info.description = "New description.") | # modified document
                                           # goes to next step
# Using processing when setting value
(.info.version = (.info.version | sub("-snapshot";"-no-snapshot"))) |
# Any type of value can be provided on right side of =
(
  .info.contact = { 
                    name: "The Awesome Banking API Team", 
                    email: "www.bankingcompany.com"
                  }
) |
# |= can also be used to only work on what is provided on left side
(.info.version |= sub("-no-snapshot";"")) |
# It works on objects too
(
  .info.contact |= { 
                    name, # unmodified 
                    url: .email, # path of value inside .info.contact
                    email: "[email protected]"
                  }
)

Adding elements

JQ does not only allow to replace existing values, it allows also to add elements thanks to the += operator which can be used on many kind of value and |= which can be used on objects.

JQ Operators
`.info.description += "More description"` `.info.description += .info.contact.name`	Increments a number value, concatenates string or add properties to object

This section’s content is also available as an asciinema bash session:

Adding elements

Appending to a string with +=

The += operator is used in various programming language; a+=x usually means a=a+x and jq is no exception. But if such operator is usually to be used with numbers, jq allows to use it with other types such as string as shown in the listing below. As comparison, line 3 and 5 shows how to add the “ is awesome” string to .info.contact.name using = and |=. And line 5 shows how to do the same modification using +=.

jq '.info.contact.name' demo-api-openapi.json 
jq '.info.contact.name = .info.contact.name + " is awesome"' demo-api-openapi.json | jq '.info.contact.name'
jq '.info.contact.name |= . + " is awesome"' demo-api-openapi.json | jq '.info.contact.name'
jq '.info.contact.name += " is awesome"' demo-api-openapi.json | jq '.info.contact.name'

Appending to a string

[apihandyman.io]$ jq '.info.contact.name' demo-api-openapi.json 
"The Banking API team"
[apihandyman.io]$ jq '.info.contact.name = .info.contact.name + " is awesome"' demo-api-openapi.json | jq '.info.contact.name'
"The Banking API team is awesome"
[apihandyman.io]$ jq '.info.contact.name |= . + " is awesome"' demo-api-openapi.json | jq '.info.contact.name'
"The Banking API team is awesome"
[apihandyman.io]$ jq '.info.contact.name += " is awesome"' demo-api-openapi.json | jq '.info.contact.name'
"The Banking API team is awesome"

Adding properties to object with +=

More interesting, += can be used on objects; that means it can be used to add properties to existing objects. The following listing shows how a Slack channel name can be added to .info.contact. All that is needed is to put an object with the desired properties on the right side of +=.

Note that this new property name is prefixed with x-; indeed the standard OpenAPI Contact object does not have such property but the OpenAPI specification allows to add custom ones. They must be prefixed by x- so parsers can detect them and do not consider them as errors. Note also that as this name contains a - is must b quoted.

jq '.info.contact' demo-api-openapi.json
jq '.info.contact += {"x-slack": "apiteam" }' demo-api-openapi.json | jq '.info.contact'

Adding properties

[apihandyman.io]$ jq '.info.contact' demo-api-openapi.json
{
  "name": "The Banking API team",
  "email": "[email protected]",
  "url": "developer.bankingcompany.com"
}
[apihandyman.io]$ jq '.info.contact += {"x-slack": "api-team" }' demo-api-openapi.json | jq '.info.contact'
{
  "name": "The Banking API team",
  "email": "[email protected]",
  "url": "developer.bankingcompany.com",
  "x-slack": "apiteam"
}

Adding and updating properties with +=

Another interesting aspect of += is that it can be used to replace values inside an object as shown below. The jq command on line 7 still adds a x-slack property to .info.contact but it also updates the existing name to “The Awesome Banking API team”.

jq '.info.contact' demo-api-openapi.json
jq '.info.contact += { name: "The Awesome Banking API team", "x-slack": "apiteam" }' demo-api-openapi.json | jq '.info.contact'

Adding and updating properties

[apihandyman.io]$ jq '.info.contact' demo-api-openapi.json
{
  "name": "The Banking API team",
  "email": "[email protected]",
  "url": "developer.bankingcompany.com"
}
[apihandyman.io]$ jq '.info.contact += { name: "updated name", "x-slack": "apiteam" }' demo-api-openapi.json | jq '.info.contact'
{
  "name": "The Awesome Banking API team",
  "email": "[email protected]",
  "url": "developer.bankingcompany.com",
  "x-slack": "apiteam"
}

JQ module

Before going further and to add real stuff on an OpenAPI file, here’s a jq module summarizing what we have learned about +=.

add.jq

# += adds anything to an existing value
# String concatenation
# Equivalent to 
#   .info.contact.name = .info.contact.name + " is awesome"
#   .info.contact.name |= . + " is awesome"
(.info.contact.name += " is awesome") | # Equivalent to .info.contact.name = .info.contact.name + " is awesome"
# Adding a property
(.info.contact += {"x-slack": "api-team" }) |
# Adding a property and updating existing name
(.info.contact += {"x-fax": "555-06-777", name: "The Awesome Banking API team"})

Adding missing 500 response when needed with +=

So far, we have replaced/add to known elements (.info.version) inside the demo OpenAPI file. Whatever the operator used (=, |= or +=,), we only provided them a static reference to update. But jq is more clever than that, we already have seen that we can but anything on the right side of an operator, it’s the same on the left side.

Let’s say we have an OpenAPI file in which some operations lack the definition of 500 (unexpected server error) responses. We could use += on their responses list to add the missing 500 response.

The path to each operations responses list is .paths.<some path>.<some http method>.responses, unfortunately, path and http method will never be the same and some operations may have a 500 response defined. We could manually and painfully list all the response without 500 and then write a jq file to update them all… Hopefully we won’t. We can let jq search and update the responses lists when needed.

First we need to identify the operation missing a 500 reponse. This is done with the following jq filters.

Selecting responses list without 500

.paths[][] |
select(type == "object") |
select(has("responses")) | 
.responses |
select(has("500") | not)

Note that we cannot simply use paths[][].responses because of path parameters list and some x-tension not containing a responses property, hence the two select. Without select(type == "object"), we have have a Cannot index array with string “responses” error because the path parameters list. And without select(has("responses"))we would have some null elements in out final list.

Once we get, at last, to the responses we keep only the one not having already a 500 property using select, has and not.

The following listing shows the result of this filters combination:

jq '.paths[][] | select(type=="object") | select(has("responses")) | .responses | select(has("500") | not) ' demo-api-openapi.json

Selecting responses list without 500 command line

[apihandyman.io]$ jq '.paths[][] | select(type=="object") | select(has("responses")) | .responses | select(has("500") | not) ' demo-api-openapi.json 
...
{
  "200": {
    "description": "The money transfer has been update"
  }
}
{
  "204": {
    "description": "The money transfer has been deleted"
  },
  "404": {
    "description": "The money transfer does not exist"
  }
}

Now that we are able to list the elements to fix, let’s put these filters on the left side of += and put the missing data on the right as we have just learned :

add-missing-500.jq

# First we select the elements to modify
(
  .paths[][] |
  select(type == "object") |
  select(has("responses")) | 
  .responses |
  select(has("500") | not)
)
# Then each of them is modified 
+= {
  "500":{
    description: "Unexpected error",
    content: {
      "application/json": {
        schema: {
          "$ref": "#components/schemas/ProviderError"
        }
      }
    }
  }
}
# The fully modified document is returned

As shown in the two following listings, the get /accounts operation does not have a 500 but it is hopefully easily fixed by applying add-missing-500.jq on the file. Note that all other operations lacking a 500 are also fixed as the new 500 property is added to each element identified by the filters on the left side of +=.

jq '.paths["/accounts"].get.responses' demo-api-openapi.json

Before

[apihandyman.io]$ jq '.paths["/accounts"].get.responses' demo-api-openapi.json 
{
  "200": {
    "description": "User's accounts",
    "content": {
      "application/json": {
        "schema": {
          "required": [
            "properties"
          ],
          "properties": {
            "items": {
              "type": "array",
              "items": {
                "$ref": "#/components/schemas/Account"
              }
            }
          }
        }
      }
    }
  }
}

jq -f add-missing-500.jq demo-api-openapi.json | jq '.paths["/accounts"].get.responses'

After

[apihandyman.io]$ jq -f add-missing-500.jq demo-api-openapi.json | jq '.paths["/accounts"].get.responses'
{
  "200": {
    "description": "User's accounts",
    "content": {
      "application/json": {
        "schema": {
          "required": [
            "properties"
          ],
          "properties": {
            "items": {
              "type": "array",
              "items": {
                "$ref": "#/components/schemas/Account"
              }
            }
          }
        }
      }
    }
  },
  "500": {
    "description": "Unexpected error",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#components/schemas/ProviderError"
        }
      }
    }
  }
}

Adding missing response content when needed with |= and complex update filter

In the demo-api-openapi.json file some operations may have basic 401 and 403 responses but without any schema (missing content property). Let’s fix that with a jq module inpired by previous one but using |= with a more complex filters on the left and accepting a regex HTTP status code and a schema model name.

The following jq module is composed of a |= main statement. On its left side, it selects all responses list. On its right side, it works on each key/value of all responses list using with_entry. If the key (HTTP status code) matches the regex ($coderegex) provided as a command line argument and does not already contain a content property, it add a content to the value. If not, it leaves the value as is is by returning ..

add-missing-response-content.jq

# Selecting all responses list
( 
  .paths[][] |
  select(type == "object") |
  select(has("responses")) | 
  .responses
) 
|= # Updating selected values
  with_entries( # transforms key: value into { key: key, value: value}
  # $coderegex is provided with --arg coderegex 40.
  if (.key | test($coderegex)) and (.value | has("content") | not) then
    .value += { # Actually updating the value
      content: {
        "application/json": {
          schema: { # $schema is provided with --arg schema ConsumerError 
            "$ref": ("#/components/schemas/" + $schema)
          }
        }
      }
  }
  else
    . # unmodified element
  end
)

Here’s the responses list of post /beneficiaries before modification:

jq '.paths["/beneficiaries"].post.responses' demo-api-openapi.json

Before

[apihandyman.io]$ jq '.paths["/beneficiaries"].post.responses' demo-api-openapi.json 
{
  "201": {
    "description": "Beneficiary added"
  },
  "401": {
    "description": "Unauthorized"
  },
  "403": {
    "description": "Forbidden"
  }
}

Once add-missing-response-content.jq is applied on the OpenAPI file, the responses matching the 40. regex (coderegex command line argument) have been modified to add a content with application/json media type referencing the schema provided with schema command line argument. And as seen previously that has been done to all operations responses.

jq --arg coderegex 40. --arg schema ConsumerError -f add-missing-response-content.jq demo-api-openapi.json | jq '.paths["/beneficiaries"].post.responses'

Before

[apihandyman.io]$ jq --arg coderegex 40. --arg schema ConsumerError -f add-missing-response-content.jq demo-api-openapi.json | jq '.paths["/beneficiaries"].post.responses'
{
  "201": {
    "description": "Beneficiary added"
  },
  "401": {
    "description": "Unauthorized",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/ConsumerError"
        }
      }
    }
  },
  "403": {
    "description": "Forbidden",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/ConsumerError"
        }
      }
    }
  }
}

Note that the selection of elements to modify with $coderegex could be fully made on right side (you could try to modify the module to do so).

Deleting elements

Last but not least, deleting elements. We’ll learn to use the following jq operators and function:

JQ Operators and Functions
`.info.description = "New description."` `.info.description = .info.contact.name`	Replaces a value, the input to the right-hand-side (rhs) is the same as the input to the left-hand-side (lhs) rather than the value at the lhs path
`.info.contact \|= .name` `.info.version \|= sub("-snapshot";"")`	Replaces a value, it takes a filter on the right-hand side and works out the new value for the property of `.` being assigned to by running the old value through this expression
`.info.contact \|= del(.contact)` `del(.info.contact)`	Removes a key and its corresponding value from an object
`walk(f)`	Applies a filter recursively to every component of the input entity
`delpaths(PATHS)`	Deletes an array of paths. each path is an array of string and numbers.

This section’s content is also available as an asciinema bash session:

Deleting elements

Dumbly deleting contact with = or |=

If we want to get rid of the contact property in info, we can do a replacement using = or a more clever |= as we have seen before. The idea is to keep all other properties (title, version, and description). Note how the |= syntax is simpler.

jq '.info' demo-api-openapi.json
jq '.info = { info: .info.title, version: .info.version, description: .info.description }' demo-api-openapi.json | jq '.info'
jq '.info |= { title, version, description }' demo-api-openapi.json | jq '.info'

Keeping title, version and description to delete contact

[apihandyman.io]$ jq '.info' demo-api-openapi.json
{
  "title": "Banking API",
  "version": "1.0.0-snapshot",
  "description": "The Banking API provides access ...",
  "contact": {
    "name": "The Banking API team",
    "email": "[email protected]",
    "url": "developer.bankingcompany.com"
  }
}
[apihandyman.io]$ jq '.info = { info: .info.title, version: .info.version, description: .info.description }' demo-api-openapi.json | jq '.info'
{
  "info": "Banking API",
  "version": "1.0.0-snapshot",
  "description": "The Banking API provides access ..."
}
[apihandyman.io]$ jq '.info |= { title, version, description }' demo-api-openapi.json | jq '.info'
{
  "info": "Banking API",
  "version": "1.0.0-snapshot",
  "description": "The Banking API provides access ..."
}

Efficiently deleting contact with del

That works because there’s not too many properties to keep inside info. But if you work on a more complex object this could become quite boring to delete an element. Hopefully, there is a del function that can be used to actually delete an element. It can be used with |= but also alone. The del function takes a path expressions so it can be directly used on .info.contact to delete this property.

jq '.info |= del(.contact)' demo-api-openapi.json | jq '.info'
jq 'del(.info.contact)' demo-api-openapi.json | jq '.info'

Actually deleting contact

[apihandyman.io]$ jq '.info |= del(.contact)' demo-api-openapi.json | jq '.info'
{
  "title": "Banking API",
  "version": "1.0.0-snapshot",
  "description": "The Banking API provides access ..."
}
[apihandyman.io]$ jq 'del(.info.contact)' demo-api-openapi.json | jq '.info'
{
  "title": "Banking API",
  "version": "1.0.0-snapshot",
  "description": "The Banking API provides access ..."
}

deleting deprecated operations or properties with walk and del

The OpenAPI 3 specification allows to tell that some operations ans even properties are deprecated by adding a deprecated: true. It can be useful to be able to delete all those deprecated elements. That can be done quite easily with walk and del functions.

JQ Functions
`.info.contact \|= del(.contact)` `del(.info.contact)`	Removes a key and its corresponding value from an object
`walk(f)`	Applies a filter recursively to every component of the input entity

The walk function is very useful, it will basically walk through all nodes inside the provided document and apply the filters provided as parameter. The following jq module is split in 3 steps and uses walk on each of them:

First we delete all object values having a deprecated property set to true
Then we delete the keys having a null value created by first step
And eventually, we delete all keys whose value became an empty object because of second step

delete-deprecated.jq

# Removes all object having a deprecated property set to true

# Before: "objectProperty": { "deprecated": true }
# After: "objectProperty": null
walk(
  if type=="object" and .deprecated == true then 
    del(.) 
  else 
    . 
  end
) | 
# Removes all property set to null ("nullProperty": null) created 
# when deleted objects containing deprecated set to true
walk(
  if type=="object" then 
    with_entries( 
      select( .value != null)
    ) 
  else  # Not an object, just keep it
    .
  end
) |
# Removes all empty property ("emptyProperty": {}) that may have
# been created when removing the null ones
walk(
  if type=="object" then 
    with_entries(
      select(.value != {} )
    )
  else # Not an object, just keep it
    . 
  end
)

Here’s the module in action (note for this demonstration we focus only on deprecated operations and reuse the search operations module from part 2):

jq -r --arg deprecated true -f search-operations.jq demo-api-openapi.json
jq -f delete-deprecated.jq demo-api-openapi.json | jq -r -f search-operations.jq --arg deprecated true

Deleting deprecated elements

[apihandyman.io]$ jq -r --arg deprecated true -f search-operations.jq demo-api-openapi.json
[demo-api-openapi.jso]  delete  /beneficiaries/{id}     Delete a beneficiary (deprecated)
[demo-api-openapi.jso]  patch   /beneficiaries/{id}     Updates a beneficiary (deprecated)

[apihandyman.io]$ jq -f delete-deprecated.jq demo-api-openapi.json | jq -r -f search-operations.jq --arg deprecated true

Deleting x-tensions with paths and delpaths

Another simple way of deleting elements is to use paths and delpaths, we can use them to remove all x-tensions (custom properties starting whose names start with x-) from an OpenAPI file.

JQ Functions
`paths`	Lists all possible paths in documents, each path is represented as an array
`delpaths(PATHS)`	Deletes an array of paths. each path is an array of string and numbers.

First we list all paths to those x-tensions. The paths function returns an array of all paths to all elements inside a document. Each path is an array, for example the path to contact is ["info","contact"] (hence paths returns an array of array). As we only want x-tensions, we filter this lists of paths to keep only the ones having their leafs prefixed by x-. Once we have this array of paths, we can use it as a parameter for delpaths and we’re done!

delete-xtensions.jq

# Lists all available x-tensions' paths
[(
  paths | # Lists ALL possible paths in documents 
          # (each path is represented as an array)
  select( # Keeps only the values for which what follows return true
    .[-1] | # Gets the path leaf (last item in array)
            # Equivalent to .[.|length-1]
    tostring | # Converts to string for next step
    test("^x-") # Matches "^x-" regex (starts with x-)
  )
)] as $xtensions |
# Delete all found x-tensions using their paths
delpaths($xtensions)

jq -r -f list-xtension-paths.jq demo-api-openapi.json
jq -f delete-xtensions.jq demo-api-openapi.json | jq -r -f list-xtension-paths.jq

Deleting x-tensions

[apihandyman.io]$ jq -r -f list-xtension-paths.jq demo-api-openapi.json 
[
  [
    "paths",
    "/accounts/{id}",
    "get",
    "x-implementation"
  ],
  ...
]

[apihandyman.io]$ jq -f delete-xtensions.jq demo-api-openapi.json | jq -r -f list-xtension-paths.jq 
[]

Deleting unsused schemas with delpaths

We’re under no obligation to use paths when using delpaths, we can build the paths ourselves.

JQ Filters and Functions
`..`	Returns every value recursively
`delpaths(PATHS)`	Deletes an array of paths. each path is an array of string and numbers.

The following module removes unused schemas from an OpenAPI file. Unused schemas are defined in components.schemas but never referenced in a $ref property.

To build the paths list, we first create a list of all possible $ref values using the keys of components.schemas. Then we substract all actually used $ref. Note the use of ..to get all document’s nodes and unique to keep only one occurrence of each used $ref. Once we have a list of #/components/schemas/name strings, we transform them in paths by splitting on / and removing #. And eventually we use delpaths on the resulting array of paths. Note that delpaths accept a parameter which is not strictly an array of array, indeed the unsused schema paths list is only a succession of individual arrays not enclosed inside an array.

delete-unused-schemas.jq

(
  # Defined schemas
  (
    .components.schemas | # select reusable schemas structure
    keys | # keeps only the schema names
    map("#/components/schemas/" + .) # return an array with schema refs
  )
  # Minus operator to substract used schemas from defined schemas
  -
  # Actually used schemas
  ([ # Creating an array
    .. | # selects all nodes
    select(type=="object") | # keeps only object
    select(has("$ref")) | # keeps only object having $ref property
    .["$ref"] # the $ref property (.$ref connot be used because of $) 
  ] | unique) # Keeps only one occurence
  |
  map(split("/") - ["#"])
) as $unused |
delpaths($unused)

jq -r -f list-unused-schemas.jq demo-api-openapi.json
jq -f delete-unused-schemas.jq demo-api-openapi.json | jq -r -f list-unused-schemas.jq

Deleting unused schemas

[apihandyman.io]$ jq -r -f list-unused-schemas.jq demo-api-openapi.json 
[
  "components",
  "schemas",
  "ProviderError"
]
[
  "components",
  "schemas",
  "UselessSchema"
]

[apihandyman.io]$ jq -f delete-unused-schemas.jq demo-api-openapi.json | jq -r -f list-unused-schemas.jq

Summary

And we’re done with part 3, you should now be able to modify any OpenAPI or any JSON file as you like using the following operators and functions:

JQ Filters, Operators and Functions
`.info.description = "New description."` `.info.description = .info.contact.name`	Replaces a value, the input to the right-hand-side (rhs) is the same as the input to the left-hand-side (lhs) rather than the value at the lhs path
`.info.contact \|= .name` `.info.version \|= sub("-snapshot";"")`	Replaces a value, it takes a filter on the right-hand side and works out the new value for the property of `.` being assigned to by running the old value through this expression
`.info.description += "More description"` `.info.description += .info.contact.name`	Increments a number value, concatenates string or add properties to object
`.info.contact \|= del(.contact)` `del(.info.contact)`	Removes a key and its corresponding value from an object
`walk(f)`	Applies a filter recursively to every component of the input entity
`paths`	Lists all possible paths in documents, each path is represented as an array
`..`	Returns every value recursively
`delpaths(PATHS)`	Deletes an array of paths. each path is an array of string and numbers.

What’s next

With what you have learned in the first 3 parts, you should be able to achieve anything you want with jq. And known that we only scratched the surface, check jq’s documentation to discover all of its features. The next and final part is a little bonus in which you’ll learn to colorize jq terminal output just for fun.