---
name: processing-json
description: Creates, reads, modifies, validates, and queries JSON and JSON Lines files. Use when the user asks to read, write, parse, edit, update, merge, validate, pretty-print, or query a .json or .jsonl file, mentions JSON data or JSON Lines, or asks to fix invalid JSON. Do not use for YAML/TOML config files or for designing APIs.
---

<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# Processing JSON

## When to use / when NOT to use
- **Use for:** any task where a `.json` or `.jsonl` file is the input or output — creating, parsing, editing, validating, querying.
- **Do NOT use for:** YAML/TOML config files or API design discussions.

## Quick reference — one default per operation

**Read / write — Python stdlib `json`:**
```python
import json
with open("data.json", encoding="utf-8") as f:
    data = json.load(f)                       # load() IS the syntax validator

with open("data.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)
```

**JSON Lines — one object per line, never json.load the whole file:**
```python
records = [json.loads(line) for line in open("data.jsonl", encoding="utf-8") if line.strip()]
with open("out.jsonl", "w", encoding="utf-8") as f:
    for r in records:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
```

**Modify — always atomically** (temp file + rename; a crash mid-write can't corrupt the original):
```python
import json, os, tempfile
with open("data.json", encoding="utf-8") as f:
    data = json.load(f)
data["version"] = 2
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath("data.json")))
with os.fdopen(fd, "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)
os.replace(tmp, "data.json")
```

**Query — escape hatch for large files:** `jq '.items[] | select(.active)' big.json` (requires jq installed: `brew install jq`). For everything else, load and filter in Python.

**Schema validation:** `pip install jsonschema`, then `jsonschema.validate(instance=data, schema=schema)`.

## Rules
- **Never edit JSON with regex or string replacement.** Parse → mutate → dump. Always.
- Preserve key order: Python dicts keep insertion order; do not pass `sort_keys=True` unless asked.
- `indent=2, ensure_ascii=False` for human-facing files; single-line compact only for machine-to-machine output.

## Workflow
1. Load the input (`json.load` / line-by-line for `.jsonl`). A parse error here is a finding, not a failure — see edge cases.
2. Apply the change/query in Python on the parsed structure.
3. Write atomically (recipe above).
4. **Validate:** re-open and `json.load` the written file; for `.jsonl`, re-parse every line. Only then report success.
5. Report the output path and what changed (keys touched, records added/removed).

## Edge cases & failure modes
- **Malformed JSON** → `json.JSONDecodeError` includes line and column; report it verbatim (e.g. "Expecting ',' delimiter: line 12 column 3") and show the offending line. Common causes: trailing commas, single quotes, comments — fix precisely, don't guess.
- **`NaN`/`Infinity` in input** → stdlib accepts them but they are NOT valid JSON; re-emit with `json.dump(..., allow_nan=False)` after replacing them with `null` (confirm with the user).
- **Missing dependency** → only third-party need: `pip install jsonschema` (schema validation) or `brew install jq`.
- **Huge file (>500 MB)** → if it's `.jsonl`, stream line-by-line; if a single JSON document, use `jq` rather than loading into Python.
- **Empty file** → report "file is empty — not valid JSON (an empty JSON file should contain `{}` or `[]`)" and ask which the user wants.
- **Duplicate keys** → `json.load` silently keeps the last one; when auditing, parse with `object_pairs_hook=list` to detect them.

## References
Deeper recipes (merging, diffing, flattening, jsonl↔json, encoding traps): see [references/recipes.md](references/recipes.md).