<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# JSON Recipes

## Contents
- Create
- Read / query
- Modify
- Convert (json ↔ jsonl, json → CSV)
- Validate
- Gotchas

## Create

**New file with non-ASCII content kept readable:**
```python
import json
data = {"name": "Café Müller", "items": [1, 2, 3]}
with open("out.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)
    f.write("\n")                     # trailing newline: POSIX-friendly diffs
```

**Compact machine-to-machine output:**
```python
json.dumps(data, separators=(",", ":"), ensure_ascii=False)
```

## Read / query

**Safe nested access:**
```python
value = data.get("config", {}).get("db", {}).get("host")   # None if any level missing
```

**Filter a list of records:**
```python
active = [r for r in data["users"] if r.get("active")]
```

**jq equivalents for large files (jq must be installed):**
```bash
jq '.users[] | select(.active) | .email' big.json      # filter + project
jq 'length' big.json                                   # count
jq -r '.items[].id' big.json                           # raw strings, no quotes
```

**Detect duplicate keys while parsing:**
```python
def no_dupes(pairs):
    keys = [k for k, _ in pairs]
    dupes = {k for k in keys if keys.count(k) > 1}
    if dupes:
        raise ValueError(f"duplicate keys: {sorted(dupes)}")
    return dict(pairs)

data = json.load(open("in.json", encoding="utf-8"), object_pairs_hook=no_dupes)
```

## Modify

**Atomic in-place update (the only sanctioned write pattern for existing files):**
```python
import json, os, tempfile

def update_json(path, mutate):
    with open(path, encoding="utf-8") as f:
        data = json.load(f)
    mutate(data)
    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(path)))
    with os.fdopen(fd, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2, ensure_ascii=False)
        f.write("\n")
    os.replace(tmp, path)             # atomic on POSIX

update_json("config.json", lambda d: d.setdefault("features", {}).update(dark_mode=True))
```

**Deep merge (dict-over-dict, right side wins):**
```python
def deep_merge(base, override):
    out = dict(base)
    for k, v in override.items():
        out[k] = deep_merge(out[k], v) if isinstance(out.get(k), dict) and isinstance(v, dict) else v
    return out
```

**Shallow diff of two objects:**
```python
def diff(a, b):
    keys = a.keys() | b.keys()
    return {k: (a.get(k), b.get(k)) for k in keys if a.get(k) != b.get(k)}
```

## Convert

**jsonl → json array:**
```python
records = [json.loads(l) for l in open("in.jsonl", encoding="utf-8") if l.strip()]
json.dump(records, open("out.json", "w", encoding="utf-8"), indent=2, ensure_ascii=False)
```

**json array → jsonl (streams better, appends safely):**
```python
with open("out.jsonl", "w", encoding="utf-8") as f:
    for r in json.load(open("in.json", encoding="utf-8")):
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
```

**Flat records → CSV** (nested values must be flattened or stringified first):
```python
import csv
rows = json.load(open("in.json", encoding="utf-8"))
fields = sorted({k for r in rows for k in r})
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=fields)
    w.writeheader(); w.writerows(rows)
```

**Flatten nested keys with dots:**
```python
def flatten(d, prefix=""):
    out = {}
    for k, v in d.items():
        key = f"{prefix}{k}"
        out.update(flatten(v, key + ".")) if isinstance(v, dict) else out.setdefault(key, v)
    return out
```

## Validate

**Syntax:** `python3 -m json.tool file.json > /dev/null` — prints the error with line/column, exit 1 on failure.

**Schema:**
```python
# pip install jsonschema
from jsonschema import validate, ValidationError
try:
    validate(instance=data, schema=schema)
except ValidationError as e:
    print(f"invalid at {list(e.absolute_path)}: {e.message}")
```

## Gotchas
- **Trailing commas, single quotes, comments** are the top three causes of `JSONDecodeError` — they are JavaScript habits, not JSON. Fix the exact character the error points at.
- **`NaN`, `Infinity`**: `json.dumps` emits them by default but no strict parser accepts them. Use `allow_nan=False` to force the error at write time, then substitute `null`.
- **Encoding:** JSON files are UTF-8 by spec; always pass `encoding="utf-8"` — Windows defaults to cp1252 and corrupts round-trips.
- **Float precision:** `json.load` gives you binary floats (`0.1 + 0.2 != 0.3`); for money, parse with `json.load(f, parse_float=decimal.Decimal)`.
- **Large ints** round-trip fine in Python but break JavaScript beyond 2^53 − 1; stringify IDs above that when the consumer is JS.
- **`sort_keys=True` rewrites the whole file's order** — a huge diff for a one-key change. Leave order alone unless asked.
- **BOM:** files from Windows tools may start with U+FEFF; open with `encoding="utf-8-sig"` if `json.load` fails on character 0.