---
name: handling-file-uploads
description: Designs safe, scalable file-upload handling — presigned direct-to-storage URLs for large files, multipart for small ones, magic-byte content validation, server-generated storage names, streaming instead of buffering, and resumable chunked uploads. Use when the user asks to implement or review file, image, video, or document uploads, add an upload endpoint, validate uploaded files, or generate presigned S3/GCS upload URLs. Do not use for serving or downloading files, image processing/resizing, or CDN configuration.
---

<!--
Author: Simon-Pierre Boucher
Contact: contact@spboucher.ai
-->

# Handling File Uploads

## When to use / when NOT to use
- **Use for:** upload endpoint design, presigned-URL flows, upload validation and limits, storage naming, resumable uploads, virus-scanning hook points.
- **Do NOT use for:** download/serving paths, thumbnailing or transcoding pipelines, CDN/cache setup — handle those separately.

## Core rules

1. **Files >5 MB go direct to storage via presigned URL — never through the app.** The API issues a short-lived presigned PUT (15 min expiry, content-length-range enforced), the client uploads to S3/GCS, then confirms. App servers proxy only small multipart uploads.
   - ✅ `POST /uploads` → `{upload_url, file_id}` → client PUTs to storage → `POST /uploads/{file_id}/complete`
   - ❌ 500 MB video buffered through the web server's memory
2. **Enforce size limits before buffering.** Reject on `Content-Length` first, and hard-cap the stream anyway (clients lie). Set limits per file type: images 10 MB, documents 25 MB, video via resumable only — and put the numbers in the error message.
3. **Validate content, not just extension.** Check magic bytes (e.g. `%PDF`, `\x89PNG`) against the claimed type; reject mismatches. Extension and `Content-Type` header are client-controlled hints, nothing more.
4. **Never trust the client filename.** Generate the storage key server-side (`uploads/2026/08/{uuid4}.pdf`); keep the original name as escaped metadata only. This kills path traversal (`../../etc/cron.d/x`) and collision attacks.
5. **Stream, don't buffer.** Small-path multipart parsing goes chunk-by-chunk to disk/storage; memory use must be O(chunk), not O(file).
6. **Uploads are pending until scanned and confirmed.** New objects land in a quarantine prefix/bucket with a `pending` status; a scanning hook (ClamAV, provider malware scan) promotes to `clean` or deletes. Serve nothing from quarantine.
7. **Very large files use chunked/resumable uploads** (S3 multipart or tus): 8 MB parts, per-part checksums and retries, abort-and-clean incomplete uploads older than 24 h (lifecycle rule).

## Workflow

1. Classify expected uploads: types, size ranges, volume → pick path per rule 1/7.
2. Define the presigned flow endpoints (`create`, `complete`) or the multipart endpoint with limits (rule 2).
3. Implement validation: magic bytes (rule 3), server-side naming (rule 4).
4. Wire the quarantine + scan hook (rule 6) and lifecycle cleanup (rule 7).
5. **Validate:** upload (a) an oversized file → clean 413 with the limit named; (b) an `.exe` renamed `.png` → rejected by magic bytes; (c) a filename `../../x` → stored under the generated UUID key with metadata escaped; (d) a happy-path file → reaches `clean` status and only then is retrievable.

## Edge cases & failure modes
- **Client never calls `complete`** → the `pending` record and quarantine object are garbage-collected by the 24 h lifecycle rule.
- **Duplicate uploads** → hash the content (SHA-256) at scan time; either dedupe by hash or at least record it for later dedup/audit.
- **Zip/archive uploads** → beware zip bombs: cap decompressed size and entry count before extracting anything.
- **SVG uploads** → they are executable XML (scripts, external entities); sanitize or serve with `Content-Disposition: attachment` + strict CSP, never inline from your origin.
- **Presigned URL leaks** → 15-minute expiry plus `content-length-range` bounds the damage; never sign without both.

## References
Deeper patterns (presigned flow code, magic-byte table, tus/multipart setup, lifecycle rules): see [references/patterns.md](references/patterns.md).