spb/ultra-sharp-agent-skills Public

1<!--2Author: Simon-Pierre Boucher3Contact: contact@spboucher.ai4-->56# Patterns — Handling File Uploads78## Contents9- Presigned direct-to-storage flow10- Magic-byte validation table11- Streaming multipart limit enforcement12- Quarantine and scan promotion13- Resumable uploads and lifecycle cleanup14- Gotchas1516## Presigned direct-to-storage flow1718```python19# 1) Client asks to upload20# POST /v1/uploads {"filename": "report.pdf", "content_type": "application/pdf", "size": 8123456}21import boto3, uuid2223# 15 min: enough for a slow client on the declared size, short enough that a24# leaked URL is a bounded liability.25PRESIGN_TTL = 9002627def create_upload(req):28    validate_declared(req.content_type, req.size)          # rule 2 pre-check29    file_id = str(uuid.uuid4())30    key = f"quarantine/{file_id}.pdf"                      # rule 4: server key31    url = s3.generate_presigned_post(32        Bucket="uploads", Key=key,33        Fields={"Content-Type": req.content_type},34        Conditions=[35            {"Content-Type": req.content_type},36            ["content-length-range", 1, 26_214_400],       # 25 MB doc cap37        ],38        ExpiresIn=PRESIGN_TTL)39    db.insert("uploads", id=file_id, key=key, status="pending",40              original_name=req.filename, declared_size=req.size)41    return {"file_id": file_id, "upload": url}4243# 2) Client PUT/POSTs the file to `url`44# 3) POST /v1/uploads/{file_id}/complete → server HEADs the object, verifies45#    size matches, kicks the scan job, returns {"status": "scanning"}46```4748## Magic-byte validation table4950```python51MAGIC = {52    "image/png":  [b"\x89PNG\r\n\x1a\n"],53    "image/jpeg": [b"\xff\xd8\xff"],54    "image/webp": [b"RIFF"],            # + b"WEBP" at offset 855    "application/pdf": [b"%PDF"],56    "application/zip": [b"PK\x03\x04"], # also docx/xlsx/pptx containers57}5859def sniff_ok(claimed: str, head: bytes) -> bool:60    sigs = MAGIC.get(claimed)61    return bool(sigs) and any(head.startswith(s) for s in sigs)62# Read the first 16 bytes from storage (ranged GET) — never rely on the63# client's Content-Type alone.64```6566## Streaming multipart limit enforcement6768```python69# FastAPI/Starlette example — same idea in any framework: consume the stream70# in chunks and abort the moment the cap is crossed.71CHUNK = 64 * 102472MAX_IMAGE = 10 * 1024 * 10247374async def save_stream(stream, dest, cap=MAX_IMAGE):75    written = 076    with open(dest, "wb") as f:77        async for chunk in stream:78            written += len(chunk)79            if written > cap:80                f.close(); os.unlink(dest)81                raise Payload413(f"file exceeds {cap // 1_048_576} MB limit")82            f.write(chunk)83    return written84```8586## Quarantine and scan promotion8788```89quarantine/{uuid}.{ext}   status=pending   (not servable)90        │  scan job (ClamAV / provider malware scan)91        ├── clean    → copy to files/{uuid}.{ext}, status=clean, delete quarantine copy92        └── infected → delete object, status=rejected, notify uploader93```94Record `sha256` at scan time for dedup/audit. The serving layer reads only95`files/` and only rows with `status=clean`.9697## Resumable uploads and lifecycle cleanup9899```python100# S3 multipart: 8 MB parts — large enough to keep part count low (10k max),101# small enough that a retry wastes little.102PART_SIZE = 8 * 1024 * 1024103```104105Bucket lifecycle rules (Terraform-style):106107```hcl108rule { id = "abort-incomplete"  abort_incomplete_multipart_upload_days = 1 }109rule { id = "purge-quarantine"  prefix = "quarantine/"  expiration_days = 1 }110```111112For browser-based resumable uploads prefer tus (tusd server or provider113equivalent) over hand-rolled chunk endpoints.114115## Gotchas116117- **`Content-Length` is a claim, not a fact** — always cap the actual stream118  too (rule 2 belt-and-braces).119- **docx/xlsx/pptx sniff as zip** (`PK\x03\x04`) — accept the zip signature,120  then check the internal `[Content_Types].xml` if you must distinguish.121- **Filename header injection:** original filenames go into122  `Content-Disposition` later; store them escaped and serve with123  `filename*=UTF-8''...` encoding, or attackers smuggle header content.124- **Presigning with no Content-Type condition** lets an attacker upload125  `text/html` to your bucket and phish from your domain.126- **EXIF in images** may carry GPS/PII — strip metadata at scan/processing127  time if images are re-served publicly.128- **Multipart ETag ≠ MD5** on S3 multipart uploads — use explicit per-part129  checksums (`ChecksumSHA256`) for integrity, not ETag comparison.130