SPB Git forge

spb/doc-api

Public
2commits 1branches 0releases
15.7 MBsize
maindefault branch
13 days agolast push
Python 88.3% TypeScript 7.6% Shell 4.1%
12.1 KB · 148 lines markdown
Rendered Raw Blame History
1# Gemini — File Search (`tools[].fileSearch` + `fileSearchStores` API, Preview)23**Status:** DOCUMENTED + LIVE_VERIFIED (2026-09-18 run — see "Live verification" section at the end)45Sources:6- https://ai.google.dev/gemini-api/docs/generate-content/file-search · https://ai.google.dev/gemini-api/docs/file-search7- https://ai.google.dev/api/file-search/file-search-stores · https://ai.google.dev/api/file-search/documents · https://ai.google.dev/api/all-methods8- https://ai.google.dev/api/generate-content (#FileSearch #RetrievedContext #CustomMetadata) · https://ai.google.dev/gemini-api/docs/pricing · changelog (public preview 2025-11; multimodal 2026-02)9- Discovery v1beta rev. 20260918: resources `fileSearchStores`, `media.uploadToFileSearchStore`1011Last verified: 2026-09-18 (docs only)1213## 1. Concepts and lifecycle1415```16 file bytes ──uploadToFileSearchStore (LRO)──▶ FileSearchStore ──▶ Document ──▶ Chunks + embeddings17 Files API File (48 h) ──importFile (LRO)────▶      │18                                                     └── generateContent tools[].fileSearch ──▶ groundingMetadata.retrievedContext19```20- **FileSearchStore**: persistent container (no TTL; deleted only by you or model deprecation). Names are globally scoped, server-generated: `fileSearchStores/<displayName-slug>-<12 random chars>`.21- **Document**: one ingested file; `state` `STATE_PENDING` → `STATE_ACTIVE` | `STATE_FAILED`; up to 20 `customMetadata`.22- Embedding model fixed at store creation: default text model (`gemini-embedding-001` per guide) or `models/gemini-embedding-2` for **multimodal** (PNG/JPEG ≤ 4K×4K).23- Semantic search at query time; retrieved chunks are injected as context and cited in `groundingMetadata`.2425## 2. Endpoints2627| Method | Path | Body / params | Returns | SDK (python / node) |28|---|---|---|---|---|29| POST | `/v1beta/fileSearchStores` | `{displayName?, embeddingModel?}` | `FileSearchStore` | `client.file_search_stores.create(config={'display_name':…,'embedding_model':…})` / `ai.fileSearchStores.create({config})` |30| GET | `/v1beta/fileSearchStores` | `pageSize` (≤10), `pageToken` | `{fileSearchStores[], nextPageToken}` | `.list()` |31| GET | `/v1beta/fileSearchStores/{id}` | — | `FileSearchStore` | `.get(name=…)` |32| DELETE | `/v1beta/fileSearchStores/{id}` | `force` (bool) | `{}` | `.delete(name=…, config={'force': True})` |33| POST | **`/upload/v1beta/fileSearchStores/{id}:uploadToFileSearchStore`** (media URI; also `/resumable/upload/v1beta/…`; metadata-only `/v1beta/…:uploadToFileSearchStore`) | multipart/resumable: metadata `{displayName?, mimeType?, customMetadata[]?, chunkingConfig?}` + bytes (≤100 MB, any MIME) | `Operation` (`fileSearchStores/{id}/upload/operations/{op}`) | `.upload_to_file_search_store(file=…, file_search_store_name=…, config={…})` / `ai.fileSearchStores.uploadToFileSearchStore({file, fileSearchStoreName, config})` |34| POST | `/v1beta/fileSearchStores/{id}:importFile` | `{fileName: "files/…", customMetadata[]?, chunkingConfig?}` | `Operation` (`fileSearchStores/{id}/operations/{op}`) | `.import_file(file_search_store_name=…, file_name=…, custom_metadata=[…])` / `ai.fileSearchStores.importFile({…})` |35| GET | `/v1beta/fileSearchStores/{id}/documents` | `pageSize` (≤20), `pageToken` | `{documents[], nextPageToken}` | `.documents.list(parent=…)` |36| GET | `/v1beta/fileSearchStores/{id}/documents/{doc}` | — | `Document` | `.documents.get(name=…)` |37| DELETE | `/v1beta/fileSearchStores/{id}/documents/{doc}` | `force` | `{}` | `.documents.delete(name=…, config={'force': True})` |38| GET | `/v1beta/fileSearchStores/{id}/operations/{op}` | — | `Operation` | `client.operations.get(op)` / `ai.operations.get({operation})` |39| GET | `/v1beta/fileSearchStores/{id}/upload/operations/{op}` | — | `Operation` | same |40| GET | `/v1beta/fileSearchStores/{id}/media/{blob}` (guide shows `/v1/`) | — | image bytes of a cited `mediaId` | `.download_media(media_id=…)` / `ai.fileSearchStores.downloadMedia(id)` — not in discovery (UNVERIFIED) |4142Delete semantics: `force=false` (default) → `FAILED_PRECONDITION` if the store has Documents / the Document has Chunks; `force=true` cascades.4344## 3. Upload protocols (REST)4546Resumable (guide example): `POST https://generativelanguage.googleapis.com/upload/v1beta/fileSearchStores/{id}:uploadToFileSearchStore` with headers `X-Goog-Upload-Protocol: resumable`, `X-Goog-Upload-Command: start`, `X-Goog-Upload-Header-Content-Length: <bytes>`, `X-Goog-Upload-Header-Content-Type: <mime>`, JSON body `{"displayName":"sample.txt"}` → read `X-Goog-Upload-URL` from response headers → `POST <upload url>` with `X-Goog-Upload-Offset: 0`, `X-Goog-Upload-Command: upload, finalize`, `--data-binary @file` → Operation JSON. Simple/multipart: `multipart/related` body on the same `/upload/v1beta/` URI (discovery `mediaUpload.protocols.simple.multipart: true`). Poll `GET /v1beta/<operation.name>` until `done: true`.4748## 4. Chunking, metadata, filters4950| Item | Shape |51|---|---|52| `chunkingConfig.whiteSpaceConfig` | `{maxTokensPerChunk (words; ≤512 recommended → ≈2,560 tokens), maxOverlapTokens}` — only algorithm exposed |53| `customMetadata[]` | `{key, stringValue \| numericValue \| stringListValue{values[]}}`; ≤20 per Document |54| `tools[].fileSearch.metadataFilter` | **string**, AIP-160 list-filter syntax (google.aip.dev/160), e.g. `author = "Robert Graves"`, `year > 1930`; operators (Condition enum): `LESS`, `LESS_EQUAL`, `EQUAL`, `GREATER_EQUAL`, `GREATER`, `NOT_EQUAL`, `INCLUDES`, `EXCLUDES` (string lists) |55| `tools[].fileSearch.topK` | integer, REST reference only |56| `tools[].fileSearch.fileSearchStoreNames[]` | required; several stores allowed |5758## 5. Query response5960```json61"groundingMetadata": {62  "groundingChunks": [{"retrievedContext": {"title": "sample.txt", "uri": "...", "text": "chunk text",63      "fileSearchStore": "fileSearchStores/my-store-123", "pageNumber": 3,64      "mediaId": "fileSearchStores/my-store-123/media/BlobId-456",65      "customMetadata": [{"key": "author", "stringValue": "Robert Graves"}, {"key": "year", "numericValue": 1934}]}}],66  "groundingSupports": [{"segment": {"startIndex": 0, "endIndex": 50, "text": "..."}, "groundingChunkIndices": [0]}]67}68```69`pageNumber` for paged documents (PDF); `mediaId` for image chunks (persistent across searches). With `includeServerSideToolInvocations`: `toolCall{toolType: FILE_SEARCH}` (no user-visible args). Structured outputs + File Search: Gemini 3.7071## 6. Supported file types (summary)7273Application: dart, ecmascript, json, ms-java, msword, **pdf**, sql, typescript, vnd.curl, vnd.dart, vnd.ibm.secure-container, vnd.jupyter, vnd.ms-excel, vnd.oasis.opendocument.text, OOXML pptx/xlsx/docx/dotx, x-csh, x-hwp(-v5), x-latex, x-php, x-powershell, x-sh, x-shellscript, x-tex, x-zsh, xml, zip. Text: ~150 `text/*` types incl. plain, html, css, csv, tsv, markdown, javascript, jsx, tsx, x-python, x-java, x-c, x-go, x-rust, yaml, xml… (full list in the guide). Images (PNG, JPEG) only with `models/gemini-embedding-2`. **Audio and video not supported.**7475## 7. Limits and tiers7677| Limit | Value |78|---|---|79| Max file / document size | 100 MB (discovery `maxSize` 104857600) |80| Total store size per project | Free 1 GB · Tier 1 10 GB · Tier 2 100 GB · Tier 3 1 TB (backend ≈3× input incl. embeddings) |81| Recommended store size | < 20 GB for retrieval latency |82| `displayName` | ≤512 chars; store id ≤40 chars |83| Live API | not supported |84| Tool combinations | cannot be combined with Google Search, URL context, etc.; Gemini 3 can combine with function calling |8586## 8. Pricing8788| Item | Price |89|---|---|90| Indexing (embedding at ingest) | **$0.15 / 1M tokens** (pricing tools table; the gemini-embedding-2 table lists $0.20 standard / $0.10 batch text input — inconsistent) |91| Storage | free |92| Query-time embedding | free |93| Retrieved chunks | billed as regular input tokens of the generating model |94| Free tier | free of charge |9596## 9. Supported models9798Guide table: gemini-3.8/3.7/3.6/3.5-flash, 3.5-flash-lite, 3.1-pro-preview, 3.1-flash-lite, 3-flash-preview, 2.5-pro, 2.5-flash-lite. Model pages add gemini-2.5-flash, gemini-3-pro-preview, 3.1-flash-lite-preview, robotics ER 2; gemini-3.1-pro-preview page says "Supported (AI Studio only)".99100## 10. Examples101102```bash103# create store104curl -s -X POST "https://generativelanguage.googleapis.com/v1beta/fileSearchStores" \105  -H "x-goog-api-key: $GEMINI_API_KEY" -H "Content-Type: application/json" \106  -d '{"displayName":"atlas-test","embeddingModel":"models/gemini-embedding-2"}'107# query108curl -s "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash-lite:generateContent" \109  -H "x-goog-api-key: $GEMINI_API_KEY" -H "Content-Type: application/json" \110  -d '{"contents":[{"parts":[{"text":"What is in the uploaded file?"}]}],111       "tools":[{"fileSearch":{"fileSearchStoreNames":["fileSearchStores/STORE_ID"],"metadataFilter":"author = \"X\""}}]}'112```113```python114store = client.file_search_stores.create(config={"display_name": "atlas-test"})115op = client.file_search_stores.upload_to_file_search_store(file="sample.txt", file_search_store_name=store.name,116        config={"display_name": "sample.txt", "custom_metadata": [{"key": "author", "string_value": "X"}]})117while not op.done: time.sleep(3); op = client.operations.get(op)118r = client.models.generate_content(model="gemini-3.5-flash-lite", contents="What is in the file?",119        config=types.GenerateContentConfig(tools=[types.Tool(file_search=types.FileSearch(file_search_store_names=[store.name]))]))120print(r.candidates[0].grounding_metadata.grounding_chunks)121client.file_search_stores.delete(name=store.name, config={"force": True})122```123```ts124const store = await ai.fileSearchStores.create({ config: { displayName: 'atlas-test' } });125let op = await ai.fileSearchStores.uploadToFileSearchStore({ file: 'sample.txt', fileSearchStoreName: store.name, config: { displayName: 'sample.txt' } });126while (!op.done) op = await ai.operations.get({ operation: op });127const r = await ai.models.generateContent({ model: 'gemini-3.5-flash-lite', contents: 'What is in the file?',128  config: { tools: [{ fileSearch: { fileSearchStoreNames: [store.name] } }] } });129await ai.fileSearchStores.delete({ name: store.name, config: { force: true } });130```131132## Live verification (2026-09-18)133**LIVE_VERIFIED** end-to-end (`tmp-live/gemini-tools/g*.json`, `examples/gemini/file-search/file_search_lifecycle.py`, `file_search.ts`, `tests/gemini/test_file_search.py`):134135| Step | Call | Observed |136|---|---|---|137| create | `POST /v1beta/fileSearchStores {displayName}` | 200 `{name:"fileSearchStores/atlasprobe20260918-…", displayName, createTime, updateTime, embeddingModel:"models/gemini-embedding-001"}` |138| upload (resumable) | `POST /upload/v1beta/{store}:uploadToFileSearchStore` headers `X-Goog-Upload-Protocol: resumable`, `-Command: start`, JSON `{displayName, customMetadata, chunkingConfig}` | 200 + `X-Goog-Upload-URL` (+ `X-Goog-Upload-Control-URL`, `-Chunk-Granularity`, `-Status`) |139| finalize | `POST <upload url>` `X-Goog-Upload-Command: upload, finalize`, 200 bytes | 200 `Operation{name:".../upload/operations/…", response}` (`done` omitted) |140| poll | `GET /v1beta/{operation}` | `done:true`, `response{@type, documentName, mimeType, parent, sizeBytes}` |141| documents | `GET …/documents`, `GET …/documents/{doc}` | `{name, displayName, mimeType, sizeBytes, state:"STATE_ACTIVE", customMetadata, createTime, updateTime}` |142| query | `tools:[{fileSearch:{fileSearchStoreNames:[…], metadataFilter:"kind=probe"}}]` | correct answer; `groundingMetadata.groundingChunks[{retrievedContext{title, text, fileSearchStore}}]` + `groundingSupports` in 2 of 3 runs, **absent once** (only `usageMetadata.toolUsePromptTokenCount` 834–985 proved retrieval) |143| store get / list | `GET /v1beta/{store}`, `GET /v1beta/fileSearchStores?pageSize=5` | `activeDocumentsCount:"1"`, `sizeBytes:"200"`; `{fileSearchStores:[…]}` |144| delete document | `DELETE /v1beta/{doc}` | **400** `FAILED_PRECONDITION "Cannot delete non-empty Document"`; with `?force=true` → 200 |145| delete store | `DELETE /v1beta/{store}?force=true` | 200 `{}` |146147Not exercised: `importFile`, `/operations/{op}` poller for imports, `media/{blob}` download.148