Repository Public
1# Batch API23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Learn how to use OpenAI's Batch API to send asynchronous groups of requests with 50% lower costs, a separate pool of significantly higher rate limits, and a clear 24-hour turnaround time. The service is ideal for processing jobs that don't require immediate responses. You can also [explore the API reference directly here](https://developers.openai.com/api/reference/resources/batches).67## Overview89While some uses of the OpenAI Platform require you to send synchronous requests, there are many cases where requests do not need an immediate response or [rate limits](https://developers.openai.com/api/docs/guides/rate-limits) prevent you from executing a large number of queries quickly. Batch processing jobs are often helpful in use cases like:10111. Running evaluations122. Classifying large datasets133. Embedding content repositories144. Queuing large offline video-render jobs1516The Batch API offers a straightforward set of endpoints that allow you to collect a set of requests into a single file, kick off a batch processing job to execute these requests, query for the status of that batch while the underlying requests execute, and eventually retrieve the collected results when the batch is complete.1718Compared to using standard endpoints directly, Batch API has:19201. **Better cost efficiency:** 50% cost discount compared to synchronous APIs212. **Higher rate limits:** [Substantially more headroom](https://platform.openai.com/settings/organization/limits) compared to the synchronous APIs223. **Fast completion times:** Each batch completes within 24 hours (and often more quickly)2324## Getting started2526### 1. Prepare your batch file2728Batches start with a `.jsonl` file where each line contains the details of an individual request to the API. For now, the available endpoints are:2930- `/v1/responses` ([Responses API](https://developers.openai.com/api/reference/resources/responses))31- `/v1/chat/completions` ([Chat Completions API](https://developers.openai.com/api/reference/resources/chat))32- `/v1/embeddings` ([Embeddings API](https://developers.openai.com/api/reference/resources/embeddings))33- `/v1/completions` ([Completions API](https://developers.openai.com/api/reference/resources/completions))34- `/v1/moderations` ([Moderation guide](https://developers.openai.com/api/docs/guides/moderation))35- `/v1/images/generations` ([Images API](https://developers.openai.com/api/reference/resources/images))36- `/v1/images/edits` ([Images API](https://developers.openai.com/api/reference/resources/images))37- `/v1/videos` ([Video generation guide](https://developers.openai.com/api/docs/guides/video-generation))3839For a given input file, the parameters in each line's `body` field are the same as the parameters for the underlying endpoint. Each request must include a unique `custom_id` value, which you can use to reference results after completion. Here's an example of an input file with 2 requests. Note that each input file can only include requests to a single model.4041For video generation in Batch:4243- Batch currently supports `POST /v1/videos` only.44- Batch requests for videos must use JSON, not multipart.45- Upload assets ahead of time and pass supported asset references in the request body rather than using multipart uploads.46- Use `input_reference` for image-guided generations in Batch. In JSON requests, pass `input_reference` as an object with either `file_id` or `image_url`.47- Multipart `input_reference` uploads, including video reference inputs, aren't supported in Batch.48- Batch-generated videos are available for download for up to `24` hours after the batch completes.4950When targeting `/v1/moderations`, include an `input` field in every request body. Batch accepts plain-text inputs and content arrays with text or image inputs using `omni-moderation-latest`. The Batch worker rejects requests that set `stream=true`, matching the synchronous moderation endpoint.5152```jsonl53{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}54{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}55```5657#### Moderation input examples5859Text-only request:6061```jsonl62{63 "custom_id": "moderation-text-1",64 "method": "POST",65 "url": "/v1/moderations",66 "body": {67 "model": "omni-moderation-latest",68 "input": "This is a harmless test sentence."69 }70}71```7273Request with text and image input:7475```jsonl76{77 "custom_id": "moderation-mm-1",78 "method": "POST",79 "url": "/v1/moderations",80 "body": {81 "model": "omni-moderation-latest",82 "input": [83 {84 "type": "text",85 "text": "Describe this image"86 },87 {88 "type": "image_url",89 "image_url": {90 "url": "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg"91 }92 }93 ]94 }95}96```9798Prefer referencing remote assets with `image_url` (instead of base64 blobs) to99 keep your `.jsonl` files well below the 200 MB Batch upload limit,100 especially for multimodal Moderations requests.101102### 2. Upload your batch input file103104Similar to our [Fine-tuning API](https://developers.openai.com/api/docs/guides/model-optimization), you must first upload your input file so that you can reference it correctly when kicking off batches. Upload your `.jsonl` file using the [Files API](https://developers.openai.com/api/reference/resources/files).105106Upload files for Batch API107108```javascript109import fs from "fs";110import OpenAI from "openai";111const openai = new OpenAI();112113const file = await openai.files.create({114 file: fs.createReadStream("fixtures/batchinput.jsonl"),115 purpose: "batch",116});117118console.log(file);119```120121```python122from openai import OpenAI123124client = OpenAI()125126batch_input_file = client.files.create(127 file=open("batchinput.jsonl", "rb"), purpose="batch"128)129130print(batch_input_file)131```132133```go134package main135136import (137 "context"138 "fmt"139 "os"140141 "github.com/openai/openai-go/v3"142)143144func main() {145 client := openai.NewClient()146 file, err := os.Open("batchinput.jsonl")147 if err != nil {148 panic(err)149 }150 defer file.Close()151152 uploaded, err := client.Files.New(context.Background(), openai.FileNewParams{153 File: file,154 Purpose: openai.FilePurposeBatch,155 })156 if err != nil {157 panic(err)158 }159 fmt.Println(uploaded.ID)160}161```162163```java164import com.openai.client.OpenAIClient;165import com.openai.client.okhttp.OpenAIOkHttpClient;166import com.openai.models.files.FileCreateParams;167import com.openai.models.files.FilePurpose;168import java.nio.file.Path;169170var file =171 client172 .files()173 .create(174 FileCreateParams.builder()175 .file(Path.of(System.getenv("OPENAI_EXAMPLE_FILE_PATH")))176 .purpose(FilePurpose.BATCH)177 .build());178179System.out.println(file.id());180```181182```ruby183require "openai"184require "pathname"185186client = OpenAI::Client.new187file = Pathname("batchinput.jsonl")188uploaded = client.files.create(file: file, purpose: :batch)189puts(uploaded.id)190```191192```bash193curl https://api.openai.com/v1/files \194 -H "Authorization: Bearer $OPENAI_API_KEY" \195 -F purpose="batch" \196 -F file="@batchinput.jsonl"197```198199```bash200openai files create \201 --file batchinput.jsonl \202 --purpose batch203```204205206### 3. Create the batch207208Once you've successfully uploaded your input file, you can use the input File object's ID to create a batch. In this case, let's assume the file ID is `file-abc123`. For now, the completion window can only be set to `24h`. You can also provide custom metadata via an optional `metadata` parameter.209210Create the Batch211212```javascript213import OpenAI from "openai";214const openai = new OpenAI();215216const batch = await openai.batches.create({217 input_file_id: "file-abc123",218 endpoint: "/v1/chat/completions",219 completion_window: "24h",220});221222console.log(batch);223```224225```python226batch = client.batches.create(227 input_file_id=batch_input_file.id,228 endpoint="/v1/chat/completions",229 completion_window="24h",230 metadata={"description": "nightly eval job"},231)232print(batch)233```234235```go236package main237238import (239 "context"240 "fmt"241242 "github.com/openai/openai-go/v3"243)244245func main() {246 client := openai.NewClient()247 batch, err := client.Batches.New(context.Background(), openai.BatchNewParams{248 InputFileID: "file-abc123",249 Endpoint: openai.BatchNewParamsEndpointV1ChatCompletions,250 CompletionWindow: openai.BatchNewParamsCompletionWindow24h,251 })252 if err != nil {253 panic(err)254 }255 fmt.Println(batch.ID)256}257```258259```java260import com.openai.client.OpenAIClient;261import com.openai.client.okhttp.OpenAIOkHttpClient;262import com.openai.models.batches.BatchCreateParams;263264String fileId = "file-abc123";265266var batch =267 client268 .batches()269 .create(270 BatchCreateParams.builder()271 .inputFileId(fileId)272 .endpoint(BatchCreateParams.Endpoint.V1_RESPONSES)273 .completionWindow(BatchCreateParams.CompletionWindow._24H)274 .build());275276System.out.println(batch.id());277```278279```ruby280require "openai"281282client = OpenAI::Client.new283batch = client.batches.create(input_file_id: "file-abc123", endpoint: "/v1/responses", completion_window: "24h")284puts(batch.id)285```286287```bash288curl https://api.openai.com/v1/batches \289 -H "Authorization: Bearer $OPENAI_API_KEY" \290 -H "Content-Type: application/json" \291 -d '{292 "input_file_id": "file-abc123",293 "endpoint": "/v1/chat/completions",294 "completion_window": "24h"295 }'296```297298```bash299openai batches create \300 --input-file-id file-abc123 \301 --endpoint /v1/chat/completions \302 --completion-window 24h303```304305306This request will return a [Batch object](https://developers.openai.com/api/reference/resources/batches) with metadata about your batch:307308```json309{310 "id": "batch_abc123",311 "object": "batch",312 "endpoint": "/v1/chat/completions",313 "errors": null,314 "input_file_id": "file-abc123",315 "completion_window": "24h",316 "status": "validating",317 "output_file_id": null,318 "error_file_id": null,319 "created_at": 1714508499,320 "in_progress_at": null,321 "expires_at": 1714536634,322 "completed_at": null,323 "failed_at": null,324 "expired_at": null,325 "request_counts": {326 "total": 0,327 "completed": 0,328 "failed": 0329 },330 "metadata": null331}332```333334### 4. Check the status of a batch335336You can check the status of a batch at any time, which will also return a Batch object.337338Check the status of a batch339340```javascript341import OpenAI from "openai";342const openai = new OpenAI();343344const batch = await openai.batches.retrieve("batch_abc123");345console.log(batch);346```347348```python349batch = client.batches.retrieve(batch.id)350print(batch)351```352353```go354package main355356import (357 "context"358 "fmt"359360 "github.com/openai/openai-go/v3"361)362363func main() {364 client := openai.NewClient()365 batch, err := client.Batches.Get(context.Background(), "batch_abc123")366 if err != nil {367 panic(err)368 }369 fmt.Println(batch.Status)370}371```372373```java374import com.openai.client.OpenAIClient;375import com.openai.client.okhttp.OpenAIOkHttpClient;376377String batchId = "batch_abc123";378379var batch = client.batches().retrieve(batchId);380381System.out.println(batch.status());382```383384```ruby385require "openai"386387client = OpenAI::Client.new388batch = client.batches.retrieve("batch_abc123")389puts(batch.status)390```391392```bash393curl https://api.openai.com/v1/batches/batch_abc123 \394 -H "Authorization: Bearer $OPENAI_API_KEY" \395 -H "Content-Type: application/json"396```397398```bash399openai batches retrieve \400 --batch-id batch_abc123401```402403404The status of a given Batch object can be any of the following:405406| Status | Description |407| ------------- | ------------------------------------------------------------------------------ |408| `validating` | the input file is being validated before the batch can begin |409| `failed` | the input file has failed the validation process |410| `in_progress` | the input file was successfully validated and the batch is currently being run |411| `finalizing` | the batch has completed and the results are being prepared |412| `completed` | the batch has been completed and the results are ready |413| `expired` | the batch was not able to be completed within the 24-hour time window |414| `cancelling` | the batch is being cancelled (may take up to 10 minutes) |415| `cancelled` | the batch was cancelled |416417### 5. Retrieve the results418419Once the batch is complete, you can download the output by making a request against the [Files API](https://developers.openai.com/api/reference/resources/files) via the `output_file_id` field from the Batch object and writing it to a file on your machine, in this case `batch_output.jsonl`420421Retrieving the batch results422423```javascript424import OpenAI from "openai";425const openai = new OpenAI();426427const fileResponse = await openai.files.content("file-xyz123");428const fileContents = await fileResponse.text();429430console.log(fileContents);431```432433```python434# Replace the illustrative IDs and URLs below with your own resource values.435436from openai import OpenAI437438output_file_id = "file_123"439client = OpenAI()440441file_response = client.files.content(output_file_id)442print(file_response.text)443```444445```go446package main447448import (449 "context"450 "fmt"451 "io"452453 "github.com/openai/openai-go/v3"454)455456func main() {457 client := openai.NewClient()458 response, err := client.Files.Content(context.Background(), "file-xyz123")459 if err != nil {460 panic(err)461 }462 defer response.Body.Close()463 contents, err := io.ReadAll(response.Body)464 if err != nil {465 panic(err)466 }467 fmt.Println(string(contents))468}469```470471```java472import com.openai.client.OpenAIClient;473import com.openai.client.okhttp.OpenAIOkHttpClient;474import com.openai.core.http.HttpResponse;475import java.io.IOException;476import java.nio.file.Files;477import java.nio.file.Path;478import java.nio.file.StandardCopyOption;479480String fileId = "file-xyz123";481482try (HttpResponse content = client.files().content(fileId)) {483 Files.copy(484 content.body(), Path.of("batch_output.jsonl"), StandardCopyOption.REPLACE_EXISTING);485}486```487488```ruby489require "openai"490491client = OpenAI::Client.new492content = client.files.content("file-xyz123")493puts(content.read)494```495496```bash497curl https://api.openai.com/v1/files/file-xyz123/content \498 -H "Authorization: Bearer $OPENAI_API_KEY" > batch_output.jsonl499```500501```bash502openai files content \503 --file-id file-xyz123 \504 --output batch_output.jsonl505```506507508The output `.jsonl` file will have one response line for every successful request line in the input file. Any failed requests in the batch will have their error information written to an error file that can be found via the batch's `error_file_id`.509510For `/v1/videos`, a completed batch result contains video objects that have already reached a terminal state such as `completed`, `failed`, or `expired`. You can use the returned video IDs to download final assets immediately after the batch finishes.511512Note that the output line order **may not match** the input line order.513 Instead of relying on order to process your results, use the custom_id field514 which will be present in each line of your output file and allow you to map515 requests in your input to results in your output.516517```jsonl518{"id": "batch_req_123", "custom_id": "request-2", "response": {"status_code": 200, "request_id": "req_123", "body": {"id": "chatcmpl-123", "object": "chat.completion", "created": 1711652795, "model": "gpt-3.5-turbo-0125", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello."}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 2, "total_tokens": 24}, "system_fingerprint": "fp_123"}}, "error": null}519{"id": "batch_req_456", "custom_id": "request-1", "response": {"status_code": 200, "request_id": "req_789", "body": {"id": "chatcmpl-abc", "object": "chat.completion", "created": 1711652789, "model": "gpt-3.5-turbo-0125", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I assist you today?"}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 20, "completion_tokens": 9, "total_tokens": 29}, "system_fingerprint": "fp_3ba"}}, "error": null}520```521522The output file will automatically be deleted 30 days after the batch is complete.523524### 6. Cancel a batch525526If necessary, you can cancel an ongoing batch. The batch's status will change to `cancelling` until in-flight requests are complete (up to 10 minutes), after which the status will change to `cancelled`.527528Cancelling a batch529530```javascript531import OpenAI from "openai";532const openai = new OpenAI();533534const batch = await openai.batches.cancel("batch_abc123");535console.log(batch);536```537538```python539# Replace the illustrative IDs and URLs below with your own resource values.540541from openai import OpenAI542543batch_id = "batch_123"544client = OpenAI()545546client.batches.cancel(batch_id)547```548549```go550package main551552import (553 "context"554 "fmt"555556 "github.com/openai/openai-go/v3"557)558559func main() {560 client := openai.NewClient()561 batch, err := client.Batches.Cancel(context.Background(), "batch_abc123")562 if err != nil {563 panic(err)564 }565 fmt.Println(batch.Status)566}567```568569```java570import com.openai.client.OpenAIClient;571import com.openai.client.okhttp.OpenAIOkHttpClient;572573String batchId = "batch_abc123";574575System.out.println(client.batches().cancel(batchId).status());576```577578```ruby579require "openai"580581client = OpenAI::Client.new582batch = client.batches.cancel("batch_abc123")583puts(batch.status)584```585586```bash587curl https://api.openai.com/v1/batches/batch_abc123/cancel \588 -H "Authorization: Bearer $OPENAI_API_KEY" \589 -H "Content-Type: application/json" \590 -X POST591```592593```bash594openai batches cancel \595 --batch-id batch_abc123596```597598599### 7. Get a list of all batches600601At any time, you can see all your batches. For users with many batches, you can use the `limit` and `after` parameters to paginate your results.602603Getting a list of all batches604605```javascript606import OpenAI from "openai";607const openai = new OpenAI();608609const list = await openai.batches.list();610611for await (const batch of list) {612 console.log(batch);613}614```615616```python617from openai import OpenAI618619client = OpenAI()620621client.batches.list(limit=10)622```623624```go625package main626627import (628 "context"629 "fmt"630631 "github.com/openai/openai-go/v3"632)633634func main() {635 client := openai.NewClient()636 list := client.Batches.ListAutoPaging(context.Background(), openai.BatchListParams{Limit: openai.Int(10)})637 for list.Next() {638 fmt.Println(list.Current().ID)639 }640 if err := list.Err(); err != nil {641 panic(err)642 }643}644```645646```java647import com.openai.client.OpenAIClient;648import com.openai.client.okhttp.OpenAIOkHttpClient;649import com.openai.models.batches.BatchListParams;650651client652 .batches()653 .list(BatchListParams.builder().limit(10).build())654 .autoPager()655 .forEach(batch -> System.out.println(batch.id()));656```657658```ruby659require "openai"660661client = OpenAI::Client.new662client.batches.list(limit: 10).auto_paging_each do |batch|663 puts(batch.id)664end665```666667```bash668curl https://api.openai.com/v1/batches?limit=10 \669 -H "Authorization: Bearer $OPENAI_API_KEY" \670 -H "Content-Type: application/json"671```672673```bash674openai batches list \675 --limit 10676```677678679## Model availability680681The Batch API is widely available across most of our models, but not all. Please refer to the [model reference docs](https://developers.openai.com/api/docs/models) to ensure the model you're using supports the Batch API.682683## Rate limits684685Batch API rate limits are separate from existing per-model rate limits. The Batch API has three types of rate limits:6866871. **Per-batch limits:** A single batch may include up to 50,000 requests, and a batch input file can be up to 200 MB in size. Note that `/v1/embeddings` batches are also restricted to a maximum of 50,000 embedding inputs across all requests in the batch.6882. **Queued prompt tokens per model:** Each model has a maximum number of prompt tokens that can be queued for batch processing. You can find these limits on the [Platform Settings page](https://platform.openai.com/settings/organization/limits).6893. **Batch creation rate limit:** You can create up to 2,000 batches per hour. If you need to submit more requests, increase the number of requests per batch.690691The Batch API currently has no output-token limit. Because Batch API rate limits are a new, separate pool, **using the Batch API will not consume tokens from your standard per-model rate limits**, thereby offering you a convenient way to increase the number of requests and processed tokens you can use when querying our API.692693## Batch expiration694695Batches that do not complete in time eventually move to an `expired` state; unfinished requests within that batch are cancelled, and any responses to completed requests are made available via the batch's output file. You will be charged for tokens consumed from any completed requests.696697Expired requests will be written to your error file with the message as shown below. You can use the `custom_id` to retrieve the request data for expired requests.698699```jsonl700{"id": "batch_req_123", "custom_id": "request-3", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}701{"id": "batch_req_123", "custom_id": "request-7", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}702```