SPB Git

Repository Public

vision-coordinates.md 34.5 KB 681 lines markdown

1---2title: Coordinates and bounding boxes3url: https://platform.claude.com/docs/en/build-with-claude/vision-coordinates4description: How Claude resizes images, and how to work with the pixel coordinates it returns for bounding boxes, points, and UI elements.5---67Claude can locate and label regions of an image (for example, returning bounding boxes for tables, form fields, chart elements, or UI components). This guide covers how Claude resizes images before processing them and how to work with the pixel coordinates it returns, so that boxes and points line up with your original image.89You'll need this for OCR pipelines, form extraction, chart parsing, UI element location, and any task where you act on a specific region of an image. For sending images, supported formats, and per-model resolution limits, see [Vision](https://platform.claude.com/docs/en/build-with-claude/vision).1011<Note>12  **Claude works best with absolute pixel coordinates.** Ask for them explicitly in your prompt. For example: *"Return the bounding box of each table as `[x1, y1, x2, y2]` (top-left and bottom-right corners) in pixel coordinates."* Claude does not work well when you ask for normalized coordinates, for example: *"Return bounding box coordinates between `0` and `1000`."* Always ask for pixel coordinates and normalize in your own code if you need to. To get coordinates as machine-readable JSON instead of prose, define a schema with [structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs), for example an object with an `[x1, y1, x2, y2]` array per detected element.13</Note>1415Coordinates follow the standard image convention: the origin `(0, 0)` is the top-left corner of the image, with x increasing to the right and y increasing downward. The coordinates Claude returns are pixel positions in the image Claude sees: your image after Claude resizes it to fit the model's native resolution (see [How Claude resizes and pads images](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#how-claude-resizes-and-pads-images)). To get coordinates you can use directly, either pre-resize your image so the coordinates map one-to-one onto the image you have (see [Resize your image before uploading](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#resize-your-image-before-uploading)), or rescale the coordinates Claude returns (see [Rescale coordinates when you cannot pre-resize](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#rescale-coordinates-when-you-cannot-pre-resize)).1617<Note>18  Claude's spatial reasoning has limits (see [Limitations](https://platform.claude.com/docs/en/build-with-claude/vision#limitations)). Coordinate accuracy is best when you state the expected coordinate format in your prompt and spot-check results visually before processing at scale. Small elements lose precision when an image is downscaled: for fine targets, crop the region of interest and send the crop (offset returned coordinates by the crop origin), or use a high-resolution-tier model. For [PDF support](https://platform.claude.com/docs/en/build-with-claude/pdf-support), pages are rasterized to images server-side at dimensions you don't control, so the returned coordinates can't be reliably mapped back onto the page. To work with coordinates on PDF content, rasterize the pages to images yourself and use the pre-resize approach.19</Note>2021## How Claude resizes and pads images2223Claude finds the largest aspect-preserving size that satisfies both of the model's image limits:24251. **Edge limit:** neither side exceeds the maximum edge length (1568 px on the standard tier, 2576 px on the high-resolution tier).262. **Visual token limit:** the image's token cost `⌈width / 28⌉ × ⌈height / 28⌉` does not exceed the model's visual token budget (1568 tokens on the standard tier, 4784 on the high-resolution tier).2728See [Resolution and token cost](https://platform.claude.com/docs/en/build-with-claude/vision#evaluate-image-size) for which models are in which tier.2930For nearly all photos and screenshots, the visual token limit is what determines the final size. The edge limit takes over only for elongated images such as panoramas or tall phone screenshots. Compute the size with the [reference implementation](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#resize-your-image-before-uploading) rather than scaling to the edge length by hand: a 1920×1080 screenshot resizes to 1456×819, not 1568×882, and assuming the edge limit puts every coordinate noticeably off target.3132The token limit can also trigger a resize when neither side exceeds the edge limit. Overlooking this is the most common cause of misaligned coordinates. For example, an A4 page scanned at 130 DPI is 1075×1520 pixels: both sides are under 1568 px, but it costs `39 × 55 = 2145` visual tokens, so Claude resizes it to 924×1307.3334<Note>35  This example assumes a model on the standard resolution tier. A high-resolution-tier model doesn't resize the same scan: 2145 tokens is within its 4784-token budget, so the coordinates it returns map directly onto the 1075×1520 original. Model tiers are listed in [Resolution and token cost](https://platform.claude.com/docs/en/build-with-claude/vision#evaluate-image-size).36</Note>3738Claude then pads every image, resized or not, up to the next multiple of 28 pixels on the bottom and right edges (924×1307 becomes 924×1316 in the example). The padding contains no content: Claude perceives the padded image, but the page content only ever occupies the un-padded resized region. **Always normalize or rescale by the resized dimensions, not the padded dimensions**; dividing by the padded dimensions scales every coordinate by a small amount.3940## Resize your image before uploading4142The most reliable approach is to resize your image yourself before uploading, so the image you have is exactly the image Claude sees and the coordinates Claude returns need no conversion.4344First check which resolution tier your model is on (see [Resolution and token cost](https://platform.claude.com/docs/en/build-with-claude/vision#evaluate-image-size)) and pass the matching edge and token limits. The following reference implementation computes the exact size Claude resizes an image to:4546<CodeGroup>47  ```bash cURL48  # This reference implementation is local math that makes no API request, so49  # there's nothing to show for cURL. See the SDK tabs.50  ```5152  ```bash CLI53  # This reference implementation is local math that makes no API request, so54  # there's nothing to show for the CLI. See the SDK tabs.55  ```5657  ```python Python58  import math596061  def count_image_tokens(width: int, height: int) -> int:62      """Visual tokens consumed by an image: one token per 28x28 pixel patch."""63      return math.ceil(width / 28) * math.ceil(height / 28)646566  def resized_size(67      width: int,68      height: int,69      max_edge: int = 1568,70      max_tokens: int = 1568,71  ) -> tuple[int, int]:72      """The size Claude resizes an image to before padding.7374      Defaults are for the standard resolution tier. For high-resolution-tier75      models, use max_edge=2576 and max_tokens=4784. Returns (width, height).76      Images that already fit within the limits are returned unchanged.77      """7879      def fits(w: int, h: int) -> bool:80          return (81              math.ceil(w / 28) * 28 <= max_edge82              and math.ceil(h / 28) * 28 <= max_edge83              and count_image_tokens(w, h) <= max_tokens84          )8586      if fits(width, height):87          return (width, height)88      if height > width:89          resized_h, resized_w = resized_size(height, width, max_edge, max_tokens)90          return (resized_w, resized_h)9192      # Binary search along the long edge for the largest aspect-preserving93      # size that fits.94      aspect_ratio = width / height95      lo, hi = 1, width  # lo always fits; hi never fits96      while lo + 1 < hi:97          mid = (lo + hi) // 298          if fits(mid, max(round(mid / aspect_ratio), 1)):99              lo = mid100          else:101              hi = mid102      return (lo, max(round(lo / aspect_ratio), 1))103104105  # The A4 example from "How Claude resizes and pads images":106  print(resized_size(1075, 1520))  # (924, 1307)107108  # To apply the resize, use your image library, for example Pillow:109  # image.resize(resized_size(*image.size))110  ```111112  ```typescript TypeScript113  /** Visual tokens consumed by an image: one token per 28x28 pixel patch. */114  function countImageTokens(width: number, height: number): number {115    return Math.ceil(width / 28) * Math.ceil(height / 28);116  }117118  /**119   * Round half to even (banker's rounding), matching Python's round(). The120   * live API resolves exact .5 ties toward the even neighbor, so Math.round121   * (which rounds halves up) would compute a different size for some images.122   */123  function roundTiesToEven(value: number): number {124    const floor = Math.floor(value);125    if (value - floor !== 0.5) return Math.round(value);126    return floor % 2 === 0 ? floor : floor + 1;127  }128129  /**130   * The size Claude resizes an image to before padding.131   *132   * Defaults are for the standard resolution tier. For high-resolution-tier133   * models, use maxEdge = 2576 and maxTokens = 4784. Returns [width, height].134   * Images that already fit within the limits are returned unchanged.135   */136  function resizedSize(137    width: number,138    height: number,139    maxEdge = 1568,140    maxTokens = 1568141  ): [number, number] {142    const fits = (w: number, h: number): boolean =>143      Math.ceil(w / 28) * 28 <= maxEdge &&144      Math.ceil(h / 28) * 28 <= maxEdge &&145      countImageTokens(w, h) <= maxTokens;146147    if (fits(width, height)) return [width, height];148    if (height > width) {149      const [resizedH, resizedW] = resizedSize(height, width, maxEdge, maxTokens);150      return [resizedW, resizedH];151    }152153    // Binary search along the long edge for the largest aspect-preserving154    // size that fits.155    const aspectRatio = width / height;156    let lo = 1; // lo always fits157    let hi = width; // hi never fits158    while (lo + 1 < hi) {159      const mid = Math.floor((lo + hi) / 2);160      if (fits(mid, Math.max(roundTiesToEven(mid / aspectRatio), 1))) {161        lo = mid;162      } else {163        hi = mid;164      }165    }166    return [lo, Math.max(roundTiesToEven(lo / aspectRatio), 1)];167  }168169  // The A4 example from "How Claude resizes and pads images":170  console.log(resizedSize(1075, 1520)); // [ 924, 1307 ]171172  // To apply the resize, use your image library, for example sharp:173  // await sharp(input).resize(width, height).toBuffer()174  ```175176  ```csharp C#177  // Visual tokens consumed by an image: one token per 28x28 pixel patch.178  static int CountImageTokens(int width, int height)179  {180      return (width + 27) / 28 * ((height + 27) / 28); // ceil(w/28) * ceil(h/28)181  }182183  // The size Claude resizes an image to before padding. Defaults are for the184  // standard resolution tier; for high-resolution-tier models, pass185  // maxEdge: 2576, maxTokens: 4784. Images that already fit within the limits186  // are returned unchanged.187  static (int Width, int Height) ResizedSize(188      int width, int height, int maxEdge = 1568, int maxTokens = 1568)189  {190      bool Fits(int w, int h) =>191          (w + 27) / 28 * 28 <= maxEdge192          && (h + 27) / 28 * 28 <= maxEdge193          && CountImageTokens(w, h) <= maxTokens;194195      if (Fits(width, height))196      {197          return (width, height);198      }199      if (height > width)200      {201          (int resizedH, int resizedW) = ResizedSize(height, width, maxEdge, maxTokens);202          return (resizedW, resizedH);203      }204205      // Binary search along the long edge for the largest aspect-preserving206      // size that fits. The short edge rounds half to even, matching the live207      // API at exact .5 ties (MidpointRounding.ToEven, Math.Round's default).208      double aspectRatio = (double)width / height;209      int lo = 1; // lo always fits210      int hi = width; // hi never fits211      while (lo + 1 < hi)212      {213          int mid = (lo + hi) / 2;214          if (Fits(mid, ShortEdge(mid)))215          {216              lo = mid;217          }218          else219          {220              hi = mid;221          }222      }223      return (lo, ShortEdge(lo));224225      int ShortEdge(int longEdge) =>226          Math.Max((int)Math.Round(longEdge / aspectRatio, MidpointRounding.ToEven), 1);227  }228229  // The A4 example from "How Claude resizes and pads images":230  Console.WriteLine(ResizedSize(1075, 1520)); // (924, 1307)231  ```232233  ```go Go234  // countImageTokens is the visual tokens consumed by an image: one token per235  // 28x28 pixel patch.236  func countImageTokens(width, height int) int {237  	return ((width + 27) / 28) * ((height + 27) / 28) // ceil(w/28) * ceil(h/28)238  }239240  // resizedSize is the size Claude resizes an image to before padding, as241  // (width, height). Pass maxEdge 1568 and maxTokens 1568 for the standard242  // resolution tier, or 2576 and 4784 for the high-resolution tier. Images243  // that already fit within the limits are returned unchanged.244  // The A4 example from "How Claude resizes and pads images":245  // resizedSize(1075, 1520, 1568, 1568) returns (924, 1307).246  func resizedSize(width, height, maxEdge, maxTokens int) (int, int) {247  	fits := func(w, h int) bool {248  		return ((w+27)/28)*28 <= maxEdge &&249  			((h+27)/28)*28 <= maxEdge &&250  			countImageTokens(w, h) <= maxTokens251  	}252253  	if fits(width, height) {254  		return width, height255  	}256  	if height > width {257  		resizedH, resizedW := resizedSize(height, width, maxEdge, maxTokens)258  		return resizedW, resizedH259  	}260261  	// Binary search along the long edge for the largest aspect-preserving262  	// size that fits. The short edge rounds half to even (math.RoundToEven),263  	// matching the live API at exact .5 ties; math.Round would round them up.264  	aspectRatio := float64(width) / float64(height)265  	lo, hi := 1, width // lo always fits; hi never fits266  	for lo+1 < hi {267  		mid := (lo + hi) / 2268  		short := max(int(math.RoundToEven(float64(mid)/aspectRatio)), 1)269  		if fits(mid, short) {270  			lo = mid271  		} else {272  			hi = mid273  		}274  	}275  	return lo, max(int(math.RoundToEven(float64(lo)/aspectRatio)), 1)276  }277278  ```279280  ```java Java281  /** A resized image size, as returned by resizedSize. */282  record Size(int width, int height) {}283284  /** Visual tokens consumed by an image: one token per 28x28 pixel patch. */285  static int countImageTokens(int width, int height) {286      return Math.ceilDiv(width, 28) * Math.ceilDiv(height, 28);287  }288289  /**290   * The size Claude resizes an image to before padding.291   *292   * <p>Pass maxEdge 1568 and maxTokens 1568 for the standard resolution tier,293   * or 2576 and 4784 for the high-resolution tier. Images that already fit294   * within the limits are returned unchanged.295   *296   * <p>The A4 example from "How Claude resizes and pads images":297   * resizedSize(1075, 1520, 1568, 1568) returns new Size(924, 1307).298   */299  static Size resizedSize(int width, int height, int maxEdge, int maxTokens) {300      if (fits(width, height, maxEdge, maxTokens)) {301          return new Size(width, height);302      }303      if (height > width) {304          Size rotated = resizedSize(height, width, maxEdge, maxTokens);305          return new Size(rotated.height(), rotated.width());306      }307308      // Binary search along the long edge for the largest aspect-preserving309      // size that fits. The short edge rounds half to even (Math.rint),310      // matching the live API at exact .5 ties; Math.round would round them up.311      double aspectRatio = (double) width / height;312      int lo = 1; // lo always fits313      int hi = width; // hi never fits314      while (lo + 1 < hi) {315          int mid = (lo + hi) / 2;316          if (fits(mid, shortEdge(mid, aspectRatio), maxEdge, maxTokens)) {317              lo = mid;318          } else {319              hi = mid;320          }321      }322      return new Size(lo, shortEdge(lo, aspectRatio));323  }324325  private static boolean fits(int width, int height, int maxEdge, int maxTokens) {326      return Math.ceilDiv(width, 28) * 28 <= maxEdge327              && Math.ceilDiv(height, 28) * 28 <= maxEdge328              && countImageTokens(width, height) <= maxTokens;329  }330331  private static int shortEdge(int longEdge, double aspectRatio) {332      return Math.max((int) Math.rint(longEdge / aspectRatio), 1);333  }334  ```335336  ```php PHP337  // Visual tokens consumed by an image: one token per 28x28 pixel patch.338  function countImageTokens(int $width, int $height): int339  {340      return intdiv($width + 27, 28) * intdiv($height + 27, 28);341  }342343  /**344   * The size Claude resizes an image to before padding, as [width, height].345   *346   * Defaults are for the standard resolution tier. For high-resolution-tier347   * models, pass maxEdge: 2576, maxTokens: 4784. Images that already fit348   * within the limits are returned unchanged.349   */350  function resizedSize(int $width, int $height, int $maxEdge = 1568, int $maxTokens = 1568): array351  {352      $fits = fn (int $w, int $h): bool =>353          intdiv($w + 27, 28) * 28 <= $maxEdge354          && intdiv($h + 27, 28) * 28 <= $maxEdge355          && countImageTokens($w, $h) <= $maxTokens;356357      if ($fits($width, $height)) {358          return [$width, $height];359      }360      if ($height > $width) {361          [$resizedH, $resizedW] = resizedSize($height, $width, $maxEdge, $maxTokens);362          return [$resizedW, $resizedH];363      }364365      // Binary search along the long edge for the largest aspect-preserving366      // size that fits. The short edge rounds half to even367      // (PHP_ROUND_HALF_EVEN), matching the live API at exact .5 ties.368      $aspectRatio = $width / $height;369      $lo = 1; // lo always fits370      $hi = $width; // hi never fits371      while ($lo + 1 < $hi) {372          $mid = intdiv($lo + $hi, 2);373          $short = max((int) round($mid / $aspectRatio, 0, PHP_ROUND_HALF_EVEN), 1);374          if ($fits($mid, $short)) {375              $lo = $mid;376          } else {377              $hi = $mid;378          }379      }380381      return [$lo, max((int) round($lo / $aspectRatio, 0, PHP_ROUND_HALF_EVEN), 1)];382  }383384  // The A4 example from "How Claude resizes and pads images":385  [$resizedWidth, $resizedHeight] = resizedSize(1075, 1520);386  echo "({$resizedWidth}, {$resizedHeight})\n"; // (924, 1307)387  ```388389  ```ruby Ruby390  # Visual tokens consumed by an image: one token per 28x28 pixel patch.391  def count_image_tokens(width, height)392    width.ceildiv(28) * height.ceildiv(28)393  end394395  # The size Claude resizes an image to before padding, as [width, height].396  #397  # Defaults are for the standard resolution tier. For high-resolution-tier398  # models, pass max_edge: 2576, max_tokens: 4784. Images that already fit399  # within the limits are returned unchanged.400  def resized_size(width, height, max_edge = 1568, max_tokens = 1568)401    fits = lambda do |w, h|402      w.ceildiv(28) * 28 <= max_edge &&403        h.ceildiv(28) * 28 <= max_edge &&404        count_image_tokens(w, h) <= max_tokens405    end406407    return [width, height] if fits.call(width, height)408409    if height > width410      resized_h, resized_w = resized_size(height, width, max_edge, max_tokens)411      return [resized_w, resized_h]412    end413414    # Binary search along the long edge for the largest aspect-preserving415    # size that fits. The short edge rounds half to even (round(half: :even)),416    # matching the live API at exact .5 ties.417    aspect_ratio = width.fdiv(height)418    lo = 1      # lo always fits419    hi = width  # hi never fits420    while lo + 1 < hi421      mid = (lo + hi) / 2422      short = [(mid / aspect_ratio).round(half: :even), 1].max423      if fits.call(mid, short)424        lo = mid425      else426        hi = mid427      end428    end429430    [lo, [(lo / aspect_ratio).round(half: :even), 1].max]431  end432433  # The A4 example from "How Claude resizes and pads images":434  p resized_size(1075, 1520) # => [924, 1307]435  ```436</CodeGroup>4374381. Resize the image to the dimensions returned by the resize helper. If the image already fits within the model's limits, the helper returns its dimensions unchanged and no resize is needed.4392. [Send the resized image](https://platform.claude.com/docs/en/build-with-claude/vision#send-images-to-claude) to the API. Don't pad it yourself. Claude handles padding, and padding doesn't shift the coordinate origin.4403. In your prompt, ask explicitly for pixel coordinates. For example: *"Return the click point for the Submit button as `[x, y]` in pixel coordinates."*4414. Use the returned coordinates directly against the image you sent. If you need normalized coordinates, divide by the dimensions of the image you sent, not by the original image's dimensions and not by the padded dimensions.442443<Note>444  The [Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting) endpoint estimates an image's token cost from its dimensions without fully processing it, so a successful count doesn't mean the image is within the Messages API's [request limits](https://platform.claude.com/docs/en/build-with-claude/vision#request-limits). An image can count successfully and still be rejected when you send it.445</Note>446447## Turn resizing into an error with `transformations`448449Pre-resizing only protects your coordinates while your pipeline keeps producing the right sizes. A new image source or a switch to a model on a different resolution tier can quietly reintroduce server-side resizing. To turn that silent drift into a visible error, set the optional `transformations` field on an image content block in a [Messages](https://platform.claude.com/docs/en/api/messages) request:450451```json452{453  "type": "image",454  "source": { "type": "base64", "media_type": "image/png", "data": "..." },455  "transformations": { "oversized_image": "error" }456}457```458459A request whose marked image (any block that sets `"oversized_image": "error"`) would be resized is rejected with a 400 `invalid_request_error` naming the image's dimensions and the largest dimensions that fit. Whether an image triggers the rejection depends on the [limits of every model the request names](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#how-claude-resizes-and-pads-images): the 1920×1080 example below is rejected by a standard-tier model but fits within the high-resolution tier:460461```text wrap462messages.0.content.0: image dimensions 1920x1080 exceed the maximum image size of a model named on this request and would be downsized to 1456x819; scale the image to at most 1456x819 or set the image's oversized_image setting to "downsize"463```464465Rescale to the reported target and resend: the target is the largest size, at your image's aspect ratio, that every model named on the request accepts. How marked images interact with the [server-side fallback beta](https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback#server-side-fallback) is described with that feature; in every mode, a marked image is never served resized.466467The setting is per image. `"oversized_image": "downsize"` (the default when the field is omitted) keeps automatic resizing as described on this page. Each image block is checked only against its own setting, so one request can mix images whose dimensions are load-bearing (a screenshot you'll click on) with images where resizing is harmless (a logo). What the setting does and does not change:468469* Padding ([which never discards content](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#how-claude-resizes-and-pads-images)), format conversion, and orientation correction proceed as usual.470* The hard limits (8000 px on the longest side, and the stricter per-image limit on [many-image requests](https://platform.claude.com/docs/en/build-with-claude/vision#request-limits)) are separate rejections; this setting never lets an image past them.471* Images supplied by URL or file ID are checked once their bytes have been fetched; those rejections carry the same message without the leading position, so they don't identify which image failed; only embedded base64 images are named by position in the error.472* [PDF pages](https://platform.claude.com/docs/en/build-with-claude/pdf-support) are rasterized server-side at dimensions you don't control; the `document` block does not accept the field (an image block nested inside a document's content accepts it like any other).473* A marked image whose dimensions cannot be determined is rejected rather than passed through: that rejection reports that the image's source dimensions could not be determined, not the resize message quoted above. No image that sets `"error"` reaches the model resized.474475The [Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting) endpoint honors `transformations` too, rejecting an embedded image exactly as the Messages API would, so you can check whether an embedded image fits without being resized, before running inference. Counting rejects images supplied by URL or file ID rather than fetching them, so a marked image from those sources is checked only at Messages time.476477## Rescale coordinates when you cannot pre-resize478479If you cannot pre-resize (for example, when the image comes from an upstream system you can't modify), use the resize helper from [Resize your image before uploading](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#resize-your-image-before-uploading) to recover the dimensions Claude saw, then map the coordinates Claude returns into normalized coordinates or back onto your original image. Unless an image [opts into an error instead](https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#oversized-image-error), Claude resizes oversized images rather than rejecting them, up to the API's [request limits](https://platform.claude.com/docs/en/build-with-claude/vision#request-limits). Beyond those limits the request fails with a validation error instead. Pass the tier limits that match the model you called: the wrong tier's limits recover the wrong resized dimensions and silently shift every coordinate. This approach requires knowing the pixel dimensions of the image you uploaded, so it does not apply to PDF uploads.480481Screenshots and zoom images that you return to the [computer use](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool#handle-coordinate-scaling-for-higher-resolutions) and [browser use](https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool#targets-and-coordinates) toolsets are an exception to automatic resizing. The API rejects a `tool_result` image that exceeds the model's limits with a validation error instead of resizing it. Resize those images in your application before returning them, then scale the coordinates Claude returns back to your screen's dimensions.482483<CodeGroup>484  ```bash cURL485  # This local coordinate conversion makes no API request, so there's nothing486  # to show for cURL. See the SDK tabs.487  ```488489  ```bash CLI490  # This local coordinate conversion makes no API request, so there's nothing491  # to show for the CLI. See the SDK tabs.492  ```493494  ```python Python495  # This helper calls resized_size from the resize example on this page.496  def to_relative_coordinates(497      x: float,498      y: float,499      original_width: int,500      original_height: int,501      max_edge: int = 1568,502      max_tokens: int = 1568,503  ) -> tuple[float, float]:504      """Map a pixel coordinate returned by Claude to relative coordinates in [0, 1].505506      Pass the dimensions of the image you uploaded. For high-resolution-tier507      models, use max_edge=2576 and max_tokens=4784.508      """509      resized_w, resized_h = resized_size(510          original_width, original_height, max_edge, max_tokens511      )512      return (x / resized_w, y / resized_h)513514515  # A table corner Claude returns at (462, 653.5) on the resized A4 page maps516  # back onto the 1075x1520 original like this:517  rel_x, rel_y = to_relative_coordinates(462, 653.5, 1075, 1520)518  print((rel_x * 1075, rel_y * 1520))  # (537.5, 760.0)519  ```520521  ```typescript TypeScript522  // This helper calls resizedSize from the resize example on this page.523  /**524   * Map a pixel coordinate returned by Claude to relative coordinates in [0, 1].525   *526   * Pass the dimensions of the image you uploaded. For high-resolution-tier527   * models, use maxEdge = 2576 and maxTokens = 4784.528   */529  function toRelativeCoordinates(530    x: number,531    y: number,532    originalWidth: number,533    originalHeight: number,534    maxEdge = 1568,535    maxTokens = 1568536  ): [number, number] {537    const [resizedW, resizedH] = resizedSize(538      originalWidth,539      originalHeight,540      maxEdge,541      maxTokens542    );543    return [x / resizedW, y / resizedH];544  }545546  // A table corner Claude returns at (462, 653.5) on the resized A4 page maps547  // back onto the 1075x1520 original like this:548  const [relX, relY] = toRelativeCoordinates(462, 653.5, 1075, 1520);549  console.log([relX * 1075, relY * 1520]); // [ 537.5, 760 ]550  ```551552  ```csharp C#553  // This helper calls ResizedSize from the resize example on this page.554  // Map a pixel coordinate returned by Claude to relative coordinates in555  // [0, 1]. Pass the dimensions of the image you uploaded, and the same tier556  // limits used for ResizedSize.557  static (double X, double Y) ToRelativeCoordinates(558      double x, double y, int originalWidth, int originalHeight,559      int maxEdge = 1568, int maxTokens = 1568)560  {561      (int resizedW, int resizedH) =562          ResizedSize(originalWidth, originalHeight, maxEdge, maxTokens);563      return (x / resizedW, y / resizedH);564  }565566  // A table corner Claude returns at (462, 653.5) on the resized A4 page maps567  // back onto the 1075x1520 original like this:568  (double relX, double relY) = ToRelativeCoordinates(462, 653.5, 1075, 1520);569  Console.WriteLine((relX * 1075, relY * 1520)); // (537.5, 760)570  ```571572  ```go Go573  // This helper calls resizedSize from the resize example on this page.574575  // toRelativeCoordinates maps a pixel coordinate returned by Claude to576  // relative coordinates in [0, 1]. Pass the dimensions of the image you577  // uploaded, and the same tier limits used for resizedSize.578  func toRelativeCoordinates(579  	x, y float64,580  	originalWidth, originalHeight, maxEdge, maxTokens int,581  ) (float64, float64) {582  	resizedW, resizedH := resizedSize(originalWidth, originalHeight, maxEdge, maxTokens)583  	return x / float64(resizedW), y / float64(resizedH)584  }585586  // To map back to your original image's pixel space, multiply by the original587  // dimensions: a table corner returned at (462, 653.5) on the resized A4 page588  // is (relX*1075, relY*1520) = (537.5, 760) on the 1075x1520 original.589  ```590591  ```java Java592  // This helper calls resizedSize from the resize example on this page.593  /** A coordinate scaled into the [0, 1] range on both axes. */594  record RelativeCoordinate(double x, double y) {}595596  /**597   * Map a pixel coordinate returned by Claude to relative coordinates in598   * [0, 1]. Pass the dimensions of the image you uploaded, and the same tier599   * limits used for resizedSize.600   */601  static RelativeCoordinate toRelativeCoordinates(602          double x, double y, int originalWidth, int originalHeight, int maxEdge, int maxTokens) {603      Size resized = resizedSize(originalWidth, originalHeight, maxEdge, maxTokens);604      return new RelativeCoordinate(x / resized.width(), y / resized.height());605  }606607  // To map back to your original image's pixel space, multiply by the original608  // dimensions: a table corner returned at (462, 653.5) on the resized A4 page609  // is (relative.x() * 1075, relative.y() * 1520) = (537.5, 760) on the610  // 1075x1520 original.611  ```612613  ```php PHP614  // This helper calls resizedSize() from the resize example on this page.615  /**616   * Map a pixel coordinate returned by Claude to relative coordinates in617   * [0, 1], as [x, y]. Pass the dimensions of the image you uploaded, and the618   * same tier limits used for resizedSize.619   */620  function toRelativeCoordinates(621      float $x,622      float $y,623      int $originalWidth,624      int $originalHeight,625      int $maxEdge = 1568,626      int $maxTokens = 1568,627  ): array {628      [$resizedW, $resizedH] = resizedSize($originalWidth, $originalHeight, $maxEdge, $maxTokens);629630      return [$x / $resizedW, $y / $resizedH];631  }632633  // A table corner Claude returns at (462, 653.5) on the resized A4 page maps634  // back onto the 1075x1520 original like this:635  [$relX, $relY] = toRelativeCoordinates(462, 653.5, 1075, 1520);636  echo '(' . $relX * 1075 . ', ' . $relY * 1520 . ")\n"; // (537.5, 760)637  ```638639  ```ruby Ruby640  # This helper calls resized_size from the resize example on this page.641  # Map a pixel coordinate returned by Claude to relative coordinates in642  # [0, 1], as [x, y]. Pass the dimensions of the image you uploaded, and the643  # same tier limits used for resized_size.644  def to_relative_coordinates(645    x, y, original_width, original_height, max_edge = 1568, max_tokens = 1568646  )647    resized_w, resized_h = resized_size(original_width, original_height, max_edge, max_tokens)648    [x.fdiv(resized_w), y.fdiv(resized_h)]649  end650651  # A table corner Claude returns at (462, 653.5) on the resized A4 page maps652  # back onto the 1075x1520 original like this:653  rel_x, rel_y = to_relative_coordinates(462, 653.5, 1075, 1520)654  p [rel_x * 1075, rel_y * 1520] # => [537.5, 760.0]655  ```656</CodeGroup>657658Padding is applied only to the bottom and right edges, so the origin doesn't shift and a per-axis linear rescale is sufficient. Clamp returned coordinates to the resized dimensions before rescaling, so a point slightly outside the image can't map outside your original.659660The relative coordinates multiply against whatever surface you act on: the original image, a full-resolution scan, or a screen. When you act on a screen and screenshot pixels differ from logical coordinates (HiDPI displays), also divide by the display scale factor. The [Computer use tool's scaling guidance](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool#handle-coordinate-scaling-for-higher-resolutions) covers that pattern.661662## Next steps663664<CardGroup cols={2}>665  <Card title="Agent Skills" icon="stack" href="https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview">666    Agent Skills are modular capabilities that extend Claude's functionality. Each Skill packages instructions, metadata, and optional resources (scripts, templates) that Claude uses automatically when relevant.667  </Card>668669  <Card title="Computer use tool" icon="computer" href="https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool">670    Give Claude screenshot, mouse, and keyboard control of a desktop environment with the computer use tool.671  </Card>672673  <Card title="PDF support" icon="file" href="https://platform.claude.com/docs/en/build-with-claude/pdf-support">674    Process PDFs with Claude. Extract text, analyze charts, and understand visual content from your documents.675  </Card>676677  <Card title="Token counting" icon="calculator" href="https://platform.claude.com/docs/en/build-with-claude/token-counting">678    Count the tokens in a message before you send it to Claude. Use token counts to manage rate limits and costs, make model routing decisions, and fit prompts to a target length.679  </Card>680</CardGroup>681
Files main
Menu

Theme