SPB Git forge

spb/rareindex

Public
54commits 1branches 0releases
7.1 MBsize
maindefault branch
10 days agolast push
TypeScript 61.9% HTML 37.2% SQL 0.7%
5.2 KB · 63 lines json
Raw Blame History
1{2  "id": "etsy",3  "displayName": "Etsy Open API v3 (vintage & collectibles listings) — gated",4  "sourceId": "etsy",5  "sourceName": "Etsy",6  "sourceType": "marketplace",7  "sourceUrl": "https://www.etsy.com",8  "module": "api/etsy",9  "enginePriority": ["api"],10  "categories": ["vintage_toys", "cameras", "pens", "clocks", "design_furniture", "antiques", "porcelain", "glass_crystal", "silver", "jewelry", "other_watches", "movie_posters", "music", "books", "postcards", "advertising", "pokemon", "sports_cards", "comics", "typewriters", "scientific_instruments"],11  "regions": ["US", "GB", "CA", "DE", "FR", "AU"],12  "languages": ["en"],13  "currency": ["USD", "GBP", "EUR", "CAD", "AUD"],14  "supportsListings": true,15  "supportsSold": false,16  "supportsAuctions": false,17  "supportsImages": true,18  "supportsCatalog": false,19  "supportsPopulation": false,20  "supportsLookup": true,21  "refreshFrequencyMinutes": 360,22  "priority": "medium",23  "trustScore": 0.7,24  "attributionRequired": true,25  "termsUrl": "https://www.etsy.com/legal/api",26  "acquisitionMethod": "official API (x-api-key)",27  "historicalDepth": "none",28  "requires": ["ETSY_API_KEY"],29  "accessNotes": "Official Etsy Open API v3 (developers.etsy.com/documentation, OpenAPI contract at etsy.com/openapi/generated/oas/3.0.0.json). Auth: the application keystring in the x-api-key header (Etsy's request guide shows the 'keystring:shared_secret' form; either value works in ETSY_API_KEY — we send it verbatim). No OAuth is needed for the application-scoped read endpoints we use: GET /v3/application/listings/active (findAllListingsActive: keywords, taxonomy_id, min_price/max_price, sort_on=created, sort_order=desc, limit ≤ 100, offset), GET /v3/application/listings/batch?listing_ids=…&includes=Images,Shop (one call per page to attach images and the public shop name/country/review score) and GET /v3/application/seller-taxonomy/nodes (taxonomy ids resolved by node-name path and cached in the cursor). Prices are Money {amount, divisor, currency_code} in each shop's own currency → native currency listings; images url_fullxfull; created/ending epoch timestamps → listedAt/endsAt. Etsy publishes no sold prices, so this is a listings-only source; supplies, made-to-order and non-active listings are skipped and 'when_made' eras are kept as metadata. Rate limits (Etsy default): 10 requests/second and 10,000 requests/day per app — a run uses ≤ 8 queries × (1 search + 1 batch) calls every 6 h. Credentials: register at https://www.etsy.com/developers/register, create an app (personal-access apps are approved instantly; commercial access requires Etsy's review under the API Terms of Use https://www.etsy.com/legal/api) and copy the keystring into ETSY_API_KEY. Until then the framework reports the connector DISABLED (requires) and the crawler exits with a missing_credentials anomaly. Fixtures are synthesised from the official OpenAPI schema (ShopListing, Money, ListingImage), not live captures — see the fixture note. Category slugs come from the configured query seeds (keywords + taxonomy path), confidence 0.7 (title-level identification only).",30  "enabled": true,31  "schemaVersion": "1.0",32  "config": {33    "limit": 100,34    "pagesPerQuery": 1,35    "backfillPages": 5,36    "queriesPerRun": 8,37    "fetchImages": true,38    "sortOn": "created",39    "sortOrder": "desc",40    "queries": [41      { "keywords": "vintage pokemon card", "taxonomyPath": ["Toys & Games", "Games & Puzzles", "Card Games"], "categorySlug": "pokemon" },42      { "keywords": "vintage baseball card", "taxonomyPath": ["Art & Collectibles", "Collectibles"], "categorySlug": "sports_cards" },43      { "keywords": "vintage comic book", "taxonomyPath": ["Books, Movies & Music", "Books", "Comics & Graphic Novels"], "categorySlug": "comics" },44      { "keywords": "vintage film camera", "taxonomyPath": ["Electronics & Accessories", "Cameras & Equipment"], "categorySlug": "cameras" },45      { "keywords": "vintage fountain pen", "categorySlug": "pens" },46      { "keywords": "antique mantel clock", "categorySlug": "clocks" },47      { "keywords": "mid century modern chair", "taxonomyPath": ["Home & Living", "Furniture"], "categorySlug": "design_furniture" },48      { "keywords": "antique porcelain vase", "categorySlug": "porcelain" },49      { "keywords": "murano glass vase", "categorySlug": "glass_crystal" },50      { "keywords": "sterling silver antique", "categorySlug": "silver" },51      { "keywords": "vintage wristwatch", "taxonomyPath": ["Accessories", "Watches"], "categorySlug": "other_watches" },52      { "keywords": "original movie poster", "categorySlug": "movie_posters" },53      { "keywords": "vintage vinyl record first pressing", "categorySlug": "music" },54      { "keywords": "first edition book", "taxonomyPath": ["Books, Movies & Music", "Books"], "categorySlug": "books" },55      { "keywords": "antique postcard", "categorySlug": "postcards" },56      { "keywords": "vintage advertising sign", "categorySlug": "advertising" },57      { "keywords": "vintage typewriter", "categorySlug": "typewriters" },58      { "keywords": "antique microscope", "categorySlug": "scientific_instruments" },59      { "keywords": "vintage tin toy", "taxonomyPath": ["Toys & Games", "Toys"], "categorySlug": "vintage_toys" }60    ]61  }62}63