SPB Git forge

spb/rareindex

Public
54commits 1branches 0releases
7.1 MBsize
maindefault branch
10 days agolast push
TypeScript 61.9% HTML 37.2% SQL 0.7%
5.8 KB · 119 lines json
Raw Blame History
1{2  "id": "martian-toys",3  "displayName": "Martian Toys (US collector-toy store, USD)",4  "sourceId": "martian-toys",5  "sourceName": "Martian Toys",6  "sourceType": "dealer",7  "sourceUrl": "https://martiantoys.com",8  "module": "api/martian-toys",9  "enginePriority": [10    "api"11  ],12  "categories": [13    "designer_toys"14  ],15  "regions": [16    "US"17  ],18  "languages": [19    "en"20  ],21  "currency": [22    "USD"23  ],24  "supportsListings": true,25  "supportsSold": false,26  "supportsAuctions": false,27  "supportsImages": true,28  "supportsCatalog": false,29  "supportsPopulation": false,30  "supportsLookup": true,31  "refreshFrequencyMinutes": 720,32  "priority": "medium",33  "trustScore": 0.75,34  "attributionRequired": true,35  "termsUrl": "https://martiantoys.com/policies/terms-of-service",36  "accessNotes": "Martian Toys (martiantoys.com) runs on Shopify. The connector reads ONLY the public, unauthenticated storefront JSON that every Shopify shop serves to any visitor: /collections/<handle>/products.json?limit=250&page=N for the 13 configured collections (designer-toys, designer-vinyl, blind-boxes, customs, urban-toys, martian-exclusives, all-martian-toys, pop-mart … (+5 more, see config.collections)) and /products/<handle>.json for URL lookups (~7k products; 720 designer toys, 350 customs, 150 blind boxes, Pop Mart/Superplastic/tokidoki). robots.txt (verified 2026-09-08, User-agent *): standard Shopify rules — Disallow /admin, /cart, /checkout(s), /orders, /account, /services, /sf_*, /cart.js, /recommendations/products, /cdn/wpm/*.js and the filter/sort crawl traps (/collections/*sort_by*, /collections/*+*, /collections/*filter*&*filter*); no Crawl-delay for *. The /collections/<handle>/products.json and /products/<handle>.json paths we read carry none of those patterns and are allowed. Sitemap declared at /sitemap.xml (not needed). Access behaviour: plain HTTP 200 JSON with the honest RareIndexBot UA, no anti-bot challenge, no login, no key. Market pinning: Shopify Markets localises products.json prices by the requester IP as soon as an Accept-Language header is present (observed on Markets-enabled shops: a Canadian crawler receives CAD-converted prices with content-language en-CA while the JSON carries no currency field). Every request therefore sends the public localization=US cookie — the shop's own home market, exactly what a US visitor gets — so prices are always the declared USD (verified 2026-09-08: 30.00 vs 43.00 on a Markets shop). Politeness: 1 request per 1.5 s, concurrency 1 per host (connectors/domains.d/g3-shops-na.json), crawlDepth pages per collection per incremental run, backfill bounded by backfillMaxPages. Data: asking prices in USD (the shop's declared currency) — these are LISTINGS, never sales (§111); one listing per variant (condition/size/edition), variant SKU kept as identifier, out-of-stock variants kept as 'ended' listings, 0.00-priced placeholders dropped. Dates: published_at only; no fetch time is ever used as a sale date. Deliberately NOT fetched: cart/checkout/account/customer endpoints, per-product .js barcode enrichment (fetchBarcodes=false: one extra request per product is not worth it for listings), the whole-shop /products.json, and unmapped collections — merch/shirts/prints/stickers/pins, event umbrella collections (NYCC, DCon…), \"the vault\"/\"all\" umbrella collections. No personal data is collected; seller = the store itself.",37  "enabled": true,38  "schemaVersion": "1.0",39  "acquisitionMethod": "Shopify storefront products.json",40  "historicalDepth": "none",41  "requires": [],42  "config": {43    "currency": "USD",44    "market": "US",45    "seller": "Martian Toys",46    "location": "Philadelphia, PA, USA",47    "collections": [48      {49        "handle": "designer-toys",50        "categorySlug": "designer_toys"51      },52      {53        "handle": "designer-vinyl",54        "categorySlug": "designer_toys"55      },56      {57        "handle": "blind-boxes",58        "categorySlug": "designer_toys"59      },60      {61        "handle": "customs",62        "categorySlug": "designer_toys",63        "series": "Custom"64      },65      {66        "handle": "urban-toys",67        "categorySlug": "designer_toys"68      },69      {70        "handle": "martian-exclusives",71        "categorySlug": "designer_toys",72        "brand": "Martian Toys"73      },74      {75        "handle": "all-martian-toys",76        "categorySlug": "designer_toys",77        "brand": "Martian Toys"78      },79      {80        "handle": "pop-mart",81        "categorySlug": "designer_toys",82        "brand": "Pop Mart"83      },84      {85        "handle": "super-plastic",86        "categorySlug": "designer_toys",87        "brand": "Superplastic"88      },89      {90        "handle": "tokidoki",91        "categorySlug": "designer_toys",92        "brand": "tokidoki"93      },94      {95        "handle": "kozik",96        "categorySlug": "designer_toys",97        "brand": "Frank Kozik"98      },99      {100        "handle": "3d-retro",101        "categorySlug": "designer_toys",102        "brand": "3DRetro"103      },104      {105        "handle": "madl",106        "categorySlug": "designer_toys",107        "series": "MAD*L"108      }109    ],110    "rules": [],111    "defaultCategory": null,112    "exclude": "gift card|mystery|\\| (Apparel|T-Shirts?|Hats?|Accessories|Pins?|Enamel Pins?|Patches?|Socks|Hoodies?|Bags?|Stationery|Stickers?|Prints?|Posters?|Merch|Keychains?|Lanyards?|Pins? Lanyard Set|Books?|Toy Displays?|Home Goods|Drinkware) \\||t-shirt|\\btee\\b|hoodie|sweatshirt|\\bhat\\b|\\bcap\\b|beanie|\\bsocks?\\b|enamel pin|\\bpins? set\\b|\\bpatch\\b|sticker|\\bkeychain\\b|lanyard|\\bmug\\b|\\btote\\b|tumbler|display case|acrylic case|pop protector only|art print|\\bposter\\b|\\bzine\\b|\\bbook\\b|\\bgame\\b|puzzle|blanket|towel|\\bwallet\\b|backpack|\\bshirt\\b",113    "keepOutOfStock": true,114    "fetchBarcodes": false,115    "wholeShop": false,116    "pageSize": 250117  }118}119