SPB Git forge

spb/rareindex

Public
54commits 1branches 0releases
7.1 MBsize
maindefault branch
10 days agolast push
TypeScript 61.9% HTML 37.2% SQL 0.7%
4.4 KB · 73 lines json
Raw Blame History
1{2  "id": "waxwork-records",3  "displayName": "Waxwork Records (US vinyl store, USD)",4  "sourceId": "waxwork-records",5  "sourceName": "Waxwork Records",6  "sourceType": "dealer",7  "sourceUrl": "https://waxworkrecords.com",8  "module": "api/waxwork-records",9  "enginePriority": [10    "api"11  ],12  "categories": [13    "music"14  ],15  "regions": [16    "US"17  ],18  "languages": [19    "en"20  ],21  "currency": [22    "USD"23  ],24  "supportsListings": true,25  "supportsSold": false,26  "supportsAuctions": false,27  "supportsImages": true,28  "supportsCatalog": false,29  "supportsPopulation": false,30  "supportsLookup": true,31  "refreshFrequencyMinutes": 1440,32  "priority": "low",33  "trustScore": 0.75,34  "attributionRequired": true,35  "termsUrl": "https://waxworkrecords.com/policies/terms-of-service",36  "accessNotes": "Waxwork Records (waxworkrecords.com) runs on Shopify. The connector reads ONLY the public, unauthenticated storefront JSON that every Shopify shop serves to any visitor: /collections/<handle>/products.json?limit=250&page=N for the 3 configured collections (vinyl, cd, cassettes) and /products/<handle>.json for URL lookups (~250 vinyl soundtracks, 30 CDs, cassettes). robots.txt (verified 2026-09-08, User-agent *): standard Shopify rules — Disallow /admin, /cart, /checkout(s), /orders, /account, /services, /sf_*, /cart.js, /recommendations/products, /cdn/wpm/*.js and the filter/sort crawl traps (/collections/*sort_by*, /collections/*+*, /collections/*filter*&*filter*); no Crawl-delay for *. The /collections/<handle>/products.json and /products/<handle>.json paths we read carry none of those patterns and are allowed. Sitemap declared at /sitemap.xml (not needed). Access behaviour: plain HTTP 200 JSON with the honest RareIndexBot UA, no anti-bot challenge, no login, no key. Market pinning: Shopify Markets localises products.json prices by the requester IP as soon as an Accept-Language header is present (observed on Markets-enabled shops: a Canadian crawler receives CAD-converted prices with content-language en-CA while the JSON carries no currency field). Every request therefore sends the public localization=US cookie — the shop's own home market, exactly what a US visitor gets — so prices are always the declared USD (verified 2026-09-08: 30.00 vs 43.00 on a Markets shop). Politeness: 1 request per 1.5 s, concurrency 1 per host (connectors/domains.d/g3-shops-na.json), crawlDepth pages per collection per incremental run, backfill bounded by backfillMaxPages. Data: asking prices in USD (the shop's declared currency) — these are LISTINGS, never sales (§111); one listing per variant (condition/size/edition), variant SKU kept as identifier, out-of-stock variants kept as 'ended' listings, 0.00-priced placeholders dropped. Dates: published_at only; no fetch time is ever used as a sale date. Deliberately NOT fetched: cart/checkout/account/customer endpoints, per-product .js barcode enrichment (fetchBarcodes=false: one extra request per product is not worth it for listings), the whole-shop /products.json, and unmapped collections — merch/apparel/pins, digital albums, wholesale collections. No personal data is collected; seller = the store itself.",37  "enabled": true,38  "schemaVersion": "1.0",39  "acquisitionMethod": "Shopify storefront products.json",40  "historicalDepth": "none",41  "requires": [],42  "config": {43    "currency": "USD",44    "market": "US",45    "seller": "Waxwork Records",46    "location": "New Orleans, LA, USA",47    "collections": [48      {49        "handle": "vinyl",50        "categorySlug": "music",51        "brand": "Waxwork Records"52      },53      {54        "handle": "cd",55        "categorySlug": "music",56        "brand": "Waxwork Records"57      },58      {59        "handle": "cassettes",60        "categorySlug": "music",61        "brand": "Waxwork Records"62      }63    ],64    "rules": [],65    "defaultCategory": null,66    "exclude": "gift card|mystery|\\| (Apparel|T-Shirts?|Merch|Accessories|Turntables?|Speakers?|Headphones?|Cartridges?|Cables?|Posters?|Digital) \\||t-shirt|hoodie|\\bshirt\\b|\\bhat\\b|\\btote\\b|\\bmug\\b|poster|slipmat|slip mat|cartridge|stylus|turntable|speaker|headphone|\\bcable\\b|cleaner|cleaning|\\bbrush\\b|record weight|inner sleeves?|outer sleeves?|\\bcrate\\b|\\bstand\\b|\\bpin\\b|enamel pin|digital (album|download)|\\bsocks?\\b|\\bpatch\\b|\\bsticker",67    "keepOutOfStock": true,68    "fetchBarcodes": false,69    "wholeShop": false,70    "pageSize": 25071  }72}73