SPB Git forge

spb/rareindex

Public
54commits 1branches 0releases
7.1 MBsize
maindefault branch
10 days agolast push
TypeScript 61.9% HTML 37.2% SQL 0.7%
2.7 KB · 42 lines json
Raw Blame History
1{2  "id": "mycomicshop",3  "displayName": "MyComicShop (raw & graded comic listings)",4  "sourceId": "mycomicshop",5  "sourceName": "MyComicShop",6  "sourceType": "dealer",7  "sourceUrl": "https://www.mycomicshop.com",8  "module": "scrapfly/mycomicshop",9  "enginePriority": ["scrapfly"],10  "categories": ["comics", "marvel_comics", "dc_comics", "independent_comics", "manga"],11  "regions": ["US"],12  "languages": ["en"],13  "currency": ["USD"],14  "supportsListings": true,15  "supportsSold": false,16  "supportsAuctions": true,17  "supportsImages": true,18  "supportsCatalog": false,19  "supportsPopulation": false,20  "supportsLookup": true,21  "refreshFrequencyMinutes": 1440,22  "priority": "high",23  "trustScore": 0.8,24  "attributionRequired": true,25  "termsUrl": "https://www.mycomicshop.com/help/terms",26  "accessNotes": "Reads only public title pages: https://www.mycomicshop.com/search?TID=<titleId>&mingr=0 (all issues of one series with their in-stock copies on a single page, ~1–3 MB). Each copy is schema.org/Product microdata (name incl. grade, sku = ItemID, canonical url, USD price, availability) followed by bullet notes (paper quality, 'Label #<cert>' for CGC/CBCS slabs, 'CGC Signature Series', 'Consignment. 3% buyer's premium charged at checkout', consignor store). Auction copies on the same page ('Auction Item: CGC 8.5', current bid, bid count, time left / 'Auction opens <date>') become auction_lot records — the /auctions section itself is disallowed by robots.txt and is never fetched, nor are /newlistings, /restocked, /account, /cart, /wantlist (all disallowed). robots.txt allows /search for `User-agent: *`. The site is fronted by Imperva Incapsula, which serves an interstitial to every non-browser client (bot and browser UAs alike), so pages are fetched through Scrapfly without JS rendering (~1 credit per title page, 3 s politeness, concurrency 1, 6 titles per daily run over a configurable list of series ids). Prices: the microdata price of consignment copies already includes the 3% buyer's premium; we store the displayed asking price and keep the premium-inclusive figure in metadata. Asking prices are listings, never sales (SPEC §111). No customer/bidder data exists on these pages.",27  "enabled": true,28  "schemaVersion": "1.0",29  "acquisitionMethod": "Scrapfly (no JS) — public title pages, schema.org microdata",30  "historicalDepth": "none",31  "requires": ["SCRAPFLY_API_KEY"],32  "config": {33    "titlesPerRun": 6,34    "titles": [35      { "tid": "78991", "name": "Amazing Spider-Man (1963 1st Series)" },36      { "tid": "177841", "name": "Amazing Spider-Man (1963 1st Series) Annual" },37      { "tid": "78851", "name": "Alpha Flight (1983 1st Series)" },38      { "tid": "169291", "name": "Amazing Heroes (1981)" }39    ]40  }41}42