1# Writing a connector23A connector is a Python class under `src/aiatlas/connectors/<group>/<name>.py`, listed in that module's `CONNECTORS = [Cls]`.4The registry auto-discovers it; `aia seed` registers it in the database; `aia run <name>` runs it; the scheduler runs it on its interval.56## Skeleton78```python9from aiatlas.registry import org_ref, provider_ref10from aiatlas.sdk.connector import BaseConnector, Parsed, RunContext11from aiatlas.sdk.facts import Facts, Target12from aiatlas.sdk.fetch import FetchResult1314class ExampleConnector(BaseConnector):15 name = "example" # unique, snake_case, stable (used in documents/claims/events)16 label = "Example Lab — models & news"17 description = "Official docs and blog of Example Lab."18 source_key = "example.com" # must exist in registry/sources.yaml (tier, rate limit, org)19 version = "1"; parser_version = "1" # bump parser_version when extraction improves → `aia reprocess example`20 interval_seconds = 3600; min_interval_seconds = 1800; max_interval_seconds = 8640021 rate_per_min = 15; tier = 1; priority = 0 | 1 | 222 expected_min_records = 5 # breakage detection: fewer entities/prices/results ⇒ run "suspect", nothing deleted23 concurrency = 22425 async def discover(self, ctx: RunContext) -> list[Target]:26 return [Target(url="https://example.com/models", doc_type="model_docs", key="models", min_bytes=2000),27 Target(url="https://example.com/blog/rss.xml", doc_type="feed", key="feed")]2829 async def extract(self, ctx: RunContext, target: Target, res: FetchResult, parsed: Parsed) -> Facts:30 facts = Facts()31 org = org_ref("example") # registry organization → deterministic identifiers32 facts.entities.append(org)33 if target.key == "models" and parsed.html:34 for row in parsed.html.tables[0]["rows"]: ...35 model = facts.entity("model", name, identifiers={"example_model_id": api_id}, organization=org)36 facts.relate(org, "develops", model)37 facts.claim(model, "context_length", 128000, unit="tokens")38 facts.price(model=model, provider=provider_ref("example"), input_per_mtok=1.0, output_per_mtok=4.0)39 facts.follow(detail_url, doc_type="model_page", entity=model, needs_llm=True, meta={"llm_task": "model_passport"})40 elif target.key == "feed" and parsed.kind == "feed":41 announcement_events(facts, org, parsed.feed_items, source_name="example.com/blog") # from connectors/labs/_common.py42 return facts4344CONNECTORS = [ExampleConnector]45```4647## What `Parsed` gives you4849| `parsed.kind` | fields |50|---|---|51| `html` | `parsed.html: HtmlDoc` — `title, canonical, description, meta, og, json_ld, embedded_json (__NEXT_DATA__, data-props…), headings, tables [{headers, rows}], links [(href, text)], text`, `css()`, `links_matching()` ; `find_in_json(obj, key)` helper |52| `markdown` | `parsed.markdown: MarkdownDoc` — `front_matter, headings, tables, links, text, section(pattern)` |53| `feed` | `parsed.feed_items: list[FeedItem]` (`url, title, summary, published_at, categories, authors`) |54| `json` | `parsed.json` |55| `pdf` / `text` / `xml` | `parsed.text` |5657Helpers: `aiatlas.sdk.extract.numbers` (`parse_param_count("70B")`, `parse_active_params("235B-A22B")`, `parse_context_length("128K")`,58`parse_money_per_mtok("$3 / 1M tokens")`, `parse_percent`), `aiatlas.sdk.extract.dates` (`parse_datetime`, `parse_date` with precision),59`connectors/labs/_common.py` (`announcement_events`, `transpose_feature_table`, `kv_tables`, `clean_cell`, `money`, `tokens`, `month_year`, `parse_retirement`, `model_ref`).6061## Rules62631. **Direct mode first.** Use official pages, feeds, sitemaps, Markdown/raw files, embedded JSON. Public JSON files (e.g. `openrouter.ai/api/v1/models`,64 `pypi.org/pypi/<pkg>/json`, arXiv Atom) are fine — they are public documents, not commercial APIs. Never require an API key.65 Set `Target(escalate=True)` only for pages known to block bots *and* only if a key is configured; otherwise let the document be `blocked`.662. **Never fabricate.** Only emit a claim when the page states it. Unknown → no claim. Don't infer parameter counts from names unless the67 name literally contains them (`Qwen3-235B-A22B` → 235e9 / 22e9 is fine; "Large" is not).683. **Identifiers make resolution deterministic.** Give models the provider's API id (`{provider}_model_id`), HF repo (`hf_repo`), arXiv id69 (`arxiv`), GitHub repo (`github_repo`), PyPI name (`pypi`). Organizations always come from `org_ref(<registry key>)` (add missing orgs to70 `registry/organizations.yaml` with a `source_url`).714. **Properties are shared vocabulary** (see below). Add new ones sparingly; prefix metrics with `metric.` (downloads, likes, stars) so they72 never generate events.735. **Events**: NEW_*, PRICE_CHANGED, CONTEXT_CHANGED… are emitted automatically by the writer. Emit `ANNOUNCEMENT`/`RELEASE` events yourself for74 feed items (`announcement_events`) with `effective_at` = publication date and a `dedupe_key` (URL).756. **Follow-ups** (`facts.follow`) let a listing discover detail pages; keep `max_targets` reasonable (`ctx.max_targets`, default 2000).767. **Every connector has a fixture test**: save real responses under `tests/fixtures/<connector>/…` and assert extracted facts77 (see `tests/test_anthropic.py`). Run connectors with `--file key=path` to use fixtures instead of the network.788. Respect `rate_per_min` from `registry/sources.yaml`; arXiv ≤ 4/min; Hugging Face ≤ 30/min; GitHub ≤ 20/min.7980## Property vocabulary (entities.attributes)8182**model**: `family, version, release_date (ISO, may be YYYY-MM), status (active|preview|deprecated|retired|announced|limited-availability), openness83(open-weights|open-source|proprietary|restricted), license, architecture, parameter_count (int), active_parameter_count, is_moe, modalities84(list of text|image|audio|video|code|embedding), modalities_input, modalities_output, context_length (tokens), max_output_tokens, knowledge_cutoff85(YYYY-MM), training_data_cutoff, languages, tool_calling, structured_output, reasoning, vision, audio, fine_tuning_available, tokenizer,86api_model_id, api_alias, official_url, model_card_url, paper_url, repository_url, hf_repo, pipeline_tag, base_model, quantization, quant_format87(gguf|mlx|awq|gptq|fp8), file_size_gb, deprecation_date, retirement_date, retirement_tentative, metric.downloads, metric.likes`8889**company / organization / lab**: `country (ISO-2), headquarters, founded, website, domains, hf_org, github_org, org_kind, legal_name, founders, leadership, employee_count`9091**provider**: `website, pricing_url, docs_url, regions, features` · **paper**: `authors, published_at, updated_at, abstract, arxiv_id, doi, categories, primary_category, pdf_url, code_url, venue`92· **benchmark**: `category, task, metric, unit, creator, website, paper, known_limitations, methodology` · **hardware**: `kind, architecture, release_date, memory_gb,93memory_type, memory_bandwidth_gbs, tdp_watts, runtimes, manufacturer, spec_url, price_usd, compute_fp16_tflops` · **framework / repository**: `repository_url, latest_version,94latest_release_at, license, language, description, topics, pypi, metric.stars, metric.forks` · **dataset**: `license, modality, size, publisher, task, hf_repo`9596Relations: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, distilled_from,97merged_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates, published_by`.98