# Writing a connector

A connector is a Python class under `src/aiatlas/connectors/<group>/<name>.py`, listed in that module's `CONNECTORS = [Cls]`.
The registry auto-discovers it; `aia seed` registers it in the database; `aia run <name>` runs it; the scheduler runs it on its interval.

## Skeleton

```python
from aiatlas.registry import org_ref, provider_ref
from aiatlas.sdk.connector import BaseConnector, Parsed, RunContext
from aiatlas.sdk.facts import Facts, Target
from aiatlas.sdk.fetch import FetchResult

class ExampleConnector(BaseConnector):
    name = "example"                       # unique, snake_case, stable (used in documents/claims/events)
    label = "Example Lab — models & news"
    description = "Official docs and blog of Example Lab."
    source_key = "example.com"             # must exist in registry/sources.yaml (tier, rate limit, org)
    version = "1"; parser_version = "1"    # bump parser_version when extraction improves → `aia reprocess example`
    interval_seconds = 3600; min_interval_seconds = 1800; max_interval_seconds = 86400
    rate_per_min = 15; tier = 1; priority = 0 | 1 | 2
    expected_min_records = 5               # breakage detection: fewer entities/prices/results ⇒ run "suspect", nothing deleted
    concurrency = 2

    async def discover(self, ctx: RunContext) -> list[Target]:
        return [Target(url="https://example.com/models", doc_type="model_docs", key="models", min_bytes=2000),
                Target(url="https://example.com/blog/rss.xml", doc_type="feed", key="feed")]

    async def extract(self, ctx: RunContext, target: Target, res: FetchResult, parsed: Parsed) -> Facts:
        facts = Facts()
        org = org_ref("example")            # registry organization → deterministic identifiers
        facts.entities.append(org)
        if target.key == "models" and parsed.html:
            for row in parsed.html.tables[0]["rows"]: ...
                model = model_ref(facts, name, org, api_id=api_id, provider_key="example")   # from connectors/labs/_common.py: identity + family hint
                facts.claim(model, "context_length", 128000, unit="tokens")
                facts.price(model=model, provider=provider_ref("example"), input_per_mtok=1.0, output_per_mtok=4.0)
                facts.follow(detail_url, doc_type="model_page", entity=model, needs_llm=True, meta={"llm_task": "model_passport"})
        elif target.key == "feed" and parsed.kind == "feed":
            announcement_events(facts, org, parsed.feed_items, source_name="example.com/blog")   # from connectors/labs/_common.py
        return facts

CONNECTORS = [ExampleConnector]
```

## What `Parsed` gives you

| `parsed.kind` | fields |
|---|---|
| `html` | `parsed.html: HtmlDoc` — `title, canonical, description, meta, og, json_ld, embedded_json (__NEXT_DATA__, data-props…), headings, tables [{headers, rows}], links [(href, text)], text`, `css()`, `links_matching()` ; `find_in_json(obj, key)` helper |
| `markdown` | `parsed.markdown: MarkdownDoc` — `front_matter, headings, tables, links, text, section(pattern)` |
| `feed` | `parsed.feed_items: list[FeedItem]` (`url, title, summary, published_at, categories, authors`) |
| `json` | `parsed.json` |
| `pdf` / `text` / `xml` | `parsed.text` |

Helpers: `aiatlas.sdk.extract.numbers` (`parse_param_count("70B")`, `parse_active_params("235B-A22B")`, `parse_context_length("128K")`,
`parse_money_per_mtok("$3 / 1M tokens")`, `parse_percent`), `aiatlas.sdk.extract.dates` (`parse_datetime`, `parse_date` with precision),
`connectors/labs/_common.py` (`announcement_events`, `transpose_feature_table`, `kv_tables`, `clean_cell`, `money`, `tokens`, `month_year`, `parse_retirement`,
`model_ref`, `claim_license`, `claim_status`, `claim_modalities`, `claim_api_aliases`, `normalize_capabilities`),
`connectors/_identity.py` (see below).

## Rules

1. **Direct mode first.** Use official pages, feeds, sitemaps, Markdown/raw files, embedded JSON. Public JSON files (e.g. `openrouter.ai/api/v1/models`,
   `pypi.org/pypi/<pkg>/json`, arXiv Atom) are fine — they are public documents, not commercial APIs. Never require an API key.
   Set `Target(escalate=True)` only for pages known to block bots *and* only if a key is configured; otherwise let the document be `blocked`.
2. **Never fabricate.** Only emit a claim when the page states it. Unknown → no claim. Don't infer parameter counts from names unless the
   name literally contains them (`Qwen3-235B-A22B` → 235e9 / 22e9 is fine; "Large" is not). No hard-coded "all models of this lab do X" constants:
   read the comparison table, else omit.
3. **Identifiers make resolution deterministic.** Give models the provider's API id (`{provider}_model_id`), HF repo (`hf_repo`), arXiv id
   (`arxiv`), GitHub repo (`github_repo`), PyPI name (`pypi`). Organizations always come from `org_ref(<registry key>)` (add missing orgs to
   `registry/organizations.yaml` with a `source_url`) — **never guess a developer from a display name**; the `_identity` helper only maps
   first-party prefixes and family words (`claude-` → Anthropic, `gemini-` → Google, `openrouter/x-ai/…` → xAI).
4. **Properties are shared vocabulary** (see below) and **values are canonical**: run licences through `ontology.normalize_license`, statuses through
   `normalize_status`, modalities through `normalize_modalities`, capabilities through `_common.normalize_capabilities`; keep the source label in
   `<property>_raw` when it differed. Prefix metrics with `metric.` (downloads, likes, stars) so they never generate events.
5. **Events**: NEW_*, PRICE_CHANGED, CONTEXT_CHANGED… are emitted automatically by the writer. Emit `ANNOUNCEMENT`/`RELEASE` events yourself for
   feed items (`announcement_events`) with `effective_at` = publication date and a `dedupe_key` (URL).
6. **Follow-ups** (`facts.follow`) let a listing discover detail pages; keep `max_targets` reasonable (`ctx.max_targets`, default 2000).
7. **Every connector has a fixture test**: save real responses under `tests/fixtures/<connector>/…` and assert extracted facts
   (see `tests/test_anthropic.py`). Run connectors with `--file key=path` to use fixtures instead of the network.
8. Respect `rate_per_min` from `registry/sources.yaml`; arXiv ≤ 4/min; Hugging Face ≤ 30/min; GitHub ≤ 20/min.
9. **Results carry provenance**: every `ResultObs` sets `trust_level` (`ontology.benchmarks.trust_level(source_key, config)`), `variant`
   (GPQA Diamond, SWE-bench Verified, a LiveBench category…) and `run_group` (the evaluation run the row belongs to: LiveBench release,
   aider run date, AA index version). Bookkeeping (cost, wall time, command lines, harness versions) stays out of `config`.

## Identity: model families, models, artifacts, configurations

The canonical hierarchy is **model_family → model → artifact → deployment** (`aiatlas.ontology.models`). Connectors describe it with hints on
`EntityRef`; the writer materialises `entities.family_id`, `canonical_id`, `artifact_kind`, `identity_confidence` and the relations
`member_of_family` / `artifact_of`.

| hint | meaning | who sets it |
|---|---|---|
| `family=EntityRef("model_family", "Qwen3.6", organization=org)` | the versioned family the model belongs to (`_identity.family_ref(name, org)` or the lab's own label via `model_ref(family=…)`) | every model ref with a detectable family |
| `canonical=EntityRef("model", …)` | for an **artifact**: the model it packages; for an alias entity: the canonical entity | Hugging Face (quantisations, conversions, mirrors) |
| `artifact_kind` | `checkpoint` \| `quantization` \| `conversion` \| `packaging` | Hugging Face |
| `identity_confidence` | `high` (vendor id / official repo) · `medium` (leaderboard label, analysed base name) · `low` | everyone |

**Entity types**: `model` (a release: Qwen3-8B, Claude Opus 5), `model_family` (Qwen3, Claude), `artifact` (`bartowski/Qwen3.8-27B-GGUF`,
`zai-org/GLM-5-FP8`, `mlx-community/Kimi-K2.5` — never a model of its own; keeps the full repo id as name, `parameter_count` = the packaged size),
`agent` (coding/browser agents from the repositories registry, attribute `agent_kind`), `tool` (applications, MCP servers), `framework`
(libraries, SDKs, inference engines… attribute `kind` canonical), `repository` (model-code drops), `researcher` (**only with an identifier**:
ORCID or OpenReview profile id — never from a bare author name; authors stay a claim on the paper).

**Effort variants are configurations, not models.** `claude-opus-5-xhigh`, "GPT-5.5 (xhigh)", `deepseek-v3-1-reasoning`,
`…-thinking-64k-high-effort`, "Mistral Small 4 (Non-reasoning)" all point at the base model; the setting goes into `ResultObs.config`
(`reasoning_effort`, `reasoning: on|off|adaptive`, `thinking_budget`) and the evaluator's slug into `config.aa_slug` / `livebench_model_id`.
Size tiers and official products are not efforts: `mistral-medium`, `qwen3.7-max`, `sonar-reasoning`, `kimi-k2-thinking`, `grok-4-1-fast` stay models
(`_identity.strip_effort` only strips an ambiguous suffix when the stem still carries a version digit).

### `connectors/_identity.py`

`model_identity(api_id, trusted=False)` analyses an API id from a third-party source (`anthropic/claude-3-7-sonnet-20250219`,
`gemini/gemini-2.5-pro-preview-05-06`, `openrouter/x-ai/grok-4`, `gpt-4o-2024-08-06`, `Qwen/Qwen2.5-Coder-32B-Instruct`,
`fireworks_ai/accounts/fireworks/models/qwq-32b`) and returns the developer organisation key, resolver-friendly aliases (raw id, id without
provider prefix, id without effort suffixes), the effort configuration and — **only when `trusted=True`** (the string really is the vendor's own
API id, e.g. the `--model` argument aider passed) — the vendor identifier (`anthropic_model_id`, `gemini_model_id`, `openai_model_id`,
`xai_model_id`, `deepseek_model_id`, `mistral_model_id`, `cohere_model_id`, `hf_repo`, `openrouter`, `fireworks_model_id`). Free-text tags
(SWE-bench "Model:" tags, LiveBench ids, OpenRouter slugs) never become identifiers: the resolver refuses an alias match when the entity already
carries a *different* value for the same scheme, so a wrong id would split an entity instead of linking it. Rolling ids
(`deepseek/deepseek-chat`, `codestral-latest`) are aliases only even when trusted — they name whichever snapshot served that day.
`model_ref_from_api_id(facts, api_id, name=…, trusted=…)` builds the deduplicated model `EntityRef` (+ family hint) and returns the effort config.
`family_ref`, `split_effort_label`, `strip_effort`, `org_key_for_vendor`, `org_ref_in` are the building blocks.

## Property vocabulary (entities.attributes)

Canonical enums live in `src/aiatlas/ontology/` — connectors write canonical values and keep the source label in `<property>_raw`.

**model**: `family` (label of the `model_family` hint), `version, release_date (ISO, may be YYYY-MM), status` (`ontology.taxonomy.MODEL_STATUSES`:
`announced | preview | active | limited-availability | deprecated | retired | archived | unknown`), `openness` (`ontology.openness.OPENNESS_CATEGORIES`:
`open-source | open-weights | restricted-weights | proprietary | unknown` — **derived** with `derive_openness` from `weights_available` + the licence,
asserted directly only when the source literally states it), `weights_available` (bool), `access` (`gated | open`, Hugging Face gating — access,
not a licence property), `gated_mode`, `license` (`ontology.licenses` key: `Apache-2.0`, `MIT`, `Llama-3.1-Community`, `Mistral-Research`,
`Gemma-Terms`… — SPDX id when one exists) + `license_raw`, `architecture, parameter_count (int), active_parameter_count, is_moe, modalities /
modalities_input / modalities_output` (`ontology.taxonomy.MODALITIES`: `text | image | audio | video | document | code | embedding | 3d | structured | action`,
sorted), `context_length (tokens), max_output_tokens, knowledge_cutoff (YYYY-MM), training_data_cutoff, languages, capabilities` (canonical slugs:
`function_calling, structured_output, reasoning, vision, audio_input, audio_output, image_generation, video_generation, code_execution,
search_grounding, caching, batch, fine_tuning, streaming, live_api, computer_use, file_search, url_context, mcp…` via `_common.normalize_capabilities`)
+ `capabilities_raw`, `tool_calling, structured_output, reasoning, vision, audio, fine_tuning_available, tokenizer, api_model_id, api_aliases` (always a
list; `api_alias` = the first one, kept for compatibility), `official_url, model_card_url, paper_url, repository_url, hf_repo, pipeline_tag, base_model,
quantization, quant_format (gguf|mlx|awq|gptq|fp8|…), artifact_kind, is_quantized, quantized_by, file_size_gb, deprecation_date, retirement_date,
retirement_tentative, openrouter_listed_at` (listing date — never `release_date`), `aa_release_date / aa_openness / aa_context_window` (second-hand
facts from Artificial Analysis), `metric.downloads, metric.likes`

**company / organization / lab**: `country (ISO-2), headquarters, founded, website, domains, hf_org, github_org, org_kind`
(`ontology.taxonomy.ORG_KINDS`: `company | lab | university | nonprofit | government | community | consortium | individual`), `legal_name, founders, leadership, employee_count`

**provider**: `website, pricing_url, docs_url, regions, features` · **paper**: `authors, published_at, updated_at, abstract, arxiv_id, doi, categories, primary_category, pdf_url, code_url, venue`
· **researcher**: `openreview_profile_url` (+ identifiers `orcid` / `openreview_profile`)
· **benchmark** (registry `registry/benchmarks.yaml` + `.d/`, see below): `family, variant, version, family_head, category, task, metric` (canonical:
`accuracy | pass@1 | pass^1 | resolved | pass_rate_2 | percent_cases_well_formed | global_average | average score | mean score | index | elo …`),
`metric_label, metric_raw, unit, metric_min, metric_max, higher_is_better, harness, comparability_note, creator, website, paper, known_limitations, methodology`
· **hardware**: `kind` (`ontology.taxonomy.HARDWARE_KINDS`), `architecture, release_date, memory_gb, memory_type, memory_bandwidth_gbs, tdp_watts, runtimes, manufacturer, spec_url,
price_usd, compute_fp16_tflops` · **framework / tool / agent / repository**: `kind` (`ontology.taxonomy.FRAMEWORK_KINDS`: `training-framework | inference-engine |
serving-engine | library | runtime | agent-framework | orchestration | evaluation-harness | sdk | tool | application | agent | mcp-server | vector-database |
observability | data-tooling`) + `kind_raw`, `agent_kind` (`coding | browser | research`), `repository_url, latest_version, latest_release_at, license (+ license_raw),
language, description, topics, pypi, metric.stars, metric.forks` · **dataset**: `license, modality, size, publisher, task, hf_repo`

Relations: `develops, published_by` (artifact → its publishing org; converters never `develop`), `owns, operates, available_through, evaluated_on,
described_by, derived_from, fine_tuned_from, quantized_from, distilled_from, merged_from, superseded_by, variant_of` (benchmark variant → family head),
`runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates`. The writer adds `member_of_family` and
`artifact_of` from the hints.

## Prices

`PriceObs(model, provider, provider_model_id, …)` — one row per model × provider × provider id. Aggregator prices belong to the aggregator:
OpenRouter rows are booked on `provider_ref("openrouter")` with `features.upstream_provider` = the routed lab's registry provider key (never on
the lab's own provider entity, which would create two "current" prices). Variants (`:free`, `:thinking`, `:nitro`) are separate rows of the same
model (`features.variant`). Identifier schemes for provider catalogues: `openrouter`, `groq_model_id`, `together_ai_model_slug`,
`fireworks_model_id` (the historical `fireworks-ai_model_id` is still emitted alongside).

## Benchmark registry (`registry/benchmarks.yaml` + `registry/benchmarks.d/*.yaml`)

Every entry declares `family`, `variant`, optional `version`, `family_head: true` on the representative member, canonical `metric`
(`ontology.benchmarks.normalize_metric`) with `metric_min` / `metric_max` / `higher_is_better`, `harness` and `comparability_note`, plus the
aliases evaluators use ("GPQA Diamond", "HLE", "SWE-Bench Verified", "τ²-Bench Telecom"…). `aia seed` writes them as claims and a `variant_of`
relation from each member to its family head. One benchmark entity per measured thing: LiveBench categories are `livebench-<category>` entities
(family `livebench`), aider's well-formed rate is `aider-polyglot-well-formed`, Artificial Analysis' GPQA results land on `gpqa-diamond`.
Connectors address benchmarks with `benchmark_ref(key)` (identifier `registry_benchmark`, `slug_hint` = key). Slugs are only hints: when an
organisation already owns the slug (`livebench` the org), the resolver appends a collision suffix (`livebench-2` in production) — identity is
the `registry_benchmark` identifier, and existing production slugs never change.