# Screenshots

Home — search, filters, live ticker	Program page — details first

# Why Forma-Ka?

Looking for training in Québec means opening dozens of websites — universities, CEGEPs, training firms, event organizers — each with its own navigation and format. Forma-Ka flips the problem: a dedicated connector per institution visits each site, normalizes every program into a single schema, and detects changes continuously.

Training sites don't offer webhooks. Forma-Ka reproduces the equivalent: periodic sync + content hashing → additions, updates and removals detected automatically. A program that disappears from the source site disappears from Forma-Ka (after a 2-sync grace period).

Philosophy: unlike a product aggregator, price is optional (university courses don't display one) — what matters are the details of each program: full description, learning objectives, course outline, prerequisites, target audience, duration, credits/CEUs, delivery mode, offered dates.

# Architecture in 30 seconds

flowchart LR
    subgraph Sources["18 training institutions"]
        S1["ÉTS Formation · TÉLUQ · ULaval<br/>McGill · HEC · UQAM · Technologia<br/>AFI · Cégep à distance · Les Affaires<br/>… one connector per site"]
    end
    subgraph FormaKa["Forma-Ka"]
        C["Connectors<br/><i>1 adapter / site</i>"] --> N["Normalization<br/><i>single Formation schema</i>"]
        N --> D[("SQLite<br/>hash + diff")]
        D --> A["FastAPI<br/>/api/formations · /api/facets"]
        A --> F["React 18 + Vite<br/>mobile PWA · light theme"]
    end
    W["⏱ Periodic watcher<br/>(PM2)"] -.-> C
    S1 --> C
    F --> U["🔑 Learner"]

# The three fetch backends

Each connector picks one (or chains them via the automatic fetch_html() fallback):

1. direct requests — server-rendered sites (fast, free);
2. Scrapfly (SCRAPFLY_API_KEY) — robust backend: anti-bot bypass (asp), JavaScript rendering (render_js), Canadian geolocation;
3. Firecrawl (FIRECRAWL_API_KEY) — fallback JS rendering, html/markdown formats.

Detail pages are cached in the database (detail_cache) with a weekly key: each page is revisited only when new, changed, or when the ISO week rolls over.

# Aggregated sources

# Quick start

git clone https://git.spboucher.ai/forma-ka.git && cd forma-ka

# Backend
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt

# Frontend
cd frontend && npm install && npm run build && cd ..

# Scraping backend keys (JavaScript / anti-bot sites)
cat > .env <<EOF
FIRECRAWL_API_KEY=fc-your-key
SCRAPFLY_API_KEY=scp-live-your-key
EOF

# Ingest, then serve
.venv/bin/python run.py sync            # all sources (or: run.py sync ets_formation teluq)
.venv/bin/python run.py serve 8080      # API + frontend -> http://localhost:8080
.venv/bin/python run.py watch 360       # sync loop (default: every 6 h)

# Adding a connector

1. Create formaka/connectors/<source_id>.py: a class inheriting from BaseConnector, define source_id and implement fetch() -> list[Formation]. The registry is auto-discovering — nothing else to edit.
2. Add the matching entry to data/sources.json.
3. Test: .venv/bin/python run.py sync <source_id>.

class MySchoolConnector(BaseConnector):
    source_id = "my_school"

    def fetch(self) -> list[Formation]:
        html = self.fetch_html(LIST_URL)          # direct -> Scrapfly -> Firecrawl
        ...
        return [Formation(source=self.source_id, external_id=..., url=...,
                          title=..., description=..., objectives=[...], ...)]

# API

# Tests

.venv/bin/python -m pytest tests/ -q