spb/forma-ka Public
Python 65.1%
TypeScript 17.9%
CSS 16.4%
HTML 0.5%
1<div align="center">23# Forma·Ka45### Every training program in Québec. One place.67**[www.forma-ka.com](https://www.forma-ka.com)**89101112131415161718192021*Independent aggregator of training programs — online courses, university and22college courses, seminars, workshops, certifications and bootcamps — every23program with its full standardized details and a direct link to the original24page. Always up to date, automatically.*2526<br/>2728**Author : Simon-Pierre Boucher — [contact@spboucher.ai](mailto:contact@spboucher.ai)**2930</div>3132---3334## Screenshots3536| Home — search, filters, live ticker | Program page — details first |37|---|---|38|  |  |3940<details>41<summary><b>Stats page — live portrait of Québec's training offer</b></summary>42434445</details>4647## Why Forma-Ka?4849Looking for training in Québec means opening dozens of websites — universities,50CEGEPs, training firms, event organizers — each with its own navigation and51format. **Forma-Ka flips the problem**: a dedicated connector per institution52visits each site, normalizes every program into a single schema, and detects53changes continuously.5455> Training sites don't offer webhooks. Forma-Ka reproduces the equivalent:56> **periodic sync + content hashing** → additions, updates and removals detected57> automatically. A program that disappears from the source site disappears from58> Forma-Ka (after a 2-sync grace period).5960**Philosophy**: unlike a product aggregator, **price is optional** (university61courses don't display one) — what matters are the **details** of each program:62full description, learning objectives, course outline, prerequisites, target63audience, duration, credits/CEUs, delivery mode, offered dates.6465## Architecture in 30 seconds6667```mermaid68flowchart LR69 subgraph Sources["18 training institutions"]70 S1["ÉTS Formation · TÉLUQ · ULaval<br/>McGill · HEC · UQAM · Technologia<br/>AFI · Cégep à distance · Les Affaires<br/>… one connector per site"]71 end72 subgraph FormaKa["Forma-Ka"]73 C["Connectors<br/><i>1 adapter / site</i>"] --> N["Normalization<br/><i>single Formation schema</i>"]74 N --> D[("SQLite<br/>hash + diff")]75 D --> A["FastAPI<br/>/api/formations · /api/facets"]76 A --> F["React 18 + Vite<br/>mobile PWA · light theme"]77 end78 W["⏱ Periodic watcher<br/>(PM2)"] -.-> C79 S1 --> C80 F --> U["🔑 Learner"]81```8283| Layer | Role | Files |84|---|---|---|85| **Connectors** | 1 Python module per institution: server-rendered HTML, internal JSON APIs (Shopify products.json, WordPress REST, Gatsby page-data, Destiny One…), schema.org JSON-LD (Course/Event), **Scrapfly** (robust anti-bot) or **Firecrawl** (JS rendering) for hard sites | `formaka/connectors/*.py` |86| **Schema** | Standardized `Formation`: type, category, mode, description, objectives, outline, prerequisites, duration, credits/CEUs, sessions, price *(optional)* | `formaka/schema.py` |87| **Diff engine** | Content-hash upsert — new / changed / gone (grace period), connector drift detection | `formaka/db.py` |88| **API** | Filters: type / category / mode / city / language / level / free / price / source / search, facets, stats, sync trigger | `formaka/web.py` |89| **Frontend** | "Editorial sharp" design: Space Grotesk, offset shadows, amber accent, live ticker, mobile bottom sheet, 12-per-page pagination, installable PWA | `frontend/` |9091## The three fetch backends9293Each connector picks one (or chains them via the automatic `fetch_html()` fallback):94951. **direct requests** — server-rendered sites (fast, free);962. **Scrapfly** (`SCRAPFLY_API_KEY`) — robust backend: anti-bot bypass (`asp`),97 JavaScript rendering (`render_js`), Canadian geolocation;983. **Firecrawl** (`FIRECRAWL_API_KEY`) — fallback JS rendering, html/markdown formats.99100Detail pages are cached in the database (`detail_cache`) with a weekly key:101each page is revisited only when new, changed, or when the ISO week rolls over.102103## Aggregated sources104105| Institution | Offer | Programs |106|---|---|---|107| Université Laval — Distance | University courses (distance/hybrid) | ~1,760 |108| Technologia | Professional training — IT, AI, management | ~590 |109| Université TÉLUQ | University courses, 100 % online | ~510 |110| AFI by Edgenda | Professional training — IT, leadership | ~360 |111| ÉTS Formation | Continuing education + CEUs | ~300 |112| Versalys | Office tools, IT, languages | ~230 |113| McGill School of Continuing Studies | Continuing education (English) | ~200 |114| Isarta Formations | Marketing, communications, HR | ~190 |115| CRHA — Espace Formation | HR training and events | ~175 |116| Événements Les Affaires | Business conferences and webinars | ~135 |117| Cégep à distance | College courses at a distance | ~120 |118| HEC Montréal — École des dirigeant(e)s | Executive seminars and certifications | ~90 |119| ITHQ | Wine/food workshops + hospitality training | ~80 |120| École des entrepreneurs du Québec | Entrepreneur training (mostly free) | ~45 |121| Institut de leadership | Leadership certifications and programs | ~35 |122| AlphaNumérique | Free digital-literacy courses | ~30 |123| Formation continue UQAM | Continuing education + CEUs | ~50 |124| Le Wagon Montréal | Web dev / data / AI bootcamps | 4 |125126## Quick start127128```bash129git clone https://git.spboucher.ai/forma-ka.git && cd forma-ka130131# Backend132python3 -m venv .venv && .venv/bin/pip install -r requirements.txt133134# Frontend135cd frontend && npm install && npm run build && cd ..136137# Scraping backend keys (JavaScript / anti-bot sites)138cat > .env <<EOF139FIRECRAWL_API_KEY=fc-your-key140SCRAPFLY_API_KEY=scp-live-your-key141EOF142143# Ingest, then serve144.venv/bin/python run.py sync # all sources (or: run.py sync ets_formation teluq)145.venv/bin/python run.py serve 8080 # API + frontend -> http://localhost:8080146.venv/bin/python run.py watch 360 # sync loop (default: every 6 h)147```148149## Adding a connector1501511. Create `formaka/connectors/<source_id>.py`: a class inheriting from152 `BaseConnector`, define `source_id` and implement `fetch() -> list[Formation]`.153 The registry is **auto-discovering** — nothing else to edit.1542. Add the matching entry to `data/sources.json`.1553. Test: `.venv/bin/python run.py sync <source_id>`.156157```python158class MySchoolConnector(BaseConnector):159 source_id = "my_school"160161 def fetch(self) -> list[Formation]:162 html = self.fetch_html(LIST_URL) # direct -> Scrapfly -> Firecrawl163 ...164 return [Formation(source=self.source_id, external_id=..., url=...,165 title=..., description=..., objectives=[...], ...)]166```167168## API169170| Endpoint | Description |171|---|---|172| `GET /api/formations` | Filterable list (`training_type`, `category`, `mode`, `city`, `language`, `level`, `source`, `free`, `price_max`, `starts_after`, `q`, `sort`, `limit`, `offset`) |173| `GET /api/formations/{uid}` | Full program page + price history + similar programs |174| `GET /api/facets` | Distinct values for building filters |175| `GET /api/sources` | Institution registry + sync state |176| `GET /api/stats` | Global portrait + sync log |177| `POST /api/sync` | Trigger a background sync |178179## Tests180181```bash182.venv/bin/python -m pytest tests/ -q183```184185---186187<div align="center">188189© 2026 **Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai)190191</div>192