SPB Git

spb/qc-election

Public

QC Élection Forecast — plateforme de prévision électorale du Québec 2026

Modèle bayésien d'état latent (Kalman/alr) + house effects + Monte Carlo
corrélé (erreurs systémique/régionale/locale), backtesting 2022 calibré,
carte officielle DGEQ (127 circonscriptions), simulateur what-if, sentiment
médiatique et marchés prédictifs en signaux auxiliaires. FastAPI + SQLite,
frontend ECharts thèmes sombre/clair, responsive mobile. Déployée sur M2U64
→ https://www.qc-election.com (PM2 + ngrok).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Aug 28, 2026) Browse files

59 changed files +7,265 −0

59 changed files
added .env.example +11 −0
@@ -0,0 +1,11 @@
1 +# QC Élection Forecast — configuration d'environnement
2 +# Jeton d'administration (obligatoire en production)
3 +ADMIN_TOKEN=changez-moi
4 +
5 +# Clé du gateway LLM MacLustr (optionnel — scoreur de stance LLM)
6 +# MACLUSTR_API_KEY=
7 +
8 +# Surcharges du modèle (préfixe QCE_)
9 +# QCE_PORT=8140
10 +# QCE_N_SIMULATIONS=25000
11 +# QCE_REFRESH_INTERVAL_HOURS=6
added .gitignore +9 −0
@@ -0,0 +1,9 @@
1 +__pycache__/
2 +*.pyc
3 +.venv/
4 +.env
5 +backend/qc_election.db
6 +backend/data/raw/
7 +.pytest_cache/
8 +.DS_Store
9 +logs/
added METHODOLOGY.md +132 −0
@@ -0,0 +1,132 @@
1 +<!-- QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 -->
2 +<!-- Auteur : Simon-Pierre Boucher -->
3 +<!-- Contact : contact@spboucher.ai -->
4 +<!-- https://www.qc-election.com -->
5 +
6 +# Méthodologie — QC Élection Forecast
7 +
8 +Version du modèle : **1.0.0** · Élection cible : générale québécoise du 5 octobre 2026
9 +(127 circonscriptions, majorité à 64).
10 +
11 +Ce document est la référence auditable du système. La version grand public est servie sur
12 +`/methodologie`; les snapshots de chaque exécution sont conservés en base
13 +(`forecast_runs`) et consultables par l'API.
14 +
15 +## 1. Sources de données et provenance
16 +
17 +| Source | Contenu | Rôle |
18 +|---|---|---|
19 +| DGEQ (données ouvertes officielles) | GeoJSON carte 2026 (127 circ.), liste officielle, candidatures (drapeau « député sortant »), résultats en direct le soir du vote | référence géographique, candidats, incumbency |
20 +| Wikipédia (agrégation, liens vers chaque maison) | sondages 2022→2026 (85+), sondages campagne 2022 (106), résultats 2022 par circonscription, variations 2018→2022 | série de sondages, baselines locales, backtest |
21 +| Polymarket (API Gamma publique) | probabilités implicites « plus de sièges » | **signal auxiliaire affiché, hors modèle** |
22 +| Presse québécoise (RSS publics) | Radio-Canada, Le Devoir, La Presse | **sentiment auxiliaire, hors modèle** |
23 +
24 +Chaque enregistrement conserve : source, URL, date d'accès, valeur brute, valeur normalisée.
25 +Les HTML bruts sont archivés dans `backend/data/raw/`.
26 +
27 +Dédoublonnage des sondages : clé (maison, fin de terrain, taille d'échantillon[, parti]);
28 +un même sondage publié à plusieurs endroits n'est compté qu'une fois.
29 +
30 +## 2. Pondération des sondages
31 +
32 +- **Récence** : poids `0,5^(âge/12 j)`, l'âge étant mesuré à la **fin du terrain**, pas à la
33 + publication.
34 +- **Précision** : taille effective `n_eff = min(n/deff, 2500)` avec design effects par mode
35 + (web 1,30; téléphone 1,20; IVR 1,40; SMS 1,45; mixte 1,25; inconnu 1,35). Le plafond de
36 + 2 500 empêche un mégasondage de dominer.
37 +- **Qualité de la maison** : multiplicateur 0,6–1,4 selon l'EAM historique de ses sondages
38 + finaux; maison sans historique : ×0,9 + erreur excédentaire accrue (+25 %).
39 +- **Erreur totale d'enquête** : ±1,5 pp ajoutés en variance à chaque sondage (cadre,
40 + non-réponse, pondérations internes du sondeur).
41 +
42 +## 3. House effects
43 +
44 +Sur chaque élection passée à résultat connu, biais moyen par parti des sondages des 21
45 +derniers jours de chaque maison, **rétréci de 50 % vers zéro** et borné à ±3 pp, soustrait
46 +avant l'ajustement du modèle. Recalculé à chaque cycle (`pollster_ratings`).
47 +
48 +## 4. Modèle latent (Bayesian state space)
49 +
50 +- Composition : 6 composantes (CAQ, PCQ, PLQ, PQ, QS, Autres), transformées en
51 + log-ratios additifs `y_k = log(p_k/p_CAQ)` (5 dimensions) — la somme fait toujours 100 %
52 + et la corrélation négative entre partis est structurelle.
53 +- Dynamique : marche aléatoire gaussienne `x_t = x_{t-1} + w_t`, `w_t ~ N(0, q·I)`.
54 +- Observation : `y_sondage ~ N(x_t, R)`, `R` = covariance multinomiale (méthode delta) sur
55 + `n_eff` × cote maison + erreur excédentaire diagonale.
56 +- Inférence : filtre de Kalman exact + lisseur RTS; `q` estimé par maximum de vraisemblance
57 + (grille log-uniforme, borné à [0,004², 0,030²]).
58 +
59 +## 5. Nowcast vs Forecast
60 +
61 +- **Nowcast** : distribution de `softmax(x_aujourd'hui)` — « si l'élection avait lieu aujourd'hui ».
62 +- **Forecast** : même moyenne (martingale), variance augmentée de
63 + `q × jours_restants × 1,6` (volatilité de campagne) **plus l'erreur systémique de
64 + sondage** `σ_industrie = 0,12` (voir §7).
65 +
66 +## 6. Modèle de circonscription
67 +
68 +`part(circ, parti) = softmax( alr(référence 2022 locale) + swing_national_alr
69 ++ ε_région + ε_circonscription − pénalité_retraite )`
70 +
71 +- **Swing proportionnel** en espace log-ratio (jamais de swing uniforme).
72 +- Référence locale : résultat 2022 transposé sur la carte 2026 (renommages appliqués;
73 + Bellefeuille et Marie-Lacoste-Gérin-Lajoie = moyenne des circonscriptions parentes).
74 +- Pénalité de 0,06 (log-ratio, ≈1,3 pp) au parti sortant si le député ne se représente pas
75 + (48 retraits; drapeau raffiné par le registre officiel des candidatures DGEQ).
76 +- Partial pooling : les régions sans sondage propre héritent du national + couche régionale.
77 +
78 +## 7. Monte Carlo et corrélation des erreurs (25 000 simulations)
79 +
80 +| Couche | Distribution | Justification |
81 +|---|---|---|
82 +| Opinion | posterior Kalman + dérive | incertitude d'estimation et d'évolution |
83 +| **Systémique de sondage** | `N(0, 0,12²)` nationale corrélée | 2022 : toute l'industrie a sous-estimé la CAQ (~3 pp); 2018 idem |
84 +| Régionale | `N(0, 0,14²)` × 13 régions | corrélation intrarégionale |
85 +| Locale | `N(0, 0,20²)` × 127 circ. | candidats, enjeux locaux |
86 +
87 +σ régional/local **calibrés sur les résidus du backtest 2022** (swing proportionnel
88 +2018→2022), atténués car 2022 (émergence PCQ) fut un réalignement atypique.
89 +
90 +Sorties : distribution complète des sièges par parti (P5–P95, histogramme), P(plus de
91 +sièges), P(majorité ≥64), P(gouvernement minoritaire), catégorie par circonscription
92 +(Solide ≥95 %, Probable ≥80 %, Favorisé ≥60 %, sinon Chaudement disputé).
93 +
94 +## 8. Backtesting (élection 2022, information à date seulement)
95 +
96 +| Horizon | EAM vote (pp) | Couverture IC 95 % | EAM sièges | Brier | Vainqueurs corrects |
97 +|---|---|---|---|---|---|
98 +| J−120 | 2,65 | 100 % | 3,1 | 0,147 | 92 % |
99 +| J−60 | 2,23 | 100 % | 2,2 | 0,133 | 91 % |
100 +| J−30 | 4,15 | 80 % | 4,0 | 0,179 | 88 % |
101 +| J−7 | 2,76 | 60 % | 4,2 | 0,178 | 86 % |
102 +| J−1 | 2,19 | **100 %** | 4,3 | 0,179 | 86 % |
103 +
104 +Calibration des probabilités locales (tous horizons confondus) : suit la diagonale
105 +(ex. bac 70–80 % → observé 84 %; bac 90–100 % → observé 98 %). L'ajout de la couche
106 +systémique a fait passer la couverture J−1 de 20 % à 100 % sans dégrader le Brier.
107 +Aucune correction maison n'est appliquée en backtest (pas de données antérieures à 2018) —
108 +il mesure le modèle de tendance + simulation seuls.
109 +
110 +## 9. Signaux auxiliaires (poids nul dans le forecast)
111 +
112 +- **Sentiment médiatique** : RSS → extraction d'entités → stance (pro/anti/neutre/ambigu)
113 + avec négation et attaques dirigées; scoreur LLM optionnel (MacLustr) sinon lexique;
114 + regroupement des nouvelles par grappes; anomalies de volume (z ≥ 2,5) → événements.
115 +- **Marchés prédictifs** : probabilités implicites Polymarket affichées face au modèle.
116 +- **Momentum** : Δ vote modèle 14 j + sentiment 7 j, borné [−100, 100].
117 +
118 +## 10. Limites connues
119 +
120 +1. Pas de sondages régionaux publics structurés — la variation régionale vient de la
121 + structure 2022 + couche d'erreur régionale.
122 +2. Transposition 2022→2026 approximative pour les circonscriptions dont les limites ont bougé.
123 +3. Historique des maisons limité à 2022 (les cotes s'affineront après 2026).
124 +4. Participation : outil de scénario (multiplicateurs par électorat), pas un modèle
125 + individuel de probabilité de vote.
126 +5. `AUT` agrège les tiers partis (PVQ, etc.) — aucun siège projeté pour eux.
127 +
128 +## 11. Reproductibilité
129 +
130 +- Chaque run : `forecast_runs` (posterior complet, sièges, diagnostics, version du modèle).
131 +- Graines aléatoires fixes; simulations vectorisées NumPy; tests unitaires du cœur
132 + statistique (`backend/tests/`).
added README.md +65 −0
@@ -0,0 +1,65 @@
1 +<!-- QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 -->
2 +<!-- Auteur : Simon-Pierre Boucher -->
3 +<!-- Contact : contact@spboucher.ai -->
4 +<!-- https://www.qc-election.com -->
5 +
6 +# QC Élection Forecast
7 +
8 +Plateforme de **prévision probabiliste** de l'élection générale québécoise du 5 octobre 2026
9 +(127 circonscriptions, majorité à 64). Ce n'est pas un agrégateur de sondages : c'est un moteur
10 +statistique complet — modèle d'état latent bayésien, effets maison, simulations Monte Carlo à
11 +erreurs corrélées, backtesting et calibration.
12 +
13 +**Production : https://www.qc-election.com** (nœud M2U64, PM2 + ngrok)
14 +
15 +## Architecture
16 +
17 +```
18 +backend/
19 + app/
20 + config.py paramètres (élection, partis, hyperparamètres du modèle)
21 + models.py schéma SQLAlchemy (provenance complète de chaque donnée)
22 + seed.py chargement CSV → base (idempotent, dédoublonnage)
23 + pipeline.py collect → validate → normalize → dedupe → model → simulate → publish
24 + ingest/
25 + wikipedia.py sondages 2026 + 2022, résultats 2022 par circonscription, carte 2025
26 + news_rss.py sentiment médiatique (RSS) : entités → stance → anomalies de volume
27 + modeling/
28 + compositions.py transformations log-ratio (alr) + covariance multinomiale
29 + weights.py pondération : récence, taille effective, design effect, cote maison
30 + house_effects.py biais maison rétrécis + cote de qualité dynamique
31 + trend.py filtre de Kalman (marche aléatoire en espace alr), q estimé par MV
32 + forecast.py nowcast vs forecast (incertitude d'opinion future)
33 + simulate.py Monte Carlo 25 000 élections, erreurs nationale/régionales/locales
34 + backtest.py rejoue 2022 à J-120…J-1 : MAE, Brier, log loss, couverture, calibration
35 + explain.py « Interroger le modèle » — réponses traçables aux données
36 + api/ API publique + administration (Bearer token)
37 + data/ CSV de seed + copies brutes HTML (provenance) + rapport de backtest
38 + tests/ pytest (14 tests du cœur statistique)
39 +frontend/ pages statiques premium (ECharts) servies par FastAPI
40 +```
41 +
42 +## Démarrage local
43 +
44 +```bash
45 +cd backend
46 +python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
47 +.venv/bin/python -m app.ingest.wikipedia # collecte + CSV de seed
48 +.venv/bin/python -m app.seed # base SQLite
49 +.venv/bin/python -m app.pipeline # premier forecast
50 +.venv/bin/uvicorn app.main:app --port 8140
51 +```
52 +
53 +Variables d'environnement (`.env`) : voir `.env.example`. `ADMIN_TOKEN` est requis pour `/admin`.
54 +
55 +## Tests
56 +
57 +```bash
58 +cd backend && .venv/bin/python -m pytest tests/ -q
59 +```
60 +
61 +## Méthodologie
62 +
63 +Documentée en détail dans [METHODOLOGY.md](METHODOLOGY.md) et sur `/methodologie`.
64 +Principes : incertitude d'abord, neutralité politique, traçabilité de chaque valeur,
65 +calibration avant tout.
added backend/app/__init__.py +4 −0
@@ -0,0 +1,4 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
added backend/app/api/__init__.py +4 −0
@@ -0,0 +1,4 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
added backend/app/api/admin.py +142 −0
@@ -0,0 +1,142 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""API d'administration — protégée par jeton (Authorization: Bearer <ADMIN_TOKEN>)."""
6 +from __future__ import annotations
7 +
8 +from datetime import date
9 +
10 +from fastapi import APIRouter, BackgroundTasks, Depends, Header, HTTPException
11 +from pydantic import BaseModel, Field
12 +from sqlalchemy.orm import Session
13 +
14 +from ..config import settings
15 +from ..db import get_db
16 +from .. import models as Mo
17 +from ..pipeline import run_forecast, run_pipeline
18 +from ..seed import get_or_create
19 +
20 +router = APIRouter(prefix="/api/admin")
21 +
22 +
23 +def require_admin(authorization: str = Header(default="")):
24 + if authorization != f"Bearer {settings.admin_token}":
25 + raise HTTPException(401, "Jeton d'administration invalide")
26 +
27 +
28 +class PollIn(BaseModel):
29 + pollster: str
30 + field_start: date | None = None
31 + field_end: date
32 + published: date | None = None
33 + sample_size: int | None = Field(None, ge=50, le=100000)
34 + population: str = "adults"
35 + mode: str = "unknown"
36 + moe: float | None = None
37 + sponsor: str | None = None
38 + source_name: str = "ajout manuel"
39 + source_url: str | None = None
40 + shares: dict[str, float] # {party: %}
41 +
42 +
43 +@router.post("/poll", dependencies=[Depends(require_admin)])
44 +def add_poll(p: PollIn, db: Session = Depends(get_db)):
45 + unknown = [k for k in p.shares if k not in settings.parties]
46 + if unknown:
47 + raise HTTPException(422, f"Partis inconnus: {unknown}")
48 + if not (60 <= sum(p.shares.values()) <= 110):
49 + raise HTTPException(422, "La somme des intentions doit être plausible (60–110)")
50 + el = db.query(Mo.Election).filter_by(is_target=True).one()
51 + pollster, _ = get_or_create(db, Mo.Pollster, name=p.pollster)
52 + dupe = (db.query(Mo.Poll).filter_by(pollster_id=pollster.id, field_end=p.field_end,
53 + sample_size=p.sample_size).first())
54 + if dupe:
55 + raise HTTPException(409, f"Doublon probable du sondage #{dupe.id}")
56 + poll = Mo.Poll(election_id=el.id, pollster_id=pollster.id, field_start=p.field_start,
57 + field_end=p.field_end, published=p.published, sample_size=p.sample_size,
58 + population=p.population, mode=p.mode, moe=p.moe, sponsor=p.sponsor,
59 + source_name=p.source_name, source_url=p.source_url)
60 + total = sum(p.shares.values())
61 + for party, v in p.shares.items():
62 + poll.results.append(Mo.PollResult(party=party, raw_value=v,
63 + normalized_value=round(v * 100 / total, 3)))
64 + db.add(poll)
65 + db.commit()
66 + return {"id": poll.id, "status": "créé"}
67 +
68 +
69 +class PollPatch(BaseModel):
70 + excluded: bool | None = None
71 + exclusion_reason: str | None = None
72 + sample_size: int | None = None
73 + mode: str | None = None
74 + population: str | None = None
75 +
76 +
77 +@router.patch("/poll/{poll_id}", dependencies=[Depends(require_admin)])
78 +def patch_poll(poll_id: int, patch: PollPatch, db: Session = Depends(get_db)):
79 + poll = db.get(Mo.Poll, poll_id)
80 + if poll is None:
81 + raise HTTPException(404, "Sondage introuvable")
82 + for k, v in patch.model_dump(exclude_none=True).items():
83 + setattr(poll, k, v)
84 + db.commit()
85 + return {"id": poll.id, "status": "modifié"}
86 +
87 +
88 +@router.post("/refresh", dependencies=[Depends(require_admin)])
89 +def refresh(background: BackgroundTasks):
90 + background.add_task(run_pipeline, True)
91 + return {"status": "pipeline lancé en arrière-plan"}
92 +
93 +
94 +@router.post("/forecast", dependencies=[Depends(require_admin)])
95 +def rerun_forecast(db: Session = Depends(get_db)):
96 + run = run_forecast(db)
97 + return {"run_id": run.id, "as_of": run.as_of.isoformat()}
98 +
99 +
100 +def _backtest_job():
101 + from ..db import SessionLocal
102 + from ..modeling.backtest import run_backtest
103 + db = SessionLocal()
104 + try:
105 + run_backtest(db)
106 + finally:
107 + db.close()
108 +
109 +
110 +@router.post("/backtest", dependencies=[Depends(require_admin)])
111 +def rerun_backtest(background: BackgroundTasks):
112 + background.add_task(_backtest_job)
113 + return {"status": "backtest lancé en arrière-plan"}
114 +
115 +
116 +@router.get("/logs", dependencies=[Depends(require_admin)])
117 +def logs(db: Session = Depends(get_db), limit: int = 100):
118 + rows = (db.query(Mo.PipelineLog).order_by(Mo.PipelineLog.at.desc()).limit(limit).all())
119 + return [{"at": r.at.isoformat(), "step": r.step, "status": r.status,
120 + "message": (r.message or "")[:500]} for r in rows]
121 +
122 +
123 +@router.get("/runs", dependencies=[Depends(require_admin)])
124 +def runs(db: Session = Depends(get_db), limit: int = 50):
125 + rows = (db.query(Mo.ForecastRun).order_by(Mo.ForecastRun.run_at.desc()).limit(limit).all())
126 + return [{"id": r.id, "run_at": r.run_at.isoformat(), "as_of": r.as_of.isoformat(),
127 + "label": r.label, "is_backtest": r.is_backtest,
128 + "n_polls": r.n_polls_used, "n_sims": r.n_simulations,
129 + "model_version": r.model_version} for r in rows]
130 +
131 +
132 +@router.get("/sources", dependencies=[Depends(require_admin)])
133 +def sources(db: Session = Depends(get_db)):
134 + return [{"name": s.name, "url": s.url, "kind": s.kind,
135 + "last_fetch": s.last_fetch.isoformat() if s.last_fetch else None,
136 + "last_status": s.last_status} for s in db.query(Mo.DataSource).all()]
137 +
138 +
139 +@router.get("/versions", dependencies=[Depends(require_admin)])
140 +def versions(db: Session = Depends(get_db)):
141 + return [{"version": v.version, "released": v.released.isoformat(),
142 + "changelog": v.changelog} for v in db.query(Mo.ModelVersion).all()]
added backend/app/api/public.py +354 −0
@@ -0,0 +1,354 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""API publique — lecture du forecast, sondages, circonscriptions, sentiment."""
6 +from __future__ import annotations
7 +
8 +import json
9 +from datetime import date, datetime, timedelta, timezone
10 +
11 +import numpy as np
12 +from fastapi import APIRouter, Depends, HTTPException, Query
13 +from pydantic import BaseModel, Field
14 +from sqlalchemy.orm import Session
15 +
16 +from ..config import DATA_DIR, settings
17 +from ..db import get_db
18 +from .. import models as Mo
19 +from ..modeling import simulate as SIM
20 +from ..modeling.explain import compose_answer, gather_facts
21 +from ..modeling.house_effects import PollsterProfile, adjust_shares, profile_for
22 +from ..modeling.weights import display_weight
23 +from ..pipeline import _district_inputs
24 +
25 +router = APIRouter(prefix="/api")
26 +
27 +
28 +def _latest_run(db: Session) -> Mo.ForecastRun:
29 + run = (db.query(Mo.ForecastRun).filter(Mo.ForecastRun.is_backtest.is_(False))
30 + .order_by(Mo.ForecastRun.as_of.desc(), Mo.ForecastRun.id.desc()).first())
31 + if run is None:
32 + raise HTTPException(503, "Aucun forecast disponible — lancez le pipeline.")
33 + return run
34 +
35 +
36 +@router.get("/meta")
37 +def meta(db: Session = Depends(get_db)):
38 + target = db.query(Mo.Election).filter_by(is_target=True).first()
39 + parties = db.query(Mo.Party).all()
40 + last = (db.query(Mo.ForecastRun).filter(Mo.ForecastRun.is_backtest.is_(False))
41 + .order_by(Mo.ForecastRun.run_at.desc()).first())
42 + return {
43 + "app": settings.app_name,
44 + "election": {"name": target.name if target else settings.election_name,
45 + "date": settings.election_date.isoformat(),
46 + "total_seats": settings.total_seats,
47 + "majority_seats": settings.majority_seats,
48 + "days_remaining": (settings.election_date - date.today()).days},
49 + "model_version": settings.model_version,
50 + "last_update": last.run_at.isoformat() if last else None,
51 + "parties": {p.code: {"name": p.name, "leader": p.leader, "color": p.color}
52 + for p in parties},
53 + }
54 +
55 +
56 +@router.get("/forecast/latest")
57 +def forecast_latest(db: Session = Depends(get_db)):
58 + run = _latest_run(db)
59 + return {"run_id": run.id, "as_of": run.as_of.isoformat(),
60 + "run_at": run.run_at.isoformat(), "model_version": run.model_version,
61 + "n_polls_used": run.n_polls_used, "n_simulations": run.n_simulations,
62 + "national": run.national, "seats": run.seats, "diagnostics": run.diagnostics}
63 +
64 +
65 +@router.get("/forecast/history")
66 +def forecast_history(db: Session = Depends(get_db)):
67 + """Évolution des probabilités et votes à travers les runs (Forecast Explorer)."""
68 + runs = (db.query(Mo.ForecastRun).filter(Mo.ForecastRun.is_backtest.is_(False))
69 + .order_by(Mo.ForecastRun.as_of.asc(), Mo.ForecastRun.id.asc()).all())
70 + seen: dict[str, Mo.ForecastRun] = {}
71 + for r in runs:
72 + seen[r.as_of.isoformat()] = r # dernier run de chaque journée
73 + out = []
74 + for iso, r in sorted(seen.items()):
75 + out.append({
76 + "as_of": iso, "run_id": r.id,
77 + "prob_most": {p: r.seats["per_party"][p]["prob_most"] for p in settings.parties},
78 + "prob_majority": {p: r.seats["per_party"][p]["prob_majority"] for p in settings.parties},
79 + "seats_mean": {p: r.seats["per_party"][p]["mean"] for p in settings.parties},
80 + "vote_mean": {p: r.national["forecast"][p]["mean"] for p in settings.parties},
81 + })
82 + return out
83 +
84 +
85 +@router.get("/forecast/run/{run_id}")
86 +def forecast_run(run_id: int, db: Session = Depends(get_db)):
87 + run = db.get(Mo.ForecastRun, run_id)
88 + if run is None:
89 + raise HTTPException(404, "Run introuvable")
90 + return {"run_id": run.id, "as_of": run.as_of.isoformat(), "label": run.label,
91 + "is_backtest": run.is_backtest, "national": run.national,
92 + "seats": run.seats, "diagnostics": run.diagnostics}
93 +
94 +
95 +@router.get("/districts")
96 +def districts(db: Session = Depends(get_db)):
97 + run = _latest_run(db)
98 + static = {d.name: d for d in db.query(Mo.District).all()}
99 + out = []
100 + for fr in run.district_results:
101 + d = static.get(fr.district_name)
102 + out.append({
103 + **fr.detail,
104 + "incumbent_party": d.incumbent_party if d else None,
105 + "incumbent_running": d.incumbent_running if d else None,
106 + "baseline_2022": d.baseline_shares if d else None,
107 + "turnout_2022": d.turnout_2022 if d else None,
108 + "is_new_2026": d.is_new_2026 if d else False,
109 + })
110 + return {"as_of": run.as_of.isoformat(), "run_id": run.id, "districts": out}
111 +
112 +
113 +@router.get("/districts/{name}")
114 +def district_detail(name: str, db: Session = Depends(get_db)):
115 + d = db.query(Mo.District).filter(Mo.District.name == name).first()
116 + if d is None:
117 + raise HTTPException(404, "Circonscription introuvable")
118 + run = _latest_run(db)
119 + fr = next((x for x in run.district_results if x.district_name == name), None)
120 + hist = (db.query(Mo.HistoricalDistrictResult)
121 + .filter_by(district_name=name).all())
122 + candidates = db.query(Mo.Candidate).filter_by(district_id=d.id).all()
123 + return {"name": d.name, "region": d.region, "baseline_2022": d.baseline_shares,
124 + "baseline_source": d.baseline_source, "incumbent_party": d.incumbent_party,
125 + "incumbent_running": d.incumbent_running, "turnout_2022": d.turnout_2022,
126 + "is_new_2026": d.is_new_2026,
127 + "forecast": fr.detail if fr else None,
128 + "candidates": [{"name": c.name, "party": c.party, "party_full": c.party_full,
129 + "is_incumbent": c.is_incumbent} for c in candidates],
130 + "history": [{"party": h.party, "pct": h.pct} for h in hist]}
131 +
132 +
133 +@router.get("/geo")
134 +def geo():
135 + """Carte électorale officielle 2026 (GeoJSON DGEQ, propriétés minimales)."""
136 + f = DATA_DIR / "carte_2026.geojson"
137 + if not f.exists():
138 + raise HTTPException(404, "Carte non téléchargée — lancez le pipeline.")
139 + from fastapi.responses import FileResponse
140 + return FileResponse(f, media_type="application/geo+json",
141 + headers={"Cache-Control": "public, max-age=86400"})
142 +
143 +
144 +@router.get("/markets")
145 +def markets(db: Session = Depends(get_db)):
146 + """Marchés prédictifs (signal auxiliaire) + comparaison au modèle."""
147 + from ..ingest.markets import fetch_winner_market
148 + data = fetch_winner_market()
149 + if data is None:
150 + raise HTTPException(503, "Marchés indisponibles")
151 + try:
152 + run = _latest_run(db)
153 + data["model_prob_most_seats"] = {
154 + p: run.seats["per_party"][p]["prob_most"]
155 + for p in settings.parties if p in data["implied_prob_most_seats"]}
156 + except HTTPException:
157 + pass
158 + return data
159 +
160 +
161 +@router.get("/polls")
162 +def polls(db: Session = Depends(get_db),
163 + date_from: date | None = None, date_to: date | None = None,
164 + pollster: str | None = None, election: str = "2026",
165 + limit: int = Query(500, le=2000)):
166 + el = (db.query(Mo.Election)
167 + .filter(Mo.Election.name.contains(election)).first())
168 + q = db.query(Mo.Poll).filter(Mo.Poll.election_id == el.id) if el else db.query(Mo.Poll)
169 + if date_from:
170 + q = q.filter(Mo.Poll.field_end >= date_from)
171 + if date_to:
172 + q = q.filter(Mo.Poll.field_end <= date_to)
173 + if pollster:
174 + q = q.join(Mo.Pollster).filter(Mo.Pollster.name == pollster)
175 + rows = q.order_by(Mo.Poll.field_end.desc()).limit(limit).all()
176 +
177 + ratings = {r.pollster.name: r for r in db.query(Mo.PollsterRating).all()}
178 + as_of = date.today()
179 + out, max_w = [], 1e-9
180 + for poll in rows:
181 + rating = ratings.get(poll.pollster.name)
182 + prof = PollsterProfile(
183 + name=poll.pollster.name,
184 + mae_pp=rating.mae_pp if rating else None,
185 + house_effects=rating.house_effects if rating else {},
186 + weight_multiplier=rating.weight_multiplier if rating else 0.9)
187 + shares = {r.party: r.normalized_value for r in poll.results}
188 + adj = adjust_shares(shares, prof) if shares else {}
189 + w = display_weight(poll.field_end, as_of, poll.sample_size, poll.mode,
190 + prof.weight_multiplier)
191 + max_w = max(max_w, w)
192 + out.append({
193 + "id": poll.id, "pollster": poll.pollster.name, "sponsor": poll.sponsor,
194 + "field_start": poll.field_start.isoformat() if poll.field_start else None,
195 + "field_end": poll.field_end.isoformat(),
196 + "sample_size": poll.sample_size, "moe": poll.moe, "mode": poll.mode,
197 + "population": poll.population, "source_url": poll.source_url,
198 + "source_name": poll.source_name, "excluded": poll.excluded,
199 + "raw": {r.party: r.raw_value for r in poll.results},
200 + "normalized": shares,
201 + "adjusted": {k: round(v, 1) for k, v in adj.items()},
202 + "weight": w,
203 + })
204 + for r in out:
205 + r["weight"] = round(r["weight"] / max_w, 4)
206 + return out
207 +
208 +
209 +@router.get("/pollsters")
210 +def pollsters(db: Session = Depends(get_db)):
211 + out = []
212 + for p in db.query(Mo.Pollster).all():
213 + rating = (db.query(Mo.PollsterRating).filter_by(pollster_id=p.id)
214 + .order_by(Mo.PollsterRating.computed_at.desc()).first())
215 + n = db.query(Mo.Poll).filter_by(pollster_id=p.id).count()
216 + out.append({"name": p.name, "polls_in_db": n,
217 + "rating": {"n_final_polls": rating.n_polls, "mae_pp": rating.mae_pp,
218 + "house_effects": rating.house_effects,
219 + "weight_multiplier": rating.weight_multiplier,
220 + "detail": rating.detail} if rating else None})
221 + return sorted(out, key=lambda x: -(x["polls_in_db"]))
222 +
223 +
224 +class WhatIfRequest(BaseModel):
225 + national_pp: dict[str, float] = Field(default_factory=dict)
226 + regional_pp: dict[str, dict[str, float]] = Field(default_factory=dict)
227 + turnout_mult: dict[str, float] = Field(default_factory=dict)
228 + poll_error_pp: dict[str, float] = Field(default_factory=dict)
229 + n_sims: int = Field(8000, ge=1000, le=40000)
230 +
231 +
232 +@router.post("/whatif")
233 +def whatif(req: WhatIfRequest, db: Session = Depends(get_db)):
234 + for d in (req.national_pp, req.poll_error_pp):
235 + for k, v in d.items():
236 + if abs(v) > 20:
237 + raise HTTPException(422, f"Choc trop grand pour {k} (max ±20 pp)")
238 + run = _latest_run(db)
239 + result_ref = run.national.get("baseline_national")
240 + inp = _district_inputs(db, result_ref)
241 + inp.x_mean = np.array(run.national["x_forecast"])
242 + inp.P = np.array(run.national["P_forecast"])
243 + scenario = {"national_pp": req.national_pp, "regional_pp": req.regional_pp,
244 + "turnout_mult": req.turnout_mult, "poll_error_pp": req.poll_error_pp}
245 + sim = SIM.run_simulation(inp, n_sims=req.n_sims, scenario=scenario)
246 + return {"base_run_id": run.id, "scenario": scenario, "n_sims": sim.n_sims,
247 + "seats": sim.seats, "national_vote": sim.national_vote,
248 + "summary": sim.seat_matrix_summary,
249 + "districts": sim.districts}
250 +
251 +
252 +@router.get("/backtest")
253 +def backtest_report():
254 + f = DATA_DIR / "backtest_report.json"
255 + if not f.exists():
256 + raise HTTPException(404, "Backtest non exécuté")
257 + return json.loads(f.read_text())
258 +
259 +
260 +@router.get("/sentiment")
261 +def sentiment(db: Session = Depends(get_db), days: int = Query(30, le=120)):
262 + cutoff = datetime.now(timezone.utc) - timedelta(days=days)
263 + docs = (db.query(Mo.SentimentDocument)
264 + .filter(Mo.SentimentDocument.fetched_at >= cutoff).all())
265 + per_party: dict[str, dict] = {}
266 + daily: dict[str, dict[str, list]] = {}
267 + for doc in docs:
268 + d = (doc.published or doc.fetched_at).date().isoformat()
269 + for s in doc.scores:
270 + pp = per_party.setdefault(s.entity, {"volume": 0, "sum": 0.0,
271 + "stances": {"pro": 0, "anti": 0,
272 + "neutre": 0, "ambigu": 0}})
273 + pp["volume"] += 1
274 + pp["sum"] += s.sentiment
275 + pp["stances"][s.stance] = pp["stances"].get(s.stance, 0) + 1
276 + daily.setdefault(s.entity, {}).setdefault(d, []).append(s.sentiment)
277 +
278 + now = datetime.now(timezone.utc)
279 + def _window(entity, hours):
280 + vals = [s.sentiment for doc in docs for s in doc.scores
281 + if s.entity == entity and (doc.published or doc.fetched_at)
282 + and (now - (doc.published or doc.fetched_at).replace(tzinfo=timezone.utc)
283 + ).total_seconds() < hours * 3600]
284 + return {"volume": len(vals),
285 + "moyenne": round(float(np.mean(vals)), 3) if vals else None}
286 +
287 + out = {}
288 + for party, pp in per_party.items():
289 + out[party] = {
290 + "volume": pp["volume"],
291 + "sentiment_moyen": round(pp["sum"] / pp["volume"], 3),
292 + "stances": pp["stances"],
293 + "h24": _window(party, 24), "j7": _window(party, 168),
294 + "timeline": [{"date": d, "sentiment": round(float(np.mean(v)), 3), "volume": len(v)}
295 + for d, v in sorted(daily.get(party, {}).items())],
296 + }
297 + n_docs = len(docs)
298 + return {"note": ("Signal auxiliaire très bruité — n'entre PAS dans le calcul "
299 + "des probabilités électorales."),
300 + "documents": n_docs, "parties": out}
301 +
302 +
303 +@router.get("/momentum")
304 +def momentum(db: Session = Depends(get_db)):
305 + """Indicateur maison (auxiliaire) : tendance sondages + sentiment + volume."""
306 + run = _latest_run(db)
307 + series = run.national["trend_series"]
308 + sent = sentiment(db=db, days=14)
309 + out = {}
310 + for p in [x for x in settings.parties if x != "AUT"]:
311 + means = series["mean"][p]
312 + d14 = means[-1] - means[max(0, len(means) - 15)]
313 + s = sent["parties"].get(p, {})
314 + s7 = (s.get("j7") or {}).get("moyenne") or 0.0
315 + score = float(np.clip(d14 * 12 + s7 * 25, -100, 100))
316 + out[p] = {"score": round(score, 1), "d_vote_14j_pp": round(d14, 2),
317 + "sentiment_7j": s7, "volume_7j": (s.get("j7") or {}).get("volume", 0)}
318 + return {"note": "Indicateur auxiliaire — n'est PAS une probabilité électorale.",
319 + "momentum": out}
320 +
321 +
322 +@router.get("/events")
323 +def events(db: Session = Depends(get_db)):
324 + rows = db.query(Mo.NewsEvent).order_by(Mo.NewsEvent.event_date.asc()).all()
325 + return [{"date": e.event_date.isoformat(), "title": e.title, "kind": e.kind,
326 + "description": e.description, "parties": e.parties,
327 + "importance": e.importance, "detected_by": e.detected_by} for e in rows]
328 +
329 +
330 +@router.get("/news")
331 +def news(db: Session = Depends(get_db), limit: int = Query(40, le=200)):
332 + """Nouvelles regroupées par grappe (une entrée par nouvelle, pas 50 doublons)."""
333 + docs = (db.query(Mo.SentimentDocument)
334 + .order_by(Mo.SentimentDocument.fetched_at.desc()).limit(400).all())
335 + clusters: dict[str, dict] = {}
336 + for doc in docs:
337 + c = clusters.setdefault(doc.cluster_key or doc.url, {
338 + "title": doc.title, "url": doc.url, "source": doc.source,
339 + "published": (doc.published or doc.fetched_at).isoformat(),
340 + "n_articles": 0, "parties": {}})
341 + c["n_articles"] += 1
342 + for s in doc.scores:
343 + c["parties"][s.entity] = {"sentiment": s.sentiment, "stance": s.stance}
344 + return list(clusters.values())[:limit]
345 +
346 +
347 +class AskRequest(BaseModel):
348 + question: str = Field(min_length=3, max_length=500)
349 +
350 +
351 +@router.post("/ask")
352 +def ask(req: AskRequest, db: Session = Depends(get_db)):
353 + facts = gather_facts(db)
354 + return compose_answer(req.question, facts)
added backend/app/config.py +99 −0
@@ -0,0 +1,99 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""Configuration centrale de la plateforme QC-Élection."""
6 +from __future__ import annotations
7 +
8 +import os
9 +from datetime import date
10 +from pathlib import Path
11 +
12 +from pydantic_settings import BaseSettings
13 +
14 +BASE_DIR = Path(__file__).resolve().parent.parent # backend/
15 +DATA_DIR = BASE_DIR / "data"
16 +FRONTEND_DIR = BASE_DIR.parent / "frontend"
17 +
18 +
19 +class Settings(BaseSettings):
20 + # --- Général ---
21 + app_name: str = "QC Élection Forecast"
22 + model_version: str = "1.0.0"
23 + database_url: str = f"sqlite:///{BASE_DIR / 'qc_election.db'}"
24 + admin_token: str = os.environ.get("ADMIN_TOKEN", "change-me")
25 + port: int = 8140
26 +
27 + # --- Élection cible ---
28 + election_date: date = date(2026, 10, 5)
29 + election_name: str = "Élection générale québécoise de 2026"
30 + total_seats: int = 127
31 + majority_seats: int = 64
32 +
33 + # --- Partis (ordre alphabétique du sigle — neutralité) ---
34 + parties: list[str] = ["CAQ", "PCQ", "PLQ", "PQ", "QS", "AUT"]
35 + party_names: dict[str, str] = {
36 + "CAQ": "Coalition avenir Québec",
37 + "PCQ": "Parti conservateur du Québec",
38 + "PLQ": "Parti libéral du Québec",
39 + "PQ": "Parti québécois",
40 + "QS": "Québec solidaire",
41 + "AUT": "Autres",
42 + }
43 +
44 + # --- Modèle : pondération des sondages ---
45 + # Décroissance temporelle appliquée en plus de la structure Kalman
46 + # (utilisée pour les poids affichés et la variance d'observation).
47 + time_decay_halflife_days: float = 12.0
48 + max_effective_n: int = 2500 # plafond anti-domination des très gros sondages
49 + design_effects: dict[str, float] = { # deff par mode de collecte
50 + "web": 1.30, "ivr": 1.40, "phone": 1.20, "sms": 1.45, "mixed": 1.25, "unknown": 1.35,
51 + }
52 + # Variance excédentaire (erreur non-échantillonnale) ajoutée à chaque sondage,
53 + # en points de pourcentage au carré (≈ (1.5 pp)²) — cf. littérature "total survey error".
54 + excess_poll_sd_pp: float = 1.5
55 +
56 + # --- Modèle : marche aléatoire latente (espace log-ratio) ---
57 + # Écart-type quotidien de la marche aléatoire (estimé + borné).
58 + rw_daily_sd_min: float = 0.004
59 + rw_daily_sd_max: float = 0.030
60 + # Inflation d'incertitude pour le forecast (opinion future) : multiplicateur
61 + # appliqué à la variance de dérive jusqu'au jour du vote (volatilité de campagne).
62 + campaign_drift_inflation: float = 1.6
63 +
64 + # --- House effects ---
65 + house_effect_shrinkage: float = 0.5 # rétrécissement partiel vers 0
66 + house_effect_max_pp: float = 3.0 # borne de correction (pp)
67 +
68 + # --- Simulation Monte Carlo ---
69 + n_simulations: int = 25000
70 + # Erreur systémique de sondage (« industry error ») : les maisons se trompent
71 + # ensemble (2022 : CAQ sous-estimée de ~3 pp par toute l'industrie). Couche
72 + # corrélée ajoutée au forecast du jour du vote, calibrée sur 2018/2022.
73 + industry_error_sd: float = 0.12
74 + # Écarts-types calibrés sur les résidus du backtest 2022 (swing proportionnel
75 + # 2018→2022), atténués car 2022 fut un réalignement atypique (émergence PCQ).
76 + regional_error_sd: float = 0.14 # écart-type régional (espace log-ratio)
77 + riding_error_sd: float = 0.20 # écart-type propre à la circonscription
78 + incumbent_retirement_penalty: float = 0.06 # log-ratio (~1.3 pp) si le sortant ne se représente pas
79 +
80 + # --- Sentiment (signal auxiliaire, poids nul dans le forecast) ---
81 + sentiment_feeds: list[str] = [
82 + "https://ici.radio-canada.ca/rss/92", # Radio-Canada — Politique
83 + "https://www.ledevoir.com/rss/section/politique.xml",
84 + "https://www.lapresse.ca/actualites/politique/rss",
85 + ]
86 + maclustr_api_key: str = os.environ.get("MACLUSTR_API_KEY", "")
87 + maclustr_llm_url: str = "https://llm.maclustr.io/v1/chat/completions"
88 + maclustr_llm_model: str = "maclustr-fast"
89 +
90 + # --- Pipeline ---
91 + refresh_interval_hours: float = 6.0
92 +
93 + class Config:
94 + env_prefix = "QCE_"
95 + env_file = ".env"
96 + extra = "ignore"
97 +
98 +
99 +settings = Settings()
added backend/app/db.py +34 −0
@@ -0,0 +1,34 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +from __future__ import annotations
6 +
7 +from sqlalchemy import create_engine
8 +from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
9 +
10 +from .config import settings
11 +
12 +
13 +class Base(DeclarativeBase):
14 + pass
15 +
16 +
17 +engine = create_engine(
18 + settings.database_url,
19 + connect_args={"check_same_thread": False} if settings.database_url.startswith("sqlite") else {},
20 +)
21 +SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
22 +
23 +
24 +def get_db():
25 + db: Session = SessionLocal()
26 + try:
27 + yield db
28 + finally:
29 + db.close()
30 +
31 +
32 +def init_db() -> None:
33 + from . import models # noqa: F401
34 + Base.metadata.create_all(engine)
added backend/app/ingest/__init__.py +4 −0
@@ -0,0 +1,4 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
added backend/app/ingest/dgeq.py +144 −0
@@ -0,0 +1,144 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""Ingestion des données ouvertes officielles d'Élections Québec (DGEQ).
6 +
7 +Sources (https://dgeq.org/donnees.html) :
8 + * GeoJSON des 127 circonscriptions 2026 (carte électorale 2025, sans plans d'eau);
9 + * liste officielle des circonscriptions 2026;
10 + * candidatures officielles (avec drapeau « député sortant »);
11 + * résultats en direct le soir du scrutin (resultats.json — provider prêt).
12 +"""
13 +from __future__ import annotations
14 +
15 +import json
16 +import unicodedata
17 +from datetime import datetime, timezone
18 +
19 +import httpx
20 +from sqlalchemy.orm import Session
21 +
22 +from ..config import DATA_DIR, settings
23 +from .. import models as Mo
24 +
25 +BASE = "https://donnees.electionsquebec.qc.ca"
26 +URL_GEOJSON = f"{BASE}/autres/provincial/circonscriptions_electorales_sans_eau_2026.json"
27 +URL_LISTE = f"{BASE}/autres/provincial/liste_circonscriptions2026.csv"
28 +URL_CANDIDATURES = f"{BASE}/production/provincial/candidatures/candidatures.json"
29 +URL_RESULTATS = f"{BASE}/production/provincial/resultats/resultats.json"
30 +
31 +PARTY_BY_NAME = {
32 + "parti québécois": "PQ",
33 + "québec solidaire": "QS",
34 + "coalition avenir québec": "CAQ",
35 + "parti conservateur du québec": "PCQ",
36 + "parti libéral du québec": "PLQ",
37 +}
38 +
39 +
40 +def _norm(s: str) -> str:
41 + """Clé de rapprochement de noms (accents conservés, tirets/apostrophes unifiés)."""
42 + s = unicodedata.normalize("NFC", str(s)).strip()
43 + return s.replace("–", "-").replace("—", "-").replace("’", "'").lower()
44 +
45 +
46 +def party_code(nom_parti: str | None) -> str:
47 + if not nom_parti:
48 + return "AUT"
49 + n = _norm(nom_parti)
50 + for key, code in PARTY_BY_NAME.items():
51 + if key in n:
52 + return code
53 + return "AUT"
54 +
55 +
56 +def _mark(db: Session, name: str, url: str, status: str) -> None:
57 + src = db.query(Mo.DataSource).filter_by(url=url).first()
58 + if src is None:
59 + src = Mo.DataSource(name=name, url=url, kind="officiel")
60 + db.add(src)
61 + src.last_fetch = datetime.now(timezone.utc)
62 + src.last_status = status[:200]
63 + db.commit()
64 +
65 +
66 +def fetch_geojson(db: Session | None = None) -> dict:
67 + """Télécharge la carte officielle et l'écrit (propriétés minimales)."""
68 + r = httpx.get(URL_GEOJSON, timeout=120)
69 + r.raise_for_status()
70 + g = r.json()
71 + slim = {"type": "FeatureCollection",
72 + "features": [{"type": "Feature",
73 + "properties": {"name": f["properties"]["NM_CEP"]},
74 + "geometry": f["geometry"]} for f in g["features"]]}
75 + out = DATA_DIR / "carte_2026.geojson"
76 + out.write_text(json.dumps(slim, ensure_ascii=False))
77 + if db:
78 + _mark(db, "DGEQ — carte électorale 2026 (GeoJSON)", URL_GEOJSON,
79 + f"ok ({len(slim['features'])} circonscriptions)")
80 + return {"features": len(slim["features"]), "path": str(out)}
81 +
82 +
83 +def validate_names(db: Session) -> dict:
84 + """Rapproche les noms officiels DGEQ et les circonscriptions en base."""
85 + f = DATA_DIR / "carte_2026.geojson"
86 + if not f.exists():
87 + return {"error": "GeoJSON absent"}
88 + geo_names = {_norm(x["properties"]["name"]) for x in json.loads(f.read_text())["features"]}
89 + db_names = {_norm(d.name): d.name for d in db.query(Mo.District).all()}
90 + missing_geo = sorted(n for n in db_names if n not in geo_names)
91 + missing_db = sorted(n for n in geo_names if n not in db_names)
92 + return {"non_trouvées_dans_geo": missing_geo, "non_trouvées_en_base": missing_db}
93 +
94 +
95 +def fetch_candidatures(db: Session) -> dict:
96 + """Candidatures officielles : stockées et utilisées pour raffiner l'incumbency."""
97 + try:
98 + r = httpx.get(URL_CANDIDATURES, timeout=60)
99 + r.raise_for_status()
100 + data = r.json()
101 + except Exception as e:
102 + _mark(db, "DGEQ — candidatures officielles", URL_CANDIDATURES, f"error: {e}")
103 + raise
104 + districts = {_norm(d.name): d for d in db.query(Mo.District).all()}
105 + db.query(Mo.Candidate).delete()
106 + n, sortants = 0, 0
107 + for c in data:
108 + dname = c.get("nom_circonscription")
109 + if not dname:
110 + continue
111 + d = districts.get(_norm(dname))
112 + if d is None:
113 + continue
114 + code = party_code(c.get("nom_parti"))
115 + is_inc = c.get("depute_sortant") == "O"
116 + db.add(Mo.Candidate(
117 + district_id=d.id, party=code,
118 + name=f"{c.get('prenom_bulletin_vote') or ''} {c.get('nom_bulletin_vote') or ''}".strip(),
119 + party_full=c.get("nom_parti"), is_incumbent=is_inc,
120 + source_url=URL_CANDIDATURES))
121 + n += 1
122 + if is_inc:
123 + sortants += 1
124 + # confirmation officielle que le sortant se représente
125 + if d.incumbent_party == code:
126 + d.incumbent_running = True
127 + db.commit()
128 + _mark(db, "DGEQ — candidatures officielles", URL_CANDIDATURES,
129 + f"ok ({n} candidatures, {sortants} sortants)")
130 + return {"candidatures": n, "sortants_confirmés": sortants}
131 +
132 +
133 +def refresh_dgeq(db: Session) -> dict:
134 + report = {}
135 + try:
136 + report["geojson"] = fetch_geojson(db)
137 + except Exception as e:
138 + report["geojson"] = f"échec: {e}"
139 + try:
140 + report["candidatures"] = fetch_candidatures(db)
141 + except Exception as e:
142 + report["candidatures"] = f"échec: {e}"
143 + report["validation_noms"] = validate_names(db)
144 + return report
added backend/app/ingest/markets.py +67 −0
@@ -0,0 +1,67 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""Marchés prédictifs (Polymarket, API publique Gamma) — signal AUXILIAIRE.
6 +
7 +Les prix des marchés reflètent l'opinion agrégée de parieurs, pas un modèle
8 +statistique : ils sont affichés en regard du forecast pour comparaison, avec
9 +provenance, et n'entrent jamais dans le calcul des probabilités du modèle.
10 +"""
11 +from __future__ import annotations
12 +
13 +import json
14 +import time
15 +from datetime import datetime, timezone
16 +
17 +import httpx
18 +
19 +GAMMA = "https://gamma-api.polymarket.com/events"
20 +SLUG_WINNER = "quebec-general-election-winner"
21 +
22 +PARTY_IN_QUESTION = {
23 + "coalition avenir": "CAQ", "parti québécois": "PQ", "parti libéral": "PLQ",
24 + "québec solidaire": "QS", "parti conservateur": "PCQ",
25 +}
26 +
27 +_cache: dict = {"at": 0.0, "data": None}
28 +CACHE_TTL = 900 # 15 min
29 +
30 +
31 +def fetch_winner_market() -> dict | None:
32 + now = time.time()
33 + if _cache["data"] is not None and now - _cache["at"] < CACHE_TTL:
34 + return _cache["data"]
35 + try:
36 + r = httpx.get(GAMMA, params={"slug": SLUG_WINNER}, timeout=20)
37 + r.raise_for_status()
38 + events = r.json()
39 + if not events:
40 + return None
41 + ev = events[0]
42 + probs = {}
43 + for m in ev.get("markets", []):
44 + q = (m.get("question") or "").lower()
45 + prices = m.get("outcomePrices")
46 + if isinstance(prices, str):
47 + prices = json.loads(prices)
48 + if not prices:
49 + continue
50 + for frag, code in PARTY_IN_QUESTION.items():
51 + if frag in q:
52 + probs[code] = float(prices[0]) # prix du « Yes »
53 + data = {
54 + "source": "Polymarket (marché prédictif)",
55 + "event": ev.get("title"),
56 + "url": f"https://polymarket.com/event/{SLUG_WINNER}",
57 + "question": "Quel parti remportera le plus de sièges ?",
58 + "implied_prob_most_seats": probs,
59 + "volume_usd": round(float(ev.get("volume") or 0)),
60 + "fetched_at": datetime.now(timezone.utc).isoformat(),
61 + "note": ("Prix implicites d'un marché de paris — signal auxiliaire, "
62 + "n'entre pas dans le modèle."),
63 + }
64 + _cache.update(at=now, data=data)
65 + return data
66 + except Exception:
67 + return _cache["data"]
added backend/app/ingest/news_rss.py +210 −0
@@ -0,0 +1,210 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""Moteur de sentiment politique — signal AUXILIAIRE, poids nul dans le forecast.
6 +
7 +Pipeline : document → extraction d'entités (partis, chefs) → stance/sentiment
8 +→ clustering (déduplication de la même nouvelle) → agrégats + anomalies de volume.
9 +
10 +Deux scoreurs :
11 + * « lexicon » : lexique français + négation + heuristique de cible
12 + (une attaque de X contre Y est négative envers Y, pas envers X);
13 + * « llm » (optionnel, si MACLUSTR_API_KEY) : classification de stance par LLM.
14 +
15 +Le sentiment Web n'est PAS un sondage : il est affiché comme signal bruité et
16 +n'entre jamais dans le calcul des probabilités électorales.
17 +"""
18 +from __future__ import annotations
19 +
20 +import hashlib
21 +import re
22 +from datetime import date, datetime, timedelta, timezone
23 +
24 +import numpy as np
25 +from sqlalchemy.orm import Session
26 +
27 +from ..config import settings
28 +from .. import models as Mo
29 +
30 +ENTITY_ALIASES: dict[str, list[str]] = {
31 + "CAQ": ["caq", "coalition avenir québec", "christine fréchette", "fréchette"],
32 + "PLQ": ["plq", "parti libéral du québec", "libéral", "libéraux", "charles milliard",
33 + "milliard"],
34 + "PQ": ["parti québécois", " pq ", "st-pierre plamondon", "plamondon", "pspp",
35 + "péquiste"],
36 + "QS": ["québec solidaire", " qs ", "ruba ghazal", "sol zanetti", "solidaire"],
37 + "PCQ": ["parti conservateur du québec", " pcq ", "éric duhaime", "duhaime",
38 + "conservateur du québec"],
39 +}
40 +
41 +POS_WORDS = {"gagne", "succès", "hausse", "progresse", "populaire", "appui", "appuis",
42 + "victoire", "monte", "remonte", "convainc", "salue", "félicite", "record",
43 + "confiance", "promet", "espoir", "solide", "fort", "avance", "favori",
44 + "endosse", "rallie", "applaudit", "réussit", "améliore"}
45 +NEG_WORDS = {"scandale", "chute", "recul", "critique", "critiqué", "controverse",
46 + "démission", "échec", "perd", "baisse", "colère", "crise", "attaque",
47 + "accuse", "accusé", "dérape", "conflit", "corruption", "enquête",
48 + "polémique", "déçoit", "impopulaire", "fragile", "faible", "menace",
49 + "dénonce", "erreur", "gaffe", "tollé", "démissionne", "poursuite"}
50 +NEGATIONS = {"ne", "pas", "jamais", "sans", "aucun", "aucune", "ni", "non"}
51 +ATTACK_VERBS = {"accuse", "attaque", "dénonce", "critique", "blâme", "reproche",
52 + "fustige", "tacle", "éreinte", "qualifie"}
53 +
54 +STOP = {"le", "la", "les", "de", "des", "du", "un", "une", "et", "en", "au", "aux",
55 + "pour", "sur", "dans", "avec", "que", "qui", "est", "sont", "a", "à", "d",
56 + "l", "se", "son", "sa", "ses", "plus", "pas", "ne", "il", "elle", "on"}
57 +
58 +
59 +def _tokens(text: str) -> list[str]:
60 + return re.findall(r"[a-zàâäéèêëîïôöùûüç'-]+", text.lower())
61 +
62 +
63 +def cluster_key(title: str) -> str:
64 + words = sorted(w for w in _tokens(title) if w not in STOP and len(w) > 3)[:6]
65 + return hashlib.sha1(" ".join(words).encode()).hexdigest()[:16]
66 +
67 +
68 +def detect_entities(text: str) -> list[str]:
69 + t = " " + text.lower() + " "
70 + return [code for code, aliases in ENTITY_ALIASES.items()
71 + if any(a in t for a in aliases)]
72 +
73 +
74 +def lexicon_score(text: str, entity: str) -> tuple[float, str]:
75 + """Sentiment [-1,1] + stance dirigés vers l'entité (heuristique de cible)."""
76 + toks = _tokens(text)
77 + aliases = ENTITY_ALIASES[entity]
78 + positions = [i for i, tok in enumerate(toks)
79 + if any(a.strip().split()[0] in tok for a in aliases if a.strip())]
80 + score, n = 0.0, 0
81 + for i, tok in enumerate(toks):
82 + val = 1.0 if tok in POS_WORDS else (-1.0 if tok in NEG_WORDS else 0.0)
83 + if val == 0.0:
84 + continue
85 + if any(toks[j] in NEGATIONS for j in range(max(0, i - 3), i)):
86 + val = -val
87 + # attaque politique : le sentiment négatif vise l'entité APRÈS le verbe
88 + if tok in ATTACK_VERBS:
89 + after = detect_entities(" ".join(toks[i:i + 12]))
90 + if entity not in after:
91 + continue # l'attaque vise quelqu'un d'autre
92 + dist = min((abs(i - p) for p in positions), default=99)
93 + if dist > 25:
94 + continue # trop loin de l'entité — probablement une autre cible
95 + score += val
96 + n += 1
97 + if n == 0:
98 + return 0.0, "neutre"
99 + s = float(np.clip(score / max(n, 1), -1, 1))
100 + stance = "pro" if s > 0.25 else ("anti" if s < -0.25 else ("neutre" if abs(s) < 0.1 else "ambigu"))
101 + return s, stance
102 +
103 +
104 +def llm_score(text: str, entity: str) -> tuple[float, str] | None:
105 + """Stance par LLM (MacLustr) — optionnel; retourne None en cas d'échec."""
106 + if not settings.maclustr_api_key:
107 + return None
108 + import httpx, json
109 + prompt = (
110 + "Tu analyses la couverture politique québécoise. Pour le texte suivant, "
111 + f"détermine la position (stance) envers {settings.party_names.get(entity, entity)}. "
112 + "Attention : sarcasme, négation, citations et attaques dirigées vers un adversaire. "
113 + "Réponds UNIQUEMENT en JSON: {\"stance\": \"pro|anti|neutre|ambigu\", "
114 + "\"sentiment\": nombre entre -1 et 1}.\n\nTexte: " + text[:1500])
115 + try:
116 + r = httpx.post(settings.maclustr_llm_url,
117 + headers={"Authorization": f"Bearer {settings.maclustr_api_key}"},
118 + json={"model": settings.maclustr_llm_model, "temperature": 0,
119 + "messages": [{"role": "user", "content": prompt}],
120 + "max_tokens": 100},
121 + timeout=25)
122 + r.raise_for_status()
123 + content = r.json()["choices"][0]["message"]["content"]
124 + m = re.search(r"\{.*\}", content, re.S)
125 + data = json.loads(m.group())
126 + return float(np.clip(data["sentiment"], -1, 1)), str(data["stance"])
127 + except Exception:
128 + return None
129 +
130 +
131 +def ingest_feeds(db: Session, feeds: list[str] | None = None) -> dict:
132 + import feedparser
133 + feeds = feeds or settings.sentiment_feeds
134 + added, scored = 0, 0
135 + for url in feeds:
136 + src = db.query(Mo.DataSource).filter_by(url=url).first()
137 + if src is None:
138 + src = Mo.DataSource(name=f"RSS {url.split('/')[2]}", url=url, kind="sentiment")
139 + db.add(src); db.flush()
140 + try:
141 + parsed = feedparser.parse(url)
142 + for e in parsed.entries[:60]:
143 + link = getattr(e, "link", None)
144 + title = getattr(e, "title", "") or ""
145 + if not link or not title:
146 + continue
147 + if db.query(Mo.SentimentDocument).filter_by(url=link).first():
148 + continue
149 + summary = re.sub(r"<[^>]+>", " ", getattr(e, "summary", "") or "")[:2000]
150 + pub = None
151 + if getattr(e, "published_parsed", None):
152 + pub = datetime(*e.published_parsed[:6], tzinfo=timezone.utc)
153 + doc = Mo.SentimentDocument(
154 + source=parsed.feed.get("title", url.split("/")[2]),
155 + url=link, title=title, summary=summary,
156 + published=pub, cluster_key=cluster_key(title))
157 + text = f"{title}. {summary}"
158 + entities = detect_entities(text)
159 + for ent in entities:
160 + res = llm_score(text, ent)
161 + method = "llm"
162 + if res is None:
163 + res, method = lexicon_score(text, ent), "lexicon"
164 + s, stance = res
165 + doc.scores.append(Mo.SentimentScore(
166 + entity=ent, sentiment=s, stance=stance, method=method))
167 + scored += 1
168 + db.add(doc)
169 + added += 1
170 + src.last_fetch = datetime.now(timezone.utc)
171 + src.last_status = "ok"
172 + except Exception as e:
173 + src.last_fetch = datetime.now(timezone.utc)
174 + src.last_status = f"error: {e}"[:200]
175 + db.commit()
176 + detect_volume_anomalies(db)
177 + return {"documents_ajoutés": added, "scores": scored}
178 +
179 +
180 +def detect_volume_anomalies(db: Session, z_threshold: float = 2.5) -> int:
181 + """Anomalies de volume médiatique par parti (z-score vs 14 jours) → événements."""
182 + today = date.today()
183 + created = 0
184 + for party in ENTITY_ALIASES:
185 + counts = []
186 + for back in range(15):
187 + d = today - timedelta(days=back)
188 + n = (db.query(Mo.SentimentScore).join(Mo.SentimentDocument)
189 + .filter(Mo.SentimentScore.entity == party,
190 + Mo.SentimentDocument.published >= datetime(d.year, d.month, d.day, tzinfo=timezone.utc),
191 + Mo.SentimentDocument.published < datetime(d.year, d.month, d.day, tzinfo=timezone.utc) + timedelta(days=1))
192 + .count())
193 + counts.append(n)
194 + base = counts[1:]
195 + if len(base) < 7 or np.std(base) == 0:
196 + continue
197 + z = (counts[0] - np.mean(base)) / np.std(base)
198 + if z >= z_threshold and counts[0] >= 5:
199 + exists = (db.query(Mo.NewsEvent)
200 + .filter_by(event_date=today, kind="anomalie-volume").count())
201 + if not exists:
202 + db.add(Mo.NewsEvent(
203 + event_date=today, kind="anomalie-volume",
204 + title=f"Pic de couverture médiatique — {settings.party_names[party]}",
205 + description=f"Volume {counts[0]} vs moyenne {np.mean(base):.1f} (z={z:.1f})",
206 + parties=[party], importance=min(0.9, 0.4 + z / 10),
207 + detected_by="anomaly-detector"))
208 + created += 1
209 + db.commit()
210 + return created
added backend/app/ingest/wikipedia.py +422 −0
@@ -0,0 +1,422 @@
1 +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026
2 +# Auteur : Simon-Pierre Boucher
3 +# Contact : contact@spboucher.ai
4 +# https://www.qc-election.com
5 +"""Ingestion des données électorales depuis Wikipédia (sondages, résultats 2022).
6 +
7 +Chaque exécution sauvegarde une copie brute du HTML dans data/raw/ (provenance)
8 +et produit des CSV normalisés dans data/. Le pipeline recharge ensuite ces CSV
9 +dans la base. Toute valeur conserve : source, URL, date d'accès, valeur brute.
10 +"""
11 +from __future__ import annotations
12 +
13 +import re
14 +import unicodedata
15 +from datetime import date, datetime, timedelta, timezone
16 +from io import StringIO
17 +from pathlib import Path
18 +
19 +import httpx
20 +import pandas as pd
21 +
22 +from ..config import DATA_DIR
23 +
24 +URL_2026 = "https://en.wikipedia.org/wiki/2026_Quebec_general_election"
25 +URL_2022 = "https://en.wikipedia.org/wiki/2022_Quebec_general_election"
26 +UA = {"User-Agent": "Mozilla/5.0 (qc-election-forecast; research/aggregation)"}
27 +
28 +# Renommages carte électorale 2025 (ancienne → nouvelle)
29 +RENAMES_2025 = {
30 + "Johnson": "Daniel-Johnson",
31 + "Arthabaska": "Arthabaska–L'Érable",
32 + "Laporte": "Pierre-Laporte",
33 + "Matane-Matapédia": "Matane-Matapédia-Mitis",
34 + "Rivière-du-Loup–Témiscouata": "Rivière-du-Loup–Témiscouata–Les Basques",
35 + "Vimont": "Vimont-Auteuil",
36 +}
37 +
38 +# Nouvelles circonscriptions 2026 : (nom, région, circonscriptions parentes)
39 +NEW_DISTRICTS_2026 = [
40 + ("Bellefeuille", "Laurentides", ["Saint-Jérôme", "Mirabel", "Argenteuil"]),
41 + ("Marie-Lacoste-Gérin-Lajoie", "Chaudière-Appalaches et Centre-du-Québec",
42 + ["Johnson", "Nicolet-Bécancour", "Drummond–Bois-Francs"]),
43 +]
44 +
45 +# Régions de modélisation (13 groupes — navbox Wikipédia fr, arbitrages documentés
46 +# pour les circonscriptions chevauchant deux régions administratives).
47 +REGION_RIDINGS = {
48 + "Bas-Saint-Laurent–Gaspésie": [
49 + "Bonaventure", "Côte-du-Sud", "Gaspé", "Îles-de-la-Madeleine",
50 + "Matane-Matapédia-Mitis", "Rimouski", "Rivière-du-Loup–Témiscouata–Les Basques"],
51 + "Saguenay–Lac-Saint-Jean et Côte-Nord": [
52 + "Chicoutimi", "Dubuc", "Duplessis", "Jonquière", "Lac-Saint-Jean",
53 + "René-Lévesque", "Roberval"],
54 + "Capitale-Nationale": [
55 + "Charlesbourg", "Charlevoix–Côte-de-Beaupré", "Chauveau", "Jean-Lesage",
56 + "Jean-Talon", "La Peltrie", "Louis-Hébert", "Montmorency", "Portneuf",
57 + "Taschereau", "Vanier-Les Rivières"],
58 + "Mauricie": ["Champlain", "Laviolette–Saint-Maurice", "Maskinongé", "Trois-Rivières"],
59 + "Chaudière-Appalaches et Centre-du-Québec": [
60 + "Arthabaska–L'Érable", "Beauce-Nord", "Beauce-Sud", "Bellechasse",
61 + "Chutes-de-la-Chaudière", "Drummond–Bois-Francs", "Daniel-Johnson", "Lévis",
62 + "Lotbinière-Frontenac", "Marie-Lacoste-Gérin-Lajoie", "Mégantic",
63 + "Nicolet-Bécancour"],
64 + "Estrie": ["Brome-Missisquoi", "Granby", "Orford", "Richmond", "Saint-François",
65 + "Sherbrooke"],
66 + "Montérégie": [
67 + "Beauharnois", "Borduas", "Chambly", "Châteauguay", "Huntingdon", "Iberville",
68 + "La Pinière", "Pierre-Laporte", "La Prairie", "Marie-Victorin", "Montarville",
69 + "Richelieu", "Saint-Hyacinthe", "Saint-Jean", "Sanguinet", "Soulanges",
70 + "Taillon", "Vachon", "Vaudreuil", "Verchères"],
71 + "Montréal": [
72 + "Acadie", "Anjou–Louis-Riel", "Bourassa-Sauvé", "Camille-Laurin",
73 + "D'Arcy-McGee", "Gouin", "Hochelaga-Maisonneuve", "Jacques-Cartier",
74 + "Jeanne-Mance–Viger", "LaFontaine", "Laurier-Dorion", "Marguerite-Bourgeoys",
75 + "Marquette", "Maurice-Richard", "Mercier", "Mont-Royal–Outremont", "Nelligan",
76 + "Notre-Dame-de-Grâce", "Pointe-aux-Trembles", "Robert-Baldwin", "Rosemont",
77 + "Saint-Henri–Sainte-Anne", "Sainte-Marie–Saint-Jacques", "Saint-Laurent",
78 + "Verdun", "Viau", "Westmount–Saint-Louis"],
79 + "Laval": ["Chomedey", "Fabre", "Laval-des-Rapides", "Mille-Îles", "Sainte-Rose",
80 + "Vimont-Auteuil"],
81 + "Lanaudière": ["Berthier", "Joliette", "L'Assomption", "Les Plaines", "Masson",
82 + "Repentigny", "Rousseau", "Terrebonne"],
83 + "Laurentides": ["Argenteuil", "Bellefeuille", "Bertrand", "Blainville",
84 + "Deux-Montagnes", "Groulx", "Labelle", "Mirabel", "Prévost",
85 + "Saint-Jérôme"],
86 + "Outaouais": ["Chapleau", "Gatineau", "Hull", "Papineau", "Pontiac"],
87 + "Abitibi-Témiscamingue et Nord-du-Québec": [
88 + "Abitibi-Est", "Abitibi-Ouest", "Rouyn-Noranda–Témiscamingue", "Ungava"],
89 +}
90 +
91 +POLLSTER_ALIASES = {
92 + "leger": "Léger", "léger": "Léger", "leger marketing": "Léger",
93 + "mainstreet research": "Mainstreet", "mainstreet": "Mainstreet",
94 + "mainstreet (exit poll)": "Mainstreet",
95 + "pallas data": "Pallas Data", "pallas": "Pallas Data",
96 + "research co.": "Research Co.", "research co": "Research Co.",
97 + "angus reid institute": "Angus Reid", "angus reid": "Angus Reid",
98 + "liaison strategies": "Liaison Strategies",
99 + "segma recherche": "SEGMA", "segma": "SEGMA",
100 + "ekos": "EKOS",
101 + "synopsis recherche": "Synopsis", "synopsis": "Synopsis",
102 + "innovative research group": "Innovative Research",
103 + "innovative research": "Innovative Research",
104 +}
105 +
106 +
107 +def norm_name(s: str) -> str:
108 + """Normalise un nom de circonscription (apostrophes, tirets, espaces)."""
109 + s = unicodedata.normalize("NFC", str(s)).strip()
110 + s = s.replace("’", "'").replace("–", "–").replace("—", "–")
111 + s = re.sub(r"\s+", " ", s)
112 + s = re.sub(r"\s*\[.*?\]\s*$", "", s) # notes wiki [a]
113 + return s
114 +
115 +
116 +def _dash_key(s: str) -> str:
117 + return norm_name(s).replace("–", "-").lower()
118 +
119 +
120 +_RENAMES_BY_KEY = {}
121 +
122 +
123 +def canon_district(s: str) -> str:
124 + if not _RENAMES_BY_KEY:
125 + _RENAMES_BY_KEY.update({_dash_key(k): v for k, v in RENAMES_2025.items()})
126 + n = norm_name(s)
127 + return _RENAMES_BY_KEY.get(_dash_key(n), n)
128 +
129 +
130 +def _district_key(s: str) -> str:
131 + return _dash_key(canon_district(s))
132 +
133 +
134 +def norm_pollster(s: str) -> str:
135 + key = norm_name(s).lower()
136 + key = re.sub(r"\s*\(.*\)$", "", key).strip()
137 + return POLLSTER_ALIASES.get(key, norm_name(s))
138 +
139 +
140 +def fetch(url: str, tag: str) -> str:
141 + r = httpx.get(url, headers=UA, timeout=60, follow_redirects=True)
142 + r.raise_for_status()
143 + raw_dir = DATA_DIR / "raw"
144 + raw_dir.mkdir(parents=True, exist_ok=True)
145 + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
146 + (raw_dir / f"{tag}_{stamp}.html").write_text(r.text)
147 + return r.text
148 +
149 +
150 +def _num(x) -> float | None:
151 + if x is None or (isinstance(x, float) and pd.isna(x)):
152 + return None
153 + s = str(x).strip().replace("%", "").replace(",", "")
154 + s = re.sub(r"\[.*?\]", "", s)
155 + m = re.search(r"-?\d+(\.\d+)?", s)
156 + return float(m.group()) if m else None
157 +
158 +
159 +def _parse_date(x, default_year: int | None = None) -> date | None:
160 + from dateutil import parser as dp
161 + s = re.sub(r"\[.*?\]", "", str(x)).strip()
162 + if not s or s.lower() in ("nan", "none"):
163 + return None
164 + # plages "September 30 – October 2, 2022" → prendre la fin
165 + s = re.split(r"[–—]|-(?=\s*[A-Z])", s)[-1].strip()
166 + try:
167 + d = dp.parse(s, default=datetime(default_year or 2000, 6, 15))
168 + return d.date()
169 + except (ValueError, OverflowError):
170 + return None
171 +
172 +
173 +def _flat_cols(df: pd.DataFrame) -> list[str]:
174 + out = []
175 + for c in df.columns:
176 + if isinstance(c, tuple):
177 + parts = [str(p) for p in c if not str(p).startswith("Unnamed")]
178 + out.append(parts[-1] if parts else str(c[0]))
179 + else:
180 + out.append(str(c))
181 + return out
182 +
183 +
184 +PARTY_COL_MAP = {"caq": "CAQ", "qs": "QS", "pq": "PQ", "liberal": "PLQ", "plq": "PLQ",
185 + "pcq": "PCQ", "conservative": "PCQ", "other": "AUT"}
186 +
187 +
188 +def parse_poll_table(df: pd.DataFrame, election_year: int, source_url: str) -> list[dict]:
189 + """Table de sondages Wikipédia → liste de dicts normalisés."""
190 + df = df.copy()
191 + df.columns = _flat_cols(df)
192 + cols = {c.lower().strip(): c for c in df.columns}
193 + org_col = next((cols[k] for k in cols if "polling organisation" in k or k == "firm"), None)
194 + date_col = next((cols[k] for k in cols if "last date" in k or "date" in k), None)
195 + if org_col is None or date_col is None:
196 + return []
197 + party_cols = {}
198 + for k, orig in cols.items():
199 + if k in PARTY_COL_MAP:
200 + party_cols[PARTY_COL_MAP[k]] = orig
201 + rows = []
202 + for _, r in df.iterrows():
203 + org = r.get(org_col)
204 + if org is None or pd.isna(org):
205 + continue
206 + org = str(org)
207 + vals = [str(r.get(c)) for c in df.columns]
208 + if len(set(vals)) <= 2 and len(org) > 40: # rangée-bannière (événement)
209 + continue
210 + if "election" in org.lower(): # rangée résultat officiel
211 + continue
212 + end = _parse_date(r.get(date_col), election_year)
213 + if end is None:
214 + continue
215 + shares = {}
216 + for party, c in party_cols.items():
217 + v = _num(r.get(c))
218 + if v is not None:
219 + shares[party] = v
220 + if len([p for p in shares if p != "AUT"]) < 4:
221 + continue
222 + n = _num(r.get(cols.get("sample size", cols.get("sample", ""))))
223 + moe = _num(r.get(cols.get("moe", "")))
224 + rows.append({
225 + "pollster": norm_pollster(org),
226 + "field_end": end.isoformat(),
227 + "field_start": (end - timedelta(days=3)).isoformat(),
228 + "sample_size": int(n) if n else None,
229 + "moe": abs(moe) if moe else None,
230 + "population": "adults",
231 + "mode": "unknown",
232 + "region": "QC",
233 + "source_name": "Wikipédia (agrégation)",
234 + "source_url": source_url,
235 + "shares": shares,
236 + })
237 + return rows
238 +
239 +
240 +def parse_polls_2026(html: str) -> list[dict]:
241 + tables = pd.read_html(StringIO(html))
242 + out = []
243 + for t in tables:
244 + cols = " ".join(str(c).lower() for c in t.columns)
245 + if "polling organisation" in cols and "caq" in cols and len(t) > 20:
246 + out.extend(parse_poll_table(t, 2026, URL_2026))
247 + return out
248 +
249 +
250 +def parse_polls_2022(html: str) -> list[dict]:
251 + """Tables 'Timeline of opinion polls' de l'article 2022 (campagne + pré-campagne)."""
252 + tables = pd.read_html(StringIO(html))
253 + out = []
254 + for t in tables:
255 + cols = " ".join(str(c).lower() for c in t.columns)
256 + if "timeline of opinion polls" in cols and "caq" in cols:
257 + year = 2022
258 + parsed = parse_poll_table(t, year, URL_2022)
259 + out.extend(parsed)
260 + return out
261 +
262 +
263 +def parse_riding_results_2022(html: str) -> pd.DataFrame:
264 + """Table % par parti et par circonscription (élection 2022)."""
265 + tables = pd.read_html(StringIO(html))
266 + target = None
267 + for t in tables:
268 + raw = [c for c in t.columns]
269 + heads = {str(c[0]) if isinstance(c, tuple) else str(c) for c in raw}
270 + if {"Riding", "CAQ", "QS", "PQ", "PLQ", "PCQ"} <= heads and len(t) > 100:
271 + target = t
272 + break
273 + if target is None:
274 + raise RuntimeError("Table des résultats par circonscription introuvable")
275 + # première sous-colonne '%' et 'Change (pp)' de chaque parti
276 + pct_col: dict[str, object] = {}
277 + chg_col: dict[str, object] = {}
278 + name_col = None
279 + for c in target.columns:
280 + top = str(c[0]) if isinstance(c, tuple) else str(c)
281 + sub = str(c[-1]) if isinstance(c, tuple) else ""
282 + if top == "Riding" and name_col is None:
283 + name_col = c
284 + elif top in ("CAQ", "QS", "PQ", "PLQ", "PCQ") and sub == "%" and top not in pct_col:
285 + pct_col[top] = c
286 + elif (top in ("CAQ", "QS", "PQ", "PLQ", "PCQ") and sub.startswith("Change")
287 + and top not in chg_col):
288 + chg_col[top] = c
289 + rows = []
290 + for _, r in target.iterrows():
291 + name = r.get(name_col)
292 + if name is None or pd.isna(name):
293 + continue
294 + name = norm_name(name)
295 + if name.lower().startswith(("total", "riding", "source")):
296 + continue
297 + rec = {"district": name}
298 + n_num = 0
299 + for p, c in pct_col.items():
300 + v = _num(r.get(c))
301 + if v is not None:
302 + n_num += 1
303 + rec[p] = v if v is not None else 0.0 # parti sans candidat
304 + if n_num < 3: # rangée d'en-tête ou de total
305 + continue
306 + s = sum(rec[p] for p in pct_col)
307 + rec["AUT"] = round(max(0.0, 100.0 - s), 2)
308 + # reconstruction 2018 : % de 2022 − variation (pp)
309 + for p, c in chg_col.items():
310 + chg = _num(r.get(c))
311 + rec[f"prev_{p}"] = round(max(0.0, rec[p] - (chg if chg is not None else rec[p])), 2)
312 + s18 = sum(rec.get(f"prev_{p}", 0.0) for p in pct_col)
313 + rec["prev_AUT"] = round(max(0.0, 100.0 - s18), 2)
314 + rows.append(rec)
315 + return pd.DataFrame(rows)
316 +
317 +
318 +def parse_turnout_2022(html: str) -> dict[str, float]:
319 + tables = pd.read_html(StringIO(html))
320 + for t in tables:
321 + heads = {str(p) for c in t.columns for p in (c if isinstance(c, tuple) else (c,))}
322 + if "Turnout" not in heads or len(t) < 100:
323 + continue
324 + name_col = t.columns[0]
325 + turn_col = next((c for c in t.columns
326 + if isinstance(c, tuple) and "Turnout" in [str(p) for p in c]
327 + and str(c[-1]) == "%"), None)
328 + if turn_col is not None:
329 + out = {}
330 + for _, r in t.iterrows():
331 + n, v = r.get(name_col), _num(r.get(turn_col))
332 + if n is not None and not pd.isna(n) and v is not None and 20 < v < 95:
333 + out[norm_name(n)] = v
334 + if len(out) > 100:
335 + return out
336 + return {}
337 +
338 +
339 +def parse_retirements_2026(html: str) -> set[str]:
340 + """Circonscriptions dont le député sortant ne se représente pas."""
341 + tables = pd.read_html(StringIO(html))
342 + for t in tables:
343 + cols = [str(c) for c in t.columns]
344 + if any("Electoral District" in c for c in cols) and any("Date announced" in c for c in cols):
345 + col = next(c for c in t.columns if "Electoral District" in str(c))
346 + return {canon_district(v) for v in t[col].dropna().astype(str)}
347 + return set()
348 +
349 +
350 +def build_districts(riding_2022: pd.DataFrame, turnout: dict[str, float],
351 + retirements: set[str]) -> pd.DataFrame:
352 + """Construit les 127 circonscriptions 2026 avec référence 2022 (transposition approx.)."""
353 + parties = ["CAQ", "PCQ", "PLQ", "PQ", "QS", "AUT"]
354 + base = {norm_name(r["district"]): {p: r[p] for p in parties}
355 + for _, r in riding_2022.iterrows()}
356 + region_of = {}
357 + for reg, names in REGION_RIDINGS.items():
358 + for n in names:
359 + region_of[_district_key(n)] = reg
360 +
361 + rows = []
362 + for old_name, shares in base.items():
363 + new_name = canon_district(old_name)
364 + reg = region_of.get(_district_key(new_name))
365 + rows.append({
366 + "district": new_name, "region": reg or "Inconnue",
367 + **{f"b_{p}": shares[p] for p in parties},
368 + "turnout_2022": turnout.get(old_name),
369 + "incumbent_party": max((p for p in parties if p != "AUT"),
370 + key=lambda p: shares[p]),
371 + "incumbent_running": new_name not in retirements,
372 + "is_new_2026": False,
373 + "baseline_source": f"Résultat 2022 ({old_name}) — Wikipédia/DGEQ",
374 + })
375 + for name, reg, parents in NEW_DISTRICTS_2026:
376 + ps = [base[p] for p in parents if p in base]
377 + shares = {p: round(sum(x[p] for x in ps) / len(ps), 2) for p in parties}
378 + rows.append({
379 + "district": name, "region": reg,
380 + **{f"b_{p}": shares[p] for p in parties},
381 + "turnout_2022": None,
382 + "incumbent_party": None, "incumbent_running": False, "is_new_2026": True,
383 + "baseline_source": "Moyenne des circonscriptions parentes ("
384 + + ", ".join(parents) + ") — approximation transposition 2025",
385 + })
386 + df = pd.DataFrame(rows)
387 + missing = df[df.region == "Inconnue"]["district"].tolist()
388 + if missing:
389 + raise RuntimeError(f"Régions manquantes pour: {missing}")
390 + return df
391 +
392 +
393 +def refresh_all(out_dir: Path | None = None, offline_html: dict[str, str] | None = None) -> dict:
394 + """Télécharge et normalise toutes les données; écrit les CSV de seed."""
395 + out_dir = out_dir or DATA_DIR
396 + out_dir.mkdir(parents=True, exist_ok=True)
397 + html26 = offline_html.get("2026") if offline_html else fetch(URL_2026, "wiki2026")
398 + html22 = offline_html.get("2022") if offline_html else fetch(URL_2022, "wiki2022")
399 +
400 + polls26 = parse_polls_2026(html26)
401 + polls22 = parse_polls_2022(html22)
402 + riding = parse_riding_results_2022(html22)
403 + turnout = parse_turnout_2022(html22)
404 + retirements = parse_retirements_2026(html26)
405 + districts = build_districts(riding, turnout, retirements)
406 +
407 + accessed = datetime.now(timezone.utc).isoformat()
408 + for polls, tag in ((polls26, "polls_2026"), (polls22, "polls_2022")):
409 + recs = []
410 + for p in polls:
411 + for party, v in p["shares"].items():
412 + recs.append({**{k: p[k] for k in p if k != "shares"},
413 + "party": party, "raw_value": v, "accessed_at": accessed})
414 + pd.DataFrame(recs).to_csv(out_dir / f"{tag}.csv", index=False)
415 + districts.to_csv(out_dir / "districts_2026.csv", index=False)
416 + riding.to_csv(out_dir / "riding_results_2022.csv", index=False)
417 + return {"polls_2026": len(polls26), "polls_2022": len(polls22),
418 + "districts": len(districts), "retirements": len(retirements)}
419 +
420 +
421 +if __name__ == "__main__":
422 + print(refresh_all())
added backend/app/main.py +92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/__init__.py +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/backtest.py +152 −0
Large diff (152 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/compositions.py +61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/explain.py +122 −0
Large diff (122 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/forecast.py +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/house_effects.py +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/simulate.py +183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/trend.py +161 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/modeling/weights.py +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/models.py +238 −0
Large diff (238 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/pipeline.py +205 −0
Large diff (205 lines) is not loaded by default to keep the page fast.Load diff
added backend/app/seed.py +158 −0
Large diff (158 lines) is not loaded by default to keep the page fast.Load diff
added backend/data/backtest_report.json +412 −0
Large diff (413 lines) is not loaded by default to keep the page fast.Load diff
added backend/data/carte_2026.geojson +1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added backend/data/districts_2026.csv +128 −0
Large diff (128 lines) is not loaded by default to keep the page fast.Load diff
added backend/data/polls_2022.csv +911 −0
Large diff (911 lines) is not loaded by default to keep the page fast.Load diff
added backend/data/polls_2026.csv +407 −0
Large diff (407 lines) is not loaded by default to keep the page fast.Load diff
added backend/data/riding_results_2022.csv +126 −0
Large diff (126 lines) is not loaded by default to keep the page fast.Load diff
added backend/requirements.txt +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added backend/tests/__init__.py +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added backend/tests/test_model.py +152 −0
Large diff (152 lines) is not loaded by default to keep the page fast.Load diff
added ecosystem.config.js +34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added frontend/admin.html +177 −0
Large diff (177 lines) is not loaded by default to keep the page fast.Load diff
added frontend/assets/app.js +147 −0
Large diff (147 lines) is not loaded by default to keep the page fast.Load diff
added frontend/assets/img/leader_caq.jpg +0 −0

Binary file not shown.

added frontend/assets/img/leader_pcq.jpg +0 −0

Binary file not shown.

added frontend/assets/img/leader_plq.jpg +0 −0

Binary file not shown.

added frontend/assets/img/leader_pq.jpg +0 −0

Binary file not shown.

added frontend/assets/img/leader_qs1.jpg +0 −0

Binary file not shown.

added frontend/assets/img/leader_qs2.jpg +0 −0

Binary file not shown.

added frontend/assets/img/logo_caq.png +0 −0

Binary file not shown.

added frontend/assets/img/logo_pcq.png +0 −0

Binary file not shown.

added frontend/assets/img/logo_plq.png +0 −0

Binary file not shown.

added frontend/assets/img/logo_pq.png +0 −0

Binary file not shown.

added frontend/assets/img/logo_qs.png +0 −0

Binary file not shown.

added frontend/assets/style.css +266 −0
Large diff (266 lines) is not loaded by default to keep the page fast.Load diff
added frontend/backtest.html +106 −0
Large diff (106 lines) is not loaded by default to keep the page fast.Load diff
added frontend/carte.html +222 −0
Large diff (222 lines) is not loaded by default to keep the page fast.Load diff
added frontend/index.html +241 −0
Large diff (241 lines) is not loaded by default to keep the page fast.Load diff
added frontend/intelligence.html +169 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added frontend/methodologie.html +159 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
added frontend/simulateur.html +210 −0
Large diff (210 lines) is not loaded by default to keep the page fast.Load diff
added frontend/sondages.html +107 −0
Large diff (108 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme