QC Élection Forecast — plateforme de prévision électorale du Québec 2026
Modèle bayésien d'état latent (Kalman/alr) + house effects + Monte Carlo corrélé (erreurs systémique/régionale/locale), backtesting 2022 calibré, carte officielle DGEQ (127 circonscriptions), simulateur what-if, sentiment médiatique et marchés prédictifs en signaux auxiliaires. FastAPI + SQLite, frontend ECharts thèmes sombre/clair, responsive mobile. Déployée sur M2U64 → https://www.qc-election.com (PM2 + ngrok). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
59 changed files +7,265 −0
added
.env.example
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +# QC Élection Forecast — configuration d'environnement | |
| 2 | +# Jeton d'administration (obligatoire en production) | |
| 3 | +ADMIN_TOKEN=changez-moi | |
| 4 | + | |
| 5 | +# Clé du gateway LLM MacLustr (optionnel — scoreur de stance LLM) | |
| 6 | +# MACLUSTR_API_KEY= | |
| 7 | + | |
| 8 | +# Surcharges du modèle (préfixe QCE_) | |
| 9 | +# QCE_PORT=8140 | |
| 10 | +# QCE_N_SIMULATIONS=25000 | |
| 11 | +# QCE_REFRESH_INTERVAL_HOURS=6 | |
added
.gitignore
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +__pycache__/ | |
| 2 | +*.pyc | |
| 3 | +.venv/ | |
| 4 | +.env | |
| 5 | +backend/qc_election.db | |
| 6 | +backend/data/raw/ | |
| 7 | +.pytest_cache/ | |
| 8 | +.DS_Store | |
| 9 | +logs/ | |
added
METHODOLOGY.md
+132 −0
@@ -0,0 +1,132 @@ | ||
| 1 | +<!-- QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 --> | |
| 2 | +<!-- Auteur : Simon-Pierre Boucher --> | |
| 3 | +<!-- Contact : contact@spboucher.ai --> | |
| 4 | +<!-- https://www.qc-election.com --> | |
| 5 | + | |
| 6 | +# Méthodologie — QC Élection Forecast | |
| 7 | + | |
| 8 | +Version du modèle : **1.0.0** · Élection cible : générale québécoise du 5 octobre 2026 | |
| 9 | +(127 circonscriptions, majorité à 64). | |
| 10 | + | |
| 11 | +Ce document est la référence auditable du système. La version grand public est servie sur | |
| 12 | +`/methodologie`; les snapshots de chaque exécution sont conservés en base | |
| 13 | +(`forecast_runs`) et consultables par l'API. | |
| 14 | + | |
| 15 | +## 1. Sources de données et provenance | |
| 16 | + | |
| 17 | +| Source | Contenu | Rôle | | |
| 18 | +|---|---|---| | |
| 19 | +| DGEQ (données ouvertes officielles) | GeoJSON carte 2026 (127 circ.), liste officielle, candidatures (drapeau « député sortant »), résultats en direct le soir du vote | référence géographique, candidats, incumbency | | |
| 20 | +| Wikipédia (agrégation, liens vers chaque maison) | sondages 2022→2026 (85+), sondages campagne 2022 (106), résultats 2022 par circonscription, variations 2018→2022 | série de sondages, baselines locales, backtest | | |
| 21 | +| Polymarket (API Gamma publique) | probabilités implicites « plus de sièges » | **signal auxiliaire affiché, hors modèle** | | |
| 22 | +| Presse québécoise (RSS publics) | Radio-Canada, Le Devoir, La Presse | **sentiment auxiliaire, hors modèle** | | |
| 23 | + | |
| 24 | +Chaque enregistrement conserve : source, URL, date d'accès, valeur brute, valeur normalisée. | |
| 25 | +Les HTML bruts sont archivés dans `backend/data/raw/`. | |
| 26 | + | |
| 27 | +Dédoublonnage des sondages : clé (maison, fin de terrain, taille d'échantillon[, parti]); | |
| 28 | +un même sondage publié à plusieurs endroits n'est compté qu'une fois. | |
| 29 | + | |
| 30 | +## 2. Pondération des sondages | |
| 31 | + | |
| 32 | +- **Récence** : poids `0,5^(âge/12 j)`, l'âge étant mesuré à la **fin du terrain**, pas à la | |
| 33 | + publication. | |
| 34 | +- **Précision** : taille effective `n_eff = min(n/deff, 2500)` avec design effects par mode | |
| 35 | + (web 1,30; téléphone 1,20; IVR 1,40; SMS 1,45; mixte 1,25; inconnu 1,35). Le plafond de | |
| 36 | + 2 500 empêche un mégasondage de dominer. | |
| 37 | +- **Qualité de la maison** : multiplicateur 0,6–1,4 selon l'EAM historique de ses sondages | |
| 38 | + finaux; maison sans historique : ×0,9 + erreur excédentaire accrue (+25 %). | |
| 39 | +- **Erreur totale d'enquête** : ±1,5 pp ajoutés en variance à chaque sondage (cadre, | |
| 40 | + non-réponse, pondérations internes du sondeur). | |
| 41 | + | |
| 42 | +## 3. House effects | |
| 43 | + | |
| 44 | +Sur chaque élection passée à résultat connu, biais moyen par parti des sondages des 21 | |
| 45 | +derniers jours de chaque maison, **rétréci de 50 % vers zéro** et borné à ±3 pp, soustrait | |
| 46 | +avant l'ajustement du modèle. Recalculé à chaque cycle (`pollster_ratings`). | |
| 47 | + | |
| 48 | +## 4. Modèle latent (Bayesian state space) | |
| 49 | + | |
| 50 | +- Composition : 6 composantes (CAQ, PCQ, PLQ, PQ, QS, Autres), transformées en | |
| 51 | + log-ratios additifs `y_k = log(p_k/p_CAQ)` (5 dimensions) — la somme fait toujours 100 % | |
| 52 | + et la corrélation négative entre partis est structurelle. | |
| 53 | +- Dynamique : marche aléatoire gaussienne `x_t = x_{t-1} + w_t`, `w_t ~ N(0, q·I)`. | |
| 54 | +- Observation : `y_sondage ~ N(x_t, R)`, `R` = covariance multinomiale (méthode delta) sur | |
| 55 | + `n_eff` × cote maison + erreur excédentaire diagonale. | |
| 56 | +- Inférence : filtre de Kalman exact + lisseur RTS; `q` estimé par maximum de vraisemblance | |
| 57 | + (grille log-uniforme, borné à [0,004², 0,030²]). | |
| 58 | + | |
| 59 | +## 5. Nowcast vs Forecast | |
| 60 | + | |
| 61 | +- **Nowcast** : distribution de `softmax(x_aujourd'hui)` — « si l'élection avait lieu aujourd'hui ». | |
| 62 | +- **Forecast** : même moyenne (martingale), variance augmentée de | |
| 63 | + `q × jours_restants × 1,6` (volatilité de campagne) **plus l'erreur systémique de | |
| 64 | + sondage** `σ_industrie = 0,12` (voir §7). | |
| 65 | + | |
| 66 | +## 6. Modèle de circonscription | |
| 67 | + | |
| 68 | +`part(circ, parti) = softmax( alr(référence 2022 locale) + swing_national_alr | |
| 69 | ++ ε_région + ε_circonscription − pénalité_retraite )` | |
| 70 | + | |
| 71 | +- **Swing proportionnel** en espace log-ratio (jamais de swing uniforme). | |
| 72 | +- Référence locale : résultat 2022 transposé sur la carte 2026 (renommages appliqués; | |
| 73 | + Bellefeuille et Marie-Lacoste-Gérin-Lajoie = moyenne des circonscriptions parentes). | |
| 74 | +- Pénalité de 0,06 (log-ratio, ≈1,3 pp) au parti sortant si le député ne se représente pas | |
| 75 | + (48 retraits; drapeau raffiné par le registre officiel des candidatures DGEQ). | |
| 76 | +- Partial pooling : les régions sans sondage propre héritent du national + couche régionale. | |
| 77 | + | |
| 78 | +## 7. Monte Carlo et corrélation des erreurs (25 000 simulations) | |
| 79 | + | |
| 80 | +| Couche | Distribution | Justification | | |
| 81 | +|---|---|---| | |
| 82 | +| Opinion | posterior Kalman + dérive | incertitude d'estimation et d'évolution | | |
| 83 | +| **Systémique de sondage** | `N(0, 0,12²)` nationale corrélée | 2022 : toute l'industrie a sous-estimé la CAQ (~3 pp); 2018 idem | | |
| 84 | +| Régionale | `N(0, 0,14²)` × 13 régions | corrélation intrarégionale | | |
| 85 | +| Locale | `N(0, 0,20²)` × 127 circ. | candidats, enjeux locaux | | |
| 86 | + | |
| 87 | +σ régional/local **calibrés sur les résidus du backtest 2022** (swing proportionnel | |
| 88 | +2018→2022), atténués car 2022 (émergence PCQ) fut un réalignement atypique. | |
| 89 | + | |
| 90 | +Sorties : distribution complète des sièges par parti (P5–P95, histogramme), P(plus de | |
| 91 | +sièges), P(majorité ≥64), P(gouvernement minoritaire), catégorie par circonscription | |
| 92 | +(Solide ≥95 %, Probable ≥80 %, Favorisé ≥60 %, sinon Chaudement disputé). | |
| 93 | + | |
| 94 | +## 8. Backtesting (élection 2022, information à date seulement) | |
| 95 | + | |
| 96 | +| Horizon | EAM vote (pp) | Couverture IC 95 % | EAM sièges | Brier | Vainqueurs corrects | | |
| 97 | +|---|---|---|---|---|---| | |
| 98 | +| J−120 | 2,65 | 100 % | 3,1 | 0,147 | 92 % | | |
| 99 | +| J−60 | 2,23 | 100 % | 2,2 | 0,133 | 91 % | | |
| 100 | +| J−30 | 4,15 | 80 % | 4,0 | 0,179 | 88 % | | |
| 101 | +| J−7 | 2,76 | 60 % | 4,2 | 0,178 | 86 % | | |
| 102 | +| J−1 | 2,19 | **100 %** | 4,3 | 0,179 | 86 % | | |
| 103 | + | |
| 104 | +Calibration des probabilités locales (tous horizons confondus) : suit la diagonale | |
| 105 | +(ex. bac 70–80 % → observé 84 %; bac 90–100 % → observé 98 %). L'ajout de la couche | |
| 106 | +systémique a fait passer la couverture J−1 de 20 % à 100 % sans dégrader le Brier. | |
| 107 | +Aucune correction maison n'est appliquée en backtest (pas de données antérieures à 2018) — | |
| 108 | +il mesure le modèle de tendance + simulation seuls. | |
| 109 | + | |
| 110 | +## 9. Signaux auxiliaires (poids nul dans le forecast) | |
| 111 | + | |
| 112 | +- **Sentiment médiatique** : RSS → extraction d'entités → stance (pro/anti/neutre/ambigu) | |
| 113 | + avec négation et attaques dirigées; scoreur LLM optionnel (MacLustr) sinon lexique; | |
| 114 | + regroupement des nouvelles par grappes; anomalies de volume (z ≥ 2,5) → événements. | |
| 115 | +- **Marchés prédictifs** : probabilités implicites Polymarket affichées face au modèle. | |
| 116 | +- **Momentum** : Δ vote modèle 14 j + sentiment 7 j, borné [−100, 100]. | |
| 117 | + | |
| 118 | +## 10. Limites connues | |
| 119 | + | |
| 120 | +1. Pas de sondages régionaux publics structurés — la variation régionale vient de la | |
| 121 | + structure 2022 + couche d'erreur régionale. | |
| 122 | +2. Transposition 2022→2026 approximative pour les circonscriptions dont les limites ont bougé. | |
| 123 | +3. Historique des maisons limité à 2022 (les cotes s'affineront après 2026). | |
| 124 | +4. Participation : outil de scénario (multiplicateurs par électorat), pas un modèle | |
| 125 | + individuel de probabilité de vote. | |
| 126 | +5. `AUT` agrège les tiers partis (PVQ, etc.) — aucun siège projeté pour eux. | |
| 127 | + | |
| 128 | +## 11. Reproductibilité | |
| 129 | + | |
| 130 | +- Chaque run : `forecast_runs` (posterior complet, sièges, diagnostics, version du modèle). | |
| 131 | +- Graines aléatoires fixes; simulations vectorisées NumPy; tests unitaires du cœur | |
| 132 | + statistique (`backend/tests/`). | |
added
README.md
+65 −0
@@ -0,0 +1,65 @@ | ||
| 1 | +<!-- QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 --> | |
| 2 | +<!-- Auteur : Simon-Pierre Boucher --> | |
| 3 | +<!-- Contact : contact@spboucher.ai --> | |
| 4 | +<!-- https://www.qc-election.com --> | |
| 5 | + | |
| 6 | +# QC Élection Forecast | |
| 7 | + | |
| 8 | +Plateforme de **prévision probabiliste** de l'élection générale québécoise du 5 octobre 2026 | |
| 9 | +(127 circonscriptions, majorité à 64). Ce n'est pas un agrégateur de sondages : c'est un moteur | |
| 10 | +statistique complet — modèle d'état latent bayésien, effets maison, simulations Monte Carlo à | |
| 11 | +erreurs corrélées, backtesting et calibration. | |
| 12 | + | |
| 13 | +**Production : https://www.qc-election.com** (nœud M2U64, PM2 + ngrok) | |
| 14 | + | |
| 15 | +## Architecture | |
| 16 | + | |
| 17 | +``` | |
| 18 | +backend/ | |
| 19 | + app/ | |
| 20 | + config.py paramètres (élection, partis, hyperparamètres du modèle) | |
| 21 | + models.py schéma SQLAlchemy (provenance complète de chaque donnée) | |
| 22 | + seed.py chargement CSV → base (idempotent, dédoublonnage) | |
| 23 | + pipeline.py collect → validate → normalize → dedupe → model → simulate → publish | |
| 24 | + ingest/ | |
| 25 | + wikipedia.py sondages 2026 + 2022, résultats 2022 par circonscription, carte 2025 | |
| 26 | + news_rss.py sentiment médiatique (RSS) : entités → stance → anomalies de volume | |
| 27 | + modeling/ | |
| 28 | + compositions.py transformations log-ratio (alr) + covariance multinomiale | |
| 29 | + weights.py pondération : récence, taille effective, design effect, cote maison | |
| 30 | + house_effects.py biais maison rétrécis + cote de qualité dynamique | |
| 31 | + trend.py filtre de Kalman (marche aléatoire en espace alr), q estimé par MV | |
| 32 | + forecast.py nowcast vs forecast (incertitude d'opinion future) | |
| 33 | + simulate.py Monte Carlo 25 000 élections, erreurs nationale/régionales/locales | |
| 34 | + backtest.py rejoue 2022 à J-120…J-1 : MAE, Brier, log loss, couverture, calibration | |
| 35 | + explain.py « Interroger le modèle » — réponses traçables aux données | |
| 36 | + api/ API publique + administration (Bearer token) | |
| 37 | + data/ CSV de seed + copies brutes HTML (provenance) + rapport de backtest | |
| 38 | + tests/ pytest (14 tests du cœur statistique) | |
| 39 | +frontend/ pages statiques premium (ECharts) servies par FastAPI | |
| 40 | +``` | |
| 41 | + | |
| 42 | +## Démarrage local | |
| 43 | + | |
| 44 | +```bash | |
| 45 | +cd backend | |
| 46 | +python3 -m venv .venv && .venv/bin/pip install -r requirements.txt | |
| 47 | +.venv/bin/python -m app.ingest.wikipedia # collecte + CSV de seed | |
| 48 | +.venv/bin/python -m app.seed # base SQLite | |
| 49 | +.venv/bin/python -m app.pipeline # premier forecast | |
| 50 | +.venv/bin/uvicorn app.main:app --port 8140 | |
| 51 | +``` | |
| 52 | + | |
| 53 | +Variables d'environnement (`.env`) : voir `.env.example`. `ADMIN_TOKEN` est requis pour `/admin`. | |
| 54 | + | |
| 55 | +## Tests | |
| 56 | + | |
| 57 | +```bash | |
| 58 | +cd backend && .venv/bin/python -m pytest tests/ -q | |
| 59 | +``` | |
| 60 | + | |
| 61 | +## Méthodologie | |
| 62 | + | |
| 63 | +Documentée en détail dans [METHODOLOGY.md](METHODOLOGY.md) et sur `/methodologie`. | |
| 64 | +Principes : incertitude d'abord, neutralité politique, traçabilité de chaque valeur, | |
| 65 | +calibration avant tout. | |
added
backend/app/__init__.py
+4 −0
@@ -0,0 +1,4 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
added
backend/app/api/__init__.py
+4 −0
@@ -0,0 +1,4 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
added
backend/app/api/admin.py
+142 −0
@@ -0,0 +1,142 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""API d'administration — protégée par jeton (Authorization: Bearer <ADMIN_TOKEN>).""" | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +from datetime import date | |
| 9 | + | |
| 10 | +from fastapi import APIRouter, BackgroundTasks, Depends, Header, HTTPException | |
| 11 | +from pydantic import BaseModel, Field | |
| 12 | +from sqlalchemy.orm import Session | |
| 13 | + | |
| 14 | +from ..config import settings | |
| 15 | +from ..db import get_db | |
| 16 | +from .. import models as Mo | |
| 17 | +from ..pipeline import run_forecast, run_pipeline | |
| 18 | +from ..seed import get_or_create | |
| 19 | + | |
| 20 | +router = APIRouter(prefix="/api/admin") | |
| 21 | + | |
| 22 | + | |
| 23 | +def require_admin(authorization: str = Header(default="")): | |
| 24 | + if authorization != f"Bearer {settings.admin_token}": | |
| 25 | + raise HTTPException(401, "Jeton d'administration invalide") | |
| 26 | + | |
| 27 | + | |
| 28 | +class PollIn(BaseModel): | |
| 29 | + pollster: str | |
| 30 | + field_start: date | None = None | |
| 31 | + field_end: date | |
| 32 | + published: date | None = None | |
| 33 | + sample_size: int | None = Field(None, ge=50, le=100000) | |
| 34 | + population: str = "adults" | |
| 35 | + mode: str = "unknown" | |
| 36 | + moe: float | None = None | |
| 37 | + sponsor: str | None = None | |
| 38 | + source_name: str = "ajout manuel" | |
| 39 | + source_url: str | None = None | |
| 40 | + shares: dict[str, float] # {party: %} | |
| 41 | + | |
| 42 | + | |
| 43 | +@router.post("/poll", dependencies=[Depends(require_admin)]) | |
| 44 | +def add_poll(p: PollIn, db: Session = Depends(get_db)): | |
| 45 | + unknown = [k for k in p.shares if k not in settings.parties] | |
| 46 | + if unknown: | |
| 47 | + raise HTTPException(422, f"Partis inconnus: {unknown}") | |
| 48 | + if not (60 <= sum(p.shares.values()) <= 110): | |
| 49 | + raise HTTPException(422, "La somme des intentions doit être plausible (60–110)") | |
| 50 | + el = db.query(Mo.Election).filter_by(is_target=True).one() | |
| 51 | + pollster, _ = get_or_create(db, Mo.Pollster, name=p.pollster) | |
| 52 | + dupe = (db.query(Mo.Poll).filter_by(pollster_id=pollster.id, field_end=p.field_end, | |
| 53 | + sample_size=p.sample_size).first()) | |
| 54 | + if dupe: | |
| 55 | + raise HTTPException(409, f"Doublon probable du sondage #{dupe.id}") | |
| 56 | + poll = Mo.Poll(election_id=el.id, pollster_id=pollster.id, field_start=p.field_start, | |
| 57 | + field_end=p.field_end, published=p.published, sample_size=p.sample_size, | |
| 58 | + population=p.population, mode=p.mode, moe=p.moe, sponsor=p.sponsor, | |
| 59 | + source_name=p.source_name, source_url=p.source_url) | |
| 60 | + total = sum(p.shares.values()) | |
| 61 | + for party, v in p.shares.items(): | |
| 62 | + poll.results.append(Mo.PollResult(party=party, raw_value=v, | |
| 63 | + normalized_value=round(v * 100 / total, 3))) | |
| 64 | + db.add(poll) | |
| 65 | + db.commit() | |
| 66 | + return {"id": poll.id, "status": "créé"} | |
| 67 | + | |
| 68 | + | |
| 69 | +class PollPatch(BaseModel): | |
| 70 | + excluded: bool | None = None | |
| 71 | + exclusion_reason: str | None = None | |
| 72 | + sample_size: int | None = None | |
| 73 | + mode: str | None = None | |
| 74 | + population: str | None = None | |
| 75 | + | |
| 76 | + | |
| 77 | +@router.patch("/poll/{poll_id}", dependencies=[Depends(require_admin)]) | |
| 78 | +def patch_poll(poll_id: int, patch: PollPatch, db: Session = Depends(get_db)): | |
| 79 | + poll = db.get(Mo.Poll, poll_id) | |
| 80 | + if poll is None: | |
| 81 | + raise HTTPException(404, "Sondage introuvable") | |
| 82 | + for k, v in patch.model_dump(exclude_none=True).items(): | |
| 83 | + setattr(poll, k, v) | |
| 84 | + db.commit() | |
| 85 | + return {"id": poll.id, "status": "modifié"} | |
| 86 | + | |
| 87 | + | |
| 88 | +@router.post("/refresh", dependencies=[Depends(require_admin)]) | |
| 89 | +def refresh(background: BackgroundTasks): | |
| 90 | + background.add_task(run_pipeline, True) | |
| 91 | + return {"status": "pipeline lancé en arrière-plan"} | |
| 92 | + | |
| 93 | + | |
| 94 | +@router.post("/forecast", dependencies=[Depends(require_admin)]) | |
| 95 | +def rerun_forecast(db: Session = Depends(get_db)): | |
| 96 | + run = run_forecast(db) | |
| 97 | + return {"run_id": run.id, "as_of": run.as_of.isoformat()} | |
| 98 | + | |
| 99 | + | |
| 100 | +def _backtest_job(): | |
| 101 | + from ..db import SessionLocal | |
| 102 | + from ..modeling.backtest import run_backtest | |
| 103 | + db = SessionLocal() | |
| 104 | + try: | |
| 105 | + run_backtest(db) | |
| 106 | + finally: | |
| 107 | + db.close() | |
| 108 | + | |
| 109 | + | |
| 110 | +@router.post("/backtest", dependencies=[Depends(require_admin)]) | |
| 111 | +def rerun_backtest(background: BackgroundTasks): | |
| 112 | + background.add_task(_backtest_job) | |
| 113 | + return {"status": "backtest lancé en arrière-plan"} | |
| 114 | + | |
| 115 | + | |
| 116 | +@router.get("/logs", dependencies=[Depends(require_admin)]) | |
| 117 | +def logs(db: Session = Depends(get_db), limit: int = 100): | |
| 118 | + rows = (db.query(Mo.PipelineLog).order_by(Mo.PipelineLog.at.desc()).limit(limit).all()) | |
| 119 | + return [{"at": r.at.isoformat(), "step": r.step, "status": r.status, | |
| 120 | + "message": (r.message or "")[:500]} for r in rows] | |
| 121 | + | |
| 122 | + | |
| 123 | +@router.get("/runs", dependencies=[Depends(require_admin)]) | |
| 124 | +def runs(db: Session = Depends(get_db), limit: int = 50): | |
| 125 | + rows = (db.query(Mo.ForecastRun).order_by(Mo.ForecastRun.run_at.desc()).limit(limit).all()) | |
| 126 | + return [{"id": r.id, "run_at": r.run_at.isoformat(), "as_of": r.as_of.isoformat(), | |
| 127 | + "label": r.label, "is_backtest": r.is_backtest, | |
| 128 | + "n_polls": r.n_polls_used, "n_sims": r.n_simulations, | |
| 129 | + "model_version": r.model_version} for r in rows] | |
| 130 | + | |
| 131 | + | |
| 132 | +@router.get("/sources", dependencies=[Depends(require_admin)]) | |
| 133 | +def sources(db: Session = Depends(get_db)): | |
| 134 | + return [{"name": s.name, "url": s.url, "kind": s.kind, | |
| 135 | + "last_fetch": s.last_fetch.isoformat() if s.last_fetch else None, | |
| 136 | + "last_status": s.last_status} for s in db.query(Mo.DataSource).all()] | |
| 137 | + | |
| 138 | + | |
| 139 | +@router.get("/versions", dependencies=[Depends(require_admin)]) | |
| 140 | +def versions(db: Session = Depends(get_db)): | |
| 141 | + return [{"version": v.version, "released": v.released.isoformat(), | |
| 142 | + "changelog": v.changelog} for v in db.query(Mo.ModelVersion).all()] | |
added
backend/app/api/public.py
+354 −0
@@ -0,0 +1,354 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""API publique — lecture du forecast, sondages, circonscriptions, sentiment.""" | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +import json | |
| 9 | +from datetime import date, datetime, timedelta, timezone | |
| 10 | + | |
| 11 | +import numpy as np | |
| 12 | +from fastapi import APIRouter, Depends, HTTPException, Query | |
| 13 | +from pydantic import BaseModel, Field | |
| 14 | +from sqlalchemy.orm import Session | |
| 15 | + | |
| 16 | +from ..config import DATA_DIR, settings | |
| 17 | +from ..db import get_db | |
| 18 | +from .. import models as Mo | |
| 19 | +from ..modeling import simulate as SIM | |
| 20 | +from ..modeling.explain import compose_answer, gather_facts | |
| 21 | +from ..modeling.house_effects import PollsterProfile, adjust_shares, profile_for | |
| 22 | +from ..modeling.weights import display_weight | |
| 23 | +from ..pipeline import _district_inputs | |
| 24 | + | |
| 25 | +router = APIRouter(prefix="/api") | |
| 26 | + | |
| 27 | + | |
| 28 | +def _latest_run(db: Session) -> Mo.ForecastRun: | |
| 29 | + run = (db.query(Mo.ForecastRun).filter(Mo.ForecastRun.is_backtest.is_(False)) | |
| 30 | + .order_by(Mo.ForecastRun.as_of.desc(), Mo.ForecastRun.id.desc()).first()) | |
| 31 | + if run is None: | |
| 32 | + raise HTTPException(503, "Aucun forecast disponible — lancez le pipeline.") | |
| 33 | + return run | |
| 34 | + | |
| 35 | + | |
| 36 | +@router.get("/meta") | |
| 37 | +def meta(db: Session = Depends(get_db)): | |
| 38 | + target = db.query(Mo.Election).filter_by(is_target=True).first() | |
| 39 | + parties = db.query(Mo.Party).all() | |
| 40 | + last = (db.query(Mo.ForecastRun).filter(Mo.ForecastRun.is_backtest.is_(False)) | |
| 41 | + .order_by(Mo.ForecastRun.run_at.desc()).first()) | |
| 42 | + return { | |
| 43 | + "app": settings.app_name, | |
| 44 | + "election": {"name": target.name if target else settings.election_name, | |
| 45 | + "date": settings.election_date.isoformat(), | |
| 46 | + "total_seats": settings.total_seats, | |
| 47 | + "majority_seats": settings.majority_seats, | |
| 48 | + "days_remaining": (settings.election_date - date.today()).days}, | |
| 49 | + "model_version": settings.model_version, | |
| 50 | + "last_update": last.run_at.isoformat() if last else None, | |
| 51 | + "parties": {p.code: {"name": p.name, "leader": p.leader, "color": p.color} | |
| 52 | + for p in parties}, | |
| 53 | + } | |
| 54 | + | |
| 55 | + | |
| 56 | +@router.get("/forecast/latest") | |
| 57 | +def forecast_latest(db: Session = Depends(get_db)): | |
| 58 | + run = _latest_run(db) | |
| 59 | + return {"run_id": run.id, "as_of": run.as_of.isoformat(), | |
| 60 | + "run_at": run.run_at.isoformat(), "model_version": run.model_version, | |
| 61 | + "n_polls_used": run.n_polls_used, "n_simulations": run.n_simulations, | |
| 62 | + "national": run.national, "seats": run.seats, "diagnostics": run.diagnostics} | |
| 63 | + | |
| 64 | + | |
| 65 | +@router.get("/forecast/history") | |
| 66 | +def forecast_history(db: Session = Depends(get_db)): | |
| 67 | + """Évolution des probabilités et votes à travers les runs (Forecast Explorer).""" | |
| 68 | + runs = (db.query(Mo.ForecastRun).filter(Mo.ForecastRun.is_backtest.is_(False)) | |
| 69 | + .order_by(Mo.ForecastRun.as_of.asc(), Mo.ForecastRun.id.asc()).all()) | |
| 70 | + seen: dict[str, Mo.ForecastRun] = {} | |
| 71 | + for r in runs: | |
| 72 | + seen[r.as_of.isoformat()] = r # dernier run de chaque journée | |
| 73 | + out = [] | |
| 74 | + for iso, r in sorted(seen.items()): | |
| 75 | + out.append({ | |
| 76 | + "as_of": iso, "run_id": r.id, | |
| 77 | + "prob_most": {p: r.seats["per_party"][p]["prob_most"] for p in settings.parties}, | |
| 78 | + "prob_majority": {p: r.seats["per_party"][p]["prob_majority"] for p in settings.parties}, | |
| 79 | + "seats_mean": {p: r.seats["per_party"][p]["mean"] for p in settings.parties}, | |
| 80 | + "vote_mean": {p: r.national["forecast"][p]["mean"] for p in settings.parties}, | |
| 81 | + }) | |
| 82 | + return out | |
| 83 | + | |
| 84 | + | |
| 85 | +@router.get("/forecast/run/{run_id}") | |
| 86 | +def forecast_run(run_id: int, db: Session = Depends(get_db)): | |
| 87 | + run = db.get(Mo.ForecastRun, run_id) | |
| 88 | + if run is None: | |
| 89 | + raise HTTPException(404, "Run introuvable") | |
| 90 | + return {"run_id": run.id, "as_of": run.as_of.isoformat(), "label": run.label, | |
| 91 | + "is_backtest": run.is_backtest, "national": run.national, | |
| 92 | + "seats": run.seats, "diagnostics": run.diagnostics} | |
| 93 | + | |
| 94 | + | |
| 95 | +@router.get("/districts") | |
| 96 | +def districts(db: Session = Depends(get_db)): | |
| 97 | + run = _latest_run(db) | |
| 98 | + static = {d.name: d for d in db.query(Mo.District).all()} | |
| 99 | + out = [] | |
| 100 | + for fr in run.district_results: | |
| 101 | + d = static.get(fr.district_name) | |
| 102 | + out.append({ | |
| 103 | + **fr.detail, | |
| 104 | + "incumbent_party": d.incumbent_party if d else None, | |
| 105 | + "incumbent_running": d.incumbent_running if d else None, | |
| 106 | + "baseline_2022": d.baseline_shares if d else None, | |
| 107 | + "turnout_2022": d.turnout_2022 if d else None, | |
| 108 | + "is_new_2026": d.is_new_2026 if d else False, | |
| 109 | + }) | |
| 110 | + return {"as_of": run.as_of.isoformat(), "run_id": run.id, "districts": out} | |
| 111 | + | |
| 112 | + | |
| 113 | +@router.get("/districts/{name}") | |
| 114 | +def district_detail(name: str, db: Session = Depends(get_db)): | |
| 115 | + d = db.query(Mo.District).filter(Mo.District.name == name).first() | |
| 116 | + if d is None: | |
| 117 | + raise HTTPException(404, "Circonscription introuvable") | |
| 118 | + run = _latest_run(db) | |
| 119 | + fr = next((x for x in run.district_results if x.district_name == name), None) | |
| 120 | + hist = (db.query(Mo.HistoricalDistrictResult) | |
| 121 | + .filter_by(district_name=name).all()) | |
| 122 | + candidates = db.query(Mo.Candidate).filter_by(district_id=d.id).all() | |
| 123 | + return {"name": d.name, "region": d.region, "baseline_2022": d.baseline_shares, | |
| 124 | + "baseline_source": d.baseline_source, "incumbent_party": d.incumbent_party, | |
| 125 | + "incumbent_running": d.incumbent_running, "turnout_2022": d.turnout_2022, | |
| 126 | + "is_new_2026": d.is_new_2026, | |
| 127 | + "forecast": fr.detail if fr else None, | |
| 128 | + "candidates": [{"name": c.name, "party": c.party, "party_full": c.party_full, | |
| 129 | + "is_incumbent": c.is_incumbent} for c in candidates], | |
| 130 | + "history": [{"party": h.party, "pct": h.pct} for h in hist]} | |
| 131 | + | |
| 132 | + | |
| 133 | +@router.get("/geo") | |
| 134 | +def geo(): | |
| 135 | + """Carte électorale officielle 2026 (GeoJSON DGEQ, propriétés minimales).""" | |
| 136 | + f = DATA_DIR / "carte_2026.geojson" | |
| 137 | + if not f.exists(): | |
| 138 | + raise HTTPException(404, "Carte non téléchargée — lancez le pipeline.") | |
| 139 | + from fastapi.responses import FileResponse | |
| 140 | + return FileResponse(f, media_type="application/geo+json", | |
| 141 | + headers={"Cache-Control": "public, max-age=86400"}) | |
| 142 | + | |
| 143 | + | |
| 144 | +@router.get("/markets") | |
| 145 | +def markets(db: Session = Depends(get_db)): | |
| 146 | + """Marchés prédictifs (signal auxiliaire) + comparaison au modèle.""" | |
| 147 | + from ..ingest.markets import fetch_winner_market | |
| 148 | + data = fetch_winner_market() | |
| 149 | + if data is None: | |
| 150 | + raise HTTPException(503, "Marchés indisponibles") | |
| 151 | + try: | |
| 152 | + run = _latest_run(db) | |
| 153 | + data["model_prob_most_seats"] = { | |
| 154 | + p: run.seats["per_party"][p]["prob_most"] | |
| 155 | + for p in settings.parties if p in data["implied_prob_most_seats"]} | |
| 156 | + except HTTPException: | |
| 157 | + pass | |
| 158 | + return data | |
| 159 | + | |
| 160 | + | |
| 161 | +@router.get("/polls") | |
| 162 | +def polls(db: Session = Depends(get_db), | |
| 163 | + date_from: date | None = None, date_to: date | None = None, | |
| 164 | + pollster: str | None = None, election: str = "2026", | |
| 165 | + limit: int = Query(500, le=2000)): | |
| 166 | + el = (db.query(Mo.Election) | |
| 167 | + .filter(Mo.Election.name.contains(election)).first()) | |
| 168 | + q = db.query(Mo.Poll).filter(Mo.Poll.election_id == el.id) if el else db.query(Mo.Poll) | |
| 169 | + if date_from: | |
| 170 | + q = q.filter(Mo.Poll.field_end >= date_from) | |
| 171 | + if date_to: | |
| 172 | + q = q.filter(Mo.Poll.field_end <= date_to) | |
| 173 | + if pollster: | |
| 174 | + q = q.join(Mo.Pollster).filter(Mo.Pollster.name == pollster) | |
| 175 | + rows = q.order_by(Mo.Poll.field_end.desc()).limit(limit).all() | |
| 176 | + | |
| 177 | + ratings = {r.pollster.name: r for r in db.query(Mo.PollsterRating).all()} | |
| 178 | + as_of = date.today() | |
| 179 | + out, max_w = [], 1e-9 | |
| 180 | + for poll in rows: | |
| 181 | + rating = ratings.get(poll.pollster.name) | |
| 182 | + prof = PollsterProfile( | |
| 183 | + name=poll.pollster.name, | |
| 184 | + mae_pp=rating.mae_pp if rating else None, | |
| 185 | + house_effects=rating.house_effects if rating else {}, | |
| 186 | + weight_multiplier=rating.weight_multiplier if rating else 0.9) | |
| 187 | + shares = {r.party: r.normalized_value for r in poll.results} | |
| 188 | + adj = adjust_shares(shares, prof) if shares else {} | |
| 189 | + w = display_weight(poll.field_end, as_of, poll.sample_size, poll.mode, | |
| 190 | + prof.weight_multiplier) | |
| 191 | + max_w = max(max_w, w) | |
| 192 | + out.append({ | |
| 193 | + "id": poll.id, "pollster": poll.pollster.name, "sponsor": poll.sponsor, | |
| 194 | + "field_start": poll.field_start.isoformat() if poll.field_start else None, | |
| 195 | + "field_end": poll.field_end.isoformat(), | |
| 196 | + "sample_size": poll.sample_size, "moe": poll.moe, "mode": poll.mode, | |
| 197 | + "population": poll.population, "source_url": poll.source_url, | |
| 198 | + "source_name": poll.source_name, "excluded": poll.excluded, | |
| 199 | + "raw": {r.party: r.raw_value for r in poll.results}, | |
| 200 | + "normalized": shares, | |
| 201 | + "adjusted": {k: round(v, 1) for k, v in adj.items()}, | |
| 202 | + "weight": w, | |
| 203 | + }) | |
| 204 | + for r in out: | |
| 205 | + r["weight"] = round(r["weight"] / max_w, 4) | |
| 206 | + return out | |
| 207 | + | |
| 208 | + | |
| 209 | +@router.get("/pollsters") | |
| 210 | +def pollsters(db: Session = Depends(get_db)): | |
| 211 | + out = [] | |
| 212 | + for p in db.query(Mo.Pollster).all(): | |
| 213 | + rating = (db.query(Mo.PollsterRating).filter_by(pollster_id=p.id) | |
| 214 | + .order_by(Mo.PollsterRating.computed_at.desc()).first()) | |
| 215 | + n = db.query(Mo.Poll).filter_by(pollster_id=p.id).count() | |
| 216 | + out.append({"name": p.name, "polls_in_db": n, | |
| 217 | + "rating": {"n_final_polls": rating.n_polls, "mae_pp": rating.mae_pp, | |
| 218 | + "house_effects": rating.house_effects, | |
| 219 | + "weight_multiplier": rating.weight_multiplier, | |
| 220 | + "detail": rating.detail} if rating else None}) | |
| 221 | + return sorted(out, key=lambda x: -(x["polls_in_db"])) | |
| 222 | + | |
| 223 | + | |
| 224 | +class WhatIfRequest(BaseModel): | |
| 225 | + national_pp: dict[str, float] = Field(default_factory=dict) | |
| 226 | + regional_pp: dict[str, dict[str, float]] = Field(default_factory=dict) | |
| 227 | + turnout_mult: dict[str, float] = Field(default_factory=dict) | |
| 228 | + poll_error_pp: dict[str, float] = Field(default_factory=dict) | |
| 229 | + n_sims: int = Field(8000, ge=1000, le=40000) | |
| 230 | + | |
| 231 | + | |
| 232 | +@router.post("/whatif") | |
| 233 | +def whatif(req: WhatIfRequest, db: Session = Depends(get_db)): | |
| 234 | + for d in (req.national_pp, req.poll_error_pp): | |
| 235 | + for k, v in d.items(): | |
| 236 | + if abs(v) > 20: | |
| 237 | + raise HTTPException(422, f"Choc trop grand pour {k} (max ±20 pp)") | |
| 238 | + run = _latest_run(db) | |
| 239 | + result_ref = run.national.get("baseline_national") | |
| 240 | + inp = _district_inputs(db, result_ref) | |
| 241 | + inp.x_mean = np.array(run.national["x_forecast"]) | |
| 242 | + inp.P = np.array(run.national["P_forecast"]) | |
| 243 | + scenario = {"national_pp": req.national_pp, "regional_pp": req.regional_pp, | |
| 244 | + "turnout_mult": req.turnout_mult, "poll_error_pp": req.poll_error_pp} | |
| 245 | + sim = SIM.run_simulation(inp, n_sims=req.n_sims, scenario=scenario) | |
| 246 | + return {"base_run_id": run.id, "scenario": scenario, "n_sims": sim.n_sims, | |
| 247 | + "seats": sim.seats, "national_vote": sim.national_vote, | |
| 248 | + "summary": sim.seat_matrix_summary, | |
| 249 | + "districts": sim.districts} | |
| 250 | + | |
| 251 | + | |
| 252 | +@router.get("/backtest") | |
| 253 | +def backtest_report(): | |
| 254 | + f = DATA_DIR / "backtest_report.json" | |
| 255 | + if not f.exists(): | |
| 256 | + raise HTTPException(404, "Backtest non exécuté") | |
| 257 | + return json.loads(f.read_text()) | |
| 258 | + | |
| 259 | + | |
| 260 | +@router.get("/sentiment") | |
| 261 | +def sentiment(db: Session = Depends(get_db), days: int = Query(30, le=120)): | |
| 262 | + cutoff = datetime.now(timezone.utc) - timedelta(days=days) | |
| 263 | + docs = (db.query(Mo.SentimentDocument) | |
| 264 | + .filter(Mo.SentimentDocument.fetched_at >= cutoff).all()) | |
| 265 | + per_party: dict[str, dict] = {} | |
| 266 | + daily: dict[str, dict[str, list]] = {} | |
| 267 | + for doc in docs: | |
| 268 | + d = (doc.published or doc.fetched_at).date().isoformat() | |
| 269 | + for s in doc.scores: | |
| 270 | + pp = per_party.setdefault(s.entity, {"volume": 0, "sum": 0.0, | |
| 271 | + "stances": {"pro": 0, "anti": 0, | |
| 272 | + "neutre": 0, "ambigu": 0}}) | |
| 273 | + pp["volume"] += 1 | |
| 274 | + pp["sum"] += s.sentiment | |
| 275 | + pp["stances"][s.stance] = pp["stances"].get(s.stance, 0) + 1 | |
| 276 | + daily.setdefault(s.entity, {}).setdefault(d, []).append(s.sentiment) | |
| 277 | + | |
| 278 | + now = datetime.now(timezone.utc) | |
| 279 | + def _window(entity, hours): | |
| 280 | + vals = [s.sentiment for doc in docs for s in doc.scores | |
| 281 | + if s.entity == entity and (doc.published or doc.fetched_at) | |
| 282 | + and (now - (doc.published or doc.fetched_at).replace(tzinfo=timezone.utc) | |
| 283 | + ).total_seconds() < hours * 3600] | |
| 284 | + return {"volume": len(vals), | |
| 285 | + "moyenne": round(float(np.mean(vals)), 3) if vals else None} | |
| 286 | + | |
| 287 | + out = {} | |
| 288 | + for party, pp in per_party.items(): | |
| 289 | + out[party] = { | |
| 290 | + "volume": pp["volume"], | |
| 291 | + "sentiment_moyen": round(pp["sum"] / pp["volume"], 3), | |
| 292 | + "stances": pp["stances"], | |
| 293 | + "h24": _window(party, 24), "j7": _window(party, 168), | |
| 294 | + "timeline": [{"date": d, "sentiment": round(float(np.mean(v)), 3), "volume": len(v)} | |
| 295 | + for d, v in sorted(daily.get(party, {}).items())], | |
| 296 | + } | |
| 297 | + n_docs = len(docs) | |
| 298 | + return {"note": ("Signal auxiliaire très bruité — n'entre PAS dans le calcul " | |
| 299 | + "des probabilités électorales."), | |
| 300 | + "documents": n_docs, "parties": out} | |
| 301 | + | |
| 302 | + | |
| 303 | +@router.get("/momentum") | |
| 304 | +def momentum(db: Session = Depends(get_db)): | |
| 305 | + """Indicateur maison (auxiliaire) : tendance sondages + sentiment + volume.""" | |
| 306 | + run = _latest_run(db) | |
| 307 | + series = run.national["trend_series"] | |
| 308 | + sent = sentiment(db=db, days=14) | |
| 309 | + out = {} | |
| 310 | + for p in [x for x in settings.parties if x != "AUT"]: | |
| 311 | + means = series["mean"][p] | |
| 312 | + d14 = means[-1] - means[max(0, len(means) - 15)] | |
| 313 | + s = sent["parties"].get(p, {}) | |
| 314 | + s7 = (s.get("j7") or {}).get("moyenne") or 0.0 | |
| 315 | + score = float(np.clip(d14 * 12 + s7 * 25, -100, 100)) | |
| 316 | + out[p] = {"score": round(score, 1), "d_vote_14j_pp": round(d14, 2), | |
| 317 | + "sentiment_7j": s7, "volume_7j": (s.get("j7") or {}).get("volume", 0)} | |
| 318 | + return {"note": "Indicateur auxiliaire — n'est PAS une probabilité électorale.", | |
| 319 | + "momentum": out} | |
| 320 | + | |
| 321 | + | |
| 322 | +@router.get("/events") | |
| 323 | +def events(db: Session = Depends(get_db)): | |
| 324 | + rows = db.query(Mo.NewsEvent).order_by(Mo.NewsEvent.event_date.asc()).all() | |
| 325 | + return [{"date": e.event_date.isoformat(), "title": e.title, "kind": e.kind, | |
| 326 | + "description": e.description, "parties": e.parties, | |
| 327 | + "importance": e.importance, "detected_by": e.detected_by} for e in rows] | |
| 328 | + | |
| 329 | + | |
| 330 | +@router.get("/news") | |
| 331 | +def news(db: Session = Depends(get_db), limit: int = Query(40, le=200)): | |
| 332 | + """Nouvelles regroupées par grappe (une entrée par nouvelle, pas 50 doublons).""" | |
| 333 | + docs = (db.query(Mo.SentimentDocument) | |
| 334 | + .order_by(Mo.SentimentDocument.fetched_at.desc()).limit(400).all()) | |
| 335 | + clusters: dict[str, dict] = {} | |
| 336 | + for doc in docs: | |
| 337 | + c = clusters.setdefault(doc.cluster_key or doc.url, { | |
| 338 | + "title": doc.title, "url": doc.url, "source": doc.source, | |
| 339 | + "published": (doc.published or doc.fetched_at).isoformat(), | |
| 340 | + "n_articles": 0, "parties": {}}) | |
| 341 | + c["n_articles"] += 1 | |
| 342 | + for s in doc.scores: | |
| 343 | + c["parties"][s.entity] = {"sentiment": s.sentiment, "stance": s.stance} | |
| 344 | + return list(clusters.values())[:limit] | |
| 345 | + | |
| 346 | + | |
| 347 | +class AskRequest(BaseModel): | |
| 348 | + question: str = Field(min_length=3, max_length=500) | |
| 349 | + | |
| 350 | + | |
| 351 | +@router.post("/ask") | |
| 352 | +def ask(req: AskRequest, db: Session = Depends(get_db)): | |
| 353 | + facts = gather_facts(db) | |
| 354 | + return compose_answer(req.question, facts) | |
added
backend/app/config.py
+99 −0
@@ -0,0 +1,99 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""Configuration centrale de la plateforme QC-Élection.""" | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +import os | |
| 9 | +from datetime import date | |
| 10 | +from pathlib import Path | |
| 11 | + | |
| 12 | +from pydantic_settings import BaseSettings | |
| 13 | + | |
| 14 | +BASE_DIR = Path(__file__).resolve().parent.parent # backend/ | |
| 15 | +DATA_DIR = BASE_DIR / "data" | |
| 16 | +FRONTEND_DIR = BASE_DIR.parent / "frontend" | |
| 17 | + | |
| 18 | + | |
| 19 | +class Settings(BaseSettings): | |
| 20 | + # --- Général --- | |
| 21 | + app_name: str = "QC Élection Forecast" | |
| 22 | + model_version: str = "1.0.0" | |
| 23 | + database_url: str = f"sqlite:///{BASE_DIR / 'qc_election.db'}" | |
| 24 | + admin_token: str = os.environ.get("ADMIN_TOKEN", "change-me") | |
| 25 | + port: int = 8140 | |
| 26 | + | |
| 27 | + # --- Élection cible --- | |
| 28 | + election_date: date = date(2026, 10, 5) | |
| 29 | + election_name: str = "Élection générale québécoise de 2026" | |
| 30 | + total_seats: int = 127 | |
| 31 | + majority_seats: int = 64 | |
| 32 | + | |
| 33 | + # --- Partis (ordre alphabétique du sigle — neutralité) --- | |
| 34 | + parties: list[str] = ["CAQ", "PCQ", "PLQ", "PQ", "QS", "AUT"] | |
| 35 | + party_names: dict[str, str] = { | |
| 36 | + "CAQ": "Coalition avenir Québec", | |
| 37 | + "PCQ": "Parti conservateur du Québec", | |
| 38 | + "PLQ": "Parti libéral du Québec", | |
| 39 | + "PQ": "Parti québécois", | |
| 40 | + "QS": "Québec solidaire", | |
| 41 | + "AUT": "Autres", | |
| 42 | + } | |
| 43 | + | |
| 44 | + # --- Modèle : pondération des sondages --- | |
| 45 | + # Décroissance temporelle appliquée en plus de la structure Kalman | |
| 46 | + # (utilisée pour les poids affichés et la variance d'observation). | |
| 47 | + time_decay_halflife_days: float = 12.0 | |
| 48 | + max_effective_n: int = 2500 # plafond anti-domination des très gros sondages | |
| 49 | + design_effects: dict[str, float] = { # deff par mode de collecte | |
| 50 | + "web": 1.30, "ivr": 1.40, "phone": 1.20, "sms": 1.45, "mixed": 1.25, "unknown": 1.35, | |
| 51 | + } | |
| 52 | + # Variance excédentaire (erreur non-échantillonnale) ajoutée à chaque sondage, | |
| 53 | + # en points de pourcentage au carré (≈ (1.5 pp)²) — cf. littérature "total survey error". | |
| 54 | + excess_poll_sd_pp: float = 1.5 | |
| 55 | + | |
| 56 | + # --- Modèle : marche aléatoire latente (espace log-ratio) --- | |
| 57 | + # Écart-type quotidien de la marche aléatoire (estimé + borné). | |
| 58 | + rw_daily_sd_min: float = 0.004 | |
| 59 | + rw_daily_sd_max: float = 0.030 | |
| 60 | + # Inflation d'incertitude pour le forecast (opinion future) : multiplicateur | |
| 61 | + # appliqué à la variance de dérive jusqu'au jour du vote (volatilité de campagne). | |
| 62 | + campaign_drift_inflation: float = 1.6 | |
| 63 | + | |
| 64 | + # --- House effects --- | |
| 65 | + house_effect_shrinkage: float = 0.5 # rétrécissement partiel vers 0 | |
| 66 | + house_effect_max_pp: float = 3.0 # borne de correction (pp) | |
| 67 | + | |
| 68 | + # --- Simulation Monte Carlo --- | |
| 69 | + n_simulations: int = 25000 | |
| 70 | + # Erreur systémique de sondage (« industry error ») : les maisons se trompent | |
| 71 | + # ensemble (2022 : CAQ sous-estimée de ~3 pp par toute l'industrie). Couche | |
| 72 | + # corrélée ajoutée au forecast du jour du vote, calibrée sur 2018/2022. | |
| 73 | + industry_error_sd: float = 0.12 | |
| 74 | + # Écarts-types calibrés sur les résidus du backtest 2022 (swing proportionnel | |
| 75 | + # 2018→2022), atténués car 2022 fut un réalignement atypique (émergence PCQ). | |
| 76 | + regional_error_sd: float = 0.14 # écart-type régional (espace log-ratio) | |
| 77 | + riding_error_sd: float = 0.20 # écart-type propre à la circonscription | |
| 78 | + incumbent_retirement_penalty: float = 0.06 # log-ratio (~1.3 pp) si le sortant ne se représente pas | |
| 79 | + | |
| 80 | + # --- Sentiment (signal auxiliaire, poids nul dans le forecast) --- | |
| 81 | + sentiment_feeds: list[str] = [ | |
| 82 | + "https://ici.radio-canada.ca/rss/92", # Radio-Canada — Politique | |
| 83 | + "https://www.ledevoir.com/rss/section/politique.xml", | |
| 84 | + "https://www.lapresse.ca/actualites/politique/rss", | |
| 85 | + ] | |
| 86 | + maclustr_api_key: str = os.environ.get("MACLUSTR_API_KEY", "") | |
| 87 | + maclustr_llm_url: str = "https://llm.maclustr.io/v1/chat/completions" | |
| 88 | + maclustr_llm_model: str = "maclustr-fast" | |
| 89 | + | |
| 90 | + # --- Pipeline --- | |
| 91 | + refresh_interval_hours: float = 6.0 | |
| 92 | + | |
| 93 | + class Config: | |
| 94 | + env_prefix = "QCE_" | |
| 95 | + env_file = ".env" | |
| 96 | + extra = "ignore" | |
| 97 | + | |
| 98 | + | |
| 99 | +settings = Settings() | |
added
backend/app/db.py
+34 −0
@@ -0,0 +1,34 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +from __future__ import annotations | |
| 6 | + | |
| 7 | +from sqlalchemy import create_engine | |
| 8 | +from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker | |
| 9 | + | |
| 10 | +from .config import settings | |
| 11 | + | |
| 12 | + | |
| 13 | +class Base(DeclarativeBase): | |
| 14 | + pass | |
| 15 | + | |
| 16 | + | |
| 17 | +engine = create_engine( | |
| 18 | + settings.database_url, | |
| 19 | + connect_args={"check_same_thread": False} if settings.database_url.startswith("sqlite") else {}, | |
| 20 | +) | |
| 21 | +SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False) | |
| 22 | + | |
| 23 | + | |
| 24 | +def get_db(): | |
| 25 | + db: Session = SessionLocal() | |
| 26 | + try: | |
| 27 | + yield db | |
| 28 | + finally: | |
| 29 | + db.close() | |
| 30 | + | |
| 31 | + | |
| 32 | +def init_db() -> None: | |
| 33 | + from . import models # noqa: F401 | |
| 34 | + Base.metadata.create_all(engine) | |
added
backend/app/ingest/__init__.py
+4 −0
@@ -0,0 +1,4 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
added
backend/app/ingest/dgeq.py
+144 −0
@@ -0,0 +1,144 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""Ingestion des données ouvertes officielles d'Élections Québec (DGEQ). | |
| 6 | + | |
| 7 | +Sources (https://dgeq.org/donnees.html) : | |
| 8 | + * GeoJSON des 127 circonscriptions 2026 (carte électorale 2025, sans plans d'eau); | |
| 9 | + * liste officielle des circonscriptions 2026; | |
| 10 | + * candidatures officielles (avec drapeau « député sortant »); | |
| 11 | + * résultats en direct le soir du scrutin (resultats.json — provider prêt). | |
| 12 | +""" | |
| 13 | +from __future__ import annotations | |
| 14 | + | |
| 15 | +import json | |
| 16 | +import unicodedata | |
| 17 | +from datetime import datetime, timezone | |
| 18 | + | |
| 19 | +import httpx | |
| 20 | +from sqlalchemy.orm import Session | |
| 21 | + | |
| 22 | +from ..config import DATA_DIR, settings | |
| 23 | +from .. import models as Mo | |
| 24 | + | |
| 25 | +BASE = "https://donnees.electionsquebec.qc.ca" | |
| 26 | +URL_GEOJSON = f"{BASE}/autres/provincial/circonscriptions_electorales_sans_eau_2026.json" | |
| 27 | +URL_LISTE = f"{BASE}/autres/provincial/liste_circonscriptions2026.csv" | |
| 28 | +URL_CANDIDATURES = f"{BASE}/production/provincial/candidatures/candidatures.json" | |
| 29 | +URL_RESULTATS = f"{BASE}/production/provincial/resultats/resultats.json" | |
| 30 | + | |
| 31 | +PARTY_BY_NAME = { | |
| 32 | + "parti québécois": "PQ", | |
| 33 | + "québec solidaire": "QS", | |
| 34 | + "coalition avenir québec": "CAQ", | |
| 35 | + "parti conservateur du québec": "PCQ", | |
| 36 | + "parti libéral du québec": "PLQ", | |
| 37 | +} | |
| 38 | + | |
| 39 | + | |
| 40 | +def _norm(s: str) -> str: | |
| 41 | + """Clé de rapprochement de noms (accents conservés, tirets/apostrophes unifiés).""" | |
| 42 | + s = unicodedata.normalize("NFC", str(s)).strip() | |
| 43 | + return s.replace("–", "-").replace("—", "-").replace("’", "'").lower() | |
| 44 | + | |
| 45 | + | |
| 46 | +def party_code(nom_parti: str | None) -> str: | |
| 47 | + if not nom_parti: | |
| 48 | + return "AUT" | |
| 49 | + n = _norm(nom_parti) | |
| 50 | + for key, code in PARTY_BY_NAME.items(): | |
| 51 | + if key in n: | |
| 52 | + return code | |
| 53 | + return "AUT" | |
| 54 | + | |
| 55 | + | |
| 56 | +def _mark(db: Session, name: str, url: str, status: str) -> None: | |
| 57 | + src = db.query(Mo.DataSource).filter_by(url=url).first() | |
| 58 | + if src is None: | |
| 59 | + src = Mo.DataSource(name=name, url=url, kind="officiel") | |
| 60 | + db.add(src) | |
| 61 | + src.last_fetch = datetime.now(timezone.utc) | |
| 62 | + src.last_status = status[:200] | |
| 63 | + db.commit() | |
| 64 | + | |
| 65 | + | |
| 66 | +def fetch_geojson(db: Session | None = None) -> dict: | |
| 67 | + """Télécharge la carte officielle et l'écrit (propriétés minimales).""" | |
| 68 | + r = httpx.get(URL_GEOJSON, timeout=120) | |
| 69 | + r.raise_for_status() | |
| 70 | + g = r.json() | |
| 71 | + slim = {"type": "FeatureCollection", | |
| 72 | + "features": [{"type": "Feature", | |
| 73 | + "properties": {"name": f["properties"]["NM_CEP"]}, | |
| 74 | + "geometry": f["geometry"]} for f in g["features"]]} | |
| 75 | + out = DATA_DIR / "carte_2026.geojson" | |
| 76 | + out.write_text(json.dumps(slim, ensure_ascii=False)) | |
| 77 | + if db: | |
| 78 | + _mark(db, "DGEQ — carte électorale 2026 (GeoJSON)", URL_GEOJSON, | |
| 79 | + f"ok ({len(slim['features'])} circonscriptions)") | |
| 80 | + return {"features": len(slim["features"]), "path": str(out)} | |
| 81 | + | |
| 82 | + | |
| 83 | +def validate_names(db: Session) -> dict: | |
| 84 | + """Rapproche les noms officiels DGEQ et les circonscriptions en base.""" | |
| 85 | + f = DATA_DIR / "carte_2026.geojson" | |
| 86 | + if not f.exists(): | |
| 87 | + return {"error": "GeoJSON absent"} | |
| 88 | + geo_names = {_norm(x["properties"]["name"]) for x in json.loads(f.read_text())["features"]} | |
| 89 | + db_names = {_norm(d.name): d.name for d in db.query(Mo.District).all()} | |
| 90 | + missing_geo = sorted(n for n in db_names if n not in geo_names) | |
| 91 | + missing_db = sorted(n for n in geo_names if n not in db_names) | |
| 92 | + return {"non_trouvées_dans_geo": missing_geo, "non_trouvées_en_base": missing_db} | |
| 93 | + | |
| 94 | + | |
| 95 | +def fetch_candidatures(db: Session) -> dict: | |
| 96 | + """Candidatures officielles : stockées et utilisées pour raffiner l'incumbency.""" | |
| 97 | + try: | |
| 98 | + r = httpx.get(URL_CANDIDATURES, timeout=60) | |
| 99 | + r.raise_for_status() | |
| 100 | + data = r.json() | |
| 101 | + except Exception as e: | |
| 102 | + _mark(db, "DGEQ — candidatures officielles", URL_CANDIDATURES, f"error: {e}") | |
| 103 | + raise | |
| 104 | + districts = {_norm(d.name): d for d in db.query(Mo.District).all()} | |
| 105 | + db.query(Mo.Candidate).delete() | |
| 106 | + n, sortants = 0, 0 | |
| 107 | + for c in data: | |
| 108 | + dname = c.get("nom_circonscription") | |
| 109 | + if not dname: | |
| 110 | + continue | |
| 111 | + d = districts.get(_norm(dname)) | |
| 112 | + if d is None: | |
| 113 | + continue | |
| 114 | + code = party_code(c.get("nom_parti")) | |
| 115 | + is_inc = c.get("depute_sortant") == "O" | |
| 116 | + db.add(Mo.Candidate( | |
| 117 | + district_id=d.id, party=code, | |
| 118 | + name=f"{c.get('prenom_bulletin_vote') or ''} {c.get('nom_bulletin_vote') or ''}".strip(), | |
| 119 | + party_full=c.get("nom_parti"), is_incumbent=is_inc, | |
| 120 | + source_url=URL_CANDIDATURES)) | |
| 121 | + n += 1 | |
| 122 | + if is_inc: | |
| 123 | + sortants += 1 | |
| 124 | + # confirmation officielle que le sortant se représente | |
| 125 | + if d.incumbent_party == code: | |
| 126 | + d.incumbent_running = True | |
| 127 | + db.commit() | |
| 128 | + _mark(db, "DGEQ — candidatures officielles", URL_CANDIDATURES, | |
| 129 | + f"ok ({n} candidatures, {sortants} sortants)") | |
| 130 | + return {"candidatures": n, "sortants_confirmés": sortants} | |
| 131 | + | |
| 132 | + | |
| 133 | +def refresh_dgeq(db: Session) -> dict: | |
| 134 | + report = {} | |
| 135 | + try: | |
| 136 | + report["geojson"] = fetch_geojson(db) | |
| 137 | + except Exception as e: | |
| 138 | + report["geojson"] = f"échec: {e}" | |
| 139 | + try: | |
| 140 | + report["candidatures"] = fetch_candidatures(db) | |
| 141 | + except Exception as e: | |
| 142 | + report["candidatures"] = f"échec: {e}" | |
| 143 | + report["validation_noms"] = validate_names(db) | |
| 144 | + return report | |
added
backend/app/ingest/markets.py
+67 −0
@@ -0,0 +1,67 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""Marchés prédictifs (Polymarket, API publique Gamma) — signal AUXILIAIRE. | |
| 6 | + | |
| 7 | +Les prix des marchés reflètent l'opinion agrégée de parieurs, pas un modèle | |
| 8 | +statistique : ils sont affichés en regard du forecast pour comparaison, avec | |
| 9 | +provenance, et n'entrent jamais dans le calcul des probabilités du modèle. | |
| 10 | +""" | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import json | |
| 14 | +import time | |
| 15 | +from datetime import datetime, timezone | |
| 16 | + | |
| 17 | +import httpx | |
| 18 | + | |
| 19 | +GAMMA = "https://gamma-api.polymarket.com/events" | |
| 20 | +SLUG_WINNER = "quebec-general-election-winner" | |
| 21 | + | |
| 22 | +PARTY_IN_QUESTION = { | |
| 23 | + "coalition avenir": "CAQ", "parti québécois": "PQ", "parti libéral": "PLQ", | |
| 24 | + "québec solidaire": "QS", "parti conservateur": "PCQ", | |
| 25 | +} | |
| 26 | + | |
| 27 | +_cache: dict = {"at": 0.0, "data": None} | |
| 28 | +CACHE_TTL = 900 # 15 min | |
| 29 | + | |
| 30 | + | |
| 31 | +def fetch_winner_market() -> dict | None: | |
| 32 | + now = time.time() | |
| 33 | + if _cache["data"] is not None and now - _cache["at"] < CACHE_TTL: | |
| 34 | + return _cache["data"] | |
| 35 | + try: | |
| 36 | + r = httpx.get(GAMMA, params={"slug": SLUG_WINNER}, timeout=20) | |
| 37 | + r.raise_for_status() | |
| 38 | + events = r.json() | |
| 39 | + if not events: | |
| 40 | + return None | |
| 41 | + ev = events[0] | |
| 42 | + probs = {} | |
| 43 | + for m in ev.get("markets", []): | |
| 44 | + q = (m.get("question") or "").lower() | |
| 45 | + prices = m.get("outcomePrices") | |
| 46 | + if isinstance(prices, str): | |
| 47 | + prices = json.loads(prices) | |
| 48 | + if not prices: | |
| 49 | + continue | |
| 50 | + for frag, code in PARTY_IN_QUESTION.items(): | |
| 51 | + if frag in q: | |
| 52 | + probs[code] = float(prices[0]) # prix du « Yes » | |
| 53 | + data = { | |
| 54 | + "source": "Polymarket (marché prédictif)", | |
| 55 | + "event": ev.get("title"), | |
| 56 | + "url": f"https://polymarket.com/event/{SLUG_WINNER}", | |
| 57 | + "question": "Quel parti remportera le plus de sièges ?", | |
| 58 | + "implied_prob_most_seats": probs, | |
| 59 | + "volume_usd": round(float(ev.get("volume") or 0)), | |
| 60 | + "fetched_at": datetime.now(timezone.utc).isoformat(), | |
| 61 | + "note": ("Prix implicites d'un marché de paris — signal auxiliaire, " | |
| 62 | + "n'entre pas dans le modèle."), | |
| 63 | + } | |
| 64 | + _cache.update(at=now, data=data) | |
| 65 | + return data | |
| 66 | + except Exception: | |
| 67 | + return _cache["data"] | |
added
backend/app/ingest/news_rss.py
+210 −0
@@ -0,0 +1,210 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""Moteur de sentiment politique — signal AUXILIAIRE, poids nul dans le forecast. | |
| 6 | + | |
| 7 | +Pipeline : document → extraction d'entités (partis, chefs) → stance/sentiment | |
| 8 | +→ clustering (déduplication de la même nouvelle) → agrégats + anomalies de volume. | |
| 9 | + | |
| 10 | +Deux scoreurs : | |
| 11 | + * « lexicon » : lexique français + négation + heuristique de cible | |
| 12 | + (une attaque de X contre Y est négative envers Y, pas envers X); | |
| 13 | + * « llm » (optionnel, si MACLUSTR_API_KEY) : classification de stance par LLM. | |
| 14 | + | |
| 15 | +Le sentiment Web n'est PAS un sondage : il est affiché comme signal bruité et | |
| 16 | +n'entre jamais dans le calcul des probabilités électorales. | |
| 17 | +""" | |
| 18 | +from __future__ import annotations | |
| 19 | + | |
| 20 | +import hashlib | |
| 21 | +import re | |
| 22 | +from datetime import date, datetime, timedelta, timezone | |
| 23 | + | |
| 24 | +import numpy as np | |
| 25 | +from sqlalchemy.orm import Session | |
| 26 | + | |
| 27 | +from ..config import settings | |
| 28 | +from .. import models as Mo | |
| 29 | + | |
| 30 | +ENTITY_ALIASES: dict[str, list[str]] = { | |
| 31 | + "CAQ": ["caq", "coalition avenir québec", "christine fréchette", "fréchette"], | |
| 32 | + "PLQ": ["plq", "parti libéral du québec", "libéral", "libéraux", "charles milliard", | |
| 33 | + "milliard"], | |
| 34 | + "PQ": ["parti québécois", " pq ", "st-pierre plamondon", "plamondon", "pspp", | |
| 35 | + "péquiste"], | |
| 36 | + "QS": ["québec solidaire", " qs ", "ruba ghazal", "sol zanetti", "solidaire"], | |
| 37 | + "PCQ": ["parti conservateur du québec", " pcq ", "éric duhaime", "duhaime", | |
| 38 | + "conservateur du québec"], | |
| 39 | +} | |
| 40 | + | |
| 41 | +POS_WORDS = {"gagne", "succès", "hausse", "progresse", "populaire", "appui", "appuis", | |
| 42 | + "victoire", "monte", "remonte", "convainc", "salue", "félicite", "record", | |
| 43 | + "confiance", "promet", "espoir", "solide", "fort", "avance", "favori", | |
| 44 | + "endosse", "rallie", "applaudit", "réussit", "améliore"} | |
| 45 | +NEG_WORDS = {"scandale", "chute", "recul", "critique", "critiqué", "controverse", | |
| 46 | + "démission", "échec", "perd", "baisse", "colère", "crise", "attaque", | |
| 47 | + "accuse", "accusé", "dérape", "conflit", "corruption", "enquête", | |
| 48 | + "polémique", "déçoit", "impopulaire", "fragile", "faible", "menace", | |
| 49 | + "dénonce", "erreur", "gaffe", "tollé", "démissionne", "poursuite"} | |
| 50 | +NEGATIONS = {"ne", "pas", "jamais", "sans", "aucun", "aucune", "ni", "non"} | |
| 51 | +ATTACK_VERBS = {"accuse", "attaque", "dénonce", "critique", "blâme", "reproche", | |
| 52 | + "fustige", "tacle", "éreinte", "qualifie"} | |
| 53 | + | |
| 54 | +STOP = {"le", "la", "les", "de", "des", "du", "un", "une", "et", "en", "au", "aux", | |
| 55 | + "pour", "sur", "dans", "avec", "que", "qui", "est", "sont", "a", "à", "d", | |
| 56 | + "l", "se", "son", "sa", "ses", "plus", "pas", "ne", "il", "elle", "on"} | |
| 57 | + | |
| 58 | + | |
| 59 | +def _tokens(text: str) -> list[str]: | |
| 60 | + return re.findall(r"[a-zàâäéèêëîïôöùûüç'-]+", text.lower()) | |
| 61 | + | |
| 62 | + | |
| 63 | +def cluster_key(title: str) -> str: | |
| 64 | + words = sorted(w for w in _tokens(title) if w not in STOP and len(w) > 3)[:6] | |
| 65 | + return hashlib.sha1(" ".join(words).encode()).hexdigest()[:16] | |
| 66 | + | |
| 67 | + | |
| 68 | +def detect_entities(text: str) -> list[str]: | |
| 69 | + t = " " + text.lower() + " " | |
| 70 | + return [code for code, aliases in ENTITY_ALIASES.items() | |
| 71 | + if any(a in t for a in aliases)] | |
| 72 | + | |
| 73 | + | |
| 74 | +def lexicon_score(text: str, entity: str) -> tuple[float, str]: | |
| 75 | + """Sentiment [-1,1] + stance dirigés vers l'entité (heuristique de cible).""" | |
| 76 | + toks = _tokens(text) | |
| 77 | + aliases = ENTITY_ALIASES[entity] | |
| 78 | + positions = [i for i, tok in enumerate(toks) | |
| 79 | + if any(a.strip().split()[0] in tok for a in aliases if a.strip())] | |
| 80 | + score, n = 0.0, 0 | |
| 81 | + for i, tok in enumerate(toks): | |
| 82 | + val = 1.0 if tok in POS_WORDS else (-1.0 if tok in NEG_WORDS else 0.0) | |
| 83 | + if val == 0.0: | |
| 84 | + continue | |
| 85 | + if any(toks[j] in NEGATIONS for j in range(max(0, i - 3), i)): | |
| 86 | + val = -val | |
| 87 | + # attaque politique : le sentiment négatif vise l'entité APRÈS le verbe | |
| 88 | + if tok in ATTACK_VERBS: | |
| 89 | + after = detect_entities(" ".join(toks[i:i + 12])) | |
| 90 | + if entity not in after: | |
| 91 | + continue # l'attaque vise quelqu'un d'autre | |
| 92 | + dist = min((abs(i - p) for p in positions), default=99) | |
| 93 | + if dist > 25: | |
| 94 | + continue # trop loin de l'entité — probablement une autre cible | |
| 95 | + score += val | |
| 96 | + n += 1 | |
| 97 | + if n == 0: | |
| 98 | + return 0.0, "neutre" | |
| 99 | + s = float(np.clip(score / max(n, 1), -1, 1)) | |
| 100 | + stance = "pro" if s > 0.25 else ("anti" if s < -0.25 else ("neutre" if abs(s) < 0.1 else "ambigu")) | |
| 101 | + return s, stance | |
| 102 | + | |
| 103 | + | |
| 104 | +def llm_score(text: str, entity: str) -> tuple[float, str] | None: | |
| 105 | + """Stance par LLM (MacLustr) — optionnel; retourne None en cas d'échec.""" | |
| 106 | + if not settings.maclustr_api_key: | |
| 107 | + return None | |
| 108 | + import httpx, json | |
| 109 | + prompt = ( | |
| 110 | + "Tu analyses la couverture politique québécoise. Pour le texte suivant, " | |
| 111 | + f"détermine la position (stance) envers {settings.party_names.get(entity, entity)}. " | |
| 112 | + "Attention : sarcasme, négation, citations et attaques dirigées vers un adversaire. " | |
| 113 | + "Réponds UNIQUEMENT en JSON: {\"stance\": \"pro|anti|neutre|ambigu\", " | |
| 114 | + "\"sentiment\": nombre entre -1 et 1}.\n\nTexte: " + text[:1500]) | |
| 115 | + try: | |
| 116 | + r = httpx.post(settings.maclustr_llm_url, | |
| 117 | + headers={"Authorization": f"Bearer {settings.maclustr_api_key}"}, | |
| 118 | + json={"model": settings.maclustr_llm_model, "temperature": 0, | |
| 119 | + "messages": [{"role": "user", "content": prompt}], | |
| 120 | + "max_tokens": 100}, | |
| 121 | + timeout=25) | |
| 122 | + r.raise_for_status() | |
| 123 | + content = r.json()["choices"][0]["message"]["content"] | |
| 124 | + m = re.search(r"\{.*\}", content, re.S) | |
| 125 | + data = json.loads(m.group()) | |
| 126 | + return float(np.clip(data["sentiment"], -1, 1)), str(data["stance"]) | |
| 127 | + except Exception: | |
| 128 | + return None | |
| 129 | + | |
| 130 | + | |
| 131 | +def ingest_feeds(db: Session, feeds: list[str] | None = None) -> dict: | |
| 132 | + import feedparser | |
| 133 | + feeds = feeds or settings.sentiment_feeds | |
| 134 | + added, scored = 0, 0 | |
| 135 | + for url in feeds: | |
| 136 | + src = db.query(Mo.DataSource).filter_by(url=url).first() | |
| 137 | + if src is None: | |
| 138 | + src = Mo.DataSource(name=f"RSS {url.split('/')[2]}", url=url, kind="sentiment") | |
| 139 | + db.add(src); db.flush() | |
| 140 | + try: | |
| 141 | + parsed = feedparser.parse(url) | |
| 142 | + for e in parsed.entries[:60]: | |
| 143 | + link = getattr(e, "link", None) | |
| 144 | + title = getattr(e, "title", "") or "" | |
| 145 | + if not link or not title: | |
| 146 | + continue | |
| 147 | + if db.query(Mo.SentimentDocument).filter_by(url=link).first(): | |
| 148 | + continue | |
| 149 | + summary = re.sub(r"<[^>]+>", " ", getattr(e, "summary", "") or "")[:2000] | |
| 150 | + pub = None | |
| 151 | + if getattr(e, "published_parsed", None): | |
| 152 | + pub = datetime(*e.published_parsed[:6], tzinfo=timezone.utc) | |
| 153 | + doc = Mo.SentimentDocument( | |
| 154 | + source=parsed.feed.get("title", url.split("/")[2]), | |
| 155 | + url=link, title=title, summary=summary, | |
| 156 | + published=pub, cluster_key=cluster_key(title)) | |
| 157 | + text = f"{title}. {summary}" | |
| 158 | + entities = detect_entities(text) | |
| 159 | + for ent in entities: | |
| 160 | + res = llm_score(text, ent) | |
| 161 | + method = "llm" | |
| 162 | + if res is None: | |
| 163 | + res, method = lexicon_score(text, ent), "lexicon" | |
| 164 | + s, stance = res | |
| 165 | + doc.scores.append(Mo.SentimentScore( | |
| 166 | + entity=ent, sentiment=s, stance=stance, method=method)) | |
| 167 | + scored += 1 | |
| 168 | + db.add(doc) | |
| 169 | + added += 1 | |
| 170 | + src.last_fetch = datetime.now(timezone.utc) | |
| 171 | + src.last_status = "ok" | |
| 172 | + except Exception as e: | |
| 173 | + src.last_fetch = datetime.now(timezone.utc) | |
| 174 | + src.last_status = f"error: {e}"[:200] | |
| 175 | + db.commit() | |
| 176 | + detect_volume_anomalies(db) | |
| 177 | + return {"documents_ajoutés": added, "scores": scored} | |
| 178 | + | |
| 179 | + | |
| 180 | +def detect_volume_anomalies(db: Session, z_threshold: float = 2.5) -> int: | |
| 181 | + """Anomalies de volume médiatique par parti (z-score vs 14 jours) → événements.""" | |
| 182 | + today = date.today() | |
| 183 | + created = 0 | |
| 184 | + for party in ENTITY_ALIASES: | |
| 185 | + counts = [] | |
| 186 | + for back in range(15): | |
| 187 | + d = today - timedelta(days=back) | |
| 188 | + n = (db.query(Mo.SentimentScore).join(Mo.SentimentDocument) | |
| 189 | + .filter(Mo.SentimentScore.entity == party, | |
| 190 | + Mo.SentimentDocument.published >= datetime(d.year, d.month, d.day, tzinfo=timezone.utc), | |
| 191 | + Mo.SentimentDocument.published < datetime(d.year, d.month, d.day, tzinfo=timezone.utc) + timedelta(days=1)) | |
| 192 | + .count()) | |
| 193 | + counts.append(n) | |
| 194 | + base = counts[1:] | |
| 195 | + if len(base) < 7 or np.std(base) == 0: | |
| 196 | + continue | |
| 197 | + z = (counts[0] - np.mean(base)) / np.std(base) | |
| 198 | + if z >= z_threshold and counts[0] >= 5: | |
| 199 | + exists = (db.query(Mo.NewsEvent) | |
| 200 | + .filter_by(event_date=today, kind="anomalie-volume").count()) | |
| 201 | + if not exists: | |
| 202 | + db.add(Mo.NewsEvent( | |
| 203 | + event_date=today, kind="anomalie-volume", | |
| 204 | + title=f"Pic de couverture médiatique — {settings.party_names[party]}", | |
| 205 | + description=f"Volume {counts[0]} vs moyenne {np.mean(base):.1f} (z={z:.1f})", | |
| 206 | + parties=[party], importance=min(0.9, 0.4 + z / 10), | |
| 207 | + detected_by="anomaly-detector")) | |
| 208 | + created += 1 | |
| 209 | + db.commit() | |
| 210 | + return created | |
added
backend/app/ingest/wikipedia.py
+422 −0
@@ -0,0 +1,422 @@ | ||
| 1 | +# QC Élection Forecast — Plateforme de prévision électorale du Québec 2026 | |
| 2 | +# Auteur : Simon-Pierre Boucher | |
| 3 | +# Contact : contact@spboucher.ai | |
| 4 | +# https://www.qc-election.com | |
| 5 | +"""Ingestion des données électorales depuis Wikipédia (sondages, résultats 2022). | |
| 6 | + | |
| 7 | +Chaque exécution sauvegarde une copie brute du HTML dans data/raw/ (provenance) | |
| 8 | +et produit des CSV normalisés dans data/. Le pipeline recharge ensuite ces CSV | |
| 9 | +dans la base. Toute valeur conserve : source, URL, date d'accès, valeur brute. | |
| 10 | +""" | |
| 11 | +from __future__ import annotations | |
| 12 | + | |
| 13 | +import re | |
| 14 | +import unicodedata | |
| 15 | +from datetime import date, datetime, timedelta, timezone | |
| 16 | +from io import StringIO | |
| 17 | +from pathlib import Path | |
| 18 | + | |
| 19 | +import httpx | |
| 20 | +import pandas as pd | |
| 21 | + | |
| 22 | +from ..config import DATA_DIR | |
| 23 | + | |
| 24 | +URL_2026 = "https://en.wikipedia.org/wiki/2026_Quebec_general_election" | |
| 25 | +URL_2022 = "https://en.wikipedia.org/wiki/2022_Quebec_general_election" | |
| 26 | +UA = {"User-Agent": "Mozilla/5.0 (qc-election-forecast; research/aggregation)"} | |
| 27 | + | |
| 28 | +# Renommages carte électorale 2025 (ancienne → nouvelle) | |
| 29 | +RENAMES_2025 = { | |
| 30 | + "Johnson": "Daniel-Johnson", | |
| 31 | + "Arthabaska": "Arthabaska–L'Érable", | |
| 32 | + "Laporte": "Pierre-Laporte", | |
| 33 | + "Matane-Matapédia": "Matane-Matapédia-Mitis", | |
| 34 | + "Rivière-du-Loup–Témiscouata": "Rivière-du-Loup–Témiscouata–Les Basques", | |
| 35 | + "Vimont": "Vimont-Auteuil", | |
| 36 | +} | |
| 37 | + | |
| 38 | +# Nouvelles circonscriptions 2026 : (nom, région, circonscriptions parentes) | |
| 39 | +NEW_DISTRICTS_2026 = [ | |
| 40 | + ("Bellefeuille", "Laurentides", ["Saint-Jérôme", "Mirabel", "Argenteuil"]), | |
| 41 | + ("Marie-Lacoste-Gérin-Lajoie", "Chaudière-Appalaches et Centre-du-Québec", | |
| 42 | + ["Johnson", "Nicolet-Bécancour", "Drummond–Bois-Francs"]), | |
| 43 | +] | |
| 44 | + | |
| 45 | +# Régions de modélisation (13 groupes — navbox Wikipédia fr, arbitrages documentés | |
| 46 | +# pour les circonscriptions chevauchant deux régions administratives). | |
| 47 | +REGION_RIDINGS = { | |
| 48 | + "Bas-Saint-Laurent–Gaspésie": [ | |
| 49 | + "Bonaventure", "Côte-du-Sud", "Gaspé", "Îles-de-la-Madeleine", | |
| 50 | + "Matane-Matapédia-Mitis", "Rimouski", "Rivière-du-Loup–Témiscouata–Les Basques"], | |
| 51 | + "Saguenay–Lac-Saint-Jean et Côte-Nord": [ | |
| 52 | + "Chicoutimi", "Dubuc", "Duplessis", "Jonquière", "Lac-Saint-Jean", | |
| 53 | + "René-Lévesque", "Roberval"], | |
| 54 | + "Capitale-Nationale": [ | |
| 55 | + "Charlesbourg", "Charlevoix–Côte-de-Beaupré", "Chauveau", "Jean-Lesage", | |
| 56 | + "Jean-Talon", "La Peltrie", "Louis-Hébert", "Montmorency", "Portneuf", | |
| 57 | + "Taschereau", "Vanier-Les Rivières"], | |
| 58 | + "Mauricie": ["Champlain", "Laviolette–Saint-Maurice", "Maskinongé", "Trois-Rivières"], | |
| 59 | + "Chaudière-Appalaches et Centre-du-Québec": [ | |
| 60 | + "Arthabaska–L'Érable", "Beauce-Nord", "Beauce-Sud", "Bellechasse", | |
| 61 | + "Chutes-de-la-Chaudière", "Drummond–Bois-Francs", "Daniel-Johnson", "Lévis", | |
| 62 | + "Lotbinière-Frontenac", "Marie-Lacoste-Gérin-Lajoie", "Mégantic", | |
| 63 | + "Nicolet-Bécancour"], | |
| 64 | + "Estrie": ["Brome-Missisquoi", "Granby", "Orford", "Richmond", "Saint-François", | |
| 65 | + "Sherbrooke"], | |
| 66 | + "Montérégie": [ | |
| 67 | + "Beauharnois", "Borduas", "Chambly", "Châteauguay", "Huntingdon", "Iberville", | |
| 68 | + "La Pinière", "Pierre-Laporte", "La Prairie", "Marie-Victorin", "Montarville", | |
| 69 | + "Richelieu", "Saint-Hyacinthe", "Saint-Jean", "Sanguinet", "Soulanges", | |
| 70 | + "Taillon", "Vachon", "Vaudreuil", "Verchères"], | |
| 71 | + "Montréal": [ | |
| 72 | + "Acadie", "Anjou–Louis-Riel", "Bourassa-Sauvé", "Camille-Laurin", | |
| 73 | + "D'Arcy-McGee", "Gouin", "Hochelaga-Maisonneuve", "Jacques-Cartier", | |
| 74 | + "Jeanne-Mance–Viger", "LaFontaine", "Laurier-Dorion", "Marguerite-Bourgeoys", | |
| 75 | + "Marquette", "Maurice-Richard", "Mercier", "Mont-Royal–Outremont", "Nelligan", | |
| 76 | + "Notre-Dame-de-Grâce", "Pointe-aux-Trembles", "Robert-Baldwin", "Rosemont", | |
| 77 | + "Saint-Henri–Sainte-Anne", "Sainte-Marie–Saint-Jacques", "Saint-Laurent", | |
| 78 | + "Verdun", "Viau", "Westmount–Saint-Louis"], | |
| 79 | + "Laval": ["Chomedey", "Fabre", "Laval-des-Rapides", "Mille-Îles", "Sainte-Rose", | |
| 80 | + "Vimont-Auteuil"], | |
| 81 | + "Lanaudière": ["Berthier", "Joliette", "L'Assomption", "Les Plaines", "Masson", | |
| 82 | + "Repentigny", "Rousseau", "Terrebonne"], | |
| 83 | + "Laurentides": ["Argenteuil", "Bellefeuille", "Bertrand", "Blainville", | |
| 84 | + "Deux-Montagnes", "Groulx", "Labelle", "Mirabel", "Prévost", | |
| 85 | + "Saint-Jérôme"], | |
| 86 | + "Outaouais": ["Chapleau", "Gatineau", "Hull", "Papineau", "Pontiac"], | |
| 87 | + "Abitibi-Témiscamingue et Nord-du-Québec": [ | |
| 88 | + "Abitibi-Est", "Abitibi-Ouest", "Rouyn-Noranda–Témiscamingue", "Ungava"], | |
| 89 | +} | |
| 90 | + | |
| 91 | +POLLSTER_ALIASES = { | |
| 92 | + "leger": "Léger", "léger": "Léger", "leger marketing": "Léger", | |
| 93 | + "mainstreet research": "Mainstreet", "mainstreet": "Mainstreet", | |
| 94 | + "mainstreet (exit poll)": "Mainstreet", | |
| 95 | + "pallas data": "Pallas Data", "pallas": "Pallas Data", | |
| 96 | + "research co.": "Research Co.", "research co": "Research Co.", | |
| 97 | + "angus reid institute": "Angus Reid", "angus reid": "Angus Reid", | |
| 98 | + "liaison strategies": "Liaison Strategies", | |
| 99 | + "segma recherche": "SEGMA", "segma": "SEGMA", | |
| 100 | + "ekos": "EKOS", | |
| 101 | + "synopsis recherche": "Synopsis", "synopsis": "Synopsis", | |
| 102 | + "innovative research group": "Innovative Research", | |
| 103 | + "innovative research": "Innovative Research", | |
| 104 | +} | |
| 105 | + | |
| 106 | + | |
| 107 | +def norm_name(s: str) -> str: | |
| 108 | + """Normalise un nom de circonscription (apostrophes, tirets, espaces).""" | |
| 109 | + s = unicodedata.normalize("NFC", str(s)).strip() | |
| 110 | + s = s.replace("’", "'").replace("–", "–").replace("—", "–") | |
| 111 | + s = re.sub(r"\s+", " ", s) | |
| 112 | + s = re.sub(r"\s*\[.*?\]\s*$", "", s) # notes wiki [a] | |
| 113 | + return s | |
| 114 | + | |
| 115 | + | |
| 116 | +def _dash_key(s: str) -> str: | |
| 117 | + return norm_name(s).replace("–", "-").lower() | |
| 118 | + | |
| 119 | + | |
| 120 | +_RENAMES_BY_KEY = {} | |
| 121 | + | |
| 122 | + | |
| 123 | +def canon_district(s: str) -> str: | |
| 124 | + if not _RENAMES_BY_KEY: | |
| 125 | + _RENAMES_BY_KEY.update({_dash_key(k): v for k, v in RENAMES_2025.items()}) | |
| 126 | + n = norm_name(s) | |
| 127 | + return _RENAMES_BY_KEY.get(_dash_key(n), n) | |
| 128 | + | |
| 129 | + | |
| 130 | +def _district_key(s: str) -> str: | |
| 131 | + return _dash_key(canon_district(s)) | |
| 132 | + | |
| 133 | + | |
| 134 | +def norm_pollster(s: str) -> str: | |
| 135 | + key = norm_name(s).lower() | |
| 136 | + key = re.sub(r"\s*\(.*\)$", "", key).strip() | |
| 137 | + return POLLSTER_ALIASES.get(key, norm_name(s)) | |
| 138 | + | |
| 139 | + | |
| 140 | +def fetch(url: str, tag: str) -> str: | |
| 141 | + r = httpx.get(url, headers=UA, timeout=60, follow_redirects=True) | |
| 142 | + r.raise_for_status() | |
| 143 | + raw_dir = DATA_DIR / "raw" | |
| 144 | + raw_dir.mkdir(parents=True, exist_ok=True) | |
| 145 | + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") | |
| 146 | + (raw_dir / f"{tag}_{stamp}.html").write_text(r.text) | |
| 147 | + return r.text | |
| 148 | + | |
| 149 | + | |
| 150 | +def _num(x) -> float | None: | |
| 151 | + if x is None or (isinstance(x, float) and pd.isna(x)): | |
| 152 | + return None | |
| 153 | + s = str(x).strip().replace("%", "").replace(",", "") | |
| 154 | + s = re.sub(r"\[.*?\]", "", s) | |
| 155 | + m = re.search(r"-?\d+(\.\d+)?", s) | |
| 156 | + return float(m.group()) if m else None | |
| 157 | + | |
| 158 | + | |
| 159 | +def _parse_date(x, default_year: int | None = None) -> date | None: | |
| 160 | + from dateutil import parser as dp | |
| 161 | + s = re.sub(r"\[.*?\]", "", str(x)).strip() | |
| 162 | + if not s or s.lower() in ("nan", "none"): | |
| 163 | + return None | |
| 164 | + # plages "September 30 – October 2, 2022" → prendre la fin | |
| 165 | + s = re.split(r"[–—]|-(?=\s*[A-Z])", s)[-1].strip() | |
| 166 | + try: | |
| 167 | + d = dp.parse(s, default=datetime(default_year or 2000, 6, 15)) | |
| 168 | + return d.date() | |
| 169 | + except (ValueError, OverflowError): | |
| 170 | + return None | |
| 171 | + | |
| 172 | + | |
| 173 | +def _flat_cols(df: pd.DataFrame) -> list[str]: | |
| 174 | + out = [] | |
| 175 | + for c in df.columns: | |
| 176 | + if isinstance(c, tuple): | |
| 177 | + parts = [str(p) for p in c if not str(p).startswith("Unnamed")] | |
| 178 | + out.append(parts[-1] if parts else str(c[0])) | |
| 179 | + else: | |
| 180 | + out.append(str(c)) | |
| 181 | + return out | |
| 182 | + | |
| 183 | + | |
| 184 | +PARTY_COL_MAP = {"caq": "CAQ", "qs": "QS", "pq": "PQ", "liberal": "PLQ", "plq": "PLQ", | |
| 185 | + "pcq": "PCQ", "conservative": "PCQ", "other": "AUT"} | |
| 186 | + | |
| 187 | + | |
| 188 | +def parse_poll_table(df: pd.DataFrame, election_year: int, source_url: str) -> list[dict]: | |
| 189 | + """Table de sondages Wikipédia → liste de dicts normalisés.""" | |
| 190 | + df = df.copy() | |
| 191 | + df.columns = _flat_cols(df) | |
| 192 | + cols = {c.lower().strip(): c for c in df.columns} | |
| 193 | + org_col = next((cols[k] for k in cols if "polling organisation" in k or k == "firm"), None) | |
| 194 | + date_col = next((cols[k] for k in cols if "last date" in k or "date" in k), None) | |
| 195 | + if org_col is None or date_col is None: | |
| 196 | + return [] | |
| 197 | + party_cols = {} | |
| 198 | + for k, orig in cols.items(): | |
| 199 | + if k in PARTY_COL_MAP: | |
| 200 | + party_cols[PARTY_COL_MAP[k]] = orig | |
| 201 | + rows = [] | |
| 202 | + for _, r in df.iterrows(): | |
| 203 | + org = r.get(org_col) | |
| 204 | + if org is None or pd.isna(org): | |
| 205 | + continue | |
| 206 | + org = str(org) | |
| 207 | + vals = [str(r.get(c)) for c in df.columns] | |
| 208 | + if len(set(vals)) <= 2 and len(org) > 40: # rangée-bannière (événement) | |
| 209 | + continue | |
| 210 | + if "election" in org.lower(): # rangée résultat officiel | |
| 211 | + continue | |
| 212 | + end = _parse_date(r.get(date_col), election_year) | |
| 213 | + if end is None: | |
| 214 | + continue | |
| 215 | + shares = {} | |
| 216 | + for party, c in party_cols.items(): | |
| 217 | + v = _num(r.get(c)) | |
| 218 | + if v is not None: | |
| 219 | + shares[party] = v | |
| 220 | + if len([p for p in shares if p != "AUT"]) < 4: | |
| 221 | + continue | |
| 222 | + n = _num(r.get(cols.get("sample size", cols.get("sample", "")))) | |
| 223 | + moe = _num(r.get(cols.get("moe", ""))) | |
| 224 | + rows.append({ | |
| 225 | + "pollster": norm_pollster(org), | |
| 226 | + "field_end": end.isoformat(), | |
| 227 | + "field_start": (end - timedelta(days=3)).isoformat(), | |
| 228 | + "sample_size": int(n) if n else None, | |
| 229 | + "moe": abs(moe) if moe else None, | |
| 230 | + "population": "adults", | |
| 231 | + "mode": "unknown", | |
| 232 | + "region": "QC", | |
| 233 | + "source_name": "Wikipédia (agrégation)", | |
| 234 | + "source_url": source_url, | |
| 235 | + "shares": shares, | |
| 236 | + }) | |
| 237 | + return rows | |
| 238 | + | |
| 239 | + | |
| 240 | +def parse_polls_2026(html: str) -> list[dict]: | |
| 241 | + tables = pd.read_html(StringIO(html)) | |
| 242 | + out = [] | |
| 243 | + for t in tables: | |
| 244 | + cols = " ".join(str(c).lower() for c in t.columns) | |
| 245 | + if "polling organisation" in cols and "caq" in cols and len(t) > 20: | |
| 246 | + out.extend(parse_poll_table(t, 2026, URL_2026)) | |
| 247 | + return out | |
| 248 | + | |
| 249 | + | |
| 250 | +def parse_polls_2022(html: str) -> list[dict]: | |
| 251 | + """Tables 'Timeline of opinion polls' de l'article 2022 (campagne + pré-campagne).""" | |
| 252 | + tables = pd.read_html(StringIO(html)) | |
| 253 | + out = [] | |
| 254 | + for t in tables: | |
| 255 | + cols = " ".join(str(c).lower() for c in t.columns) | |
| 256 | + if "timeline of opinion polls" in cols and "caq" in cols: | |
| 257 | + year = 2022 | |
| 258 | + parsed = parse_poll_table(t, year, URL_2022) | |
| 259 | + out.extend(parsed) | |
| 260 | + return out | |
| 261 | + | |
| 262 | + | |
| 263 | +def parse_riding_results_2022(html: str) -> pd.DataFrame: | |
| 264 | + """Table % par parti et par circonscription (élection 2022).""" | |
| 265 | + tables = pd.read_html(StringIO(html)) | |
| 266 | + target = None | |
| 267 | + for t in tables: | |
| 268 | + raw = [c for c in t.columns] | |
| 269 | + heads = {str(c[0]) if isinstance(c, tuple) else str(c) for c in raw} | |
| 270 | + if {"Riding", "CAQ", "QS", "PQ", "PLQ", "PCQ"} <= heads and len(t) > 100: | |
| 271 | + target = t | |
| 272 | + break | |
| 273 | + if target is None: | |
| 274 | + raise RuntimeError("Table des résultats par circonscription introuvable") | |
| 275 | + # première sous-colonne '%' et 'Change (pp)' de chaque parti | |
| 276 | + pct_col: dict[str, object] = {} | |
| 277 | + chg_col: dict[str, object] = {} | |
| 278 | + name_col = None | |
| 279 | + for c in target.columns: | |
| 280 | + top = str(c[0]) if isinstance(c, tuple) else str(c) | |
| 281 | + sub = str(c[-1]) if isinstance(c, tuple) else "" | |
| 282 | + if top == "Riding" and name_col is None: | |
| 283 | + name_col = c | |
| 284 | + elif top in ("CAQ", "QS", "PQ", "PLQ", "PCQ") and sub == "%" and top not in pct_col: | |
| 285 | + pct_col[top] = c | |
| 286 | + elif (top in ("CAQ", "QS", "PQ", "PLQ", "PCQ") and sub.startswith("Change") | |
| 287 | + and top not in chg_col): | |
| 288 | + chg_col[top] = c | |
| 289 | + rows = [] | |
| 290 | + for _, r in target.iterrows(): | |
| 291 | + name = r.get(name_col) | |
| 292 | + if name is None or pd.isna(name): | |
| 293 | + continue | |
| 294 | + name = norm_name(name) | |
| 295 | + if name.lower().startswith(("total", "riding", "source")): | |
| 296 | + continue | |
| 297 | + rec = {"district": name} | |
| 298 | + n_num = 0 | |
| 299 | + for p, c in pct_col.items(): | |
| 300 | + v = _num(r.get(c)) | |
| 301 | + if v is not None: | |
| 302 | + n_num += 1 | |
| 303 | + rec[p] = v if v is not None else 0.0 # parti sans candidat | |
| 304 | + if n_num < 3: # rangée d'en-tête ou de total | |
| 305 | + continue | |
| 306 | + s = sum(rec[p] for p in pct_col) | |
| 307 | + rec["AUT"] = round(max(0.0, 100.0 - s), 2) | |
| 308 | + # reconstruction 2018 : % de 2022 − variation (pp) | |
| 309 | + for p, c in chg_col.items(): | |
| 310 | + chg = _num(r.get(c)) | |
| 311 | + rec[f"prev_{p}"] = round(max(0.0, rec[p] - (chg if chg is not None else rec[p])), 2) | |
| 312 | + s18 = sum(rec.get(f"prev_{p}", 0.0) for p in pct_col) | |
| 313 | + rec["prev_AUT"] = round(max(0.0, 100.0 - s18), 2) | |
| 314 | + rows.append(rec) | |
| 315 | + return pd.DataFrame(rows) | |
| 316 | + | |
| 317 | + | |
| 318 | +def parse_turnout_2022(html: str) -> dict[str, float]: | |
| 319 | + tables = pd.read_html(StringIO(html)) | |
| 320 | + for t in tables: | |
| 321 | + heads = {str(p) for c in t.columns for p in (c if isinstance(c, tuple) else (c,))} | |
| 322 | + if "Turnout" not in heads or len(t) < 100: | |
| 323 | + continue | |
| 324 | + name_col = t.columns[0] | |
| 325 | + turn_col = next((c for c in t.columns | |
| 326 | + if isinstance(c, tuple) and "Turnout" in [str(p) for p in c] | |
| 327 | + and str(c[-1]) == "%"), None) | |
| 328 | + if turn_col is not None: | |
| 329 | + out = {} | |
| 330 | + for _, r in t.iterrows(): | |
| 331 | + n, v = r.get(name_col), _num(r.get(turn_col)) | |
| 332 | + if n is not None and not pd.isna(n) and v is not None and 20 < v < 95: | |
| 333 | + out[norm_name(n)] = v | |
| 334 | + if len(out) > 100: | |
| 335 | + return out | |
| 336 | + return {} | |
| 337 | + | |
| 338 | + | |
| 339 | +def parse_retirements_2026(html: str) -> set[str]: | |
| 340 | + """Circonscriptions dont le député sortant ne se représente pas.""" | |
| 341 | + tables = pd.read_html(StringIO(html)) | |
| 342 | + for t in tables: | |
| 343 | + cols = [str(c) for c in t.columns] | |
| 344 | + if any("Electoral District" in c for c in cols) and any("Date announced" in c for c in cols): | |
| 345 | + col = next(c for c in t.columns if "Electoral District" in str(c)) | |
| 346 | + return {canon_district(v) for v in t[col].dropna().astype(str)} | |
| 347 | + return set() | |
| 348 | + | |
| 349 | + | |
| 350 | +def build_districts(riding_2022: pd.DataFrame, turnout: dict[str, float], | |
| 351 | + retirements: set[str]) -> pd.DataFrame: | |
| 352 | + """Construit les 127 circonscriptions 2026 avec référence 2022 (transposition approx.).""" | |
| 353 | + parties = ["CAQ", "PCQ", "PLQ", "PQ", "QS", "AUT"] | |
| 354 | + base = {norm_name(r["district"]): {p: r[p] for p in parties} | |
| 355 | + for _, r in riding_2022.iterrows()} | |
| 356 | + region_of = {} | |
| 357 | + for reg, names in REGION_RIDINGS.items(): | |
| 358 | + for n in names: | |
| 359 | + region_of[_district_key(n)] = reg | |
| 360 | + | |
| 361 | + rows = [] | |
| 362 | + for old_name, shares in base.items(): | |
| 363 | + new_name = canon_district(old_name) | |
| 364 | + reg = region_of.get(_district_key(new_name)) | |
| 365 | + rows.append({ | |
| 366 | + "district": new_name, "region": reg or "Inconnue", | |
| 367 | + **{f"b_{p}": shares[p] for p in parties}, | |
| 368 | + "turnout_2022": turnout.get(old_name), | |
| 369 | + "incumbent_party": max((p for p in parties if p != "AUT"), | |
| 370 | + key=lambda p: shares[p]), | |
| 371 | + "incumbent_running": new_name not in retirements, | |
| 372 | + "is_new_2026": False, | |
| 373 | + "baseline_source": f"Résultat 2022 ({old_name}) — Wikipédia/DGEQ", | |
| 374 | + }) | |
| 375 | + for name, reg, parents in NEW_DISTRICTS_2026: | |
| 376 | + ps = [base[p] for p in parents if p in base] | |
| 377 | + shares = {p: round(sum(x[p] for x in ps) / len(ps), 2) for p in parties} | |
| 378 | + rows.append({ | |
| 379 | + "district": name, "region": reg, | |
| 380 | + **{f"b_{p}": shares[p] for p in parties}, | |
| 381 | + "turnout_2022": None, | |
| 382 | + "incumbent_party": None, "incumbent_running": False, "is_new_2026": True, | |
| 383 | + "baseline_source": "Moyenne des circonscriptions parentes (" | |
| 384 | + + ", ".join(parents) + ") — approximation transposition 2025", | |
| 385 | + }) | |
| 386 | + df = pd.DataFrame(rows) | |
| 387 | + missing = df[df.region == "Inconnue"]["district"].tolist() | |
| 388 | + if missing: | |
| 389 | + raise RuntimeError(f"Régions manquantes pour: {missing}") | |
| 390 | + return df | |
| 391 | + | |
| 392 | + | |
| 393 | +def refresh_all(out_dir: Path | None = None, offline_html: dict[str, str] | None = None) -> dict: | |
| 394 | + """Télécharge et normalise toutes les données; écrit les CSV de seed.""" | |
| 395 | + out_dir = out_dir or DATA_DIR | |
| 396 | + out_dir.mkdir(parents=True, exist_ok=True) | |
| 397 | + html26 = offline_html.get("2026") if offline_html else fetch(URL_2026, "wiki2026") | |
| 398 | + html22 = offline_html.get("2022") if offline_html else fetch(URL_2022, "wiki2022") | |
| 399 | + | |
| 400 | + polls26 = parse_polls_2026(html26) | |
| 401 | + polls22 = parse_polls_2022(html22) | |
| 402 | + riding = parse_riding_results_2022(html22) | |
| 403 | + turnout = parse_turnout_2022(html22) | |
| 404 | + retirements = parse_retirements_2026(html26) | |
| 405 | + districts = build_districts(riding, turnout, retirements) | |
| 406 | + | |
| 407 | + accessed = datetime.now(timezone.utc).isoformat() | |
| 408 | + for polls, tag in ((polls26, "polls_2026"), (polls22, "polls_2022")): | |
| 409 | + recs = [] | |
| 410 | + for p in polls: | |
| 411 | + for party, v in p["shares"].items(): | |
| 412 | + recs.append({**{k: p[k] for k in p if k != "shares"}, | |
| 413 | + "party": party, "raw_value": v, "accessed_at": accessed}) | |
| 414 | + pd.DataFrame(recs).to_csv(out_dir / f"{tag}.csv", index=False) | |
| 415 | + districts.to_csv(out_dir / "districts_2026.csv", index=False) | |
| 416 | + riding.to_csv(out_dir / "riding_results_2022.csv", index=False) | |
| 417 | + return {"polls_2026": len(polls26), "polls_2022": len(polls22), | |
| 418 | + "districts": len(districts), "retirements": len(retirements)} | |
| 419 | + | |
| 420 | + | |
| 421 | +if __name__ == "__main__": | |
| 422 | + print(refresh_all()) | |
added
backend/app/main.py
+92 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/__init__.py
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/backtest.py
+152 −0
Large diff (152 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/compositions.py
+61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/explain.py
+122 −0
Large diff (122 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/forecast.py
+69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/house_effects.py
+89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/simulate.py
+183 −0
Large diff (183 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/trend.py
+161 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/modeling/weights.py
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/models.py
+238 −0
Large diff (238 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/pipeline.py
+205 −0
Large diff (205 lines) is not loaded by default to keep the page fast.Load diff
added
backend/app/seed.py
+158 −0
Large diff (158 lines) is not loaded by default to keep the page fast.Load diff
added
backend/data/backtest_report.json
+412 −0
Large diff (413 lines) is not loaded by default to keep the page fast.Load diff
added
backend/data/carte_2026.geojson
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
backend/data/districts_2026.csv
+128 −0
Large diff (128 lines) is not loaded by default to keep the page fast.Load diff
added
backend/data/polls_2022.csv
+911 −0
Large diff (911 lines) is not loaded by default to keep the page fast.Load diff
added
backend/data/polls_2026.csv
+407 −0
Large diff (407 lines) is not loaded by default to keep the page fast.Load diff
added
backend/data/riding_results_2022.csv
+126 −0
Large diff (126 lines) is not loaded by default to keep the page fast.Load diff
added
backend/requirements.txt
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
backend/tests/__init__.py
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
backend/tests/test_model.py
+152 −0
Large diff (152 lines) is not loaded by default to keep the page fast.Load diff
added
ecosystem.config.js
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/admin.html
+177 −0
Large diff (177 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/assets/app.js
+147 −0
Large diff (147 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/assets/img/leader_caq.jpg
+0 −0
Binary file not shown.
added
frontend/assets/img/leader_pcq.jpg
+0 −0
Binary file not shown.
added
frontend/assets/img/leader_plq.jpg
+0 −0
Binary file not shown.
added
frontend/assets/img/leader_pq.jpg
+0 −0
Binary file not shown.
added
frontend/assets/img/leader_qs1.jpg
+0 −0
Binary file not shown.
added
frontend/assets/img/leader_qs2.jpg
+0 −0
Binary file not shown.
added
frontend/assets/img/logo_caq.png
+0 −0
Binary file not shown.
added
frontend/assets/img/logo_pcq.png
+0 −0
Binary file not shown.
added
frontend/assets/img/logo_plq.png
+0 −0
Binary file not shown.
added
frontend/assets/img/logo_pq.png
+0 −0
Binary file not shown.
added
frontend/assets/img/logo_qs.png
+0 −0
Binary file not shown.
added
frontend/assets/style.css
+266 −0
Large diff (266 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/backtest.html
+106 −0
Large diff (106 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/carte.html
+222 −0
Large diff (222 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/index.html
+241 −0
Large diff (241 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/intelligence.html
+169 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/methodologie.html
+159 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/simulateur.html
+210 −0
Large diff (210 lines) is not loaded by default to keep the page fast.Load diff
added
frontend/sondages.html
+107 −0
Large diff (108 lines) is not loaded by default to keep the page fast.Load diff