Repository Public License
1# Geospatial engine (`geo`) — compatibility23Package `Packages/EngineGeo`, module `EngineGeo`, factory `makeEngines()`. Pure Swift plus two system libraries4(SQLite for GeoPackage, Compression for KMZ / zipped shapefiles). **No GDAL and no PROJ**: both are large, and PROJ5depends on data files (grids, the EPSG database) that would have to be shipped and kept current. The price is a6deliberately small set of coordinate transformations, listed below; everything else is carried or refused, never7guessed.89State on 2026-11-21 (macOS 27.0, arm64): **30 declared paths, 53 tests, all passing**10(`swift test`: `Test run with 53 tests in 11 suites passed`; two of the 53 are development aids that only run when11an environment variable is set).1213## Declared paths1415Every ordered pair among GeoJSON, KML, GPX, CSV, Shapefile and GeoPackage: 6 × 5 = 30 paths, all `supported`, no16named operation. Test for all of them: `PathTests.convertsCities(_:)` (30 cases — a hand-written three-city fixture17per source format, exact coordinates and attribute values checked on the re-read output, every validation check18required to pass).1920| Destination ↓ / fidelity, preserves | From GeoJSON, CSV, Shapefile, GeoPackage, GPX | From KML |21|---|---|---|22| GeoJSON | structural; geometry, attributes | partial (styles dropped) |23| KML | structural; geometry, attributes | — |24| GPX | **partial**; geometry only | partial |25| CSV | structural; geometry, attributes | partial |26| Shapefile | structural; geometry, attributes, coordinate system | partial |27| GeoPackage | structural; geometry, attributes, coordinate system | partial |2829Limitations shown to the user (from `ConversionPath.limitations`):3031| Applies to | Limitation |32|---|---|33| source KML | styles, icons, overlays, tours, 3D models and network links are not carried over; network links are never fetched |34| source GPX | per-point timestamps and sensor data of tracks are dropped unless “Track points as features” is on |35| source GeoPackage | only vector layers; tiles, rasters and BLOB attributes are reported and left out; the feature id column is not exported |36| source Shapefile | without a `.prj` the reference system must be stated |37| source CSV | no reference system in the format: WGS 84 is assumed only for columns named like longitude / latitude |38| destination GeoJSON, KML | always WGS 84 lon/lat: supported systems are reprojected, others refused; measures (M) dropped; KML dates become text |39| destination GPX | always WGS 84; points and lines only (polygons refused unless written as closed tracks); only the fixed GPX fields are kept |40| destination Shapefile | delivered as a folder (`.shp`, `.shx`, `.dbf`, `.cpg`, `.prj`); dBASE limits; one geometry type per file |41| destination CSV | cannot record the reference system; coordinate columns for points, WKT otherwise |42| multi-layer source → GeoJSON, CSV, Shapefile | one output per layer |4344### Formats: what is read and written4546| Format | Reading | Writing |47|---|---|---|48| GeoJSON | RFC 7946 FeatureCollection / Feature / bare geometry; newline-delimited GeoJSON and RFC 8142 sequences (detected from content); property order and integer / real distinction kept; nested values kept as JSON text; `id` kept as an `id` attribute; legacy `crs` member honoured and flagged | one FeatureCollection, one feature per line, polygons following the right-hand rule, shortest round-trip numbers |49| KML | KML 2.2; folders → layers (`Folder/Subfolder`); name, description, ExtendedData `Data` and typed `SchemaData`, TimeStamp / TimeSpan; Point, LineString, LinearRing, Polygon with holes, MultiGeometry, gx:Track / gx:MultiTrack (as lines); **KMZ**: `doc.kml` (or the shallowest `.kml`) through the engine's bounded ZIP reader | one Document, one Folder per layer, a typed `Schema` per layer, `<name>` / `<description>` from the attributes chosen in the options |50| GPX | 1.0 and 1.1; waypoints, routes, tracks → three layers; `ele` → Z; the fixed fields; leaf elements of `<extensions>` as attributes; optional `track_points` / `route_points` layers | GPX 1.1: points → waypoints, lines → tracks (or routes), multi-part lines → multi-segment tracks |51| CSV | RFC 4180, delimiter detected among `, ; tab |`; UTF-8 (Windows-1252 fallback, reported); geometry from longitude/latitude or x/y columns (by name, checked against the range of degrees) or a WKT column; explicit column options; per-column typing (text with leading zeros stays text) | `longitude,latitude[,elevation]` (geographic) or `x,y[,z]` for point layers, a `WKT` column otherwise; CRLF |52| Shapefile | `.shp` + `.dbf` + `.prj` + `.cpg`, siblings found case-insensitively; a `.zip` holding one or several shapefiles; Point, MultiPoint, PolyLine, Polygon and their Z and M variants, null shapes; holes assigned to the outer ring that contains them; dBASE C / N / F / L / D; code page from the option, then `.cpg`, then the language driver id, then a reported guess | all of the above; `.shx` index; outer rings clockwise, holes counter-clockwise; `.cpg` = UTF-8; `.prj` in Esri flavour |53| GeoPackage | `gpkg_contents` feature and attribute tables; `gpkg_geometry_columns`; `gpkg_spatial_ref_sys`; GeoPackageBinary header (both byte orders, every envelope form) + ISO WKB (Z, M, ZM; EWKB flags tolerated); column types incl. BOOLEAN, DATE, DATETIME | `application_id` GPKG, `user_version` 10200, the required tables and reference-system rows, one feature table per layer, `gpkg_extensions` + R-tree index and its six triggers when the system SQLite has the rtree module (checked at run time) |5455Not read: Shapefile MultiPatch (refused by name), dBASE memo contents, GeoPackage tiles / coverages / extended56geometry types, curve geometries in WKB, KML `Model` and overlays, TopoJSON.5758## Coordinate reference systems5960**A reference system is never changed silently.** Every conversion reports three facts61(`geo.source-crs`, `geo.destination-crs`, `geo.transformation`) and a warning whenever numbers change.6263Where the system comes from: GeoJSON → WGS 84 by RFC 7946 (a legacy `crs` member is honoured and flagged);64KML and GPX → WGS 84 by definition; Shapefile → the `.prj` WKT (identified structurally when it has no EPSG65authority, as Esri files do); GeoPackage → `srs_id` → `gpkg_spatial_ref_sys`; CSV → none (see below).6667| EPSG codes | System | Transformation implemented |68|---|---|---|69| 4326 | WGS 84 geographic | identity |70| 4269, 4617 | NAD83, NAD83(CSRS) geographic | treated as coincident with WGS 84 — **caution warning “up to about 1–2 m”, no datum shift is applied** |71| 3857 | WGS 84 / Pseudo-Mercator | spherical Mercator, both directions |72| 32601–32660, 32701–32760 | WGS 84 / UTM zones 1–60 N and S | Transverse Mercator, Krüger series (Karney 2011, order n⁶), both directions |73| 26901–26923 | NAD83 / UTM zones 1–23 N | same, GRS 80 ellipsoid (+ the datum caveat towards WGS 84) |74| 2955–2962, 3154–3160, 3761 | NAD83(CSRS) / UTM zones 7–22 N | same |75| 2945–2952, 32183–32190 | NAD83(CSRS) / MTM and NAD83 / MTM zones 3–10 (Québec; 2950 = zone 8) | Transverse Mercator, scale 0.9999, false easting 304 800 m |7677179 codes. Each one is checked by `CRSTests.forwardAndInverseMatchControlPoints` against two control coordinates78per projected code (four for 3857): forward within 0.1 mm, inverse within 1e-9°. The control values were computed79once with PROJ 9.8.1 (pyproj, in a scratch environment) and are stored in `ControlPoints.swift`; the tests do not80need PROJ. `CRSTests.knownReferencePoints` adds values that follow from the definitions (±π·R for Web Mercator,81500 000 m on a UTM central meridian, the WGS 84 meridian quadrant). Files written by the engine in eight of these82systems were also opened with GDAL 3.12.4 during development, which identified each EPSG code and returned the83same coordinates.8485Rules:8687- Destination must be WGS 84 (GeoJSON, KML, GPX): a source in the table is reprojected (info warning, plus the88 datum caveat when it applies); **any other source system is refused** with an error naming it89 (`CRSTests.refusesUnknownSystemForWGS84Formats`). The advanced option `crs.unsafe-passthrough` writes the90 coordinates unchanged instead and emits a `dataLoss` warning.91- Destination can carry a system (Shapefile, GeoPackage; CSV with a warning that it cannot record it):92 coordinates are passed through unchanged with the definition carried verbatim, including systems the engine93 does not know (tested with NAD83 / Québec Lambert). `crs.target` reprojects on request, only between systems of94 the table.95- No declared system (Shapefile without `.prj`, CSV with x / y columns, GeoPackage “undefined”): `crs.source` must96 state it. Without it the engine assumes WGS 84 **only** when every coordinate is within ±180 / ±90, with a97 caution warning; projected-looking numbers are refused (`CRSTests.missingPRJMustBeStated`).98- `crs.source` overrides a declared system, with a warning.99- Axis order is per format and tested (`CRSTests.axisOrderPerFormat`): GeoJSON and KML lon, lat; GPX `lat=` /100 `lon=` attributes; CSV by column name; WKT x y; GeoPackage and Shapefile x, y. Latitude / longitude columns whose101 values are out of range are refused instead of being swapped silently.102103Not implemented: every other projection (Lambert Conformal Conic — hence NAD83 / Québec Lambert EPSG:32198 and104Lambert-93 —, Albers, State Plane, polar stereographic…), datum shifts of any kind (NAD27, grids, Helmert), vertical105systems, WKT 2 beyond what identification needs.106107## Attributes: what each destination can hold108109Each loss is counted in a warning (tests: `ShapefileLimitTests.dbfLimitsAreCountedExactly`,110`GPXTests.writingKeepsOnlyFixedFieldsAndCountsTheRest`, `KMLTests.readsFoldersStylesAndExtendedData`,111`KMLTests.controlCharactersAreRemovedAndCounted`).112113| Destination | Kept | Dropped or changed (warning code) |114|---|---|---|115| GeoJSON | every attribute, typed; nested JSON restored as structure | measures (`measures-dropped`); dates are JSON strings |116| KML | every attribute (typed schema: int, double, bool, string) | measures; dates and date-times as text; control characters XML cannot carry are removed (`xml-control-characters`, counted; also for GPX) |117| GPX | name, desc, cmt, src, sym, type, link, time, number, sat, hdop… (matched case-insensitively; `description` → `desc`, `elevation` → `<ele>`) | every other attribute value (`gpx-attributes-dropped`, count + names); features without geometry (`gpx-no-geometry`); polygons need `gpx.polygons-as-tracks` (`dataLoss`); multi-points become several waypoints |118| Shapefile | text, integers, reals (shortest round-trip text), booleans, dates | names cut to 10 bytes and made unique (`dbf-names-truncated`, with the mapping); text over 254 bytes (`dbf-text-truncated`); trailing spaces; empty text ≡ missing; date-times and nested values as text; NaN / infinity as missing; mixed-type columns as text |119| GeoPackage | every attribute, typed (TEXT, INTEGER, DOUBLE, BOOLEAN, DATE, DATETIME) | nested values as JSON text; columns named `fid` / `geom` or repeated are renamed (`gpkg-columns-renamed`) |120| CSV | every attribute as text | types are re-inferred by whoever reads the file |121122Geometry narrowing: a GeometryCollection is refused for Shapefile and GPX unless `geometry.explode-collections`123is on (each member becomes a feature, counted). A layer mixing geometry families gives one shapefile per family124(`name_point.shp`, `name_line.shp`…) with a warning. Geometries without Z in a Z shapefile get Z = 0, counted.125126## Options127128`layer` (multi-layer sources), `crs.source`, `crs.target` or `crs.unsafe-passthrough`, `precision.decimals`129(off by default; rounding emits a `dataLoss` warning and is announced by `preflight`),130`geometry.explode-collections`, `csv.x-column` / `csv.y-column` / `csv.z-column` / `csv.wkt-column` /131`csv.delimiter`, `gpx.track-points`, `shp.encoding`, `csv.geometry`, `gpx.lines-as`, `gpx.polygons-as-tracks`,132`kml.name-field`, `kml.description-field`, `gpkg.spatial-index`.133134## Inspection and preview135136`inspect` returns layers, feature counts, geometry types, reference system, bounding box, Z presence and the137attribute schema per layer (`rowCount`, `sheetNames`, `columnCount` typed). The app's preview API is138`GeoPreview.preview(url:format:limit:options:)`: per layer the name, CRS, feature count, geometry-type histogram,139bounding box, fields, and the first N features (geometry type, bounding box, vertex count, WKT cut to a bounded140length, attributes). Test: `PathTests.inspectsAndPreviews(_:)` for the six formats.141142Cheap validity checks run on every conversion and only report: unclosed rings, rings with fewer than 4 points,143lines with fewer than 2, coordinates outside ±180 / ±90 in a geographic system, non-finite coordinates.144145## Validation146147`validate` reads the output back with the engine's readers and compares it with what `convert` recorded for the148data as written (after reprojection and the destination's reductions): feature count, geometry-family histogram,149bounding box (relative tolerance 1e-9), an order-independent checksum of every non-empty attribute name and150value of every feature, and, for Shapefile and GeoPackage, the declared EPSG code.151152The readers and writers are not only tested against each other: shapefiles are parsed from bytes assembled in the153tests per the ESRI technical description (`ShapefileTests.readsEveryShapeTypeFromIndependentBytes`,154`assignsHolesToTheirOuterRings`, `readsAttributeTypesAndCodePages`) and the writer's bytes are decoded by hand155(`writerProducesSpecConformingBytes`); GeoPackages are created with raw SQL and hand-packed blobs156(`GeoPackageTests.readsRawSQLGeoPackage`) and the writer's file is examined with plain SQLite157(`writesAConformingPackage`). During development the outputs were additionally opened with GDAL 3.12.4 and pyshp158(Shapefile incl. Z / M, GeoPackage, KML, GPX, GeoJSON): same geometries, attributes and EPSG codes. Those tools159are not used by the tests and are not shipped.160161## Untrusted input162163- XML (KML, GPX): external entities are never resolved; a document that declares its own entities is refused164 (entity-expansion bombs); depth ≤ 128, ≤ 30 million elements.165- ZIP (KMZ, zipped shapefile): ≤ 10 000 entries, ≤ 1 GiB per entry, ≤ 2 GiB in total; an entry that does not166 inflate to exactly its declared size is refused; entry names are only lookup keys; ZIP64 and encrypted entries167 are refused.168- Shapefile / dBASE / WKB / GeoPackage binary: every count is checked against the bytes present before anything169 is allocated; part offsets must be ordered; nesting ≤ 16.170- GeoPackage: opened read-only with `trusted_schema` off; table and column names from the file are only ever171 used as quoted identifiers (tested with a hostile table name).172- `RobustnessTests.corruptionSweepNeverCrashes`: several thousand corrupted or truncated variants of each binary173 and text fixture end in an error or a successful read, never in a crash.174175## Measured performance176177M5 Max, release build, one generated layer of 200 000 features (150 000 points with Z, 50 000 polygons, three178attributes; GeoJSON 32.2 MiB). `ORIX_GEO_BENCH=1 swift test -c release -Xswiftc -enable-testing --filter GeoBenchmark`,179run on 2026-11-21. Conversion time excludes validation.180181| Path | Convert | Features / s | Validate | Output |182|---|---|---|---|---|183| GeoJSON → GeoPackage | 1.98 s | 101 000 | 0.27 s | 34.3 MiB |184| GeoJSON → GeoPackage, reprojected to UTM 18N | 1.71 s | 117 000 | 0.31 s | 30.6 MiB |185| GeoJSON → Shapefile (2 shapefiles) | 1.18 s | 169 000 | 0.27 s | 19.8 MiB |186| GeoJSON → KML | 0.87 s | 230 000 | 2.13 s | 56.5 MiB |187| GeoJSON → CSV | 0.98 s | 204 000 | 0.53 s | 14.1 MiB |188| GeoJSON → GPX (polygons as tracks) | 0.96 s | 208 000 | 1.20 s | 24.5 MiB |189| GeoPackage → GeoJSON | 0.51 s | 391 000 | 0.58 s | 31.8 MiB |190| GeoPackage → Shapefile | 0.96 s | 208 000 | 0.31 s | 19.8 MiB |191| Shapefile (points) → GeoPackage | 0.95 s | 158 000 | 0.19 s | 20.0 MiB |192| Shapefile (points) → GeoJSON | 0.34 s | 435 000 | 0.39 s | 20.9 MiB |193194Peak memory was not measured. Every format is read fully into memory before writing (see below), so memory grows195with the source: text and XML sources are refused above 2 GiB.196197## Evaluated and not implemented198199| Item | Why |200|---|---|201| GDAL / PROJ | size, and PROJ's dependence on data files; replaced by the small tested set above |202| Streaming conversion | the engine holds the whole dataset in memory (needed to split by geometry type, build schemas and validate). Newline-delimited GeoJSON is parsed line by line but still collected. Large sources are bounded, not streamed |203| A `kmz` format identifier | KMZ and zipped shapefiles are read when the file is handed to the engine as `kml` / `shp` (recognised from the ZIP signature), but the core catalog has no `kmz` entry and the detector reports such files as ZIP, so the app cannot route them yet. Needs a catalog / detector addition in the core |204| Writing KMZ or zipped shapefiles | not requested; the shapefile is delivered as a folder so no sibling is orphaned |205| Same-format paths (re-projection GeoJSON → GeoJSON, repair) | not declared |206| GeoPackage tiles, rasters, coverages, extended geometry types, related tables | out of scope; detected and reported |207| Shapefile MultiPatch, dBASE memo fields | refused / read as plain text, with a message |208| KML styles, overlays, models, tours, network links, regions | dropped and counted; network links are never fetched |209| GPX per-point data on lines | kept only through the `track_points` / `route_points` layers |210| Topology repair, simplification, polygon validity beyond the cheap checks | not implemented |211| Peak-memory measurement | not run |212213## Known defects214215- A soft limit rather than a defect: validation re-reads the whole output; for KML this costs more than the216 conversion itself (2.1 s vs 0.9 s above).217- `inspect` reads the entire file (it reuses the readers); it is not a header-only probe.218219## KMZ as a source (2026-11-21)220221`kmz` is a catalog format detected by content (ZIP with `doc.kml` or a top-level `.kml`). The engine declares222`kmz → geojson, kml, gpx, csv, shp, gpkg` (6 paths, 36 in total), read by the KML reader through the bounded ZIP223reader; KMZ is never written. The other files of the archive are not carried over (`kmz-resources`, data loss).224Paths whose source can hold several layers and whose destination gets one output per layer (kml / kmz / gpkg / gpx →225geojson / csv / shp) are flagged `producesMultipleOutputs`. `GeoFeaturePreview.geometry` hands the geometry to the226app for drawing (features up to 50 000 vertices). Test: `KMZSourceTests`.227