SPB Git

Repository Public License

geo.md 18.3 KB

# Geospatial engine (geo) — compatibility

Package Packages/EngineGeo, module EngineGeo, factory makeEngines(). Pure Swift plus two system libraries (SQLite for GeoPackage, Compression for KMZ / zipped shapefiles). No GDAL and no PROJ: both are large, and PROJ depends on data files (grids, the EPSG database) that would have to be shipped and kept current. The price is a deliberately small set of coordinate transformations, listed below; everything else is carried or refused, never guessed.

State on 2026-11-21 (macOS 27.0, arm64): 30 declared paths, 53 tests, all passing (swift test: Test run with 53 tests in 11 suites passed; two of the 53 are development aids that only run when an environment variable is set).

# Declared paths

Every ordered pair among GeoJSON, KML, GPX, CSV, Shapefile and GeoPackage: 6 × 5 = 30 paths, all supported, no named operation. Test for all of them: PathTests.convertsCities(_:) (30 cases — a hand-written three-city fixture per source format, exact coordinates and attribute values checked on the re-read output, every validation check required to pass).

Destination ↓ / fidelity, preserves From GeoJSON, CSV, Shapefile, GeoPackage, GPX From KML
GeoJSON structural; geometry, attributes partial (styles dropped)
KML structural; geometry, attributes —
GPX partial; geometry only partial
CSV structural; geometry, attributes partial
Shapefile structural; geometry, attributes, coordinate system partial
GeoPackage structural; geometry, attributes, coordinate system partial

Limitations shown to the user (from ConversionPath.limitations):

Applies to Limitation
source KML styles, icons, overlays, tours, 3D models and network links are not carried over; network links are never fetched
source GPX per-point timestamps and sensor data of tracks are dropped unless “Track points as features” is on
source GeoPackage only vector layers; tiles, rasters and BLOB attributes are reported and left out; the feature id column is not exported
source Shapefile without a .prj the reference system must be stated
source CSV no reference system in the format: WGS 84 is assumed only for columns named like longitude / latitude
destination GeoJSON, KML always WGS 84 lon/lat: supported systems are reprojected, others refused; measures (M) dropped; KML dates become text
destination GPX always WGS 84; points and lines only (polygons refused unless written as closed tracks); only the fixed GPX fields are kept
destination Shapefile delivered as a folder (.shp, .shx, .dbf, .cpg, .prj); dBASE limits; one geometry type per file
destination CSV cannot record the reference system; coordinate columns for points, WKT otherwise
multi-layer source → GeoJSON, CSV, Shapefile one output per layer

# Formats: what is read and written

Format Reading Writing
GeoJSON RFC 7946 FeatureCollection / Feature / bare geometry; newline-delimited GeoJSON and RFC 8142 sequences (detected from content); property order and integer / real distinction kept; nested values kept as JSON text; id kept as an id attribute; legacy crs member honoured and flagged one FeatureCollection, one feature per line, polygons following the right-hand rule, shortest round-trip numbers
KML KML 2.2; folders → layers (Folder/Subfolder); name, description, ExtendedData Data and typed SchemaData, TimeStamp / TimeSpan; Point, LineString, LinearRing, Polygon with holes, MultiGeometry, gx:Track / gx:MultiTrack (as lines); KMZ: doc.kml (or the shallowest .kml) through the engine's bounded ZIP reader one Document, one Folder per layer, a typed Schema per layer, <name> / <description> from the attributes chosen in the options
GPX 1.0 and 1.1; waypoints, routes, tracks → three layers; ele → Z; the fixed fields; leaf elements of <extensions> as attributes; optional track_points / route_points layers GPX 1.1: points → waypoints, lines → tracks (or routes), multi-part lines → multi-segment tracks
CSV RFC 4180, delimiter detected among `, ; tab `; UTF-8 (Windows-1252 fallback, reported); geometry from longitude/latitude or x/y columns (by name, checked against the range of degrees) or a WKT column; explicit column options; per-column typing (text with leading zeros stays text)
Shapefile .shp + .dbf + .prj + .cpg, siblings found case-insensitively; a .zip holding one or several shapefiles; Point, MultiPoint, PolyLine, Polygon and their Z and M variants, null shapes; holes assigned to the outer ring that contains them; dBASE C / N / F / L / D; code page from the option, then .cpg, then the language driver id, then a reported guess all of the above; .shx index; outer rings clockwise, holes counter-clockwise; .cpg = UTF-8; .prj in Esri flavour
GeoPackage gpkg_contents feature and attribute tables; gpkg_geometry_columns; gpkg_spatial_ref_sys; GeoPackageBinary header (both byte orders, every envelope form) + ISO WKB (Z, M, ZM; EWKB flags tolerated); column types incl. BOOLEAN, DATE, DATETIME application_id GPKG, user_version 10200, the required tables and reference-system rows, one feature table per layer, gpkg_extensions + R-tree index and its six triggers when the system SQLite has the rtree module (checked at run time)

Not read: Shapefile MultiPatch (refused by name), dBASE memo contents, GeoPackage tiles / coverages / extended geometry types, curve geometries in WKB, KML Model and overlays, TopoJSON.

# Coordinate reference systems

A reference system is never changed silently. Every conversion reports three facts (geo.source-crs, geo.destination-crs, geo.transformation) and a warning whenever numbers change.

Where the system comes from: GeoJSON → WGS 84 by RFC 7946 (a legacy crs member is honoured and flagged); KML and GPX → WGS 84 by definition; Shapefile → the .prj WKT (identified structurally when it has no EPSG authority, as Esri files do); GeoPackage → srs_id → gpkg_spatial_ref_sys; CSV → none (see below).

EPSG codes System Transformation implemented
4326 WGS 84 geographic identity
4269, 4617 NAD83, NAD83(CSRS) geographic treated as coincident with WGS 84 — caution warning “up to about 1–2 m”, no datum shift is applied
3857 WGS 84 / Pseudo-Mercator spherical Mercator, both directions
32601–32660, 32701–32760 WGS 84 / UTM zones 1–60 N and S Transverse Mercator, Krüger series (Karney 2011, order n⁶), both directions
26901–26923 NAD83 / UTM zones 1–23 N same, GRS 80 ellipsoid (+ the datum caveat towards WGS 84)
2955–2962, 3154–3160, 3761 NAD83(CSRS) / UTM zones 7–22 N same
2945–2952, 32183–32190 NAD83(CSRS) / MTM and NAD83 / MTM zones 3–10 (Québec; 2950 = zone 8) Transverse Mercator, scale 0.9999, false easting 304 800 m

179 codes. Each one is checked by CRSTests.forwardAndInverseMatchControlPoints against two control coordinates per projected code (four for 3857): forward within 0.1 mm, inverse within 1e-9°. The control values were computed once with PROJ 9.8.1 (pyproj, in a scratch environment) and are stored in ControlPoints.swift; the tests do not need PROJ. CRSTests.knownReferencePoints adds values that follow from the definitions (±π·R for Web Mercator, 500 000 m on a UTM central meridian, the WGS 84 meridian quadrant). Files written by the engine in eight of these systems were also opened with GDAL 3.12.4 during development, which identified each EPSG code and returned the same coordinates.

Rules:

  • Destination must be WGS 84 (GeoJSON, KML, GPX): a source in the table is reprojected (info warning, plus the datum caveat when it applies); any other source system is refused with an error naming it (CRSTests.refusesUnknownSystemForWGS84Formats). The advanced option crs.unsafe-passthrough writes the coordinates unchanged instead and emits a dataLoss warning.
  • Destination can carry a system (Shapefile, GeoPackage; CSV with a warning that it cannot record it): coordinates are passed through unchanged with the definition carried verbatim, including systems the engine does not know (tested with NAD83 / Québec Lambert). crs.target reprojects on request, only between systems of the table.
  • No declared system (Shapefile without .prj, CSV with x / y columns, GeoPackage “undefined”): crs.source must state it. Without it the engine assumes WGS 84 only when every coordinate is within ±180 / ±90, with a caution warning; projected-looking numbers are refused (CRSTests.missingPRJMustBeStated).
  • crs.source overrides a declared system, with a warning.
  • Axis order is per format and tested (CRSTests.axisOrderPerFormat): GeoJSON and KML lon, lat; GPX lat= / lon= attributes; CSV by column name; WKT x y; GeoPackage and Shapefile x, y. Latitude / longitude columns whose values are out of range are refused instead of being swapped silently.

Not implemented: every other projection (Lambert Conformal Conic — hence NAD83 / Québec Lambert EPSG:32198 and Lambert-93 —, Albers, State Plane, polar stereographic…), datum shifts of any kind (NAD27, grids, Helmert), vertical systems, WKT 2 beyond what identification needs.

# Attributes: what each destination can hold

Each loss is counted in a warning (tests: ShapefileLimitTests.dbfLimitsAreCountedExactly, GPXTests.writingKeepsOnlyFixedFieldsAndCountsTheRest, KMLTests.readsFoldersStylesAndExtendedData, KMLTests.controlCharactersAreRemovedAndCounted).

Destination Kept Dropped or changed (warning code)
GeoJSON every attribute, typed; nested JSON restored as structure measures (measures-dropped); dates are JSON strings
KML every attribute (typed schema: int, double, bool, string) measures; dates and date-times as text; control characters XML cannot carry are removed (xml-control-characters, counted; also for GPX)
GPX name, desc, cmt, src, sym, type, link, time, number, sat, hdop… (matched case-insensitively; description → desc, elevation → <ele>) every other attribute value (gpx-attributes-dropped, count + names); features without geometry (gpx-no-geometry); polygons need gpx.polygons-as-tracks (dataLoss); multi-points become several waypoints
Shapefile text, integers, reals (shortest round-trip text), booleans, dates names cut to 10 bytes and made unique (dbf-names-truncated, with the mapping); text over 254 bytes (dbf-text-truncated); trailing spaces; empty text ≡ missing; date-times and nested values as text; NaN / infinity as missing; mixed-type columns as text
GeoPackage every attribute, typed (TEXT, INTEGER, DOUBLE, BOOLEAN, DATE, DATETIME) nested values as JSON text; columns named fid / geom or repeated are renamed (gpkg-columns-renamed)
CSV every attribute as text types are re-inferred by whoever reads the file

Geometry narrowing: a GeometryCollection is refused for Shapefile and GPX unless geometry.explode-collections is on (each member becomes a feature, counted). A layer mixing geometry families gives one shapefile per family (name_point.shp, name_line.shp…) with a warning. Geometries without Z in a Z shapefile get Z = 0, counted.

# Options

layer (multi-layer sources), crs.source, crs.target or crs.unsafe-passthrough, precision.decimals (off by default; rounding emits a dataLoss warning and is announced by preflight), geometry.explode-collections, csv.x-column / csv.y-column / csv.z-column / csv.wkt-column / csv.delimiter, gpx.track-points, shp.encoding, csv.geometry, gpx.lines-as, gpx.polygons-as-tracks, kml.name-field, kml.description-field, gpkg.spatial-index.

# Inspection and preview

inspect returns layers, feature counts, geometry types, reference system, bounding box, Z presence and the attribute schema per layer (rowCount, sheetNames, columnCount typed). The app's preview API is GeoPreview.preview(url:format:limit:options:): per layer the name, CRS, feature count, geometry-type histogram, bounding box, fields, and the first N features (geometry type, bounding box, vertex count, WKT cut to a bounded length, attributes). Test: PathTests.inspectsAndPreviews(_:) for the six formats.

Cheap validity checks run on every conversion and only report: unclosed rings, rings with fewer than 4 points, lines with fewer than 2, coordinates outside ±180 / ±90 in a geographic system, non-finite coordinates.

# Validation

validate reads the output back with the engine's readers and compares it with what convert recorded for the data as written (after reprojection and the destination's reductions): feature count, geometry-family histogram, bounding box (relative tolerance 1e-9), an order-independent checksum of every non-empty attribute name and value of every feature, and, for Shapefile and GeoPackage, the declared EPSG code.

The readers and writers are not only tested against each other: shapefiles are parsed from bytes assembled in the tests per the ESRI technical description (ShapefileTests.readsEveryShapeTypeFromIndependentBytes, assignsHolesToTheirOuterRings, readsAttributeTypesAndCodePages) and the writer's bytes are decoded by hand (writerProducesSpecConformingBytes); GeoPackages are created with raw SQL and hand-packed blobs (GeoPackageTests.readsRawSQLGeoPackage) and the writer's file is examined with plain SQLite (writesAConformingPackage). During development the outputs were additionally opened with GDAL 3.12.4 and pyshp (Shapefile incl. Z / M, GeoPackage, KML, GPX, GeoJSON): same geometries, attributes and EPSG codes. Those tools are not used by the tests and are not shipped.

# Untrusted input

  • XML (KML, GPX): external entities are never resolved; a document that declares its own entities is refused (entity-expansion bombs); depth ≤ 128, ≤ 30 million elements.
  • ZIP (KMZ, zipped shapefile): ≤ 10 000 entries, ≤ 1 GiB per entry, ≤ 2 GiB in total; an entry that does not inflate to exactly its declared size is refused; entry names are only lookup keys; ZIP64 and encrypted entries are refused.
  • Shapefile / dBASE / WKB / GeoPackage binary: every count is checked against the bytes present before anything is allocated; part offsets must be ordered; nesting ≤ 16.
  • GeoPackage: opened read-only with trusted_schema off; table and column names from the file are only ever used as quoted identifiers (tested with a hostile table name).
  • RobustnessTests.corruptionSweepNeverCrashes: several thousand corrupted or truncated variants of each binary and text fixture end in an error or a successful read, never in a crash.

# Measured performance

M5 Max, release build, one generated layer of 200 000 features (150 000 points with Z, 50 000 polygons, three attributes; GeoJSON 32.2 MiB). ORIX_GEO_BENCH=1 swift test -c release -Xswiftc -enable-testing --filter GeoBenchmark, run on 2026-11-21. Conversion time excludes validation.

Path Convert Features / s Validate Output
GeoJSON → GeoPackage 1.98 s 101 000 0.27 s 34.3 MiB
GeoJSON → GeoPackage, reprojected to UTM 18N 1.71 s 117 000 0.31 s 30.6 MiB
GeoJSON → Shapefile (2 shapefiles) 1.18 s 169 000 0.27 s 19.8 MiB
GeoJSON → KML 0.87 s 230 000 2.13 s 56.5 MiB
GeoJSON → CSV 0.98 s 204 000 0.53 s 14.1 MiB
GeoJSON → GPX (polygons as tracks) 0.96 s 208 000 1.20 s 24.5 MiB
GeoPackage → GeoJSON 0.51 s 391 000 0.58 s 31.8 MiB
GeoPackage → Shapefile 0.96 s 208 000 0.31 s 19.8 MiB
Shapefile (points) → GeoPackage 0.95 s 158 000 0.19 s 20.0 MiB
Shapefile (points) → GeoJSON 0.34 s 435 000 0.39 s 20.9 MiB

Peak memory was not measured. Every format is read fully into memory before writing (see below), so memory grows with the source: text and XML sources are refused above 2 GiB.

# Evaluated and not implemented

Item Why
GDAL / PROJ size, and PROJ's dependence on data files; replaced by the small tested set above
Streaming conversion the engine holds the whole dataset in memory (needed to split by geometry type, build schemas and validate). Newline-delimited GeoJSON is parsed line by line but still collected. Large sources are bounded, not streamed
A kmz format identifier KMZ and zipped shapefiles are read when the file is handed to the engine as kml / shp (recognised from the ZIP signature), but the core catalog has no kmz entry and the detector reports such files as ZIP, so the app cannot route them yet. Needs a catalog / detector addition in the core
Writing KMZ or zipped shapefiles not requested; the shapefile is delivered as a folder so no sibling is orphaned
Same-format paths (re-projection GeoJSON → GeoJSON, repair) not declared
GeoPackage tiles, rasters, coverages, extended geometry types, related tables out of scope; detected and reported
Shapefile MultiPatch, dBASE memo fields refused / read as plain text, with a message
KML styles, overlays, models, tours, network links, regions dropped and counted; network links are never fetched
GPX per-point data on lines kept only through the track_points / route_points layers
Topology repair, simplification, polygon validity beyond the cheap checks not implemented
Peak-memory measurement not run

# Known defects

  • A soft limit rather than a defect: validation re-reads the whole output; for KML this costs more than the conversion itself (2.1 s vs 0.9 s above).
  • inspect reads the entire file (it reuses the readers); it is not a header-only probe.

# KMZ as a source (2026-11-21)

kmz is a catalog format detected by content (ZIP with doc.kml or a top-level .kml). The engine declares kmz → geojson, kml, gpx, csv, shp, gpkg (6 paths, 36 in total), read by the KML reader through the bounded ZIP reader; KMZ is never written. The other files of the archive are not carried over (kmz-resources, data loss). Paths whose source can hold several layers and whose destination gets one output per layer (kml / kmz / gpkg / gpx → geojson / csv / shp) are flagged producesMultipleOutputs. GeoFeaturePreview.geometry hands the geometry to the app for drawing (features up to 50 000 vertices). Test: KMZSourceTests.

Files main
Menu

Theme