Schema drift
The phenomenon in which the structure of a recurring data feed — column names, column order, data types, or value formats — changes over time, breaking downstream consumers that assume a fixed shape.
Schema drift is the silent enemy of reconciliation pipelines that rely on fixed-format file ingestion. A marketplace adds a new column to its settlement export. A bank renames a field in MT940 narrative. A payment gateway changes the CSV column order between minor releases. Each change breaks any consumer that hard-codes column positions.
Drift is rarely announced in advance and rarely versioned cleanly. The first sign is usually a downstream parsing error or a silent miscategorisation — for example, a 'closing fee' column being read as 'shipping fee' because the columns swapped order. By the time the error is noticed, several days of incorrect reconciliations may have been produced.
Defenses against schema drift include: header-based parsing (reading column positions from the header row rather than hard-coding) rather than positional, automatic schema detection on each import with a diff against the previous run, and explicit alerting when new columns appear or expected columns disappear. ReconPe's data-service layer applies all three by default.