13,847
Records processed
How the system works and processes today
How one messy source row becomes a clean service description. Every one of 13,847 raw records travels the same route — six engineered processing stages, roughly 200 discrete, auditable operations, with one narrow language-model stage along the way. This page traces that journey end to end, and lets you open the machinery at every stop.
The route
Each stage is code we own, version and can point to line-by-line. The model stage — the slate-blue node — performs one deliberately narrow job: composition, itself decomposed into four single-responsibility calls. Everything factual is resolved by the stages around it.
The journey, stage by stage
A representative raw marketingDescription as it arrives from the source workbook — HTML fragments, glued tokens, typos, ambiguous region codes. Follow it down. At each stop, “open the machinery” to see the actual rules, numbered line by line.
NA,MX would naively collapse into
NORTH AMERICAMX — two regions fused into a garbage token. Every punctuation-absorbing
rule emits canonical + space, so it lands as NORTH AMERICA Mexico —
on row 12 and on row 12,000, identically.Inside the layers
A sample of the actual rule families. Every rule has an ID, a rationale and a hit count against the live corpus — the long tail exists precisely because 13.8K rows hide variants no one would anticipate up front. The full rule sets are in the consoles above.
Typos, missing spaces and concatenations, fixed before anything else so downstream rules see clean tokens.
| ID | Transform | Hits |
|---|---|---|
| S15 | fastrackpurchased → fastrack purchased | 14 |
| S28 | heathcare → healthcare | 27 |
| S11 | VISIBILTYIQ → VISIBILITYIQ | 11 |
| S22 | biling → billing | 3 |
| S36 | 24/7 → 24x7 | 3 |
| S6 | CRADL → Cradle | 1 |
Ordered transforms — each feeds the next. One canonical form per concept, corpus-wide.
| ID | Transform | Family |
|---|---|---|
| C1 | 3 day TAT → 3-day TAT | TAT |
| C8 | NA → NORTH AMERICA | Region |
| C3 | NBD → next business day | TAT |
| C18 | yrs → years | Units |
| C46 | Z1C: → Z1C | Misc |
| C29 | 3 year → 3-year | Hyphen |
The corpus mined for its own vocabulary. Detection lives here once — every other layer imports it. No duplicated logic.
| Extractor | Owns |
|---|---|
| tab-14 | isProduct — product SSOT |
| tab-06 | REGION_NAMES_UPPER — region SSOT |
| tab-05 | TAT variants |
| tab-21 | OneCare variants |
| tab-18/19 | Z1C case & trigrams |
| tab-20 | non-dictionary residue |
Four typed marker classes turn extraction into deterministic verification: every tag injected must appear in the output.
| Tag class | Purpose |
|---|---|
| [product:X] | one per product, no ordinals |
| [location-positive:X] | included region |
| [location-negative:X] | excluded region |
| [brand-qualifier:X] | e.g. Fastrack |
Measuring convergence
The founding question was measurable: can we reduce the total number of unique words, variables and misspellings? Every rule’s effect was measured against the live corpus in a discovery workbook, rebuilt after every rule change. This is the summary tab — the first full rebuild against the final one. Same 13,847 records; only the rules changed between them.
| Measurement (workbook tab) | Before | After | Reduction |
|---|---|---|---|
| Unique raw termsT1_all_raw | 1,785 | 1,061 | −40.6% |
| Unique terms, case-foldedT2 | 1,580 | 933 | −40.9% |
| Model-number wildcard termsT3_xx_termsdrained | 135 | 0 | −100% |
| Hyphenated term variantsT4 | 79 | 57 | −27.8% |
| Turnaround-time spellingsT5_tat_variants | 19 | 7 | −63.2% |
| Raw location tokensT6_locationsdrained | 56 | 0 | −100% |
| Unique 3-word phrasesT9_trigrams | 6,914 | 3,823 | −44.7% |
| Digit-bearing tokensT10 | 565 | 171 | −69.7% |
| Qualifier phrase variantsT11 | 34 | 15 | −55.9% |
| Unclassified residue tokensT12b | 612 | 371 | −39.4% |
T3 and T6 hit zero — not reduced, eliminated. Ambiguous model-number wildcards like tc27xx were replaced by {{product}} placeholders, and raw region codes like NA/MX/LATAM by canonical {{location}} handling. The worksheets that existed to track those problems are now empty.
The engineering case
The deterministic layers exist because these are jobs that must be accountable and repeatable — and a language model is structurally the wrong tool for them. The model does what it’s best at; code does the rest.
Five of the six largest historical error classes were the same failure: the model lost track of what was in the source. Typed tags don’t ask the model to remember — they hand it the facts and verify they survived.
A rule corrects heathcare→healthcare the same way on row 12 and row 12,000. A prompt’s behaviour drifts across a run of that size; deterministic code does not.
When someone asks “why does this say North America?” the answer is a named rule — C8 — with a rationale and a hit count. “The model decided” is not an answer.
Re-run the pipeline and the deterministic stages produce byte-identical results. That is the property that lets us test — and the reason quality is measurable, not anecdotal.
Typo-fixing and region-mapping in code costs nothing per row and never regresses on a model update. The full 13K run stays inexpensive because the model does only what it’s best at.
Where the journey ends
The win condition was ≥80% clean. Measured on stratified samples of increasing size — the ladder is never skipped — the tagged pipeline cleared it decisively.