Signals on a grain

Problem

Buyers need one list of properties that combine facts from different public records (for example absentee owner and a court filing and a value band). Those facts already exist, but they arrive as portals, PDFs, and files — not as inventory.

A lookup (one address, one screen per source) can inspect a parcel. It cannot answer that mixed question. The product is the combined list, not a stack of document screens. A filing that cannot be matched to a parcel is kept, but it is not sold as a property.

We went with this architecture when the lookup died on the first mixed list, and a frozen “one schema on day one” died on the first ugly PDF.

Grain: one list row = one property, so signals can be layered. Match before the list: no shared key on the PDF, so identity is resolved; unmatched stays out. Gatherers: scripts become “land the file, do not decide inventory,” so a bad parse does not stop ingest and a new source does not copy list rules. Wide property row (OBT): the buyer filters one row instead of joining a maze of ugly tables. Preview, export, and masking all read that same row.

We build property lists today. The same landed file could match another grain later — not in this post.

Options considered

1. One product per source — a courts screen, a roll screen, an export per file type. Fast while the script count is small. Cannot answer absentee and filing and value on one list.
2. One schema on day one — force every feed into the same tables before we have seen the PDFs. Looks tidy. Breaks on the first ugly docket.
3. Signal layering on a grain — land raw records, extract signals, match them to the property, query one serving row. Unmatched stays out.

Decision

We chose option 3.

Option 1 was the lookup. It inspects a parcel. It does not make a mixed list. Option 2 froze the schema too early. The first ugly PDF had no place to land.

Gatherers and the wide row are not a second product. They are how option 3 survives more sources. Unmatched stays out: no parcel match, no list filter.

How it works

Land → extract → match to parcel → one serving row → list.

A gatherer lands the PDF. We extract fields, then match. A hit writes flags onto the property row. Preview, export, and masking all read that same row. No match: we keep the file. We do not put it on the list. Buyers get properties, not documents.

flowchart LR
  pdf[Court PDF] --> land[Land the file]
  land --> extract[Extract signal]
  extract --> ok[Match to parcel]
  extract --> no[Unmatched]
  ok --> row[One property row]
  row --> list[Layered property lists]

Hard parts

Disagree on the grain: we invent properties that do not exist. Identifiers fight (address, APN, legal description, parties): loose match duplicates; strict match drops real filings. Two sources disagree on the same parcel: lists need a rule, not last-write-wins. Pressure to “just show the filing”: that is the lookup again. Documents, not properties. A new extractor with no match and no unmatched state: lists slowly fill with junk.

What we’d change

We would write the grain contract sooner: which signals are list filters, what a matched event means. Extractors can stay messy. If flags on the wide row are not enough later, we would add satellites — still projected onto the grain for list paths. A later domain gets its own matching rules. Not a second philosophy.

References

• RealtyAPI — Mastering Property Data Collection Pipelines — public records as durable identity; pipeline under the UI. Their post is long; we split that territory across this series.
• Warren Group — Combine Without Duplicates — one property, many documents; matching, not a pile of files.
• Kimball — Declaring the Grain — one row means one thing. That is the list contract.