PropTech Engineering

From scripts to lists of properties — the decisions along the way

· architecture, operations

Queue grain, not raw HTML

After we match a signal to a parcel, we may queue that grain. We do not queue raw HTML or PDFs. Compose and list serving stay a separate track.

Problem

A grain here is a signal after it sits on a parcel. Raw HTML is the fetched markup. A PDF is the attached file. Both are bytes we gathered, not the origin. A source is that origin — one job per source.

Once land and match no longer fit in one process, the fork is what goes on a bus: those bytes, nothing, or only the grain after match.

Portal-shaped sources — government apps, vendor-hosted listing UIs, JavaScript tables with no JSON — still need a browser. File-shaped sources — rolls, tax files, documented downloads — do not. Both must sit on the property grain before they are list-eligible.

ETL transforms before storage. ELT loads raw first, then transforms inside the warehouse or lake. That later step is compose. Warehouse loaders such as Airbyte are ELT tools for documented APIs and files. They are not a crawler. They are not a PDF tagger.

We build property lists today. The same landed record could match another grain later — not this post.

Options considered

  1. Stay in-process — gather, extract, and match write tables in one source job forever. Simple. The database becomes a file system. Match stays a search-index heuristic with no unmatched table.
  2. Queue raw HTML and PDFs — fetched bytes on a bus, workers parse, a “ready” queue loads tables. Known pipes-and-filters. It scales crawl throughput. We cap that on purpose. Unmatched events become traffic.
  3. Queue qualified grains — land bytes (crawlers for portals, ELT for files) into object storage. Extract under the same signal contracts. Match as a named step. Emit a small message only when a signal sits on a parcel. Loaders write source tables. Compose stays a separate track.

Decision

We chose option 3.

Option 1 holds until bronze files, gold lists, and orders fight for the same disk, and until fuzzy match cannot be a first-class table. Option 2 is the wrong first pattern: a pipeline of every attachment. Orchestration still schedules sources. The queue decouples “this source finished match” from “rows landed for compose.”

Warehouse loaders do not replace portal gatherers. A document parser does not replace signal contracts. Search index is for candidates. Probabilistic linkage is for the match. Compose is for exact keys and for gold. Unmatched is stored. It is not queued. It is not a list row. The qualification gate stays match, not download.

How it works

Same lis pendens. A crawler lands the PDF in object storage. A parse engine plus our tagger emit parties and locators. Resolve tries the parcel. If it matches, a short event goes on the queue: signal, parcel, confidence, pointer to the file — not the file. A loader writes the source row. Compose builds gold as it does today. If it does not match, the file stays. The unmatched row stays. The queue does not see it. That event never puts the property on the absentee-and-filing list.

flowchart TB
  portals[Portal sources] --> crawlers[Custom crawlers]
  files[File sources] --> elt[ELT for files]
  crawlers --> lake[Object storage]
  elt --> lake
  lake --> extract[Extract then match]
  extract --> q[Qualified grain queue]
  extract --> out[Unmatched stays out]
  q --> load[Load source tables]
  load --> compose[Compose]

Hard parts

  • A “raw / ready” bus of HTML looks like progress because messages move. It burns sources and skips the gate.
  • Airbyte on a portal is still a crawler with extra parts. File ELT and portal land must share a bronze prefix, or compose cares how a file arrived.
  • The message must hold a pointer to bytes, not the PDF. Otherwise the queue becomes a second file store.
  • Loaders must be idempotent. Poison messages go aside. Otherwise one bad match blocks gold.
  • Naming Resolve without moving the gate into Gather is easy to get wrong. List eligibility stays match.

What we’d change

We would land bytes in object storage before we introduced a queue. We would not stream every PDF through workers to use the cluster. Warehouse split and list serving wait until a night refresh or an interactive query actually hurts — a separate track, not a dependency of the gather fleet.

References

← All notes