Scale sources, not requests

Problem

A source is an origin we gather from — a government app, a vendor listing UI, or a file drop. It is not a URL and not the raw HTML. An operator needs court PDFs and tax files from hundreds of sources without rewriting extract, match, or lists. A filing still joins a list of properties only after we match it to a parcel. That constraint stays. It is not the fork.

Gather already waits between calls so we are not blocked. It must not run as fast as possible. Scale is more sources, not more requests per source.

Cron wrapping a handful of scripts is honest while the job count is tiny. At hundreds of sources, one hung job stalls a shared night window, and the crontab file becomes the catalog. The fork is how we schedule that work.

We build property lists today. The same landed record could match another grain later — not this post.

Options considered

1. One crontab line per source — cheapest while the set is small. Hundreds of lines are not auditable. One hung source stalls a shared night window.
2. A queue of raw HTML — every fetched page body or PDF as a message so workers crawl faster. That scales request throughput. We cap request throughput on purpose. Unmatched filings would ride the same bus as inventory.
3. One job per source — an orchestrator (or a work queue of sources) runs gather plus extract in-process. The gatherer’s delays stay. Failures isolate. Source tables that compose already reads still get rows.

Decision

We chose option 3.

Option 1 worked until the job list became a second codebase. Option 2 looked like a pipeline and optimized the wrong axis: requests per source, not sources. Warehouse loaders such as Airbyte assume documented APIs. They do not replace portal crawlers, so they are not the in-place scale move.

The work unit is a source (or a source×day), not a URL. Six hundred polite jobs can run at once. Six hundred faster request loops cannot — the origin will block us. Match stays in-process. Unmatched stays out of lists and off any bus. The qualification gate does not move. Compose and serve stay as they are.

How it works

A new court portal is one more source job, not a faster crawl of an old one. The orchestrator starts that job. Gather waits between requests, stores the PDF, and does not decide the parcel. Extract and match run in the same job. If identifiers match a parcel, the filing can join a property list. If they do not, we keep the file. We do not enqueue the unmatched filing. A hung neighbor source retries on its own. It does not stall this job.

flowchart LR
  src[Public sources] --> orch[Orchestrator]
  orch --> job[One source job]
  job --> gather[Gather then extract]
  gather --> db[Same tables]
  db --> compose[Compose]

Hard parts

• Teams will treat “scale” as faster crawls because the hardware is idle. That burns the source.
• A source job that also writes “ready for lists” rows recreates a one-stage scraper. Land still does not mark inventory.
• Wrapping current scripts is a smaller step than naming assets. Either way, the noun must stay the source, not the URL.
• File-shaped sources already import into tables. They can keep doing that. Where bronze lives is a later fork.

What we’d change

We would name the source as the retry unit sooner, before the crontab file became the catalog. We would not put raw HTML on a queue to get there. When files want object storage and match wants a named step, that is a different control plane — queue grain, not raw HTML.

References

• Dagster — software-defined assets — the unit of work can be a source or a landed table, not a crontab stanza.