About

Why I built it

I read a lot of tech news and I never liked any of the existing aggregators. They rank by votes or by recency, so the front page is whatever a particular crowd was arguing about that morning, and the mix is someone else's — heavy on whatever that community cares about, thin on the things I actually wanted to keep up with.

I wanted one tailored to my interests. That is really the whole premise: the rubric below is a set of five things I think make a story worth reading, written down explicitly, and applied the same way to every article every day. If the mix drifts, I change the rubric rather than hoping a ranking algorithm comes around. Everything else in this project is what it took to make that idea run unattended, overnight, on one machine.

1. Collecting the day

37 RSS feeds are fetched concurrently. Each entry is filtered against the coverage window using the publish date the feed itself reports, converted from UTC into local time first so a late-evening story is not pushed into the next day. Entries with no date at all are let through rather than dropped — a missing timestamp is a worse reason to lose a story than a slightly wrong one.

Surviving entries are crawled with a headless Chromium browser, because a modern article page is mostly JavaScript, navigation and consent dialogs. The extracted DOM runs through a pruning filter that scores each block by text density and link ratio and discards the low scorers, which strips menus, related-article rails and footers and leaves the article body as markdown. Crawls are capped at five in flight — enough to get through a day's entries in reasonable time without hammering any one publisher.

An article's identity is a SHA-256 hash of its canonicalized URL, not its raw one. Canonicalizing drops the scheme so http and https collapse together, lowercases the host, strips a leading www., removes the fragment and every tracking parameter, sorts the remaining query parameters and trims a trailing slash. The same article arriving from a newsletter link, a social share and the plain feed becomes one row instead of three. Anything under 600 characters of body text is rejected outright — at that length it is a bot wall or a consent interstitial, not an article.

2. Deciding what matters

Every article scraped that day is read in full and scored against the same rubric — five dimensions, each 1 to 20. There is no pre-screening step. A cheap headline-only gate used to run first, on the theory that it would spare the expensive read; measured over several days it rejected about one headline in three hundred while costing a model call on every one of them, so it was a per-article expense pretending to be a saving and it was deleted. Filtering that rejects nothing is just latency.

• Broad interest — how many people would actually want to read this
• Scope — how far it reaches, from a household-name platform down to one small company
• Timeliness — breaking and first-reported versus rehashed
• Novelty — genuinely new information versus restated PR
• Digest fit — whether it belongs in a punchy daily read at all

The interesting part is how the output is obtained. The rubric is declared as a typed schema, converted to JSON Schema, and handed to the runtime, which compiles it into a decoding grammar. This is not a prompt asking nicely for JSON — at each step the sampler is only allowed to emit tokens the grammar permits, so a malformed response is not unlikely, it is unrepresentable. Field length limits are part of that grammar too, which means a model that starts rambling is forced closed rather than running away. Because being forced closed cuts mid-word, any field that hits its limit is trimmed back to its last complete sentence before it goes anywhere near the site.

The overall score is not taken from the model. A validator overwrites it with the arithmetic sum of the five sub-scores, so the ranking never depends on a small model doing addition correctly. Token budgets are capped per call, but the number of articles is not: every article scraped that day gets read. That is only affordable because grading and writing are separate passes — scoring costs a few seconds an article, so a three-hundred-article day is about twenty minutes of work. An article that never gets scored can never be picked, which makes a ceiling here a quiet way to lose a third of the day.

3. Two models, split by how much work each has to do

Grading uses a 3B model. Everything downstream uses an 8B one. That split is the central cost decision in the pipeline.

A 3B model is a good filter and a coarse ranker. It reliably separates the hundred articles worth considering from the nine hundred that are not, but the ordering inside that shortlist is exactly what decides the final ten — and which telling of a duplicated story survives. So the top 25 are scored a second time by the larger model, which is roughly three times slower per call and affordable only because it runs on 25 articles rather than a hundred.

The re-grade fails open: if it errors, the cheap pass's score stands. A failed re-grade should never silently drop a contender. Each row records which model scored it, so the two passes stay distinguishable after the fact.

4. One story per event

URL canonicalization cannot help here. When six outlets cover one product launch they publish six genuinely different URLs, so the hash never sees them as the same thing. Headlines for one event also share surprisingly few words — “Meta settles child-safety claims for $18B” and “Meta agrees to pay $17B, adopt kids' safety features” have almost no overlap — so plain word matching is close to useless on its own.

Three signals run in increasing order of cost:

1. A lexical check for near-identical wording, which is free and catches syndicated reprints.
2. Cosine similarity between headline embeddings. Above 0.78 the pair is a duplicate outright; below 0.62 it is distinct outright. Only the band between them costs anything more.
3. For that ambiguous band only, the model is asked directly whether two headlines describe the same event, with worked examples in the prompt to anchor what “same event” means — one announcement described two ways is the same story; two separate things involving one company are not.

Two details make it work. All the embeddings are computed in a single batched call before any of the comparisons start: the local runtime keeps one model resident, so interleaving embedding calls with chat calls makes it evict and reload between every single title, which is orders of magnitude slower than doing them together. And the model is asked about each pair in both orderings, with either “yes” counting as a duplicate, because the verdict turns out to be order-dependent — without that, the digest would depend on how tied articles happened to sort.

Every failure path keeps both stories. A repeated story is a much smaller problem than a lost one. The losing versions are not discarded either: each is recorded on the winner and becomes the “also covered by” line, which costs nothing because the comparison had to happen anyway, and a story five outlets ran is itself a signal. Scanning continues past the cutoff purely to attribute that coverage.

5. Writing it up

Only now, on the ten stories that will actually be published, does anything generate prose. The schema pins the output to exactly three paragraphs with assigned jobs — what happened, the specifics that matter, and why it matters — so the shape of a story card is guaranteed by the generation rather than hoped for.

Ordering matters here. Deduplication runs before the write-up, so no GPU time is ever spent on prose for a story that is about to be dropped as a duplicate. Splitting the write-up out of the grading pass entirely was the single largest speed win in the project: scoring reads about a hundred articles a day but only ten reach the site, so generating a summary during grading spent roughly ninety percent of that work on text nobody would read.

6. Publishing

The finished picks are pushed to a small API over an authenticated endpoint. There are two completely independent databases here — the local corpus on the scraper machine and the cloud one behind the site — and they are synchronized only through that single call. The local one keeps everything ever scraped, full text included, because the agent needs the whole corpus. The cloud one only ever receives the finished picks.

Serving is a query for the top ten of a given day. With no day requested, it serves today's digest if the run has landed and otherwise the most recent day that has one, which is what makes the archive work and what keeps the front page from being blank in the hours before a run finishes. An explicitly requested day never falls back — an archive URL has to answer for its own day or not at all.

The request path

A question hits a Lambda function URL configured for response streaming, which matters — the usual API gateway in front of a Lambda buffers the whole response, and buffering defeats the point of an agent that shows its work. Inside, an adapter layer runs an ordinary web server process and bridges it to the Lambda runtime, so the agent is a normal application rather than a handler function. The same code runs locally with no emulation.

The response is a stream of server-sent events, one frame per step: activity as it works, sources as they are found, tokens as the answer is written, and a final frame with the citations. The browser renders each frame as it arrives, which is why you watch the agent think rather than staring at a spinner.

It scales to zero. Between questions nothing is running and nothing is billed.

How a question gets answered

Verification: what earns a citation

Anything already in the corpus is trusted, because it got there by passing this same bar. Anything from the open web starts as an unverified candidate and is never cited on provenance alone. To become citable it has to survive corroboration:

• The page's own headline becomes the claim under test — at this point in the flow there is no answer yet to check, so the claim has to come from the page itself.
• An independent search runs on that claim, and results from the same site are excluded, so a publisher cannot corroborate itself.
• A separate model call with tools disabled has to find the claim supported by a different site and judge the page to be editorial reporting from an identifiable publication. Forums, job boards, layoff trackers, aggregators, wikis, press-release wires and user-generated posts are rejected by construction, and anything uncertain resolves to no.

Pages that fail are dropped entirely — not cited, not stored, not in the answer's context. Because verification costs a round-trip, only enough of it happens before the answer to make one page citable; the remaining candidates are verified after the answer has already streamed, so a reader waits on one check rather than all of them.

One deliberate consequence: when the agent concludes it does not have the information, it shows nosources at all. Listing the weak retrievals underneath “I have no information about this” would imply they were consulted and support the conclusion, when in fact they are the near-misses that caused the shortfall.

Memory that extends itself

Pages that survive verification are written into the knowledge base using the same key format the scraper uses — a hash of the canonicalized URL — so a page the agent learned from the web and the same page scraped later collapse into one document instead of two. Each is stamped with the date it was retrieved and a note that it arrived by corroboration rather than from a feed.

Indexing is fired without waiting, because the answer has already been delivered. The vector store permits only one indexing job at a time, so if one is already running the agent hands off to a small second function that retries until the slot frees. There is also a periodic sweeper as a backstop. The net effect is that a page cited from the live web this morning is an ordinary corpus source by the afternoon.

Nothing is ever evicted. Every source carries the date it reflects, retrieved passages are tagged with it, and the system prompt is rebuilt on every request with the current date so a warm instance cannot drift into thinking it is still yesterday. The model is told to prefer newer sources and to date-qualify older claims. Facts decay in weight rather than disappearing, which is what keeps historical questions answerable.

Conversation state

The agent itself is stateless — the browser holds the transcript and resends the recent turns with each request. Follow-ups are folded into the retrieval query before searching, because “what about before that?” retrieves nothing on its own. If a whole chain of follow-ups contains no topic, the query walks further back to the nearest turn that has one, so the search stays anchored to what is actually being discussed.

Rendering

The site is server-rendered with incremental static regeneration: pages are built once and revalidated on a five-minute window, which suits content that changes once a day. The digest, the archive and this page are all server components that read the API directly; the only client component is the chat, which needs to parse a live event stream.

Every fetch fails soft to an empty value rather than throwing, so a temporarily unreachable API produces a page with a section missing instead of an error.

Layout

From large screens up the whole page is pinned to exactly one viewport, and the panels scroll internally rather than the document scrolling. Two things make that hold. The layout is driven by a single fluid root font size clamped against both viewport dimensions — a 1440p monitor and a short window are the same width but not the same amount of room — and because the styling system expresses type, spacing and radii in relative units, moving that one value scales the entire design together rather than leaving fixed text inside boxes that grew.

The story card then fits the text to the box rather than the box to the text. It measures its own content against its fixed height and steps the type down until the story fits, with everything inside sized relative to that one value so the headline, bullets and spacing scale together. There is a floor, past which a very long story scrolls instead of shrinking into illegibility.

What's on the site

• The daily digest — ten stories, three paragraphs each, paged one at a time.
• The rubric— every story shows the five sub-scores that placed it and the model's own reasoning, so the ranking is inspectable rather than asserted.
• The agent — questions across the corpus and the live web, with streamed steps and verified, linked sources.
• The archive — every digest published so far, browsable by day.
• Coverage attribution — which other outlets ran the same story.
• Pipeline stats — what the last run actually did.

Last night's run

feeds polled

37/37

headlines seen

1,187

articles crawled

5

graded by the LLM

5

duplicates folded

4

stories published

10

Tech stack

Source

All of it is open: github.com/alanreyes24/the-tech-digest.

Built by Alan Reyes. Read today's digest or ask the agent something.