The AI firehose,
sifted down to signal.
Sift collects what matters from X, 小红书, Hacker News, arXiv, GitHub and your feeds, scores every item on six explainable axes, folds away the duplicates, and gives you one keyboard-driven place to read, annotate, search and synthesise it. Everything runs on your machine, in one SQLite file.
pnpm install && pnpm seed && pnpm dev
Node 24+ · zero native modules · no account · no API key required
The problem
Following AI properly is a part-time job you did not apply for.
Six timelines, forty feeds, three languages. The same launch reported twenty times. Engagement bait ranked above the benchmark table that actually answers your question. Bookmarks you will never open again because there is no way to search them.
Every existing tool solves one slice: a reader that cannot rank, an aggregator that cannot search your own highlights, a scraper with no opinion about what is worth reading. Sift is the whole loop, and it runs locally so your reading history is nobody's dataset.
20×
duplicate reports of one launch, folded to one canonical item with an echo count
6
explainable score components, every one visible and tunable
<15ms
hybrid search across a hundred thousand items, exact not approximate
0
native dependencies, accounts, trackers or required API keys
How it works
Collect, score, then actually work with it.
Three stages, one pipeline. Each one is a few hundred lines you can read and change.
Collect
Eight built-in connectors poll public, keyless endpoints on a schedule. The browser extension covers what needs your own session — X and 小红书 — by reading the JSON your browser already received.
canonicalise → filter → enrich → fingerprint
Score
Every item gets six components, a per-source trust multiplier and an engagement-bait demerit. Duplicates fold behind their canonical item. Anything below your threshold is archived, not discarded.
dedup → embed → score → store
Work
Triage by keyboard, search in either language, highlight and tag, ask questions against your own corpus, generate a cited briefing, and export everything as Markdown, JSON or CSV.
triage → search → synthesise → share
Sources out of the box
Signal scoring
One number, and it shows its work.
Every item gets a 0–100 score answering one question: should this interrupt you? Six components, each squashed so no single axis can be gamed into dominance, combined by weights you control.
The breakdown is stored with the item, so the app can always answer why is this a 92? without re-running anything. Move a slider and the whole library rescores in the background.
Measured on the demo corpus
A thread with benchmark numbers, an ablation and a repo link scores 66. A post with 32,000 likes, nine hashtags and “this will blow your mind” scores 31 — the noise penalty and the depth term both bite.
Relevance
26% default weightCosine distance to what you have actually saved, clustered so a robotics interest and an inference interest stay distinct instead of averaging into mush.
Velocity
22% default weightAttention per hour, normalised against each platform’s own distribution and log-squashed so one viral post cannot own the top of your inbox.
Depth
15% default weightRewards evidence: numbers, code, citations, artefacts, hedged language. A 4,000-word SEO page scores below a 300-word post with a benchmark table.
Novelty
14% default weightOne minus the similarity to everything already in your library. This is what stops the fortieth report of one launch from reaching you.
Authority
13% default weightReach blended with revealed preference — how often you keep this author’s work. The second half cannot be bought, so it carries most of the weight.
Recency
10% default weightExponential decay with a half-life you set. Default 36 hours: yesterday still matters, last month rarely does.
Then: multiplied by a per-source trust factor you set, and reduced by up to 30% for recognisable engagement-bait patterns — emoji density, hashtag stuffing, giveaway language, in English and Chinese. Capped at 30% because bait wrapped around real news is still real news.
Features
Built for someone who reads for a living.
Hybrid search that works in two languages
BM25 over FTS5, concept expansion through a 24-topic AI taxonomy, and exact vector KNN — fused by reciprocal rank. Chinese is indexed as character bigrams, so 「量化」 and “quantisation” both find the same note.
Two-tier deduplication
SimHash with LSH banding catches verbatim reposts; embedding similarity gated on a shared named entity and a 72-hour window catches the same story told in different words. Nothing is deleted — echoes fold behind a canonical item.
Explains every score
Press W on any item to see all six components, their weights, and the sentence-level reasons. A ranking you cannot interrogate is a ranking you cannot trust.
Ask your own library
Retrieval-augmented chat scoped strictly to what you have collected. Citations arrive before the first token and link back to the item. If the answer is not in your corpus, it says so instead of inventing one.
Keyboard-first triage
J/K to move, S to save, E to archive, ⌘K for everything. Optimistic updates with six seconds of undo, so two hundred items is a few minutes rather than an afternoon.
Trends that tell you something
Not “which topics are big” — always the same few — but which are accelerating, computed against the previous window with smoothing so a fresh install does not report 17x growth on three items.
Bilingual by construction
中文和英文，一视同仁。
Half the interesting AI writing is in Chinese, and most tools index it into oblivion — SQLite's default tokeniser treats a whole Chinese sentence as one token, so Chinese search silently returns nothing.
Sift indexes a parallel column of character bigrams and translates queries into ordered bigram phrases, which is exact substring matching. A full-sentence question has its interrogative affixes stripped and becomes overlapping windows, so 「怎么降低推理成本」 finds the note titled 「大模型推理成本优化实战」.
怎么降低推理成本
↓ strip 怎么, window, phrase-match
cjk : ("降低 低推 推理" OR "推理 理成 成本" ...)
vLLM 推理
↓ mixed script splits cleanly
cjk : "推理" AND "vllm"*
making models cheaper to run
↓ no keyword overlap, so concept expansion
topic:efficiency → 12 items, exact match first
Local-first, not local-ish
Your reading history is nobody's dataset.
One file
Everything lives in ./data/sift.db. Back it up by copying it; delete it to start over. No hidden state, no cloud mirror.
Loopback only
The server binds to 127.0.0.1. CORS admits extension origins and localhost, nothing else.
Keys are write-only
An API key you paste goes into the local database and is never returned by any endpoint — only whether one is set.
No telemetry
Not opt-out, absent. There is no analytics code in the repository. Grep for it.
Narrow permissions
The extension asks for three hosts plus loopback. No <all_urls>, no tabs, no cookie access. Page capture rides on activeTab.
Real export
Markdown with your highlights, JSON with every field, CSV for a spreadsheet, OPML for your feeds. A tool that cannot hand your data back is a silo.
Get started
Four commands, then it is yours.
Needs Node 24 or newer and pnpm. No database to provision, no key to obtain, nothing to sign up for.
git clone https://github.com/micaho26/sift && cd sift Clone it.
pnpm install Zero native modules, so this cannot fail on a compile step.
pnpm seed Optional: 37 realistic items so you can judge it before configuring anything.
pnpm dev Starts the API and the app, then opens your browser.
Then, for X and 小红书
Run pnpm build:extension, open chrome://extensions,
enable developer mode, and load apps/extension/.output/chrome-mv3 unpacked. Browse
as you normally would — posts you scroll past are scored and filed. Press
⌥⇧S on any page to capture it.
Questions
The things people actually ask.
Does anything leave my machine?
No, unless you choose it. The server binds to loopback, the database is one SQLite file in ./data, and there is no telemetry of any kind. The only outbound traffic is to the sources you configure — and to an AI provider if you paste a key, which is entirely optional.
Do I need an API key?
No. Embeddings default to a built-in hashing embedder that needs no download and works offline, so semantic search, novelty detection and duplicate folding all work on first run. Summaries fall back to local extractive summarisation. Add Anthropic, OpenAI or Ollama when you want written prose.
How does the extension get data from X?
It reads the JSON your browser already received. A MAIN-world script observes the page’s own fetch and XHR responses and forwards them to the extension — nothing is re-requested, no credentials are touched, and no scraping traffic is generated. The DOM is scraped only as a fallback when the response shape changes.
Is this against the platforms’ terms?
Sift reads what you are already logged in and looking at, at the rate you browse, for your own private reference. It does not automate accounts, bypass access controls, or redistribute anything. Automatic collection is per-site and can be turned off with one keystroke.
What are the actual dependencies?
Zero native modules. SQLite comes from Node 24’s built-in node:sqlite, so pnpm install can never fail on node-gyp. The server has four runtime dependencies; the vector index is a hand-written flat scan that does exact KNN over a contiguous Float32Array.
How big does it get?
A 100k-item library is roughly 400 MB of SQLite plus 154 MB of in-memory vectors at 384 dimensions. Search stays under 15 ms because the corpus is small enough for exact retrieval — no approximate index, no recall cliff.