Building a near real-time topic monitoring system on a VPS

Too much noise, not enough signal

Following several topics in parallel is expensive in time. Google Alerts sends half-relevant results several times a day, raw RSS feeds drown you in dozens of articles saying the same thing differently, and SaaS tools like Mention, Talkwalker or Feedly Pro cost $30 to$300/month for features you can reproduce with RSS, a script and an LLM.

The requirements:

• Multi-topic: follow 3-5 subjects in parallel, each with its own sources and keywords
• Near real-time: check every 10 minutes
• Smart filtering: not just keyword matching, real relevance scoring
• Usable summaries: 2-3 sentences giving the gist without clicking
• Zero duplicates: one event covered by 10 outlets = 1 message
• Self-hosted: the data stays with you

The architecture

A $10/month VPS, Docker, Node.js, an LLM CLI, and a cron. Two scripts:

• monitor.js: aggregates RSS feeds, filters, scores via LLM, pushes to Slack
• page-monitor.js: watches RSS-less web pages (changelogs, blogs) by comparing a SHA-256 hash of the content

The Docker stack

services:
  n8n:
    image: ghcr.io/n8n-io/n8n:latest
    ports:
      - '5678:5678'
    environment:
      - GENERIC_TIMEZONE=Europe/Paris
    volumes:
      - n8n-data:/home/node/.n8n

  changedetection:
    image: ghcr.io/dgtlmoon/changedetection.io:latest
    ports:
      - '5000:5000'
    volumes:
      - changedetection-data:/datastore

  rsshub:
    image: ghcr.io/diygod/rsshub:latest
    ports:
      - '1200:1200'
    environment:
      - CACHE_TYPE=memory
      - CACHE_EXPIRE=600

• n8n: visual automation platform, useful for adding graphical workflows later. Not required for basic monitoring.
• RSSHub: turns almost anything into an RSS feed (GitHub repos, subreddits, YouTube channels). Essential when the source has no native feed.
• changedetection.io: web UI for watching pages, handy for adding watchers without touching code.

The whole thing runs on ~1.5 GB of RAM; a 4 GB VPS handles it comfortably.

The filtering pipeline: 4 layers

The idea: strip out as much noise as possible before calling the LLM, since every call costs time and tokens.

Layer 1: freshness

const MAX_AGE_HOURS = config.max_age_hours || 6

function isRecent(item) {
  if (!item.pubDate && !item.isoDate) return true
  const pubDate = new Date(item.isoDate || item.pubDate)
  if (isNaN(pubDate.getTime())) return true
  const ageMs = Date.now() - pubDate.getTime()
  return ageMs >= 0 && ageMs < MAX_AGE_HOURS * 60 * 60 * 1000
}

Google News returns 100 articles per query, most older than 24h. With a 6h threshold, you go from 100 to 5-20. Configurable in config.json: raise it to 12 or 24h for a daily digest.

Layer 2: URL + title deduplication

The main trap: Google News generates a unique redirect URL for every result, even when two links point at the same article. "Iran strike - Reuters" and "Iran strike - BBC" have completely different news.google.com/rss/articles/CBM... URLs, so URL-only dedup doesn't work. Hence title normalisation:

function normalizeTitle(title) {
  if (!title) return ''
  return title
    .toLowerCase()
    .replace(/\s*[-–—|:]\s*(the\s+)?(reuters|ap|bbc|cnn|...).*$/i, '')
    .replace(/[^a-z0-9àâäéèêëïîôùûüÿçæœ]/g, '')
    .replace(/^(update|breaking|live|exclusive|watch|video)\s*/i, '')
    .slice(0, 60)
}

Strip the source suffix (- Reuters, | BBC), editorial prefixes (BREAKING:, LIVE:) and punctuation, then compare the first 60 normalised characters.

SQLite storage with an index on the title hash, 30-day retention and automatic cleanup:

CREATE TABLE seen_articles (
  url TEXT PRIMARY KEY,
  title TEXT,
  title_hash TEXT,
  topic TEXT,
  seen_at TEXT DEFAULT (datetime('now'))
);
CREATE INDEX idx_title_hash ON seen_articles(title_hash);

Layer 3: keyword pre-filter

Free and instant. Each topic has its keyword list in the config:

{
  "name": "My Topic",
  "keywords": ["keyword-1", "keyword-2", "exact phrase"],
  "slack_channel": "C0XXXXXXX",
  "feeds": ["https://news.google.com/rss/search?q=...", "https://github.com/org/repo/releases.atom"]
}

A plain lowercase includes() on title + excerpt, deliberately permissive. Fine filtering is the LLM's job right after.

Layer 4: LLM scoring + semantic deduplication

Surviving articles go to the LLM in batches of 25:

Score each article (0-10) on its relevance to the topic.
DEDUPLICATE: if several articles cover the same event,
keep only the best one.
2-3 sentence summary with the key facts.
Include ONLY score >= 6.

One call, three outputs: a relevance score (an article mentioning a keyword in passing lands at 3 and isn't sent), a semantic dedup (five articles on the same event, best one kept), and a usable summary.

The result is structured JSON parsed on the Node.js side. If the LLM times out or crashes, a fallback returns the articles with a default score — the system never breaks.

Why score before summarising? You could summarise everything then filter. But scoring first cuts 80-90% of the volume to process, and therefore the token bill. Free keywords first, paid LLM only on serious candidates.

Page monitoring

For sources with no RSS (changelog, docs page, feed-less blog):

const content = extractMainContent(html)
const hash = createHash('sha256').update(content).digest('hex')

const existing = db.prepare('SELECT hash FROM page_hashes WHERE url = ?').get(url)
if (existing && existing.hash !== hash) {
  // Change detected -> Slack alert
  await postToSlack(channel, `🔔 Change detected on ${pageName}`)
}

Extract the <main> content — ignoring headers, footers and ads that shift constantly — hash it, compare. Zero false positives after two weeks of use. Runs every 30 minutes via cron, with negligible consumption.

The cron

*/10 * * * * cd ~/news-monitor/app && node monitor.js >> monitor.log 2>&1
*/30 * * * * cd ~/news-monitor/app && node page-monitor.js >> monitor.log 2>&1

Another production trap: under fish, heredoc syntax (<<EOF) doesn't exist. To write config files you need bash -c or scp.

The Slack output

🔴 Geopolitics Monitor - 17/03/2026 09:50

• Article title
> 2-3 sentence summary giving the key facts.
> The reader gets the gist without clicking.
_8/10 - 17/03, 08:30_

• Another article on a different subject
> Context and important details summarised here.
> Impact and consequences mentioned.
_7/10 - 17/03, 07:15_

No duplicates, no noise. If nothing is new since the last run, nothing is sent.

The numbers

On a typical run with an active geopolitics topic:

400 articles down to 5, a 1:80 signal-to-noise ratio. First run: 26 articles pushed to Slack. Second run, ten minutes later: zero. The dedup works.

What's next

This setup covers most of the need. A few directions:

• Telegram OSINT sources over MTProto — some channels break news 15-30 minutes before mainstream outlets
• Local LLM via Ollama to remove the external API dependency. A Llama 3.2 8B runs on 8 GB of RAM and is plenty for scoring
• Web dashboard with history and stats (n8n is already deployed)
• Push alerts for scores of 9-10, instead of waiting for the next poll

The code is two files (~200 lines each), a JSON config and a Docker Compose. No framework, no exotic dependency: if the VPS dies, you redeploy in ten minutes.

Stack: Debian 13 - Docker Compose - Node.js 22 - RSSHub - changedetection.io - SQLite - LLM CLI - Slack API