NEXUS: AI-powered cyber threat intelligence platform
An end-to-end intelligence platform: discovery and scraping of open sources, LLM extraction, idempotent queue processing, entity graph with deduplication, Prometheus observability and a Next.js Security Events section — shipped as MVP on a tight deadline.
Client context
CyberRisk Alliance needed NEXUS — a system to aggregate cybersecurity news and advisories from hundreds of sources, enrich them with LLM extraction and expose structured vulnerability data to editorial teams and end users via scworld.com/security-events.
Challenge
At-least-once message delivery created duplicates; anti-bot sources returned empty content; LLM costs had to stay controlled; the MVP deadline was fixed while requirements were still evolving. The team lacked unified AI practices and production observability.
What we did
- Stood up MVP delivery with task decomposition, backend/frontend/infra coordination and an AI-first workflow using Claude with project-specific skills and security sandboxing.
- Built discovery (RSS, sitemap, listing pages) with token-bucket rate limits and a scraping pipeline with Playwright fallback for protected sources.
- Implemented idempotent canonicalization (processing ledger, DLQ), URL and content-hash deduplication and a graph worker with entity resolution and temporal versioning.
- Added Prometheus metrics across scraping, LLM latency, pipeline stages and API; precomputed event aggregations to cut DB load by an order of magnitude.
- Delivered REST API contract-first for frontend; Security Events UI with reusable React components, Storybook coverage and SSG data pipelines.
Process
- MVP scope lock and parallel tracks for scraping, graph, API and frontend.
- Iterative hardening: DLQ, dedup indexes, enrichment jobs, Grafana dashboards.
- AI-assisted delivery with documented Claude skills and CI code-review automation.
- Integration testing and smoke tests in CI/CD before production rollout.
Result and impact
MVP launched on schedule. The platform processes heterogeneous sources reliably, exposes structured security-event data and gives the team full pipeline observability — foundation for the public Security Events product surface.
This is an NDA-protected engagement. Client name and identifying details are withheld; industry, region, challenge, solution and outcome are shared in an approved form.
We went from opaque logs to a pipeline we can actually operate in production.Engineering lead (quote representative, under NDA)