Cybersecurity
SIEM Threat Intelligence Pipeline
A modular threat-intelligence pipeline that ingests, normalizes, enriches and visualizes indicators of compromise from multiple public feeds.
Problem
The problem
Threat-intelligence providers expose indicators in different formats and with different levels of context. The project explores how those feeds can be transformed into a consistent, queryable stream of security data.
Approach
What I built
- Concurrent Python producers for OTX, AbuseIPDB and ThreatFox
- Kafka-based ingestion pipeline
- A unified normalization schema
- Deterministic SHA-256 identifiers for cross-feed consistency
- ClickHouse storage using a Kafka engine, materialized views and TTL retention
- Grafana dashboards for exploration and visualization
Outcome
What it demonstrates
Streaming architecture, data normalization, security-focused data engineering and designing an end-to-end pipeline.
Future improvements
Advanced deduplication, ASN-level enrichment, real-time alerting, correlation logic and a complete Docker Compose deployment.