Data Engineering  —  2026

Real-Time Fraud
Detection Pipeline

A streaming fraud detection system that processes financial transactions in real time using Apache Kafka, applies rule-based detection, and visualises everything on an interactive live dashboard built with Streamlit and Plotly.

Python Apache Kafka Docker Streamlit Plotly Pandas KRaft Mode
System Architecture
⚡
Producer
Python script
→
📦
Kafka
Message broker
→
🔍
Consumer
Fraud detection
→
📄
CSV Store
Persistence
→
🌐
Dashboard
Streamlit app
How It Works
From stream to insight in milliseconds
The pipeline starts with a producer generating realistic financial transactions — 8% are intentionally suspicious. Each transaction flows through Apache Kafka running in KRaft mode (no Zookeeper), where a consumer reads the stream, applies multi-signal fraud rules, and flags risky activity. Results are persisted to a CSV and visualised on a live Streamlit dashboard that auto-refreshes every 2 seconds.
⚡
Real-Time Streaming
Transactions are generated and consumed in real time via Apache Kafka. No batch processing — every transaction is analysed as it arrives.
🔎
Rule-Based Detection
Multi-signal fraud detection using transaction amount, time of day, and geographic location. Flags trigger when 2+ rules fire simultaneously.
🎨
Interactive Dashboard
Streamlit-powered dashboard with risk gauges, heatmaps, sunburst charts, and live transaction feeds — all auto-refreshing.
📦
KRaft-Mode Kafka
Runs Apache Kafka 3.8 in KRaft consensus mode via Docker — no Zookeeper dependency, faster startup, simpler architecture.
🔧
One-Click Deploy
Start and stop the entire stack with batch scripts. Four components spin up in separate windows automatically.
📊
Rich Visualisations
Plotly-powered charts including spline timelines, fraud heatmaps, sunburst breakdowns, overlapping histograms, and horizontal risk rankings.
Detection Logic
How fraud is identified
A transaction is flagged when it triggers two or more of the following rules simultaneously. This reduces false positives while catching genuinely suspicious patterns.
Rule Threshold Rationale
High Amount > $4,000 Unusually large transactions are a common fraud indicator
Unusual Hour 11 PM – 6 AM Transactions outside business hours carry higher risk
Risky Location Lagos, Moscow, Unknown Locations with elevated fraud patterns in the dataset
Dashboard Preview
What the live dashboard looks like
The Streamlit dashboard auto-refreshes every 2 seconds with real-time metrics, interactive charts, and fraud alerts.
⚠ HIGH ALERT — 6.8% fraud rate detected across 342 transactions
Transactions Monitored
342
Fraud Detected
23
6.8% of total
Legitimate
319
Total Volume
$284,150
Avg Transaction
$830.85
Risk Level
6.8%
Transaction Stream
▬ Legitimate ▬ Fraudulent
Fraud Heatmap by Hour & Location
Lagos
Moscow
Unknown
0246810121416182022
Top Risky Users
USER-2847
$18,400
USER-1193
$14,250
USER-5521
$11,800
USER-3390
$8,600
USER-7762
$5,300
Recent Fraud Alerts
Risk TXN ID User Amount Type Location Triggered Rules
● Critical TXN-482931 USER-2847 $12,450.00 transfer Lagos high_amount, unusual_hour, risky_location
● High TXN-330185 USER-1193 $8,200.00 withdrawal Moscow high_amount, risky_location
● High TXN-771204 USER-5521 $6,750.00 purchase Unknown high_amount, risky_location