Real-Time Streaming Fraud Detection Engine
Sub-50ms financial transaction anomaly scoring with PySpark and Redis
A distributed real-time streaming ML pipeline processing payment streams via Apache Kafka, computing real-time feature aggregations in Redis, and executing XGBoost anomaly classification under strict latency SLAs.
Performance Metrics & Quantified Impact
The Challenge & Core Problem
Financial institutions lose millions annually to fraudulent payment activities. Traditional batch fraud reviews identify illicit behavior hours after transactions settle, making real-time interception impossible.
Key Engineering Objectives
- Ingest high-velocity transactional events from Apache Kafka with zero message loss.
- Calculate real-time streaming feature vectors using Redis sliding window data structures.
- Serve an optimized gradient-boosted decision tree model (XGBoost) for instant scoring.
- Provide an asynchronous audit trail and alerting webhook for compliance teams.
System Architecture & Pipeline Design
Kafka distributes incoming payment events across PySpark Streaming consumers, which update Redis sorted-set sliding windows, construct the feature vector, run ML inference, and publish decision outcomes back to payment gateways.
Payment Gateway → Kafka Cluster → PySpark Structured Streaming → Redis Online Feature Store → XGBoost C++ Inference Engine → Kafka Verdict Topic → PostgreSQL Audit Store & Grafana Live Dashboard
Kafka Event Stream Ingestion
Implemented a multi-partition Kafka topic cluster with exactly-once consumer semantics and Avro schema validation to ensure strictly typed transaction payloads.
Low-Latency Sliding Window Feature Store
Engineered custom Redis Lua scripts utilizing Sorted Sets (`ZADD`, `ZREMRANGEBYSCORE`) to maintain rolling 1-hour transaction frequencies and velocity counters with sub-millisecond retrieval.
XGBoost Anomaly Classification
Trained an XGBoost model with focal loss to counter severe 1:1000 class imbalance, tuned using Bayesian optimization, and compiled to native C++ shared library for zero-overhead inference.
Audit Logging & Observability
Persisted all scored records with explainability features (SHAP values) into partitioned PostgreSQL tables and visualized system metrics in Prometheus/Grafana.
Technical Obstacles & Engineering Solutions
Obstacle #1
Extreme class imbalance (fraudulent transactions represented <0.12% of total volume), causing standard classifiers to generate high false positives.
Applied SMOTE-NC on tabular categorical features, tuned decision thresholds via precision-recall curve analysis, and implemented an ensemble layer combining Isolation Forest with cost-sensitive XGBoost.
Obstacle #2
Redis network bottleneck when making multiple roundtrip queries per transaction.
Batched feature updates using Redis Pipelining and compiled transactional validation into atomic Lua scripts executed on the Redis server in a single network hop.
Impact & Key Takeaways
Delivered a production-ready streaming anomaly detection platform that satisfies stringent financial SLA requirements while offering deep explainability for compliance audits.
Interested in similar architectures for your team?
Let's discuss custom Data Pipelines, AI deployment, or high-throughput Python backends.