August 2026

Overcast

Real-Time Distributed Sports Telemetry & Processing Platform

Go 1.22+ · Apache Kafka (KRaft) · Protocol Buffers (v3) · PostgreSQL 16 · Redis 7 · Actor Model · Prometheus · Grafana · Docker Compose (9 Services) · Gin Framework · WebSockets

Source Repository

Highlights

1.37ms p50 latency on warm Redis cache hits (66.3% faster than DB fallback)
73.3% Prometheus-verified cache hit ratio across 3,000+ benchmarked requests
Per-match Actor Model with 3s idle self-destruction to eliminate global mutex contention
Full 9-container microservices architecture with auto-provisioned Grafana dashboards

System Overview

Overcast is a distributed, high-throughput, low-latency sports event processing platform in Go. The system continuously ingests live sports telemetry via asynchronous event-driven pipelines, stores state in dual storage layers (Redis & PostgreSQL), broadcasts live updates to client applications via WebSockets and REST APIs, and delivers end-to-end system observability using Prometheus and Grafana.

The Problem & Architectural Rationale

Serving high-frequency live sports scores creates massive database read spikes and lock contention under multi-client loads. Directly querying third-party APIs risks hitting rate limits and introduces cascading failure points. Managing client WebSocket connections with global mutexes causes severe thread blocking during live score broadcasts.

Architecture & Components

1. Ingestion Service (services/ingestion)

Periodically polls external cricket REST APIs, transforms raw JSON into binary Protocol Buffer structures (cricket.v1.MatchData), and publishes events to the partitioned Kafka topic 'raw-match-events' using LeastBytes partition balancing.

2. Processing Service (services/processing)

Consumes Protobuf events from Kafka via the consumer group 'processing-group' and executes a Dual-Write pattern: caching live match state in Redis (match:live:<id> with 10m TTL) and persisting durable SQL records in PostgreSQL using ON CONFLICT upserts.

3. Query API Service (services/api)

Implements a high-performance RESTful API using Gin on port 8080 with a Cache-Aside strategy: queries Redis for instant response (~1.37ms), falling back to PostgreSQL on cache miss (~4.07ms) while lazily warming the cache.

4. Notification Service (services/notification)

Serves real-time WebSockets on port 8081 using an Actor Model. The Supervisor dynamically spawns a MatchActor per live match that subscribes to Redis Pub/Sub channels (match:updates:<id>). If all clients disconnect, the actor enters an idle state and self-destructs after 3 seconds to reclaim memory.

5. Observability Stack (monitoring/)

Prometheus scrapes telemetry metrics every 15s across API (:8080), Notification (:8081), and Processing (:8082) services. Grafana is auto-provisioned with custom dashboards displaying active actors, connected WebSocket clients, and cache hit ratios.

Empirical Benchmarks

Metric	Cold DB Path	Warm Cache Hit	Improvement
p50 (Median Latency)	4.07 ms	1.37 ms	66.3% faster
p95 Latency	23.05 ms	2.81 ms	87.8% faster
p99 Tail Latency	30.68 ms	3.65 ms	88.1% faster
Mean Latency	8.02 ms	1.51 ms	81.2% faster

Core Implementation

// Processing Worker: Consuming Kafka Protobuf stream and executing Dual-Write
func (pw *ProcessingWorker) Start(ctx context.Context) {
    defer pw.reader.Close()
    for {
        msg, err := pw.reader.ReadMessage(ctx)
        if err != nil { return }

        var match cricketv1.MatchData
        if err := proto.Unmarshal(msg.Value, &match); err != nil {
            continue
        }

        // Dual Write Pattern: Volatile Redis Cache + Durable PostgreSQL
        if err := pw.redis.CacheLiveMatch(ctx, &match); err != nil {
            log.Printf("Redis error for match %s: %v", match.GetId(), err)
        }
        if err := pw.pg.UpsertMatch(&match); err != nil {
            log.Printf("Postgres error for match %s: %v", match.GetId(), err)
        }
    }
}

Processing worker Kafka Protobuf deserializer and dual-write storage engine

Outcomes & Results

Eliminated global broadcast mutex bottlenecks using the isolated Actor Model pattern.
Achieved 73.3% cache hit ratio under sustained load testing across 3,000+ benchmark requests.
Buffered ingestion bursts through Kafka partitions, insulating downstream microservices from traffic surges.