Incident Memory
Website
In modern SaaS engineering, on-call teams are the backbone of reliability — they respond to incidents at all hours, diagnose root causes under pressure, and apply fixes to keep systems running. The problem is that incident knowledge is inherently repetitive. A connection pool exhaustion that took down the payments service today will likely hit the checkout service next week. A TLS cert expiry that caused an auth outage will recur on the API gateway months later. Yet this knowledge lives scattered across Slack threads, Jira tickets, and the fading memories of individual engineers.
Client:
HackWithBanglore
Role:
Database Administration
Year:
2026
Explore the full story –
Challenge
On-call teams repeatedly face the same or similar incidents (connection pool exhaustion, cache stampedes, TLS cert expirations, memory leaks) but lack a system to learn from past resolutions. Each incident is treated as fresh, wasting time re-diagnosing known issues.
Objective
To build an AI-powered, self-compounding incident memory system that helps on-call engineering teams stop solving the same problems twice. The system takes a new incident description, embeds it using Gemini vector embeddings, and performs semantic search over a Qdrant vector database to retrieve the most similar past incidents from Postgres. An LLM (Groq llama-3.3-70b) then reasons over the retrieved evidence and suggests a concrete fix with root cause analysis. The key goal was to close the feedback loop — every approved resolution is embedded and written back into memory, so future similar incidents automatically benefit from accumulated knowledge, making the system smarter with each incident resolved.
Results
The project delivers a complete end-to-end pipeline — embed, retrieve, suggest, write-back, and postmortem — wrapped in a polished dark-themed web UI. It was seeded with 12 real-world incidents across 7 services (payments, auth, database, checkout, etc.), and every new resolution compounds the memory. A built-in performance dashboard tracks Recall@5, MRR, and latency benchmarks across runs, with a latency budget donut chart showing time breakdown across embedding, vector search, Postgres enrichment, and LLM calls. The system also includes a webhook endpoint for automated incident ingestion, soft-delete/restore/purge lifecycle management, export/import of memory snapshots, and automatic postmortem document generation — all running on Flask, Postgres, Qdrant, and free-tier AI APIs (Gemini + Groq).
