A team built a Incident Memory System on AWS to turn routine alerts into actionable knowledge. Instead of chasing every alarm, they let the system record failures, learn from manual fixes, and replay those fixes automatically. The goal: reduce on‑call fatigue and preserve operational memory for every deployment.

The architecture splits responsibilities into five layers: detection with CloudWatch alarms, routing via EventBridge, memory in DynamoDB, execution through Lambda functions, and recovery handled by AWS Systems Manager. Each layer stays isolated, so a failure in one doesn’t mask the others and the system degrades gracefully in production.

To test the system, they launched a plain EC2 instance running nginx behind an ALB and then scripted the server to crash repeatedly. CloudWatch detected the rising 5XX errors, sent an event to EventBridge, and the incident collector Lambda logged the first record in DynamoDB in production environment.

After manually restarting nginx, the alarm cleared and the auto‑resolver Lambda, guided by the stored incident, issued a Systems Manager command to restart the service again. The incident status flipped to AUTO RESOLVED, proving the engine could now replay a proven fix without human intervention for future incidents.