In the second installment of a three-part series on creating production-ready vector search for enterprise SaaS, the focus is on production resilience and monitoring. This part delves into the critical strategies that ensure a system can withstand traffic spikes and outages, using the Black Friday Incident as a case study. On November 24, 2023, a major client launched a compliance platform to 5,000 users, causing search traffic to surge from 800 to 4,200 requests per minute.

The Qdrant cluster struggled, with CPU usage hitting 98% and query latency spiking to 12,000ms. Without circuit breakers, the system would have collapsed, leading to a complete outage for all 150 clients. However, with circuit breakers in place, 99.2% of searches continued to function, thanks to cached results and a fallback to PostgreSQL full-text search.

The implementation of rate limiting per tenant is also highlighted. This approach prevents a single tenant from overwhelming the system, as seen with a tenant that triggered rate limiting 1,847 times in a minute. By isolating rate limits, the system avoids denial-of-service attacks.

The article also covers health checks and Prometheus metrics, providing real-time observability and beautiful visualizations through Grafana dashboards. These tools offer insights into search performance, throughput, cache hit rates, and circuit breaker status, ensuring the system's resilience and reliability.