At ACM FAcc T, we presented research highlighting the critical need for rigorous evaluation of AI guardrails, mirroring the scrutiny applied to the AI models themselves. Our work demonstrated a shift from static policies to dynamic, context- and language-specific evaluations.

We conducted a hands-on session showcasing how agentic guardrails, equipped with tools like web search, are essential for reliable real-world AI deployment. Our methodology involved evaluating 120 refugee and asylum-focused scenario pairs across multiple languages, scored by native speakers. This revealed that guardrails, which filter LLM outputs, often receive less attention than the models they protect, despite their significant impact on user experience.

The session revealed that while tools didn't always change the final verdict, they provided crucial supporting evidence. For instance, 90% of verdicts agreed between agentic and non-agentic judges. However, the effectiveness of agentic behavior varied by LLM, with Claude Sonnet 4.6 utilizing web search more frequently than GPT-5 Nano. Tools proved vital in verifying factual claims and identifying errors overlooked by tool-less judges, influencing outcomes and improving trustworthiness.

We utilized Mozilla AI’s open-source any-guardrail and Otari to facilitate this research, enabling flexible guardrail and LLM selection. Future work will continue testing tool access for LLM-enabled guardrails across various use cases and languages.