The next generation of autonomous AI agents is designed to work independently, accessing databases and executing tasks like drafting emails and processing invoices in real time. Because these agents operate at high speeds, a single mistake can affect business operations within seconds. To safely integrate them, business leaders need a robust plan for managing data, security, and governance, discussed during a recent Sifted Talks event in partnership with Box.
Panellists included Deliverance AI founder Mick Mc Neil, Box solutions engineer Eléonore Hébrard, Ralio cofounder Hanah-Marie Darley, and Ralio CT O Leo Rosales. The main shift from AI copilots to autonomous agents lies in the latter's ability to make decisions without human prompting. Mc Neil compared safely deploying these agents to onboarding a human employee, emphasizing that trust and autonomy must be earned over time rather than granted immediately.
Hébrard warned that when AI shifts from generating text to taking autonomous actions with database access, the potential impact of errors escalates significantly. However, humans must remain accountable even when agents act fully autonomously. Before an agent can be trusted, business data and context must be organised.
Hébrard stated, "You don't have an AI problem, you have a content problem," explaining that agents can hallucinate when fed disorganised data or outdated files. Mc Neil stressed that giving an agent knowledge requires providing context and strict boundaries, similar to defining a human role. Organisations using third-party models should approach deployment with scepticism regarding reliability, following zero trust principles and blocking agents from acting outside their mandate.
Establishing accountability through Governance, Compliance, and Risk teams before pilots begin is crucial for successful scaling.
Source: Sifted · Summarized by HeadlinesBriefing