Researchers embedded backdoors in 40MB binaries and tested AI agents like Claude Opus 4.6 and Ghidra to identify them. Surprisingly, some models detected hidden mechanisms, but results were mixed. Claude Opus 4.6 found 49% of backdoors in small/mid-size binaries, though false positive rates remained high, flagging clean files as malicious. This benchmark, called BinaryAudit, evaluates AI’s ability to analyze stripped executables without source code or debug symbols.

Binary analysis is critical for uncovering malicious code in closed-source software, where compilers strip human-readable context. Tools like Ghidra (developed by the NSA) reverse-engineer machine code into pseudo-C, but obfuscation from optimizations complicates detection. For example, a backdoor in Ghidra’s decompiled code appears as `FUN_00130550`—meaningless identifiers that obscure malicious logic.

The study used modified open-source projects like lighttpd and Dropbear, injecting simple backdoors such as command execution via undocumented HTTP headers. AI agents were given binaries and reverse-engineering tools to pinpoint backdoor locations. While AI detected some anomalies, it struggled with obfuscated code, highlighting gaps in current models’ specialized reverse-engineering skills.

Results underscore the need for hybrid approaches combining AI with human expertise. False positives and missed detections pose risks for supply chain security, where altered binaries in firmware or critical infrastructure could enable state-sponsored attacks. The research, hosted on quesmaOrg/BinaryAudit, aims to advance AI’s role in binary analysis while acknowledging its current limitations in production environments.