LLMs Fail Without HelpDespite their strengths in reasoning and prompt-following, these LLMs repeatedly failed to autonomously achieve even partial goals in complex environments using state-of-the-art prompting techniques such as PentestGPT, ReAct, and CyberSecEval3. In nine out of 10 MHBench environments, LLMs equipped with Incalmo achieved at least partial success. In five environments, they were able to fully complete complex, multistep attacks, including exfiltrating data from dozens of databases and infiltrating segmented networks. On the other hand, the technology also reveals how quickly LLMs could become credible autonomous offensive tools with minimal scaffolding. They will release MHBench and Incalmo as open source, but will restrict built-in exploit libraries to known and safe vulnerabilities.