Benchmarking Open-Source LLMs on Verilog RTL Generation Across 50 Tasks (NMIMS, IIT Roorkee, BITS Pilani)

Researchers from NMIMS Hyderabad, IIT Roorkee, and BITS Pilani published a technical paper titled “Benchmarking LLMs for Verilog Design Flows.” Abstract Excerpt: The paper presents “aa reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs,” and reports that the pipeline improved “syntax validity from 0% to a 70.43% average and simulation pass rate to 51.8% across three open-source models..”Find the technical paper here. July 2026.arXiv:2607.22759v1Chakravertty, Angshuman, Rahul Koshti, Buddhi Prakash Sharma, and Vinay Chamola. “Benchmarking LLMs for Verilog Design Flows.” Manuscript prepared for submission to IEEE Design & Test.