A Multi-Agent LLM Framework for Automated Software Testing
DOI:
https://doi.org/10.63808/tcs.v2i2.447Keywords:
Multi-agent system, Large language model, Automated software testing, Defect detection, Test case generationAbstract
Research on integrating specialized, language-model-based agents for automated test-case generation remains limited, and the detection criteria under which such systems are evaluated have not been standardized, which makes reported rates difficult to compare across studies. This paper presents a multi-agent testing framework in which requirement analysis, test-case generation, sandboxed execution, and defect detection are delegated to four distinct agents, and in which detection rests on a composite rule combining the execution signal with the semantic judgment of a dedicated diagnostic agent. The framework is evaluated on the QuixBugs dataset of forty Python programs under three criteria of increasing strictness. A two-run differential protocol against the reference implementation, recommended here as the primary indicator, yields a detection rate of 55.0%, against 90.0% under the unconditional criterion conventional in prior work and 5.0% under a strict criterion requiring the suite to pass entirely on correct code. The thirty-five-point gap is traced to language-model test artifacts, of which hallucinated oracle values and signature mismatches account for the majority. An ablation with a same-model single-agent baseline attributes the framework’s advantage to role decomposition rather than to the diagnostic agent specifically, and the distribution of detection across the fourteen defect classes defined by the benchmark is reported descriptively.
References
[1] He, J., Treude, C., & Lo, D. (2025). LLM-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5), 1–30. https://doi.org/10.1145/3712003
[2] Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., & Schmidhuber, J. (2024). MetaGPT: Meta programming for a multi-agent collaborative framework. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=VtmBAGCN7o
[3] Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024). Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8), 1–79. https://doi.org/10.1145/3695988
[4] Huang, D., Zhang, J. M., Luck, M., Lu, Q., Qing, Y., & Cui, H. (2023). AgentCoder: Multi-agent-based code generation with iterative testing and optimisation [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2312.13010
[5] Just, R., Jalali, D., & Ernst, M. D. (2014). Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (pp. 437–440). Association for Computing Machinery. https://doi.org/10.1145/2610384.2628055
[6] Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023). CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models. In Proceedings of the 45th International Conference on Software Engineering (pp. 919–931). IEEE. https://doi.org/10.1109/ICSE48619.2023.00085
[7] Lin, D., Koppel, J., Chen, A., & Solar-Lezama, A. (2017). QuixBugs: A multi-lingual program repair benchmark set based on the Quixey Challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (pp. 55–56). Association for Computing Machinery. https://doi.org/10.1145/3135932.3135941
[8] Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2024). Large language model-based agents for software engineering: A survey. ACM Transactions on Software Engineering and Methodology. Advance online publication. https://doi.org/10.1145/3695996
[9] Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Che, X., Wang, D., & Wang, Q. (2024). Make LLM a testing expert: Bringing human-like interaction to mobile GUI testing via functionality-aware decisions. In Proceedings of the 46th International Conference on Software Engineering (pp. 1–13). IEEE. https://doi.org/10.1145/3597503.3639180
[10] Ouedraogo, W. C., Kabore, K., Tian, H., Song, Y., Koyuncu, A., Klein, J., Lo, D., & Bissyandé, T. F. (2024). LLMs and prompting for unit test generation: A large-scale evaluation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (pp. 1–12). Association for Computing Machinery. https://doi.org/10.1145/3691620.3695330
[11] Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2023). An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1), 85–105. https://doi.org/10.1109/TSE.2023.3334955
[12] Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024). Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4), 911–936. https://doi.org/10.1109/TSE.2024.3368208
[13] Xia, C. S., Paltenghi, M., Tian, J. L., Pradel, M., & Zhang, L. (2024). Fuzz4All: Universal fuzzing with large language models. In Proceedings of the 46th International Conference on Software Engineering (pp. 1547–1559). IEEE. https://doi.org/10.1145/3597503.3639110
[14] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37, 50528–50652. https://doi.org/10.48550/arXiv.2405.15793
[15] Yaraghi, A. S., Bagherzadeh, M., Kahani, N., & Briand, L. C. (2022). Scalable and accurate test case prioritization in continuous integration contexts. IEEE Transactions on Software Engineering, 49(4), 1615–1639. https://doi.org/10.1109/TSE.2022.3184842
[16] Yoon, J., Feldt, R., & Yoo, S. (2024). Intent-driven mobile GUI testing with autonomous large language model agents. In Proceedings of the 2024 IEEE Conference on Software Testing, Verification and Validation (pp. 129–139). IEEE. https://doi.org/10.1109/ICST60714.2024.00020
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Yuxuan Li
This work is licensed under a Creative Commons Attribution 4.0 International License.