Hidden flaws in AI training data threaten the future of software automation

A new study from Nanjing University reveals that many AI models used for software development may be built on shaky ground—low-quality training data. Published on 15 September 2026, in Frontiers of Computer Science, this systematic literature review is the first to focus on data preparation and quality issues in code-centric generative software engineering tasks (GSETs), such as code summarization, generation, repair, and translation.

The research team analyzed 70 studies and found widespread flaws in the datasets powering these models. Common issues include noisy or outdated code, lack of diversity, imbalanced samples, and inconsistent annotations. These problems significantly impact the accuracy, generalization, and reliability of AI tools increasingly used by developers.

“Without high-quality data, even the most advanced models can’t perform well,” said Yang Feng, corresponding author of the study. “Poor data leads to poor predictions—and that’s a serious risk when AI is writing or fixing code.”

Most datasets, the study found, are collected automatically from platforms like GitHub or StackOverflow, often without proper cleaning or validation. Manual annotation is rare but more accurate. Many benchmarks are outdated, and some may have leaked into the training sets of large language models, inflating evaluation results. Meanwhile, programming languages like Ruby and JavaScript are underrepresented, limiting model versatility.

In response to these challenges, the study highlights several promising strategies that have been proposed in the literature, including automated data cleaning tools, data augmentation methods (e.g., back-translation and code mutation), multilingual training techniques, and improved dataset design standards. However, the authors note that these solutions are sporadic and rarely applied consistently across tasks. 

To guide future research, the paper offers concrete recommendations for building reliable, high-quality datasets. The authors argue that solving dataset quality issues is not just a technical concern but a necessary step toward building AI systems that developers can trust. Their findings call for a shift from model-centric to data-centric development in AI for software engineering.