Exploring the use of LLMs for Code Generation in Test-Only Development

Project description

Test-Driven Development (TDD) is an approach to building reliable and easy-to-change software, in which tests and code are written together, in lock-step, following a few key rules:

1. Changes to production code can only be made in response to a failing unit test (or a compelling refactoring opportunity).
2. Only the simplest changes to production code needed to make the failing tests pass are allowed.
3. Refactoring should be used when tests are passing to improve code quality and expand the scope of the code unit under development.

Following these rules leads to the well known Test-Code-Refactor cycle of TDD, giving a disciplined approach to software development that can also be highly productive, thanks to its ability to avoid the need for lengthy rework due to errors, once the complexity of the code being written grows beyond what can easily be checked and managed correctly by a single developer. This cycle is also sometimes referred to as the Red-Green-Green cycle, based on the results that should be seen after each step when the test suite for the code unit under construction is run.

Test-Only Development (TOD) is a variant on TDD that was proposed at the University of Manchester. In TOD, the developer focuses on writing the unit tests needed, and the second step is handled automatically by the TOD driver. That is, once the developer has written a new unit test that fails, the TOD driver takes over and generates the code needed for the second step (the "Code" step) that makes the failing unit test just written pass). The aim is to provide a technique that allows the developer to focus on capturing the requirements for the unit under construction and the TOD driver takes care of the rest.

In our original work, the code generated by the TOD driver was created using a rule-driven approach that could only handle certain types of test failure case. In this PhD project, we will explore whether LLMs trained for code generation can increase the power of TOD and cover a richer range of cases. We can also explore how well an LLM-based solution can tackle the Refactoring step, leading to a full TOD driver solution.