Assuring Future Quantum Systems
Background
The field of quantum computing has seen expansive growth recently, drawing the eyes of many researchers and computing enthusiasts into the novel capabilities of this budding technology. As interest grows, we are bound to discover an increasing number of complex quantum programs, which will require rigorous testing.
Quantum programs exploit known features of quantum mechanics such as superposition – where a qubit can be in multiple states at once. These programs can be especially difficult to understand for many programmers who have not explored the field of quantum mechanics. In addition, features such as: superposition, entanglement, and the collapse of quantum state invalidate many testing methodologies from applying to quantum programs.
Consequently, either classical testing methodologies need to be modified or quantum testing methodologies need to be developed to be able to adequately handle quantum programs. The goal of our paper is to make it easier for people to do the above.
Our approach
The core of our work is centred around a set of quantum programs, available across three quantum programming languages, intended to be used as a test-bed for evaluating quantum testing methodologies. For each of these programs, we provide a set of mutant (modified) versions of the code, as well as properties that may be used for property based testing.
For our goal, we wanted the benchmark suite to be applicable to as many people invested into quantum programming and testing as possible. To achieve this, we analysed the top 100 results in github repositories (using different sorting methods) related to quantum computing and recorded what programming languages they utilised.
Quantum programs, like the ones found in our benchmarks, are quite different from classical programs. They are composed of quantum circuits and gates which are executed and measured to retrieve an output. Some of these programs need the ‘quantum output’ to be modified using a classical (normal) computer to get a useful result, referred to as classical post-processing.
For each of these benchmark programs, we provide mutant versions in qiskit to be used for mutation testing. You can use mutation testing to evaluate the efficacy of a testing methodology using a mutation score, which is gathered by looking at what percentage of modified versions of a program are correctly identified. We manually created these mutants by adding, removing, and replacing gates in random locations within the quantum circuits.
Property based testing has been used as a classical testing methodology, but it has promise within the context of quantum testing so long as expressive assertions are developed. The complexity of quantum algorithms poses a serious challenge when developing effective test cases. Property based testing can automate this complex task, only requiring the user to provide simple properties of the program, while the concrete test cases with inputs can be automatically generated. In our repository we produce a document containing properties of each program, which may be used when implementing property-based tests for the suite of benchmarks.
While working on the benchmark suite we gained insight into the programming languages used, in the last section of our paper, the findings and observations made relevant to testing are noted. Things such as the performance difference between the languages, carrying out different tasks in each language and how they affect testing quantum programs.
Looking ahead: the next steps
We plan to expand upon this work going forward in a few ways:
- Increasing the number of programs in the benchmark suite.
- Increasing the number of programming languages that the benchmarks cover.
- Develop a robust framework for benchmark program selection, test execution, mutant generation, and property-based testing.
- Port classical testing techniques, such as property-based testing, with the help of the benchmarks.
A good set of benchmarks will reduce the time it takes to develop and improve testing methodologies for quantum programs. Improving quantum software engineering practices such as testing will allow for more quantum programs to be developed, reduce the effort required to maintain quantum programs, and make it easier to teach and learn quantum programs.
To find out more about this exciting project you can read the full paper here.
By Gabriel Pontolillo and Mohammad Reza Mousavi
