Reproducibility in High-Performance Computing and Scientific Computing

Author: Alex Wiens (Paderborn Center for Parallel Computing, Paderborn University)

Computational Science and High-Performance Computing

Let's start with a short introduction of what HPC is about. Computers are omnipresent in scientific research, however, many tasks and calculations can be done with low-performant computers. On the other side, there are calculations that are only practical, when using the most powerful computing systems.

High-scale simulations, bigdata or certain artificial intelligence applications are only realizable when using the most recent technologies and many computers in parallel. Computational sciences rely on massive computations for their research, e.g. chemists simulate molecular interactions at the atomic level before turning to real experiments. This way, advances in computational capabilities enables research that was not possible before - provided that the compute performance can be utilized efficiently. This is where High-Performance Computing (HPC) comes into the picture. HPC researches the efficient use of compute systems to achieve the highest possible performance and solve demanding scientific problems. For this, the performance engineering methodology is applied to research software. The program behavior is analyzed and the inefficient parts are identified and finally reprogrammed to make the best use of the hardware.

Additionally, very large computations are only practical, when split onto many computers and executed in parallel, for example on a high-performance cluster of computers. These compute clusters consist of many server-grade computers that communicate over low-latency high-bandwidth networks and utilize additional accelerator hardware such as GPUs. There are many layers in the hierarchy of the hardware architecture, which makes it difficult to program efficiently. The design, construction and operation of HPC clusters becomes its own field. Especially, the efficient power supply and cooling of compute clusters constitutes a challenge. HPC cluster systems are used by a wide range of users. From beginner users whose calculations outgrew their workstations up to massive big-data simulations.

Reproducibility of computations

Using computations for research comes with its own challenges for reproducibility of scientific results. Since computers are considered deterministic machines, one could think computations are easily reproducible far into the future. However, on a closer look many problems arise with the software and hardware. Some lie in the diversity and rapid development of software environments, that makes it difficult to recreate the conditions to even run the original computations. Sometimes, the used proprietary software is not available anymore, since the software vendor is long gone or the software can't be easily executed on recent computer hardware. Another problem lies in the way computers handle numbers: floating point arithmetic.

Floating point numbers are stored in a standardized format that allows for efficient calculations in hardware. However, this approach comes with limitations that change how mathematical rules apply to the calculations. The consequence are slight changes in the exact decimal value of numbers that depend on the order of computations. Therefore, the precision of a result does not only play a role in physical measurements, but also the calculations inside a computer. The result of a computation can vary with the used algorithm and the order of the input values and the operations.

Nowadays, computations are executed in parallel on multiple CPU cores. Slight differences in the manufacturing of the hardware can create a small difference in the speed of the CPU cores, which in turn leads to a different sequence of operations. Another problematic example are background processes that belong to the operating system and are executed in the background. These are necessary to operate the complex computer, but affect the timing of events during the calculations.

Bitwise reproducibility

As explained, bitwise reproducible results are very difficult to achieve. The program code has to maintain a defined order of operations independent of the conditions of program execution, e.g. independent of hardware architecture, number of CPU cores, timing issues etc. Another difficulty is the handling of random numbers. For bitwise reproducibility, the exact same random numbers have to be used, since they influence the timing and order of operations. The high parallelism of programs in HPC intensifies these problems and leads to trade-offs between reproducibility on one hand and efficiency and performance of the computation on the other hand. Additionally, if specialized hardware is used to accelerate computations, then the availability of this hardware might become a problem for later reproduction. These problems concern all computational research. Additionally, in HPC these problems take specific form.

Challenges in HPC cluster environments

On HPC clusters, the software environment opens up additional challenges. The operating system and software stack on HPC clusters is often tailored to the specific hardware for optimized execution. The software is compiled and installed by the cluster operators, therefore, the software binaries are specific to the cluster system instead of widely available precompiled versions. This makes it more difficult to document and archive the software environment for later reproduction. Another difference in computations on clusters is the way the computations are executed. Since cluster systems are shared by many users, a workload manager is used to schedule the users' computations and avoid resource conflicts. Users define the resources needed and commands to be executed as jobs and pass them to the workload manager, which decides when and on what physical machine the job is executed. Therefore, documenting the computations can be more involved, which is why there exist workflow managers that help organizing of the computations. Lastly, HPC computations are more demanding. They can take up complete compute clusters for weeks or months and process very large datasets. The large investment of energy and compute time makes it more difficult to reproduce. The data amount can be so large, that data transfer becomes problematic and it is easier to do further calculations at the facility where the data is stored, than moving the data to a different place.

Reproducibility in publications

Another kind of challenge for reproducibility lies in scientific publication culture.

Not only has the computation to be "reproducible", but the publication, including the paper, results and software, in short the research data, has to be easily accessible and usable by fellow scientists. A more structured definition can be found in the "FAIR principles", which state that research data should be: findable, accessible, interoperable and reusable. The FAIR guidelines go into more detail about how research data can be enriched with metadata and meet the FAIR principles. However, for the science culture to change, researchers also need to be incentivized to improve their publications.

For the largest of conferences in High-Performance Computing, the Super Computing (SC), the ACM (Association for Computing Machinery) introduced badges for reproducibility [0]. The artifact reproducibility is distinguished in three categories:

• repeatability
• reproducibility
• replicability

Let's take a look at the definitions and consider a computation to be an experiment with a result of certain precision and conducted by a team using an experimental setup. Then the result is repeatable if the original team can reproduce the result using the same setup. Next, the result is reproducible if a different team can reproduce the result with the original setup. Finally, the result is replicable if a different team can reproduce the result with a different setup. The results of the computation have to achieve a defined precision.

The publication artifacts are reviewed to check if they deserve one of the badges. Actually, all publications at SC are expected to include the relevant research data artifacts. This establishes necessary incentives for more reproducibility in scientific culture that will benefit all researchers in the long run. While the open availability of research artifacts is a necessary requirement, the quality and documentation of the artifacts is also important. Therefore, the necessary metadata has to be included to make the research data really useful.

Improvements in reproducibility step-by-step

At the NFDIxCS consortium we discuss and work on improvements to reproducibility in the HPC context. Since the goal of HPC is to achieve top performance, finding solutions is challenging and for many problems trade-offs have to be found. Recording the software and hardware environments during build time and when the application is executed can improve the documentation of the conditions under which HPC applications run and compute their results.

With the Research Data Management Container (RDMC) we design a time capsule for data, software and metadata that improves reusability - for all scientists, beyond high-performance computing. One idea is the utilization of the software packaging system GUIX which might also be useful in the HPC context to automate installation and documentation of software environments. The Guix package management system focuses on reproducible software distribution. In fact, the long-running subproject GUIX HPC [1] works hard on making GUIX a viable solution for HPC users. Also, containerization can make software environments more convenient and portable, but comes with additional abstractions and complexities. Improvements in documentation of HPC usage and training of existing techniques and tools are also necessary to help new HPC users.

A perfect solution might not exist, but we continue to work on improving reproducibility of science on HPC systems step-by-step.

References and further reading

[0] “Artifact Review and Badging – Current”, Association for Computing Machinery (ACM),
https://www.acm.org/publications/policies/artifact-review-and-badging-current

[1] “GuixHPC, Reproducible software deployment for high-performance computing“,
https://hpc.guix.info/

[2] “Noctua 2 Supercomputer”, Carsten Bauer et al, Journal of large-scale research facilities JLSRF, 2024,
https://doi.org/10.17815/jlsrf-8-187

[3] “Reproducibility, Replicability and Repeatability: A survey of reproducible research with a focus on high performance computing”, Benjamin Antunes, David R.C. Hill, Computer Science Review, Volume 53, 2024,
https://doi.org/10.1016/j.cosrev.2024.100655