A Guide to Making HPC Data Accessible and Reproducible: Developing Reproducible Execution Environments
High-performance computing systems run massive computational tasks, from weather simulations to genomics research, often producing and consuming vast amounts of data. These data transfers, known as input/output (I/O) operations, play a critical role in determining the efficiency of an HPC system.
A Guide to Making HPC Data Accessible and Reproducible: Developing Reproducible Execution Environments
by Reza Salkhordeh
HPC-IO traces are logs that record these I/O operations in detail. They provide a window into how applications interact with storage systems, revealing patterns, inefficiencies, or bottlenecks. For example, a poorly optimized data write operation could delay an entire simulation. By analyzing these traces, researchers can identify and address such inefficiencies, ultimately improving system performance.
The HPC-IO traces website is a platform that allows users to upload, store, and analyze these traces. It provides tools for visualization and reporting, making complex trace data accessible and actionable even for those without deep technical expertise in HPC. By enabling users to understand and optimize their systems, the platform serves as a bridge between raw I/O data and meaningful insights.
In this guide, Reza Salkhordeh, a postdoctoral researcher in the Efficient Computing and Storage Group at Johannes Gutenberg-University Mainz and member of NFDIxCS, provides a roadmap for transforming the HPC-IO traces website into a Reproducible Execution Environment (REE). An REE packages software, configurations, and data into a self-contained system that ensures portability, reproducibility, and ease of use.
Why Transform the Website into an REE?
The process of turning the HPC-IO traces website into an REE ensures that the platform is not only functional but also highly adaptable and reproducible. By packaging everything—software, configurations, and data—into a portable environment, users can deploy the system without worrying about dependencies or compatibility issues.
An REE simplifies the experience for researchers and developers by removing unnecessary components and enabling reproducibility. For example, the REE ensures that the same analysis can be replicated on any system, which is essential for scientific research and collaboration. Below is a summary, for the detailed step-by-step guide, see here[AD1]
This figure shows the different components and their dependencies on the website. This architecture includes all components and interactions as implemented on the production website.
How the Transformation Was Achieved
Step 1: Simplifying the Service
To streamline the system, unnecessary features were removed. For instance, the original backend included tools for user authentication and integration with external services like Zenodo for archiving traces. These were stripped away to reduce complexity and focus on core functionalities, such as trace analysis and visualization.
Similarly, the frontend was simplified. Pages like login and trace approval were removed, and redundant logic was eliminated. The upload form was also redesigned to make the process faster and more intuitive by removing steps like email verification.
Step 2: Packaging with Guix
Using Guix, a functional package manager, the software and its dependencies were bundled into reusable packages. Core components such as the Darshan profiling tool were packaged specifically for this project, ensuring they could be seamlessly integrated into the REE.
Step 3: Managing System Dependencies
The website relied on system-level software like databases and runtime environments. These were identified and included in the REE. For example, MongoDB, used for storing metadata, was built from source and integrated, ensuring compatibility and reproducibility across systems.
Step 4: Structuring the Environment
The REE was designed to include all software and data in a controlled directory structure. This ensures consistency and makes the environment easy to debug and extend. Data files and databases are pre-loaded where necessary, and configurations are standardized for out-of-the-box usability.
The Resulting REE
The transformation resulted in a self-contained, portable system that:
- Simplifies deployment by bundling all dependencies.
- Ensures reproducibility, enabling users to replicate analyses with identical results.
- Reduces complexity by focusing only on essential features.
By creating an REE for the HPC-IO traces website, the project has opened up new possibilities for researchers and developers, making HPC tools more accessible and widely usable. Whether you are optimizing a weather simulation or analyzing genomics data, this REE provides the tools and insights you need without the usual technical barriers.
You can find the entire sourcecode on the NFDIxCS Storage & Repositories Gitlab: https://gitlab.gwdg.de/nfdixcs/i1-storage-and-repositories/hpc-io-traces-website
If you’re interested in applying these techniques to your own projects or learning more about Guix and REEs, feel free to explore additional resources or reach out with questions via rsalkhor@uni-mainz.de
