Artifact Evaluation in the Context of NFDIxCS
Software has become an important part of research. In this regard, research software of good quality contributes towards the FAIR principles [2]. To ensure certain levels of quality, artifact evaluations (AE) have been established in conferences as a means of quality assurance
a text by Martin Armbruster
1. Introduction
Authors of accepted papers submit related artifacts, which are peer-reviewed. Based on the reviews, the artifacts are awarded badges to express met criteria. In this blog post, we describe an AE process as supported by the NFDIxCS AE Platform in more detail in section 2, by utilizing the example of a software engineering research conference. Then, we give a short outlook to our plans in section 3 and conclude the blog post in section 4.
2. The ICSA 2024 Artifacts Evaluation Track
In the following, we describe an AE process, exemplified by the corresponding track of the 21st IEEE International Conference on Software Architecture 2024 (ICSA 2024) [1].[1] The process was supported by the NFDIxCS AE Platform.[2] The AE platform is a self-hosted instance of HotCRP,[3] an open-source conference review management software.
The process consists of these steps: Setup - Submission - Bidding - Review & Clarification Period - Decision & Notification, as displayed in Figure 1. The first step starts with the PC chairs setting up the AE platform for submissions. This involves general settings (e.g., deadlines), the creation of the submission form, and the creation of the review form. The AE platform allows customizing both forms to the needs of a conference. Therefore, it provides various types for the form fields. A small excerpt from the type selection is shown in Figure 2.
Figure 1: The complete ICSA 2024 AE Track process.
Figure 2: Excerpt from the form field type selection.
In the context of the ICSA 2024 AE Track, an artifact relates to a paper and is evaluated regarding its quality and relation to the paper. As a result of a successful evaluation, an artifact can be awarded one or more of the following badges: Research Object Reviewed (ORO) Functional, ROR Reusable, Open Research Object (ORO), Results Reproduced (ROR-R), or Results Replicated (RER). The AE also includes executing the artifact. Therefore, the submission form for the ICSA 2024 AE Track requests general information, such as title, abstract, and the authors, a copy of the related paper, a link to the artifact, the badges for which the authors apply, information about system requirements for execution, and information for executing the artifact. Figure 3 and Figure 4 show excerpts for the exemplary setup of the submission form.
Figure 3: Excerpt from creating a submission form.
Figure 4: Excerpt from creating two submission form fields: one for selecting the badges for which the authors apply and one for hardware requirements.
Similarly, the review form, an excerpt of the setup is depicted in Figure 5, asks the reviewers for their assessment which badges should be awarded.
Figure 5: Excerpt from the creation of the review form. The declared field allows reviewers to recommend awarding the ROR badges.
After the AE platform is set up, authors can submit their artifacts. In case of the ICSA 2024 AE Track, all authors of accepted papers were invited. The artifacts need to be uploaded on separate file sharing or archival platforms (e.g., Zenodo[4] for long-term archival). Then, the authors can create a submission within the AE platform, as shown in Figure 6, Figure 7, and Figure 8. At the same time, the PC chairs contact potential reviewers and add them in the AE platform.
Figure 6: Excerpt from the actual submission form.
Figure 7: Excerpt from the actual submission form: The authors need to provide a link to the artifact and to select the badges for which they apply.
Figure 8: A submission draft.
After the submission deadline, there is the bidding phase. During the bidding, reviewers state a score for each artifact, which expresses how much they would like to review an artifact. Based on the bids, the PC chairs assign the artifacts to the reviewers (cf., Figure 9) so that they can start with their review. If a reviewer encounters issues or questions, they can use the clarification period to anonymously exchange comments with the authors to clarify the issues or questions. The authors, in turn, can update their artifact, if necessary, for instance, to fix a bug. The PC chairs oversee the review process. Finally, the reviewers enter their review, displayed in Figure 10. Based on the reviews, the PC chairs decide on the badges and notify the authors about the decision.
Figure 9: Example for assigning a review.
Figure 10: Excerpt from the review form.
We also compared the ICSA 2024 AE Track process to AE processes of recent software engineering conferences by an informal analysis (cf., Appendix A for a list of all analyzed conferences). In summary, we discovered only slight differences. They concern the name of phases (e.g., Kicking-the-tires instead of clarification period), the awarded badges, or the requested information. AE tracks are not limited to software engineering conferences. They are also part of conferences in other research communities, such as HPC [3] or distributed systems research [4].
3. Outlook
In this section, we shortly highlight our plans for the AE and AE platform.
- The AE platform constitutes an initial starting point to support AE tracks. Besides, we explore approaches to improve the process and support authors and reviewers in the AE. For example, we investigate how Cloud Development Environments can be leveraged for reviews. This includes executing and viewing an artifact. Moreover, we want to investigate tools for automatic quality checks.
- Independent of the additional features, we want to support further conferences in their AE tracks by providing the AE platform. At the time of writing this blog post, we already support the ICSA 2025 AE Track.
- To improve and evaluate the AE process and platform, we seek to gather feedback from authors and reviewers by conducting surveys.
- At last, the AE platform will be integrated with the Research Data Management Containers, the central artifact of NFDIxCS, to allow their evaluation.
4. Conclusion
In this blog post, we presented the process of the ICSA 2024 AE Track, which was supported by the NFDIxCS AE Platform. Based on a customized form, authors submit their artifacts for evaluation. Reviewers examine the artifacts and try to execute them. With their reviews, the PC chairs decide about badges awarded for the artifacts and expressing the met criteria.
For the future, we explore additional features to support authors and reviewers and gather feedback to improve the process. Additionally, we will integrate the AE platform with the Research Data Management Containers.
If you are interested in the AE platform or want to give feedback, you can reach out to us via mail: nfdixcs-ae@lists.kit.edu.
References
[1] Artifacts Evaluation Track. Accessed: 12.12.2024. URL: https://conf.researchr.org/track/icsa-2024/artifact-evaluation-track.
[2] Wilhelm Hasselbring et al. “From FAIR research data toward FAIR and open research software”. In: it - Information Technology 62.1 (2020), pp. 39–47. DOI: doi:10.1515/itit-2019-0040. URL: https://doi.org/10.1515/itit-2019-0040.
[3] Reproducibility Initiative. Accessed: 21.01.2025. URL: https://sc25.supercomputing.org/program/papers/reproducibility-initiative/.
[4] Mohammad Reza Saleh Sedghpour et al. Artifact Evaluation for Distributed Systems: Current Practices and Beyond. 2024. arXiv: 2406.13045 [cs.DC]. URL: https://arxiv.org/abs/2406.13045.
A. List of Analyzed Conferences
To compare the ICSA 2024 AE Track process with other AE processes of recent software engineering conferences, we considered the following conferences:
- ICSA 2021 (https://icsa-conferences.org/2021/call-for-papers/artifact-evaluation-track/)
- ICSA 2022 (https://icsa-conferences.org/2022/conference-tracks/artifact-evaluation-track/)
- ICSA 2023 (https://icsa-conferences.org/2023/call-for-papers/artifact-evaluation-track/)
- International Conference on Software Engineering (ICSE) 2019 (https://2019.icse-conferences.org/track/icse-2019-Artifact-Evaluation#Call-For-Artifact-Submissions)
- ICSE 2020 (https://2020.icse-conferences.org/track/icse-2020-Artifact-Evaluation#Call-for-Submissions)
- ICSE 2021 (https://2021.icse-conferences.org/track/icse-2021-Artifact-Evaluation
- ICSE 2022 (https://conf.researchr.org/track/icse-2022/icse-2022-artifact-evaluation
- ICSE 2023 (https://conf.researchr.org/track/icse-2023/icse-2023-artifact-evaluation
- ICSE 2024 (https://conf.researchr.org/track/icse-2024/icse-2024-artifact-evaluation#About)
- International Conference on Model-Driven Engineering Languages and Systems (MODELS) 2020 (https://conf.researchr.org/track/models-2020/models-2020-artifact-evaluation)
- MODELS 2021 (https://conf.researchr.org/track/models-2021/models-2021-artifact-evaluation)
- MODELS 2022 (https://conf.researchr.org/track/models-2022/models-2022-artifact-evaluation#About)
- MODELS 2023 (https://conf.researchr.org/track/models-2023/models-2023-artifact-evaluation#Submission-Guidelines-)
- The European Conference on Software Architecture (ECSA) hosted no AE track between 2020 and 2024 (cf., https://ecsa2020.disim.univaq.it/, https://conf.researchr.org/home/ecsa-2021,https://conf.researchr.org/home/ecsa-2022, https://conf.researchr.org/home/ecsa-2023, and https://conf.researchr.org/home/ecsa-2024).
[1] Transparency Note: For the ICSA 2024 AE Track, Anne Koziolek, co-applicant of NFDIxCS, was PC co-chair, and Martin Armbruster was member of the PC.
