LLM-Based Information Extraction to Support Scientific Literature Research and Publication Workflows

Authors: Samy Ateia, Udo Kruschwitz, Melanie Scholz, Agnes Koschmider, and Moayad Almohaishi

LLM-Based Information Extraction to Support Scientific Literature Research and Publication Workflows

The increasing volume of scholarly publications requires advanced tools for efficient knowledge discovery and management. Our ongoing work introduces a system using Large Language Models (LLMs) for the semantic extraction of key concepts from scientific documents. Conducted within the National Research Data Infrastructure for and with Computer Science (NFDIxCS) project, this research seeks to support FAIR (Findable, Accessible, Interoperable, and Reusable) principles in scientific publishing.

Motivation

The scientific publication landscape is booming, with global annual output growing by 59% and more than one million articles published each year in biomedicine and life sciences alone. This makes it increasingly difficult for scientists to stay on top of their fields and for publications to remain visible.
In computer science, papers are often accompanied by software and datasets for reproducibility, but their management frequently lacks standardization and fails to meet FAIR principles. NFDIxCS addresses this by creating infrastructure to implement FAIR principles for computer science research outputs in Germany. To make these artifacts findable, it is necessary to link them to relevant semantic information from the publication text itself—such as research questions or methods—so that related work can be more easily discovered.

Our Goal

Our project aims to develop tools that use LLMs for semantic analysis of scientific text to enhance FAIR publishing. Specifically, we aim to:

• Develop robust methods for automatically extracting key semantic concepts (e.g., research questions, methodologies, findings) from scientific papers.
• Structure these extracted concepts to improve the organization and distribution of digital content, potentially linking them to knowledge graphs.
• Design and prototype user-driven services that use this structured information to support researchers in their workflows.

Related Work

Several platforms use natural language processing to assist in navigating large publication volumes. Semantic Scholar provides AI-based summaries and citation insights, Elicit synthesizes findings from multiple papers, and Scite.aiclassifies citations as supportive or disputing. While useful, these platforms rarely offer predefined domain-specific questions and often require paid access.
In contrast, our approach aims for higher accuracy and user guidance through curated extraction targets and examples for specialized communities.

Structuring scientific knowledge in machine-readable formats has long been a goal of the research community. Initiatives such as the Open Research Knowledge Graph (ORKG) represent papers through contributions, methods, and findings, but curating such graphs is often labor-intensive. We aim to complement these efforts by automatically pre-filling templates for knowledge graphs through domain-specific extraction questions, lowering the barrier for large-scale data curation.

LLM-Based Concept Extraction

Our system uses an LLM to extract semantic information from scientific documents. A demo interface allows users to upload a paper and pose predefined or custom questions. The LLM processes the document and identifies relevant information or synthesized answers (Figure 1).

Fig.1. LLM-based demo extraction pipeline.

We leverage in-context learning for rapid domain adaptation. By providing a few examples and instructions, our method avoids the need for extensive domain-specific datasets. We tested two modes:

• Few-shot learning, using a small number of annotated examples.
• Zero-shot learning, relying only on instructions and text input.

The few-shot mode helps align the model with the style of manual annotators, while zero-shot allows flexible querying. Both can support different research workflows, such as pre-filling forms for later mapping to knowledge graphs.

Our initial evaluation used 122 papers from the Business Process Management (BPM) domain, chosen for its active community in knowledge graph construction. We manually annotated key concepts to create a gold standard for performance evaluation.

Technical Evaluation

We tested several large models: Qwen 2.5 72B Instruct, Llama 3.3 70B Instruct, and Gemini 1.5 Flash variants. Results showed that open-weight models (Qwen and Llama) generally outperformed the commercial models in our tasks.
Performance was strongest for semantic free-text extraction (BERT_F1 ≈ 0.90), moderate for binary indicators, and limited for exact categorical extraction. Few-shot examples improved results for categorical and free-text targets but slightly reduced binary classification accuracy.

User Studies and Feedback

To collect user feedback, we developed a demo using the Gradio framework. In a pilot user study, 88% of participants expressed satisfaction with the extracted concepts, noting that traceability—linking answers back to their source text—was a key desired improvement.
During a separate workshop with 30 computer science researchers, we gathered 56 user stories: most focused on literature research and comparison, others on writing support, reviewing, and software development. Users expressed a strong interest in comparing extracted information across multiple papers, not just single documents.

Discussion and Outlook

Our preliminary results demonstrate the potential of current LLMs to extract domain-specific information from scientific text. Through in-context learning, models can be quickly adapted to new fields and support the transfer of expert knowledge.
User feedback emphasized the importance of transparency—future versions of the system will highlight text passages that support extracted information.

We are exploring embedding-based retrieval over extracted information to improve semantic search, and integration with knowledge graph systems such as ORKG to assist authors in making their work more discoverable.
While similar commercial services exist, our open-source, user-centered approach provides a transparent foundation for customizable, domain-specific solutions.

Conclusion

Our work highlights the potential of LLMs to improve the publishing process and discoverability of scientific information in digital libraries and beyond. By combining technical evaluation, user-driven design, and open-source development, we demonstrate how AI can support FAIR scientific publishing and assist researchers in managing an ever-growing body of knowledge.

Find the full-length version of this paper here: https://arxiv.org/abs/2510.04749