Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation

This paper presents an empirical interview study with 13 industry practitioners that maps the real-world state of Retrieval-Augmented Generation (RAG) adoption, showing that industrial RAG deployments are still mostly prototype-stage, domain-specific QA systems shaped more by data protection and quality concerns than by cutting-edge technical innovation.

Problem Statement

RAG research has advanced rapidly on the academic and algorithmic side, but there is little empirical evidence on how organizations actually build, configure, and evaluate RAG systems in practice. This gap risks misaligning research priorities with real industry needs around data security, preprocessing complexity, and feasible evaluation, and leaves practitioners without a consolidated view of common requirements and pitfalls.

Key Novelty

One of the first systematic empirical studies of RAG adoption grounded in semi-structured interviews with 13 practitioners across different companies and domains
A consolidated, practice-derived requirements catalog showing data protection, security, and quality dominate over ethics, bias, and scalability concerns
A taxonomy of real-world RAG use cases revealing current deployments are concentrated in domain-specific QA and largely at prototype maturity
An empirical account of current RAG evaluation practices, highlighting heavy reliance on manual/human evaluation over automated metrics

Evaluation Highlights

Qualitative/thematic analysis of 13 semi-structured interviews with industry practitioners with hands-on RAG experience
Findings triangulated into four consolidated deliverables: use-case overview, requirements list, challenges/lessons learned, and evaluation-method analysis

Signal Assessment

3/10 The paper is a well-executed but methodologically standard qualitative field study (n=13) rather than a technical or algorithmic innovation, making it a useful but incremental contribution that maps industry practice rather than advancing RAG methods themselves.

Methodology

Designed a semi-structured interview protocol covering RAG use cases, system requirements, challenges, and evaluation practices
Recruited and interviewed 13 industry practitioners with practical RAG experience across varied organizations
Transcribed and qualitatively coded interview responses to identify recurring themes and patterns
Synthesized coded findings into four consolidated outputs: use-case taxonomy, requirements catalog, challenges/lessons-learned list, and evaluation-practice overview

System Components

Use Case Overview

Categorizes how companies currently apply RAG, finding concentration in domain-specific QA applications still largely in prototype stages

Requirements Catalog

Consolidates system requirements reported by practitioners, prioritizing data protection, security, and output quality over ethics, bias, and scalability

Challenges & Lessons Learned

Documents practical obstacles in building RAG systems, identifying data preprocessing as a persistent key challenge

Evaluation Practices Analysis

Examines how industry teams assess RAG system quality, finding predominant reliance on human evaluation rather than automated methods

Results

Dimension	Academic/Research Emphasis	Industry Practice (This Study)	Implication
System maturity	Novel architectures and benchmark performance	Mostly prototype-stage deployments	Research-to-production gap remains wide
Use case scope	Broad range of applications explored	Concentrated in domain-specific QA	Real-world adoption narrower than research scope
Top requirements	Accuracy and benchmark metrics emphasized	Data protection, security, and quality prioritized	Enterprise priorities differ from academic focus
Underexplored requirements	Growing research interest in ethics/fairness	Ethics, bias, and scalability receive little attention	Potential blind spots in current deployments
Evaluation approach	Automated metrics (e.g., RAGAS-style scoring)	Predominantly human/manual evaluation	Automated evaluation not yet trusted or widely adopted

Key Takeaways

When building enterprise RAG systems, prioritize data protection, security, and output quality controls, since these are industry's top current concerns, above ethics/bias mitigation or scalability engineering
Budget significant effort for data preprocessing (cleaning, chunking, structuring source documents), as this remains the most persistent practical bottleneck reported by practitioners
Do not treat automated RAG evaluation metrics as production-ready substitutes for human judgment; plan for human-in-the-loop evaluation as the current de facto industry standard
Expect most existing RAG deployments to be domain-specific QA prototypes rather than fully scaled production systems, and calibrate roadmap expectations accordingly

Abstract

Retrieval-Augmented Generation (RAG) is a well-established and rapidly evolving field within AI that enhances the outputs of large language models by integrating relevant information retrieved from external knowledge sources. While industry adoption of RAG is now beginning, there is a significant lack of research on its practical application in industrial contexts. To address this gap, we conducted a semistructured interview study with 13 industry practitioners to explore the current state of RAG adoption in real-world settings. Our study investigates how companies apply RAG in practice, providing (1) an overview of industry use cases, (2) a consolidated list of system requirements, (3) key challenges and lessons learned from practical experiences, and (4) an analysis of current industry evaluation methods. Our main findings show that current RAG applications are mostly limited to domain-specific QA tasks, with systems still in prototype stages; industry requirements focus primarily on data protection, security, and quality, while issues such as ethics, bias, and scalability receive less attention; data preprocessing remains a key challenge, and system evaluation is predominantly conducted by humans rather than automated methods.