Usage of Large Language Model for Code Generation Tasks: A Review

This paper presents a PRISMA-based systematic literature review of 66 studies (2021-2023) mapping how Large Language Models are used for code generation, synthesizing findings on preferred languages, model performance, and unresolved research gaps.

Problem Statement

Despite the rapid adoption of LLMs for coding assistance, no exhaustive synthesis existed comparing their benefits, drawbacks, and evaluation practices across the fragmented literature. This makes it hard for researchers and practitioners to know which models and languages are best suited to which tasks, and which critical issues (e.g., ethics) remain unaddressed. The review fills this gap by systematically aggregating and answering four targeted research questions from peer-reviewed work.

Key Novelty

One of the first PRISMA-compliant systematic reviews focused specifically on LLM-driven code generation across 2021-2023
Comparative cross-study synthesis of model performance (e.g., GPT-4 vs. CodeLlama) spanning tasks like debugging and multi-turn program synthesis
Explicit identification of an underexplored gap around ethical constraints in LLM-based code generation research

Evaluation Highlights

Systematic PRISMA screening process narrowed the literature to 66 relevant, peer-reviewed papers published between 2021 and 2023
Four structured research questions used to qualitatively synthesize findings on languages, models, tasks, and ethics across the corpus

Signal Assessment

2/10 As a literature review, the paper organizes and synthesizes existing knowledge rather than proposing new models, methods, or empirical benchmarks, making it useful for orientation but incremental in scientific novelty.

Methodology

Defined four Research Questions to scope the review of LLMs for code generation
Conducted systematic literature search and screening following PRISMA guidelines
Applied inclusion/exclusion criteria to filter candidate studies down to 66 papers (2021-2023)
Extracted and thematically synthesized data on languages, models, task types, and ethical considerations

System Components

PRISMA Protocol

Standardized framework for systematic identification, screening, and selection of relevant literature to ensure reproducibility

Research Questions (RQ1-RQ4)

Four guiding questions structuring the analysis of LLM benefits, drawbacks, languages, and model comparisons

Cross-Model Comparative Analysis

Synthesis of reported performance for models such as GPT-4 and CodeLlama across tasks including debugging and multi-turn program synthesis

Gap Analysis

Identification of underexplored research areas, notably ethical constraints in LLM-based code generation

Results

Review Aspect	Observation Across Corpus	Key Insight/Implication
Programming Language Usage	Python is the most frequently used language across the 66 reviewed papers	Python dominance suggests a need for broader multi-language benchmarking
Model Performance	GPT-4 and CodeLlama are the most discussed and compared models across tasks	No single model excels universally; suitability varies by task (e.g., debugging vs. synthesis)
Ethical Constraints	Only a small subset of papers address ethics in LLM code generation	Represents a significant, largely unaddressed research gap

Key Takeaways

Python remains the de facto standard language in LLM-for-code research, so tooling, benchmarks, and evaluations should prioritize strong Python support
Model choice should be task-dependent: GPT-4 and CodeLlama show differing strengths across debugging, code completion, and multi-turn program synthesis
Ethical considerations (bias, security, licensing, data provenance) are largely absent from current literature and represent a high-value area for future research
The curated set of 66 PRISMA-reviewed papers (2021-2023) offers a practical starting point for designing new LLM-for-code studies or selecting models for production use

Abstract

Large Language Models have received a lot of attention in recent years due to their outstanding performance on various Natural Language Processing tasks. They can be used for lots of applications, including assistance in code generation tasks. Actual literature lacks an exhaustive analysis of the benefits and drawbacks of using a Large Language Model for the generation of simple and complex code. This paper aims to overcome the issue: we perform a Literature Review to explore the state-of-the-art of the proposed topic, answering 4 Research Questions. Using the PRISMA methodology, we reviewed 66 papers published between 2021 and 2023. Our analysis reveals Python’s dominance as the preferred language and identifies a significant research gap in addressing ethical constraints. Additionally, we provide insights into the performance of models such as GPT-4 and CodeLlama, and their comparative utility in tasks ranging from debugging to multi-turn program synthesis. The findings offer a foundation for future research aimed at optimizing LLMs for code generation.