Strategy-Induct: Task-Level Strategy Induction for Instruction Generation
Problem Statement
Task-level prompts strongly influence LLM performance, and prior instruction-induction methods automate prompt design but depend on input-output (labeled) pairs, which are frequently expensive, impractical, or impossible to obtain for many real-world tasks. This dependency limits where automatic instruction induction can be applied. Strategy-Induct addresses this by enabling instruction induction from questions alone, broadening applicability to label-scarce or label-free scenarios.
Key Novelty
- Introduces a label-free instruction induction paradigm that operates purely on example questions rather than input-output pairs
- Proposes a two-stage pipeline that first generates explicit per-question reasoning strategies, then aggregates (strategy, question) pairs into a unified task instruction
- Investigates cross-model synergy by combining standard LLMs and Large Reasoning Models (LRMs) across the instruction-generation and inference stages
Evaluation Highlights
- Outperforms state-of-the-art instruction induction baselines under question-only (no labeled answers) settings across multiple tasks
- Shows consistent gains across different model scales, indicating robustness of the strategy-based induction approach
- Reveals that pairing LLMs and LRMs differently for instruction generation versus inference can yield further performance improvements
Signal Assessment
Methodology
- Collect a small set of unlabeled example questions representative of the target task
- Prompt the LLM to generate an explicit reasoning strategy for each individual question, producing (strategy, question) pairs
- Aggregate these pairs and prompt the model to induce a generalized task-level instruction that captures the underlying reasoning approach
- Apply the induced instruction to guide the model's reasoning and answer generation on new, unseen questions from the same task
- Experiment with different combinations of LLMs and Large Reasoning Models at the instruction-generation and inference stages
System Components
Prompts the model to produce an explicit, question-specific reasoning strategy for each example question, without needing a ground-truth answer
Synthesizes the collected (strategy, question) pairs into a single, generalized task-level instruction meant to guide reasoning on unseen examples
Applies the induced task instruction at inference time, optionally using a different model (LLM or LRM) than the one used for induction
Results
| Setting/Metric | Baseline (SOTA instruction induction) | Strategy-Induct | Delta |
|---|---|---|---|
| Question-only task accuracy (multiple tasks) | Lower accuracy without labeled examples | Higher accuracy using strategy-derived instructions | Outperforms baseline |
| Cross-model-scale generalization | Baseline performance varies with model size | Consistent improvement across model scales | Improved robustness |
| Joint LLM + LRM pipeline | Single model type for both stages | Combining LLM/LRM roles across generation and inference | Further performance gains |
Key Takeaways
- Labeled input-output pairs are not strictly necessary for effective instruction induction; extracting explicit reasoning strategies from questions alone can substitute as a useful training signal
- Decomposing instruction induction into per-example strategy generation followed by aggregation can improve generalization compared to directly inducing instructions from raw examples
- Practitioners should consider assigning different model types (standard LLM vs. Large Reasoning Model) to instruction generation versus inference stages, as this asymmetry can boost downstream performance
- This approach is especially valuable for new or niche tasks where collecting labeled answers is costly, slow, or infeasible
Abstract
Designing effective task-level prompts is crucial for improving the performance of Large Language Models (LLMs). While prior work on instruction induction demonstrates that LLMs can infer better instructions with limited examples, existing approaches often rely on input-output pairs, where obtaining labeled answers can be difficult or costly. To address this limitation, we propose Strategy-Induct, a framework that derives task-level instructions solely from a small set of example questions without requiring labeled answers. Our approach first prompts the model to generate explicit reasoning strategies for each question, forming (strategy, question) pairs. These pairs are then used to induce a task instruction that guides reasoning. Experiments across multiple tasks and model scales demonstrate that Strategy-Induct outperforms state-of-the-art methods in question-only settings. Furthermore, we observe that jointly utilizing LLMs and Large Reasoning Models across task instruction generation and inference may lead to further performance improvements.