This is the official repo for the paper "Meta Prompting for AI Systems" (https://arxiv.org/abs/2311.11482).
Abstract: We introduce Meta Prompting (MP), a framework that elevates the reasoning capabilities of large language models (LLMs) by focusing on the formal structure of a task rather than content-specific examples. We establish a theoretical foundation for this paradigm, formalizing MP as a functor that maps a category of tasks to a category of structured prompts, thereby guaranteeing that compositional problem-solving strategies can be systematically decomposed into modular prompt structures. We extend this concept to Recursive Meta Prompting (RMP), an automated process where an LLM can generate and refine its own prompts. We model this self-improvement loop formally as a monad, providing a principled framework for automated prompt engineering. Our claims are validated through several experiments demonstrating that a Qwen-72B base model, guided by a single, example-agnostic meta-prompt, achieves improved results on MATH, GSM8K, and Game of 24. These results are achieved with substantial token efficiency gains over traditional few-shot methods.
- What is Meta Prompting?
- Key Contributions
- A Formal Framework
- Multi-Modal Meta Prompting
- Experimental Results
- Getting Started: Try Meta Prompting
- Citation
Meta Prompting is a technique that provides a high-level, structural template for how to think rather than specific examples of what to think. Unlike few-shot prompting, which relies on content-rich examples, a meta-prompt is a single, example-agnostic scaffold that guides the LLM's reasoning process. It focuses on the formal procedure, syntax, and compositionality of problem-solving.
This approach teaches the model a reusable, structured method for tackling an entire category of tasks, leading to greater efficiency and robustness.
Here’s the difference in action. A Meta Prompt provides a general structure for solving any problem in a category, while a Few-Shot Prompt provides specific, solved examples.
This prompt gives the LLM a reusable blueprint for solving math problems. It's completely agnostic to the actual numbers or problem details.
Integrate step-by-step reasoning to solve mathematical problems under the following structure:
{
"Problem": "[question to be answered]",
"Solution": {
"Step 1": "Begin the response with 'Let's think step by step.'",
"Step 2": "Follow with the reasoning steps, ensuring the solution process is broken down clearly and logically.",
"Step 3": "End the solution with the final answer encapsulated in a LaTeX-formatted box, $\\boxed{...}$"
},
"Final Answer": "[final answer]"
}
For comparison, a traditional few-shot prompt provides several concrete (problem, solution) pairs. The model is expected to learn the pattern from these specific examples.
Problem: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.
Solution: The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5-x>0$, so $x<5$. Therefore, the domain is $\boxed{[2,5)}$.
Final Answer: The final answer is $[2,5)$.
---------
Problem: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
Solution: We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = \boxed{24}.$
Final Answer: The final answer is $24$.
---------
...
- Introduces Meta Prompting (MP): A new, example-agnostic prompting paradigm that focuses on the formal structure of reasoning.
- Establishes a Theoretical Foundation: We formalize MP using category theory, modeling it as a functor that guarantees compositional problem-solving.
- Defines Recursive Meta Prompting (RMP): We introduce a process for automated prompt engineering and self-improvement, which we formalize as a monad.
- Achieves Strong Performance: Using a single, zero-shot meta-prompt, our method sets new results on benchmarks like MATH (46.3%) and GSM8K (83.5%) with a base Qwen-72B model, outperforming fine-tuned models and early GPT-4 versions in zero-shot settings.
A core novelty of our work is providing a rigorous mathematical foundation for prompt engineering.
We model Meta Prompting as a functor that maps a category of tasks () to a category of prompts (). This ensures that if a complex task can be broken down into simpler sub-tasks, the corresponding prompt can also be constructed by composing simpler prompts. This provides a theoretical guarantee of modularity and compositionality.
We model the process of an LLM refining its own prompts (Recursive Meta Prompting) as a monad over the category of prompts . Specifically, we employ a Writer Monad structure to model the accumulation of refinement strategies (edit scripts). Let be the set of valid edit scripts forming a monoid .
- **Endofunctor **: For any prompt object , . This preserves the edit history while the prompt structure transforms.
- Unit (): The natural transformation lifts a prompt into the refinement context with an empty history.
- Multiplication (): The natural transformation collapses nested levels of meta-reasoning. It ensures that recursive optimization steps are linearly composed into a single, coherent edit trace:
This framework guarantees that the refinement process is algebraically consistent and stable.
Grounded in Type Theory, our framework extends naturally to multi-modal settings. Just as type systems in programming ensure that different data types are handled correctly, a meta-prompt can enforce a clear, syntactic structure on inputs and outputs across modalities (text, images, audio).
We define schemas that create explicit, typed "slots" for different data streams, enabling the model to synthesize information to generate a cohesive output.
<task_schema>
<input_slots>
<data type="image/png" name="problem_diagram">
</data>
<data type="audio/mp3" name="verbal_instructions">
</data>
<data type="model/obj" name="object_model">
</data>
</input_slots>
<output_schema>
<synthesis type="text/markdown" name="analysis_summary">
</synthesis>
<result type="video/mp4" name="solution_animation">
</result>
</output_schema>
</task_schema>
Our experiments show that Meta Prompting unlocks powerful reasoning in base language models without any fine-tuning, using only a single, example-agnostic prompt.
With a single meta-prompt, Qwen-72B (base) surpasses all other open-source models and even the initial release of GPT-4.
| Model | FT-Dataset | Tool Usage | Eval Method | MATH (%) |
|---|---|---|---|---|
| Proprietary Models | ||||
| Claude-2 | - | No | CoT | 32.5 |
| Minerva-540B | Arxiv+Web | No | CoT | 33.6 |
| PaLM-2 | - | No | CoT | 34.3 |
| GPT-4 (2023-0314) | - | No | CoT | 42.5 |
| Open-source Models | ||||
| Llama-2-70B (base) | - | No | CoT | 13.5 |
| Qwen-14B (base) | - | No | MP | 28.9 |
| Qwen-72B-MetaMathQA | MetaMathQA | No | CoT | 41.7 |
| Qwen-72B (base) | - | No | MP | 46.3 |
Our zero-shot meta-prompting approach again shows substantial improvements over both base models with CoT and powerful fine-tuned models.
| Model | FT-Dataset | Tool Usage | Eval Method | GSM8K (%) |
|---|---|---|---|---|
| Llama-2-70B (base) | - | No | CoT | 56.8 |
| Qwen-14B (base) | - | No | MP | 64.8 |
| WizardMath-70B | WizardMath | No | CoT | 81.6 |
| MetaMath-70B | MetaMathQA | No | CoT | 82.3 |
| Qwen-72B (base) | - | No | MP | 83.5 |
Meta Prompting achieves a 100% success rate by generating a single Python program to solve all puzzles, demonstrating extreme efficiency compared to iterative methods like Tree-of-Thought (ToT).
| Method | LLM Sessions (per sample) | Generate/Prompt tokens | Cost (per sample) | Success Rate |
|---|---|---|---|---|
| IO (best of 100) | 100 | 1.8k / 1.0k | $0.13 | 33% |
| CoT (best of 100) | 100 | 6.7k / 2.2k | $0.47 | 49% |
| ToT (breadth=5) | 61.72 | 5.5k / 1.4k | $0.74 | 74% |
| MP | 1/N | ≈ 1/N (8k / 1k) | ≈ $0.0003 | 100% |
The prompts used in our paper are available in the /prompts directory. You can use them as system prompts for LLMs to replicate our results or adapt them for your own tasks.
Here is the meta-prompt used for the MATH benchmark, which you can use directly.
**Problem Statement**:
- **Problem**: [question to be answered]
**Solution Structure**:
1. Begin the response with "Let's think step by step."
2. Follow with the reasoning steps, ensuring the solution process is broken down clearly and logically.
3. End the solution with the final answer encapsulated in a LaTeX-formatted box, $\boxed{...}$, for clarity and emphasis.
4. Finally, state "The answer is [final answer to the problem].", with the final answer presented in LaTeX notation.
---------
You can try out various versions of Meta Prompting agents via these custom GPTs:
- 🤖 CR Agent (Complex Reasoning): v0.1 | v0.2 (XML)
- ✍️ MP-PT (Prompt Revision): Online Demo
- 🔄 MP-ICPD (Recursive In-Context Prompt Design): Online Demo
If you find our work useful for your research, please consider citing our paper and starring this repository. Thank you!
@article{zhang2023meta,
title={Meta Prompting for AI Systems},
author={Zhang, Yifan and Yao, Andrew Chi-Chih},
journal={arXiv preprint arXiv:2311.11482},
year={2023},
note={Published in ICLR 2024 BGPT Workshop}
}