Automatic Reasoning and Tool-use (ART) 🛠️
Explore the framework that automates intermediate reasoning steps as programs and seamlessly interleaves external tool calls for complex problem solving.
This content is adapted from Prompting Guide: ART. It has been curated and organized for educational purposes on this portfolio. No copyright infringement is intended.
Introduction
Combining Chain-of-Thought (CoT) prompting with external tools is a powerful strategy for tackling complex LLM tasks. However, traditional approaches often require hand-crafting task-specific demonstrations.
Automatic Reasoning and Tool-use (ART), proposed by Paranjape et al. (2023) (opens in a new tab), is a framework that uses a frozen LLM to automatically generate reasoning steps as a program, significantly reducing the manual effort involved.
How ART Works
ART operates through a systematic retrieval-and-generation process:
- Selection: Given a new task, ART selects relevant multi-step reasoning and tool-use demonstrations from a library.
- Execution: At test time, the model pauses generation whenever an external tool is called.
- Integration: The output from the tool is integrated back into the prompt before the model resumes generation.
Image Source: Paranjape et al. (2023)
Key Advantage: Extensibility
One of the standout features of ART is that it is human-extensible. If the model makes a mistake, humans can fix the reasoning steps in the library or add new tool descriptions, allowing the system to improve over time without retraining.
Performance Results
ART substantially improves over few-shot prompting and automatic CoT on various benchmarks. It even exceeds the performance of hand-crafted CoT prompts when human feedback is incorporated into the task library.
Benchmarking BigBench and MMLU
The table below shows ART's superior performance across unseen tasks in the BigBench and MMLU benchmarks:
Image Source: Paranjape et al. (2023)
From ART to Agents: ART is a significant step toward autonomous agents. By automating the interleaving of "thought" and "action," it lays the groundwork for frameworks like ReAct.
[!TIP] ART's ability to "delegate" tasks to tools (like calculators or search engines) makes it ideal for enterprise workflows. To see how these tool-use patterns are used in fully autonomous loops, check out the ReAct Framework next.