Engineering

Program-Aided Language Models (PAL) 🐍

Learn how to offload complex logic and mathematical reasoning from the LLM to a programmatic runtime like Python, ensuring higher reliability and precision.

Driptanil DattaSoftware Developer

Mar 20268 min read

Introduction

While Chain-of-Thought (CoT) prompting allows LLMs to perform reasoning, they often struggle with precise mathematical calculations or complex logic. Program-Aided Language Models (PAL), proposed by Gao et al. (2022) (opens in a new tab), solves this by using the LLM to generate code as intermediate reasoning steps.

The final execution is then offloaded to a programmatic runtime, such as a Python interpreter, rather than relying on the model's "mental math."

Image Source: Gao et al. (2022)

PAL vs. Chain-of-Thought

In CoT, the model generates free-form text that leads to a solution. In PAL, the model generates a Python script. This shift ensures:

• Precision: Algorithmic calculations are 100% accurate.
• Redundancy: The reasoning logic is codified and can be easily audited or reused.
• Complexity: Tasks involving date arithmetic or complex symbolic logic become trivial.

Example: Date Understanding

One of PAL's strongest use cases is interpreting and calculating dates. Here's how it works using LangChain and OpenAI:

The Prompt

We provide the model with a few exemplars where reasoning is mapped to Python's datetime logic:

# Q: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
# If 2015 is coming in 36 hours, then today is 36 hours before.
today = datetime(2015, 1, 1) - relativedelta(hours=36)
# One week from today,
one_week_from_today = today + relativedelta(weeks=1)
# The answer formatted with %m/%d/%Y is
one_week_from_today.strftime('%m/%d/%Y')

The Output

When asked: "Today is 27 February 2023. I was born exactly 25 years ago. What is the date I was born in MM/DD/YYYY?", PAL generates:

# If today is 27 February 2023 and I was born 25 years ago, then I was born 25 years before.
today = datetime(2023, 2, 27)
# I was born 25 years before,
born = today - relativedelta(years=25)
# The answer formatted with %m/%d/%Y is
born.strftime('%m/%d/%Y')

Executing this code yields the precise answer: 02/27/1998.

[!TIP] PAL is particularly powerful when combined with Prompt Chaining. Use the LLM to structure the logic, and use Python to execute the math.