Evaluating OpenAI’s o1 Models: Advancements and Challenges in AI Planning

Introduction

OpenAI’s o1 preview and o1 mini models have been available for a few weeks now, introducing a significant advancement in AI capabilities. Rather than just scaling the model at training time, OpenAI has unlocked the ability to scale the model at test time as well. This essentially means giving the model the ability to plan and think long-term. But how good are these models really? A new research paper put the 01 models to the test against the GPT-4 model, and the results are quite impressive.

The Importance of Testing AI Models

Before diving into the specifics of the study, it’s crucial to understand what it means to test an AI model effectively. The ARC prize, a million-dollar award for achieving AGI (Artificial General Intelligence), provides some insight:

“Most AI benchmarks measure skill, but skill is not intelligence. General intelligence is the ability to efficiently acquire new skills.”

This research paper has created a series of tests that are easy for humans to solve but challenging for AI, focusing on spatial reasoning and logic problems.

The Research Paper: Overview

The paper, titled “On the Planning Abilities of OpenAI o1 Models: Feasibility, Optimality, and Generalizability,” introduces six different benchmarks to test the models’ capabilities. It evaluates the models across three key perspectives:

1. Plan feasibility
2. Plan optimality
3. Plan generalizability

Key Findings from the Abstract

• o1 preview shows strengths in self-evaluation and constraint following
• Identifies bottlenecks in decision-making and memory management
• Highlights challenges in tasks requiring robust spatial reasoning

Evaluating AI Planning Capabilities

The paper focuses on the use of language agents for planning in the interactive physical world, an area that remains challenging for Large Language Models (LLMs). This has implications for embodied agents (robots) operating in the physical world.

Three Key Perspectives in Evaluation

1. Feasibility: The model’s ability to come up with a workable plan within the given rules and constraints.
   • Subcategories:
     • Ability to create feasible steps
     • Ability to generate a feasible plan
     • Ability to understand the problem
2. Optimality: How efficient the plan is in achieving the goal.

3. Generalizability: The ability to apply learned knowledge to new, unfamiliar scenarios.

The Six Benchmark Tests

The research paper introduced six different tests to evaluate the models:

1. Barman
2. Blocks World
3. Grippers
4. Floor Tile
5. Termes
6. Tire World

Let’s explore each of these tests and how the models performed.

1. Barman

Task: A robot barman must prepare a series of drinks by manipulating drink dispensers, shot glasses, and a shaker.

Results:
– All models struggled significantly
– Most errors stemmed from the inability to follow specified rules (IR error)

2. Blocks World

Task: Move blocks from an initial configuration to a pre-specified goal configuration using a robot arm.

Results:
– GPT-4: 40% success rate
– o1 Mini: 60% success rate
– o1 Preview: 100% success rate, but not always optimal

3. Grippers

Task: Control robots with two grippers to move objects between rooms.

Results:
– All models performed relatively well
– o1 Preview’s main failure was misinterpreting the goal state

4. Floor Tile

Task: Paint a grid of floor tiles in black and white using a team of robots.

Results:
– All models failed to solve the test cases
– o1 Preview showed improvement in understanding rules but encountered other errors

5. Termes

Task: Control a robot to construct 3D structures by moving and manipulating blocks.

Results:
– All models failed due to shortcomings in detailed planning and failure to account for height constraints

6. Tire World

Task: Replace flat tires on vehicle hubs with intact, inflated tires using various tools.

Results:
– o1 Preview generated correct plans for all test problems
– When tested for generalization (using random symbols), 01 Preview’s success rate dropped from 100% to 80%

Key Insights and Implications

1. Complexity vs. Performance: There’s a strong correlation between problem complexity and model performance. Tasks with higher spatial and rule-based complexity (like Floor Tile and Termes) posed significant challenges.

2. Constraint Following: o1 models showed improved ability to follow constraints and manage states compared to GPT-4.

3. Optimality: While o1 Preview often generated feasible plans, it frequently failed to produce optimal solutions.

4. Generalizability: o1 Preview demonstrated better ability to generalize across tasks with consistent rule structures compared to GPT-4, but there’s still substantial room for improvement.

Areas for Improvement

The paper suggests several ways to enhance these models:

1. Develop more sophisticated decision-making mechanisms for better optimality and resource utilization
2. Improve generalization on abstract spaces through better memory management
3. Enhance constraint adherence through self-evaluation
4. Leverage multimodal inputs for better understanding of spatial relationships
5. Implement multi-agent frameworks for improved problem-solving
6. Incorporate human feedback for continuous learning

While the o1 models show significant improvements in certain areas, particularly in understanding and following rules, they still fall short of AGI-level capabilities, especially in complex spatial reasoning tasks. This study provides valuable insights into the current state of AI planning abilities and highlights areas for future research and development.

Reference:

On The Planning Abilities of OpenAI’s o1 Models: Feasibility, Optimality, and Generalizability

https://www.arxiv.org/abs/2409.19924
The writers are from the University of Texas at Austin