MIT Researchers Teach LLMs to Solve Complex Planning Challenges with Innovative Framework
Researchers at MIT have developed a novel framework that significantly enhances the ability of Large Language Models (LLMs) to tackle complex planning problems. Unlike previous approaches that attempted to modify LLMs themselves, this framework guides LLMs to break down problems in a manner similar to human problem-solving, subsequently utilizing powerful software tools to find optimal solutions.
The framework, detailed in a paper presented at the International Conference on Learning Representations, allows users to describe planning problems in natural language. The LLM then encodes this prompt into a format decipherable by an optimization solver, designed to efficiently address intricate planning challenges.
According to Yilun Hao, a graduate student at MIT’s Laboratory for Information and Decision Systems (LIDS) and lead author of the paper, this research introduces “a framework that essentially acts as a smart assistant for planning problems.” The framework identifies the best plan to meet specified needs, even when rules are complex or unusual. The co-authors of the paper include Yang Zhang, a research scientist at the MIT-IBM Watson AI Lab, and Chuchu Fan, an associate professor of aeronautics and astronautics and LIDS principal investigator.
The LLM meticulously checks its work at each intermediate step during the formulation process, ensuring accurate problem description for the solver. If an error is detected, the LLM attempts to rectify the specific broken part rather than abandoning the entire process. This iterative self-correction mechanism enhances the robustness and reliability of the planning process.
In tests conducted on nine complex challenges, including minimizing travel distance for warehouse robots, the framework achieved an impressive 85% success rate. This is a significant improvement compared to the best baseline, which only achieved a 39% success rate. The framework’s versatility makes it applicable to a wide array of multistep planning tasks, such as scheduling airline crews or managing machine time within a factory.
Chuchu Fan notes that their algorithmic solvers typically have steep learning curves, limiting their use to experts. “We thought that LLMs could allow nonexperts to use these solving algorithms. In our lab, we take a domain expert’s problem and formalize it into a problem our solver can solve. Could we teach an LLM to do the same thing?”
The developed framework, named LLM-Based Formalized Programming (LLMFP), requires a person to provide a natural language description of the problem, background information, and a query defining their goal. LLMFP then prompts the LLM to reason about the problem, identify key constraints, and determine the decision variables that shape the optimal solution.
LLMFP’s self-assessment module also permits the LLM to incorporate any implicit constraints initially overlooked. For example, while a human understands that a coffee shop cannot ship a negative quantity of roasted beans, an LLM might not inherently grasp this constraint. The self-assessment step flags such errors and prompts the model to correct them.
Furthermore, Fan indicates that an LLM can adapt to user preferences. For instance, the model can suggest modifications that align with the user’s needs if it recognizes that a particular user prefers not to alter their travel plans’ time or budget.
Future research aims to enable LLMFP to accept images as input to complement the descriptions of planning problems. This enhancement would aid the framework in solving tasks that are difficult to fully articulate using natural language alone.
