LLM4Opt

Collection on Large Language Models for Optimization (LLM4Opt)

Applying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt. The Papers are sorted by time (first publicly available). Any suggestions and pull requests are welcome.

It is far from a comprehensive list. If you want to update the list:

• Fork, Add, and Merge
• Report an issue
• Contact Fei Liu (fliu36-c@my.cityu.edu.hk)

The sharing principle of these references here is for research. If any authors do not want their paper to be listed here, please feel free to contact us.

Overview

• Platform
• Tutorial
• Competition
• Research Papers
  • [Review]
  • [Position Paper]
  • [AI, Algorithm Design]
  • [LLM as optimizer]
  • [Machine Learning]
  • [Science]
  • [...]

Platform

• EoH (Evolution of Heuristics) (optimization, mathematics, machine learning, etc)
• OpenELM (robots, image, programming puzzles, etc)

Tutorial&Workshop

• NeurIPS 2023 Workshop: Foundation Models for Decision Making
• GECCO 2024 Workshop: Large Language Models for and with Evolutionary Computation (LLMfwEC)
• GECCO 2024 Workshop: EGML-EC — 3rd GECCO workshop on Enhancing Generative Machine Learning with Evolutionary Computation (EGML-EC) 2024
• GECCO 2024 Tutorial: Using Large Language Models for Evolutionary Search || Tutorial Report: Evolving Code with A Large Language Model
• PPSN 2024 Tutorial: Large Language Models as Tools for Metaheuristic Design: Exploring Challenges and Opportunities
• KDD 2024 Tutorial: NL2Code-Reasoning and Planning with LLM for Code Development, DEEP LEARNING AND LARGE LANGUAGE MODELS FOR KNOWLEDGE GRAPHS
• ICML 2024 Workshop: AI for Math

Competition

• AAAI 2024 Global Competition on Math Problem Solving and Reasoning
• ICML 2024 Challenges on Automated Math Reasoning

Papers

Review

• Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap, Arxiv, Jan 2024, [code], paper
• A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond, Arxiv, Mar. 2024, [code], paper
• When Large Language Model Meets Optimization, Arxiv, May 2024, [code], paper

Position paper

• The Era OF Semantic Decoding, Arxiv, Mar. 2024, [code], paper
• Leveraging Foundational Models for Black-Box Optimization: Benefits, Challenges, and Future Directions, ICML 2024 , May 2024, [code], paper

Algorithm/Heuristic/Function Search

• Hypothesis Search: Inductive Reasoning with Language Models, Arxiv Sep 2023, ICLR 2024, [code] paper
• ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search, Arxiv Oct 2023, ICLR 2024 [code] paper
• Algorithm Evolution using Large Language Model, Arxiv, Nov 2023, code,paper
• Mathematical discoveries from program search with large language models, Nature, Dec 2023, code,paper
• Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model, ICML, 2024, code, paper
• ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution, Arxiv, Feb 2024, code, paper
• Discovering More Effective Tensor Network Structure Search Algorithms via Large Language Models (LLMs), [code], paper
• AutoSAT: Automatically Optimize SAT Solvers via Large Language Models, Arxiv, Feb 2024, [code], paper
• Large Language Models tO Enhance Bayesian Optimization, Arxiv Feb 2024, ICLR 2024 [code], paper
• On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks, Arxiv, Feb 2024, [code], paper
• How Can LLM Guide RL? A Value-Based Approach, Arxiv, Feb 2024, code, paper
• Evolve Cost-aware Acquisition Functions Using Large Language Models, Arxiv, April 2024, [code], paper
• Benchmarking ChatGPT on Algorithmic Reasoning, Arxiv, April 2024, [code], paper
• How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problems, Arxiv, March 2024, [code], paper
• LLM-ABR: Designing Adaptive Bitrate Algorithms via Large Language Models, Arxiv, April 2024, [code], paper
• Constrained Neural Networks for Interpretable Heuristic Creation to Optimise Computer Algebra Systems, Arxiv, April 2024, [code], paper
• GLHF: General Learned Evolutionary Algorithm Via Hyper Functions, Arxiv [code] paper

LLM as optimizer

• Large Language Models as Optimizers, Arxiv, Sep 2023, code, paper
• Large language model for multi-objective evolutionary optimization, Arxiv, Oct. 2023, code, paper
• Large Language Models as Evolutionary Optimizers, Arxiv, Oct. 2023, [code], paper
• Using Large Language Models for Hyperparameter Optimization, NeurIPS 2023, [code], paper
• Large Language Models As Evolution Strategies, Arxiv, Feb. 2024, [code], paper
• Large Language Model-based Evolutionary Optimizer: Reasoning with Elitism, Arxiv, Mar. 2024, [code], paper
• Large Language Model-Aided Evolutionary Search for Constrained Multiobjective Optimization, Arxiv, May 9, 2024, [code], paper

Code Generation

• L2MAC: LARGE LANGUAGE MODEL AUTOMATIC COMPUTER FOR EXTENSIVE CODE GENERATION, Arxiv Oct 2023, ICLR 2024, [code] paper

Prompt Opt

• Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers, Arxiv Sep 2023, ICLR 2024 [code], paper
• PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization, Arxiv Oct 2023, ICLR 2024 [code], paper

Machine Learning

• Evolution through Large Models, Arxiv, June 2022, code, arxiv paper, chapter
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph, Arxiv July 2023, ICLR 2024, [code], paper
• Evoprompting: Language models for code-level neural architecture search, NeuIPS 2023, [code], paper
• Eureka: Human-Level Reward Design via Coding Large Language Models, ICLR 2024, code, paper
• LANGUAGE MODEL DECODING AS DIRECT METRICS OPTIMIZATION, Arxiv, Oct 2023, ICLR 2024, [code], paper
• Label-free Node Classification on Graphs with Large Language Models (LLMS), Arxiv, Oct 2023, ICLR 2024, [code], paper
• L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attacks, GECCO 2024, Jan 2024 [code], paper
• Data-driven Discovery with Large Generative Models, Arxiv, Feb. 2024, [code], paper
• Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Network, Arxiv, Feb. 2024, [code], paper
• LLM Guided Evolution-The Automation of Models Advancing Models, Arxiv, Mar. 2024, [code], paper
• Identify Critical Nodes in Complex Network with Large Language Models, Arxiv, Mar. 2024, [code], paper
• Evolving Interpretable Visual Classifiers with Large Language Models, Arxiv, April 2024, [code], paper

Science

• A Prompt-Engineered Large Language Model, Deep Learning Workflow for Materials Classification, Arxiv, Jan 2024, [code], paper
• LLM-SR: Scientific Equation Discovery via Programming with Large Language Models, Arxiv, April 2024, [code], paper
• Large Language Model Agent as a Mechanical Designer, Arxiv, April 2024, [code], paper

Industry

• Large Language Models for Supply Chain Optimization, Arxiv, July 2023, [code], paper
• LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automation, Arxiv, Dec. 2023, [code], paper

Related Collections

• Foundation Models for Combinatorial Optimization
• MOO-ML-Papers
• Awesome LLM