AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data

Zifan Song^1,2*, Yudong Wang^2*, Wenwei Zhang^2*, Kuikun Liu², Chengqi Lyu², Demin Song², Qipeng Guo², Hang Yan², Dahua Lin^2,3, Kai Chen^2†, Cairong Zhao^1†

^* Equal Contribution, ^† Corresponding Author

¹ Tongji University, ² Shanghai AI Laboratory, ³ Chinese University of Hong Kong

[🤗 HuggingFace] [📃 Paper] [🌐 Project Page] [ 🚀 Quick Start] [😃 Acknowledgements] [🌟 Citation]

📢 What's New

• [2024.09.26] AlchemistCoder has been accepted by NeurIPS 2024. 🎉🎉🎉
• [2024.05.29] Paper and models for AlchemistCoder are released. 🔥🔥🔥

✨ Highlights

Abstract: Open-source Large Language Models (LLMs) and their specialized variants, particularly Code LLMs, have recently delivered impressive performance. However, previous Code LLMs are typically fine-tuned on single-source data with limited quality and diversity, which may insufficiently elicit the potential of pre-trained Code LLMs. In this paper, we present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. To achieve this, we pioneer to unveil inherent conflicts among the various styles and qualities in multi-source code corpora and introduce data-specific prompts with hindsight relabeling, termed AlchemistPrompts, to harmonize different data sources and instruction-response pairs. Additionally, we propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review. Extensive experiments demonstrate that AlchemistCoder holds a clear lead among all models of the same size (6.7B/7B) and rivals or even surpasses larger models (15B/33B/70B), showcasing the efficacy of our method in refining instruction-following capabilities and advancing the boundaries of code intelligence.

• AlchemistPrompts: Designed as data-specific prompts for harmonizing inherent conflicts in multi-source data and mitigating the instruction/response misalignment at a fined-grained level.

• Code Comprehension Tasks: Sourced from the process of data construction, consisting of instruction evolution, data filtering, and code review.

• Harmonized Multi-source Data: Instruction tuned on 200M tokens, including 6 types of high-quality data.

• Superior Model Performance: Surpassing all the open-source models of the same size (6.7/7B), and rivaling or even beating larger models (15B/33B/70B/ChatGPT) on 6 code benchmarks.

• Advanced generic capabilities: Demonstrated by the significant improvements on MMLU, BBH, and GSM8K.

📈 Evaluation Results

We adopt 9 benchmarks to evaluate our AlchemistCoder series models, including 6 code benchmarks (HumanEval, HumanEval+, MBPP, MBPP+, HumanEval-X, and DS-1000) and 3 mainstream benchmarks (MMLU for multitask language understanding, BBH for comprehensive reasoning, and GSM8K for mathematical ability). All evaluations are conducted through OpenCompass, an LLM evaluation platform, supporting a wide range of models (LLaMA, LLaMa2, ChatGLM2, ChatGPT, Claude, etc) over 80+ datasets.

We focus on comparing the pass@1 metric and detailed evaluation results are reported below:

• AlchemistCoder holds a clear lead among all models of the same size and rivals or even surpasses larger models.

• Python Code Generation Benchmark(HumanEval/HumanEval+ and MBPP/MBPP+):

• Multilingual HumanEval-X Benchmark:

• Data Science DS-1000 Benchmark:

• Mainstream Benchmarks (MMLU, BBH, and GSM8K) for Generic Capabilities:

📦 Models

All AlchemistCoder series models have been open-source on Hugging Face, including AlchemistCoder-L-7B, AlchemistCoder-CL-7B, and AlchemistCoder-DS-6.7B. You can access our models via the following link:

Dataset	Huggingface Repo
AlchemistCoder-L-7B	Model Link
AlchemistCoder-CL-7B	Model Link
AlchemistCoder-DS-6.7B	Model Link

🚀 Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("internlm/AlchemistCoder-DS-6.7B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("internlm/AlchemistCoder-DS-6.7B", trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()
model = model.eval()

input_text = "Implement the Dijkstra algorithm in Python"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The above code will generate the following result:

import heapq

def dijkstra(graph, start, end):
    """
    Find the shortest path between two nodes in a graph using Dijkstra's algorithm.

    Args:
        graph (dict): A dictionary representing the graph, where the keys are nodes and the values are dictionaries
                      containing the neighboring nodes and their edge weights.
        start: The starting node.
        end: The target node.

    Returns:
        list: The shortest path from the start node to the target node as a list of nodes.

    Raises:
        ValueError: If either the start or end node is not present in the graph.

    """

    # Check if start and end nodes are in the graph
    if start not in graph:
        raise ValueError("Start node is not present in the graph.")
    if end not in graph:
        raise ValueError("End node is not present in the graph.")

    # Initialize the distance dictionary with infinite distances for all nodes
    distances = {node: float('inf') for node in graph}

    # Set the distance of the start node to 0
    distances[start] = 0

    # Initialize the heap with the start node
    heap = [(0, start)]

    # Initialize the previous dictionary to keep track of the path
    previous = {}

    while heap:
        # Pop the node with the smallest distance from the heap
        current_distance, current_node = heapq.heappop(heap)

        # If the current node is the end node, we have found the shortest path
        if current_node == end:
            # Reconstruct the path by following the previous nodes
            path = []
            while current_node in previous:
                path.append(current_node)
                current_node = previous[current_node]
            path.append(start)
            # Reverse the path to get the correct order
            return path[::-1]

        # Iterate over the neighboring nodes and update their distances
        for neighbor, weight in graph[current_node].items():
            new_distance = current_distance + weight
            # If a shorter path is found, update the distance and previous node
            if new_distance < distances[neighbor]:
                distances[neighbor] = new_distance
                previous[neighbor] = current_node
                heapq.heappush(heap, (new_distance, neighbor))

    # If there is no path between the start and end nodes, return an empty list
    return []

The dijkstra function takes three arguments: graph, start, and end. The graph argument is a dictionary representing the graph, where the keys are nodes and the values are dictionaries containing the neighboring nodes and their edge weights. The start argument is the starting node, and the end argument is the target node.

The function first checks if the start and end nodes are present in the graph. If either node is not present, a ValueError is raised.

The function then initializes a distances dictionary with infinite distances for all nodes. It sets the distance of the start node to 0. It also initializes a heap with the start node and a previous dictionary to keep track of the path.

The algorithm then iterates over the nodes in the heap. For each node, it checks if it is the end node. If it is, the function reconstructs the path by following the previous nodes and returns the shortest path as a list of nodes in the correct order.

If the current node is not the end node, the algorithm iterates over its neighboring nodes and updates their distances if a shorter path is found. It also updates the previous dictionary to keep track of the path.

If there is no path between the start and end nodes, the function returns an empty list.

Note that this implementation assumes that the graph is a directed graph, and it uses a heap data structure to efficiently select the node with the smallest distance at each step.

🧪 Evaluation

To effortlessly evaluate AlchemistCoder series models across a wide range of benchmarks, such as HumanEval and MBPP, you can use OpenCompass with just one command. To get started, install OpenCompass, adopt the provided configuration file in eval/opencompass/configs and simply run the commands below:

cd opencompass
python run.py configs/eval_alchemistcoder_models_hf_official/eval_alchemistcoder_models_hf_code.py
python run.py configs/eval_alchemistcoder_models_hf_official/eval_alchemistcoder_models_hf_coreset.py

Make sure to adjust the directory structure and arguments according to your requirements.

📖 Fine-tune and Others

Please refer to InternLM.

😃 Acknowledgments

AlchemistCoder is built with InternLM and OpenCompass. Thanks for their awesome work!

📧 Contact

If you have any questions, please create an issue on this repository or contact us at:

• sugger@tongji.edu.cn
• zhangwenwei@pjlab.org.cn

🌟 Citation

If you find our work useful, please consider citing:

@misc{song2024alchemistcoder,
      title={AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data}, 
      author={Zifan Song and Yudong Wang and Wenwei Zhang and Kuikun Liu and Chengqi Lyu and Demin Song and Qipeng Guo and Hang Yan and Dahua Lin and Kai Chen and Cairong Zhao},
      year={2024},
      eprint={2405.19265},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}