Feature Tracker: Evaluation Benchmarks

[rmusser01]

Title.

Benchmarks:
Evaluation Methodologies

• G-Eval
• QAG

Coding Ability

• Aider Benchmark
• CodeMMLU

Confabulation Rate

• TruthfulQA
• f

Context Length

• Ruler
• InfiniteBench
• Babilong
• LongICLBench
• HelloBench
• Snorkel Working Memory Test

Creative Writing

• EQ Bench
• f

Pop Culture

• f

Reasoning

• MMLU Pro
• ARC public

Role Playing

• Conversational Relevancy
• Role Adherence
• Knowledge Retention
• Conversation Competeness

Summarization

• DeepEval
• Salesforce
• F

Tool Calling

• DeepEval
• Berkeley Function Calling https://gorilla.cs.berkeley.edu/leaderboard.html
• ToolSandbox https://arxiv.org/html/2408.04682v1

Toxicity Testing

• DeepEval
• f

Vibes

• AidanBench
• f

[rmusser01]

Using an LLM as a Response Judge

Some metrics cannot be defined objectively and are particularly useful for more subjective or complex criteria. We care about correctness, faithfulness, and relevance.

    Answer Correctness - Is the generated answer correct compared to the reference and thoroughly answers the user's query?
    Answer Relevancy - Is the generated answer relevant and comprehensive?
    Answer Factfulness - Is the generated answer factually consistent with the context document?