# BigCodeBench: The Next Generation of HumanEval

DevFeed: [BigCodeBench: The Next Generation of HumanEval](<https://devfeed.tech/articles/bigcodebench-the-next-generation-of-humaneval-7314.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/leaderboard-bigcodebench>)

Author: Terry Yue Zhuo; Jiawei Liu; Qian Liu; Binyuan Hui; Niklas Muennighoff; Daniel Fried; Harm de Vries; Leandro von Werra; Clémentine Fourrier

Published: 2024-06-18T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [LLMs](<https://devfeed.tech/topics/llms.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Code](<https://devfeed.tech/topics/code.md>), [code-completion](<https://devfeed.tech/topics/code-completion.md>), [test-coverage](<https://devfeed.tech/topics/test-coverage.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code](<https://devfeed.tech/tags/code.md>), [code-completion](<https://devfeed.tech/tags/code-completion.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [community](<https://devfeed.tech/tags/community.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [llms](<https://devfeed.tech/tags/llms.md>), [performance](<https://devfeed.tech/tags/performance.md>), [research](<https://devfeed.tech/tags/research.md>)

## AI overview

The article announces BigCodeBench, a benchmark for evaluating the programming capabilities of large language models on practical, challenging tasks. It contains 1,140 function-level tasks using 139 libraries, with detailed instructions, interactive examples, runtime test harnesses, and extensive branch coverage. The benchmark measures whether models correctly generate code and compares its complexity and tool-use diversity with existing programming benchmarks.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.