Skip to content

Issues: EleutherAI/lm-evaluation-harness

reproduce llama 3 evals
#2557 opened Dec 10, 2024 by baberabb
Open 6
Beta
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

387 Open 959 Closed
387 Open 959 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

TypeError: top_k_top_p_sampling_from_probs() missing 1 required positional argument: 'top_p'
#2936 opened Apr 28, 2025 by rangehow
np.NaN
#2935 opened Apr 27, 2025 by upunaprosk
1
how do evaluate mmlu by local-chat-completions or local-completions?
#2934 opened Apr 26, 2025 by amdslgl
Are there any Safety Alignment Benchmarks ready for use?
#2933 opened Apr 26, 2025 by JovialYip
Some LongBench scores on Llama-3.1-8B-Instruct are far off
#2932 opened Apr 25, 2025 by cameronshinn
1
llama3-8b-instruct 0-shot mbpp pass@1=0
#2931 opened Apr 25, 2025 by LafouCC
1
Support for OlympiadBench and AMC
#2930 opened Apr 25, 2025 by Edric-Zhao
What is the correct way to save result file?
#2927 opened Apr 24, 2025 by rangehow
1
Do we have plan to add BFCL into the tasks?
#2926 opened Apr 23, 2025 by zzhangncsu
eval bbh failed
#2922 opened Apr 18, 2025 by godlikehhd
clean up process results bug Something isn't working.
#2920 opened Apr 16, 2025 by baberabb
@baberabb
GPQA Preprocessing Function Results in Incorrect Physics Equations bug Something isn't working. validation For validation of task implementations.
#2907 opened Apr 14, 2025 by ShayekhBinIslam
@baberabb
1
Filter not extracting choice selection correctly validation For validation of task implementations.
#2905 opened Apr 14, 2025 by 1jamesthompson1
@baberabb
2
Hang with VLLM backend when data_parallel_size > 1
#2904 opened Apr 14, 2025 by rangehow
3
Does lm-eval currently support testing for models in the deekseek r1 category?
#2903 opened Apr 13, 2025 by Polarisamoon
Caching requests does not work for model gguf
#2901 opened Apr 11, 2025 by falkbene
How to use *chat_template* with .gguf models ?
#2900 opened Apr 11, 2025 by Bobchenyx
12
Improve the behavior of progress bars when using huggingface
#2898 opened Apr 11, 2025 by Zephyr271828
Additional data for evaluation
#2896 opened Apr 9, 2025 by harshakokel
1
RuntimeError: 500 Server Error for URL During LM-Eval with gguf Model
#2894 opened Apr 9, 2025 by amjh83
1
Injecting custom variables/values into a task via YAML
#2891 opened Apr 8, 2025 by j0ma
2
low eval accuracy with gguf
#2887 opened Apr 7, 2025 by jerryzh168
Hellaswag filenotfound error
#2886 opened Apr 7, 2025 by dsvilarkovic
3
LLaMA-4 evaluation
#2885 opened Apr 7, 2025 by jybbjybb
1
TypeError from missing yaml_path in lm_eval.utils.load_yaml_config when task uses include
#2884 opened Apr 6, 2025 by MarieRoald
@baberabb
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.