Skip to content

Issues: THUDM/LongBench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

41 Open 58 Closed
41 Open 58 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Reproducibility Issues
#111 opened Feb 7, 2025 by Hisham-Cohere
1
The difference between longbench v2 and helmet?
#110 opened Feb 6, 2025 by leileilin
w/ cot mode for "thinking" models
#109 opened Feb 6, 2025 by olive-jy-song
Potential Error in Benchmark Data – Incorrect Answer for Question ID 6701cda0bb02136c067cb6eb
#108 opened Feb 5, 2025 by Wangmerlyn
LongBench v2 Leaderboard Submission Request: Qwen2.5-14B & Gemini2.0 Flash Experimental Results
#107 opened Feb 3, 2025 by iclal07
1
What are the accurate scores of the task-level Radar graph
#106 opened Jan 23, 2025 by hzhua
bug in pred.py <<def build_chat(tokenizer, prompt, model_name)>> func
#105 opened Jan 20, 2025 by zswodegit
Error code: 400 - {'object': 'error', 'message': "This model's maximum context length is 131072 tokens. However, you requested 351430 tokens. Please reduce the length of the messages or completion.", 'type': 'BadRequestError', 'param': None, 'code': 400
#104 opened Jan 18, 2025 by guanzy2012
2
Token indices sequence length is longer than the specified maximum sequence length for this model (1113927 > 128000). Running this sequence through the model will result in indexing errors
#103 opened Jan 18, 2025 by guanzy2012
1
Error code: 404 - {'object': 'error', 'message': 'The model /dev/shm/glm-4-9b-chat/ does not exist.', 'type': 'NotFoundError', 'param': None, 'code': 404}
#102 opened Jan 18, 2025 by guanzy2012
1
Any Implementation of new models like Meta-Llama-3.1-8B , Qwen2.5-7B?
#101 opened Jan 17, 2025 by WeiweiZhang1
1
Why are empty responses ignored in LongBench v2?
#100 opened Jan 15, 2025 by ZeonfaiHo
1
[LongBench] How was Edit Sim for code tasks calculated?
#96 opened Jan 9, 2025 by cornzz
1
如何使vllm下测评结果更接近hf？
#95 opened Jan 7, 2025 by miaoyuxun
2
关于Llama-3.1-8B-Instruct在Longbench v2 测试结果和排行榜有出入的问题
#94 opened Jan 2, 2025 by chaochen99
4
针对 Paper中 Long数据集的评测配置
#93 opened Jan 2, 2025 by MaiziXiao
3
关于Claude和gemini的token处理
#92 opened Jan 1, 2025 by Violettttee
1
关于cot
#90 opened Dec 26, 2024 by Violettttee
7
DeepSeek V3 Result enhancement New feature or request
#89 opened Dec 26, 2024 by zmwang03
@bys0318
1
results.py bug
#88 opened Dec 24, 2024 by cizhenshi
1
安装完longbench后，运行报错
#86 opened Dec 14, 2024 by fxnie
Request for Mixtral-8*7B model
#84 opened Nov 18, 2024 by Aaronhuang-778
请问用GPT-3.5-turbo-16K测试一遍完整的数据集需要花费多少？
#83 opened Nov 4, 2024 by Ethereal-sakura
1
Evaluation mechanism update
#82 opened Oct 30, 2024 by cizhenshi
1
可以测试基于OpenAI接口的模型管理框架吗，比如ollama, xinference
#74 opened Aug 29, 2024 by jiusi9
1
ProTip! no:milestone will show everything without a milestone.