OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

MT-Bench evaluation of a model using pre generated model answers

  • Thread starter Thread starter Masthan
  • Start date Start date
M

Masthan

Guest
I want to find MT-Bench score of an LLM (say EleutherAI/pythia-1b).I was able to run the command

python gen_model_answer.py --model-pat EleutherAI/pythia-1b --model-id pythia-1b

to generate answers and I could see the output in the json file "data/mt_bench/model_answer/pythia-1b.jsonl". I have downloaded pre generated model answers using the command

python3 download_mt_bench_pregenerated.py

How to compare "pythia-1b" generated answer and any pre generated answer(say llama-13b) to calculate MT-Bench score for "pythia-1b" model ?
<p>I want to find MT-Bench score of an LLM (say EleutherAI/pythia-1b).I was able to run the command</p>
<blockquote>
<p>python gen_model_answer.py --model-pat EleutherAI/pythia-1b --model-id pythia-1b</p>
</blockquote>
<p>to generate answers and I could see the output in the json file "data/mt_bench/model_answer/pythia-1b.jsonl".
I have downloaded pre generated model answers using the command</p>
<blockquote>
<p>python3 download_mt_bench_pregenerated.py</p>
</blockquote>
<p>How to compare "pythia-1b" generated answer and any pre generated answer(say llama-13b) to calculate MT-Bench score for "pythia-1b" model ?</p>
 

Latest posts

Top