Skip to content

Instantly share code, notes, and snippets.

@CultriX-Github

CultriX-Github/Llama-3.2-3B-DPO-Nous.md

Created October 20, 2024 01:29
Show Gist options
  • Save CultriX-Github/3fc21423a7422eb3e55551677d0375ce to your computer and use it in GitHub Desktop.
Save CultriX-Github/3fc21423a7422eb3e55551677d0375ce to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Llama-3.2-3B-DPO-Nous.md
Model AGIEval GPT4All TruthfulQA Bigbench
Llama-3.2-3B-DPO 27.06 Error: File does not exist 58.93 34.96

AGIEval

Task Version Metric Value Stderr
agieval_aqua_rat 0 acc 18.90 ± 2.46
acc_norm 20.87 ± 2.55
agieval_logiqa_en 0 acc 26.11 ± 1.72
acc_norm 32.26 ± 1.83
agieval_lsat_ar 0 acc 16.52 ± 2.45
acc_norm 18.26 ± 2.55
agieval_lsat_lr 0 acc 27.25 ± 1.97
acc_norm 27.25 ± 1.97
agieval_lsat_rc 0 acc 29.00 ± 2.77
acc_norm 25.28 ± 2.65
agieval_sat_en 0 acc 50.00 ± 3.49
acc_norm 41.26 ± 3.44
agieval_sat_en_without_passage 0 acc 36.41 ± 3.36
acc_norm 27.18 ± 3.11
agieval_sat_math 0 acc 26.36 ± 2.98
acc_norm 24.09 ± 2.89

Average: 27.06%

GPT4All

Average: Error: File does not exist%

TruthfulQA

Task Version Metric Value Stderr
truthfulqa_mc 1 mc1 41.98 ± 1.73
mc2 58.93 ± 1.55

Average: 58.93%

Bigbench

Task Version Metric Value Stderr
bigbench_causal_judgement 0 multiple_choice_grade 47.89 ± 3.63
bigbench_date_understanding 0 multiple_choice_grade 63.14 ± 2.51
bigbench_disambiguation_qa 0 multiple_choice_grade 33.72 ± 2.95
bigbench_geometric_shapes 0 multiple_choice_grade 10.03 ± 1.59
exact_str_match 0.00 ± 0.00
bigbench_logical_deduction_five_objects 0 multiple_choice_grade 25.80 ± 1.96
bigbench_logical_deduction_seven_objects 0 multiple_choice_grade 19.43 ± 1.50
bigbench_logical_deduction_three_objects 0 multiple_choice_grade 48.33 ± 2.89
bigbench_movie_recommendation 0 multiple_choice_grade 34.20 ± 2.12
bigbench_navigate 0 multiple_choice_grade 50.00 ± 1.58
bigbench_reasoning_about_colored_objects 0 multiple_choice_grade 50.80 ± 1.12
bigbench_ruin_names 0 multiple_choice_grade 22.10 ± 1.96
bigbench_salient_translation_error_detection 0 multiple_choice_grade 25.65 ± 1.38
bigbench_snarks 0 multiple_choice_grade 47.51 ± 3.72
bigbench_sports_understanding 0 multiple_choice_grade 49.49 ± 1.59
bigbench_temporal_sequences 0 multiple_choice_grade 17.60 ± 1.20
bigbench_tracking_shuffled_objects_five_objects 0 multiple_choice_grade 20.56 ± 1.14
bigbench_tracking_shuffled_objects_seven_objects 0 multiple_choice_grade 14.74 ± 0.85
bigbench_tracking_shuffled_objects_three_objects 0 multiple_choice_grade 48.33 ± 2.89

Average: 34.96%

Average score: Not available due to errors

Elapsed time: 02:36:06

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment