Runcrate Logoruncrate

MohamedRashad/Arabic-Orpo-Llama-3-8B-Instruct

text generationtransformersartransformerssafetensorsllamatext-generationconversationalarllama3
17
HuggingFaceHuggingFace
3.4K

👳 Arabic ORPO LLAMA 3

👓 Story first

This model is the a finetuned version of meta-llama/Meta-Llama-3-8B-Instruct using ORPO on 2A2I/argilla-dpo-mix-7k-arabic.

I wanted to try ORPO and see if it will better align a biased English model like llama3 to the arabic language or it will faill.

While the evaluations favour the base llama3 over my finetune, in practice i found my finetune was much better at spitting coherent (mostly correct) arabic text which i find interesting.

I would encourage everyone to try out the model from here and share his insights with me ^^

🤔 Evaluation and Results

This result was made using lighteval with the community|arabic_mmlu tasks.

CommunityLlama-3-8B-InstructArabic-ORPO-Llama-3-8B-Instrcut
All0.3480.317
Abstract Algebra0.3100.230
Anatomy0.3850.348
Astronomy0.3880.316
Business Ethics0.4800.370
Clinical Knowledge0.3960.385
College Biology0.3470.299
College Chemistry0.1800.250
College Computer Science0.2500.190
College Mathematics0.2600.280
College Medicine0.2310.249
College Physics0.2250.216
Computer Security0.4700.440
Conceptual Physics0.3150.404
Econometrics0.2630.272
Electrical Engineering0.4140.359
Elementary Mathematics0.3200.272
Formal Logic0.2700.214
Global Facts0.3200.320
High School Biology0.3320.335
High School Chemistry0.2560.296
High School Computer Science0.3500.300
High School European History0.2240.242
High School Geography0.3230.364
High School Government & Politics0.3520.285
High School Macroeconomics0.2900.285
High School Mathematics0.2370.278
High School Microeconomics0.2310.273
High School Physics0.2520.225
High School Psychology0.3160.330
High School Statistics0.1990.176
High School US History0.2840.250
High School World History0.3120.274
Human Aging0.3690.430
Human Sexuality0.4810.321
International Law0.6030.405
Jurisprudence0.4910.370
Logical Fallacies0.3680.276
Machine Learning0.2140.312
Management0.3500.379
Marketing0.5210.547
Medical Genetics0.3200.330
Miscellaneous0.4460.443
Moral Disputes0.4220.306
Moral Scenarios0.2480.241
Nutrition0.4120.346
Philosophy0.4080.328
Prehistory0.4290.349
Professional Accounting0.3440.273
Professional Law0.3060.244
Professional Medicine0.2280.206
Professional Psychology0.3370.315
Public Relations0.3910.373
Security Studies0.4690.335
Sociology0.4980.408
US Foreign Policy0.5900.490
Virology0.4220.416
World Religions0.4040.304
Average (All Communities)0.3480.317
DEPLOY IN 60 SECONDS

Run Arabic-Orpo-Llama-3-8B-Instruct on Runcrate

Deploy on H100, A100, or RTX GPUs. Pay only for what you use. No setup required.