Arabic LLM evaluation: how to test a model before you roll it out
A practical method to test whether a model or AI assistant works in Arabic: your own tasks, dialects and Arabizi, normalised scoring, a calibrated judge and a release gate.
Arabic LLM evaluation means testing a model or AI assistant on your own Arabic tasks, in the dialects your users actually write, with scoring rules fixed before anyone looks at the outputs. A public leaderboard helps you shortlist models. Only a test set built from your own work tells you whether an assistant is ready for your customers, staff or citizens.
This guide sets out the method we use in our R&D Lab. Build the test set, normalise the text and choose metrics. Then calibrate an LLM judge against human reviewers, run a blind A/B comparison and decide against a release gate.
Why are public Arabic benchmarks not enough?
Public benchmarks are useful for a first shortlist. ArabicMMLU, published in Findings of ACL 2024, contains 14,575 multiple-choice questions in Modern Standard Arabic drawn from school exams across North Africa, the Levant and the Gulf. Its authors reported that the best Arabic-centric model they tested at the time scored 62.3%.
The Open Arabic LLM Leaderboard was launched on Hugging Face in May 2024 with compute from the Technology Innovation Institute. At launch, it ranked open models on tasks that were mostly multiple-choice or yes/no, some of them translated from English benchmarks. Dialect benchmarks are newer. The AraDiCE paper (2024) built dialect datasets through machine translation with human post-editing, and found that dialect generation and translation remained difficult for the models it tested.
None of these measure what a regional bank's customer asks on WhatsApp in Gulf Arabic, or whether a ministry's assistant declines a question it should not answer. Multiple-choice accuracy also says little about free-text answers, citations or tone. Use leaderboard scores to choose candidates, and base the go-live decision on your own test set.
Step 1: Build a test set from your own tasks
Start from real work: support tickets, chat transcripts, emails and the questions staff put to the intranet. Write test items from them after removing personal data, or ask native speakers to paraphrase them. Do not place customer records in a test set unless the data owner has approved it and the run happens in an approved environment.
Set the dialect mix by your users, not by the map. A contact centre in Riyadh may receive Gulf and Egyptian Arabic, with English mixed into the same message. A Jordanian retailer's social channels may be mostly Levantine, with Arabizi: Arabic written in Latin letters and digits, such as 3 for ع and 7 for ح.
| Variety | Typical source | What to include |
|---|---|---|
| Modern Standard Arabic | Policies, formal letters, government forms | Long questions, answers drawn from documents, formal register |
| Gulf | Customer chat in Saudi Arabia or the UAE | Short informal questions, local terms |
| Levantine | Jordan, Lebanon, Syria, Palestine | Social-media style, switching between Arabic and English |
| Egyptian | Customer messages, voice transcripts | Colloquial phrasing, common spelling variants |
| Arabizi and mixed text | WhatsApp, social media | Latin-script Arabic, Arabic-English mixing |
Have a second native speaker review each item and its reference answer. Add adversarial and refusal items: prompt-injection attempts written in Arabic, questions outside the assistant's scope, and requests for personal data. Seal about a fifth of the set as a holdout that you use only for the final decision. Plan at least 30 items per dialect and task type before you report a score for that cell, and expect wide confidence intervals at that size: only large gaps between systems will show.
Step 2: Normalise Arabic text before you score
Two correct answers can differ at the character level and still mean the same thing. Without normalisation, exact-match scoring penalises a model for spelling variants that do not change the answer. The usual steps are:
- Alef and hamza: map أ, إ and آ to bare ا, so that أحمد and احمد match.
- Alef maksura and teh marbuta: map ى to ي and ة to ه.
- Diacritics: remove harakat (fatha, damma, kasra, tanween, shadda, sukun), because a model may add them in one answer and omit them in the next.
- Tatweel: strip the stretching character (U+0640) used to lengthen words visually.
- Unicode and digits: apply Unicode normalisation and map Eastern Arabic digits to Western digits, or the reverse, consistently.
Open-source libraries such as CAMeL Tools provide functions for alef, alef maksura, teh marbuta and Unicode normalisation, with separate functions for removing diacritics. Write down every rule you apply, apply it to references and predictions alike, and keep the original text for human review. Normalise for scoring only. The answer your users see should keep correct hamza and spelling, and you should check that separately.
Step 3: Choose metrics that match the job
One score rarely tells you whether an Arabic assistant works. Pick metrics by task type and report them separately.
| Task | Metric | Notes |
|---|---|---|
| Short factual answers (dates, amounts, names) | Exact match and token F1 after normalisation | F1 gives partial credit when an answer adds or misses words |
| Answers grounded in your documents | Faithfulness and citation accuracy | Is every claim supported by the passage it cites? |
| Scope and safety | Refusal behaviour: answers when it should, declines when it should | Report false refusals and missed refusals separately |
| Customer-facing replies | Dialect and register fit, Arabic punctuation, right-to-left rendering | Formal channels usually need Modern Standard Arabic |
| Operations | Cost per correct answer, 95th-percentile latency | Measured for each system with the same settings |
Report each metric per dialect, with the number of items. A model that does well in Modern Standard Arabic and poorly on Gulf questions calls for a different decision from one that is average on both, and an overall average hides that difference.
Step 4: Use an LLM judge, then calibrate it against people
Free-text answers need judgement, and native-speaker review is slow and costly at scale. An LLM judge can score faithfulness, completeness and tone against a written rubric. The research behind the practice also records its weaknesses. Zheng et al. (2023) found that strong judges agreed with human preferences more than 80% of the time on English chat tasks, and documented position, verbosity and self-enhancement biases.
That result does not transfer automatically to Arabic or to your rubric. Before you use judge scores in a decision:
- Write a 1-5 rubric with Arabic examples for each score.
- Run the judge on answer pairs where you already know which is better. It should pick the better answer almost every time.
- Ask native-speaker reviewers to score a sample of about 50 items independently, then compare. If agreement is low, fix the rubric and repeat.
- Hide model names, randomise answer order, and use a judge from a different model family from the system under test.
Step 5: Run a blind A/B comparison and apply a release gate
Compare the candidate with what you run today, or with a second candidate, under identical conditions: the same items, system prompt, retrieval corpus and settings, pinned model versions, and several runs per item. Report the difference with a confidence interval. If the interval includes zero, the two systems cannot be told apart on this set, whatever the averages say.
Agree the gate with the business owner before anyone sees results. A short template:
Release gate for [assistant name], version [x]
- Normalised F1 on factual items at or above the agreed threshold in every dialect with 30 or more items.
- No drop against the current system larger than the agreed margin in any dialect.
- Zero missed refusals on safety items; false refusals below the agreed rate.
- Faithfulness at or above the threshold, scored by a judge calibrated against human reviewers.
- Cost per correct answer and latency within budget.
- Decision recorded as proceed, pilot, pause or stop, with the date of the next evaluation.
Re-run the same locked set whenever the vendor releases a new model version, you change the prompt or retrieval set-up, or the vendor announces the retirement of the model you use.
How to start
Pick one assistant that is close to launch and collect 100 to 200 real questions, with personal data removed, across the dialects your users write. That is enough to surface the obvious failures and to decide whether a fuller evaluation is worth the effort.
Our R&D Lab will publish Arabic and bilingual evaluations under Published Research & Arabic AI Benchmarks and tests specific use cases in the Prototype & PoC Studio. If the assistant serves customers, see our page on support and voice assistants. If your team wants to write and review test items itself, Role-Based AI Training (Arabic & English) covers it on your own tasks. To talk through where to begin, book the free AI Exposure Scan.
Frequently asked questions
How do you evaluate an LLM in Arabic?
Build a test set from your own tasks in the dialects your users write, including Modern Standard Arabic, Gulf, Levantine, Egyptian and Arabizi. Normalise alef, hamza, diacritics and tatweel before scoring. Use exact match and F1 for short answers, faithfulness for document-based answers, and track refusals separately. Calibrate any LLM judge against native-speaker reviewers, then compare systems blind against a release gate agreed in advance.
Can I rely on the Open Arabic LLM Leaderboard to choose a model?
Use it to shortlist candidates, not to decide on go-live. At launch in May 2024, the leaderboard ranked open models on tasks that were mostly multiple-choice or yes/no, some translated from English benchmarks. It does not test your documents, your users' dialects, your refusal rules or your costs. A small test set built from your own work gives you evidence the leaderboard cannot.
How many questions do I need in an Arabic test set?
Start with 100 to 200 real questions to find obvious failures. Plan at least 30 items for each dialect or task type you report on; at that size only large differences between systems will show, and keep about a fifth of the set sealed for the final decision. Report results with the number of items and a confidence interval, so small differences are not mistaken for real ones.
Is LLM-as-a-judge reliable for Arabic answers?
It can be, once you have checked it. Published research found strong judges agreed with human preferences more than 80% of the time on English chat tasks, but also documented position, verbosity and self-enhancement biases. For Arabic, write a rubric with Arabic examples, test the judge on pairs with known answers, compare it with native-speaker scores on about 50 items, and hide model names.
Why normalise Arabic text before scoring?
The same Arabic word is often written in several ways. Hamza on alef is often dropped, final ى and ي and final ة and ه are often swapped, and diacritics and tatweel are optional. These variants change the characters but usually not the answer. Without normalisation, exact-match and F1 scores penalise correct answers. Apply the same written rules to references and predictions, and check the spelling users see as a separate quality measure.
Get new insights by email
One email when we publish something useful. No spam; leave any time.
Services in this article
Related lab: R&D Lab ↗
