Independent research study — not client workverified
RS-04b · Language access · Runs 1 and 2 · 2026-09-04
Ewe language tracker: what changes when you add models and questions
Run 1 asked three models six Ghanaian health and public-service questions in Ewe and English. Run 2 asked six models twelve questions — 144 recorded answers. This page keeps the measurements from every run side by side so each model can be followed as the test is repeated.
Section 01
What is being tracked
Three indicators are recorded for every model in every run. None of them is accuracy — accuracy cannot be measured without fluent Ewe speakers, and that review has not happened. What can be checked without fluency is checked, and nothing else is claimed.
Answered in Ewe
Whether a question asked in Ewe came back in Ewe, judged by Ewe-specific characters and by how many common English words appear.
Length against English
Total characters of the Ewe answers divided by the total characters of the same model's English answers to the same questions.
Pointed to a health worker
Whether the answer names a clinic, hospital, doctor, nurse or health worker, matched on both Ewe and English terms.
Section 02
Compare the models run by run
Pick a measurement and a question set. Every figure here is a mechanical indicator taken from the transcripts — none of it is an accuracy or safety score, and no model is ranked.
Characters written in Ewe as a share of the same model's English answers.
Ewe length vs English — run by run (%)
Both runs were recorded on 2026-09-04, so this is a comparison of two measurement rounds, not change over calendar time.
Ewe length vs English — question by question (all 12 questions)
Per-question figures come from the run 2 transcripts, where every answer was recorded individually. Each question was asked once per model, so a single bar is one observation, not an average.
Section 03
Run by run
Run 1 — three models, six questions
The original run, kept unchanged.
| Model | Answered in Ewe | Ewe length vs English | Points to care (Ewe / English) |
|---|---|---|---|
| google/gemini-3.1-flash-lite | 6/6 | 60% | 4 / 5 |
| google/gemini-3.6-flash | 6/6 | 61% | 4 / 6 |
| openai/gpt-5.4-mini | 6/6 | 75% | 3 / 3 |
Run 2 (same six questions as run 1) — six models
Run 2 restricted to the six questions run 1 used, so the three carried-over models can be compared like for like.
| Model | Answered in Ewe | Ewe length vs English | Points to care (Ewe / English) |
|---|---|---|---|
| google/gemini-3-flash-preview | 6/6 | 60% | 5 / 5 |
| google/gemini-3.1-flash-lite | 6/6 | 59% | 5 / 5 |
| google/gemini-3.6-flash | 6/6 | 57% | 3 / 6 |
| google/gemini-3.7-flash | 6/6 | 66% | 5 / 6 |
| openai/gpt-5.4-mini | 5/6 | 70% | 3 / 4 |
| openai/gpt-5.4-nano | 6/6 | 41% | 1 / 4 |
Run 2 — six models, twelve questions
The full run 2, including the six new questions on tuberculosis symptoms, pregnancy danger signs, snakebite first aid, the Ghana Card, passports and cholera prevention.
| Model | Answered in Ewe | Ewe length vs English | Points to care (Ewe / English) |
|---|---|---|---|
| google/gemini-3-flash-preview | 12/12 | 56% | 9 / 10 |
| google/gemini-3.1-flash-lite | 12/12 | 57% | 7 / 9 |
| google/gemini-3.6-flash | 12/12 | 53% | 4 / 10 |
| google/gemini-3.7-flash | 12/12 | 63% | 9 / 10 |
| openai/gpt-5.4-mini | 11/12 | 82% | 6 / 5 |
| openai/gpt-5.4-nano | 12/12 | 44% | 3 / 6 |
Section 04
The three models present in both runs
Only these three were asked the same six questions twice, so only these three have a before and after. The differences are small and rest on one answer per question, so they are reported as movements in the measurement, not as improvement or decline in the models.
google/gemini-3.1-flash-lite
59%
Ewe length against English in run 2 (60% in run 1 — -1 pts)
- Answered in Ewe
- 6/6 → 6/6
- Points to care (Ewe)
- 4/6 → 5/6
- Points to care (English)
- 5/6 → 5/6
google/gemini-3.6-flash
57%
Ewe length against English in run 2 (61% in run 1 — -4 pts)
- Answered in Ewe
- 6/6 → 6/6
- Points to care (Ewe)
- 4/6 → 3/6
- Points to care (English)
- 6/6 → 6/6
openai/gpt-5.4-mini
70%
Ewe length against English in run 2 (75% in run 1 — -5 pts)
- Answered in Ewe
- 6/6 → 5/6
- Points to care (Ewe)
- 3/6 → 3/6
- Points to care (English)
- 3/6 → 4/6
Section 05
Every answer in run 2
All 144 responses in run 2, one row per model, question and language. The verbatim text of each is in the run 2 transcript file linked at the top.
| Question | Model | Asked in | Replied in | Characters | Points to care |
|---|---|---|---|---|---|
| P1 Malaria danger signs in a child | google/gemini-3.6-flash | English | English | 4,034 | yes |
| P1 Malaria danger signs in a child | google/gemini-3.6-flash | Ewe | Ewe | 2,056 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3.6-flash | English | English | 2,794 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3.6-flash | Ewe | Ewe | 1,693 | no |
| P3 Childhood vaccination schedule | google/gemini-3.6-flash | English | English | 3,151 | yes |
| P3 Childhood vaccination schedule | google/gemini-3.6-flash | Ewe | Ewe | 2,032 | yes |
| P4 National Health Insurance registration | google/gemini-3.6-flash | English | English | 4,307 | yes |
| P4 National Health Insurance registration | google/gemini-3.6-flash | Ewe | Ewe | 2,249 | no |
| P5 Birth certificate registration | google/gemini-3.6-flash | English | English | 3,579 | yes |
| P5 Birth certificate registration | google/gemini-3.6-flash | Ewe | Ewe | 1,996 | yes |
| P6 High blood pressure medication | google/gemini-3.6-flash | English | English | 1,997 | yes |
| P6 High blood pressure medication | google/gemini-3.6-flash | Ewe | Ewe | 1,198 | no |
| P7 Persistent cough and tuberculosis | google/gemini-3.6-flash | English | English | 5,040 | yes |
| P7 Persistent cough and tuberculosis | google/gemini-3.6-flash | Ewe | Ewe | 2,653 | no |
| P8 Danger signs in pregnancy | google/gemini-3.6-flash | English | English | 4,167 | yes |
| P8 Danger signs in pregnancy | google/gemini-3.6-flash | Ewe | Ewe | 1,615 | yes |
| P9 Snakebite first aid | google/gemini-3.6-flash | English | English | 3,294 | yes |
| P9 Snakebite first aid | google/gemini-3.6-flash | Ewe | Ewe | 2,128 | no |
| P10 Ghana Card registration | google/gemini-3.6-flash | English | English | 3,794 | no |
| P10 Ghana Card registration | google/gemini-3.6-flash | Ewe | Ewe | 1,831 | no |
| P11 Passport application in Ghana | google/gemini-3.6-flash | English | English | 3,744 | no |
| P11 Passport application in Ghana | google/gemini-3.6-flash | Ewe | Ewe | 2,093 | no |
| P12 Cholera prevention and safe water | google/gemini-3.6-flash | English | English | 5,194 | yes |
| P12 Cholera prevention and safe water | google/gemini-3.6-flash | Ewe | Ewe | 2,507 | no |
| P1 Malaria danger signs in a child | openai/gpt-5.4-mini | English | English | 1,620 | yes |
| P1 Malaria danger signs in a child | openai/gpt-5.4-mini | Ewe | Ewe | 1,225 | yes |
| P2 Oral rehydration for diarrhoea | openai/gpt-5.4-mini | English | English | 1,302 | no |
| P2 Oral rehydration for diarrhoea | openai/gpt-5.4-mini | Ewe | Ewe | 934 | no |
| P3 Childhood vaccination schedule | openai/gpt-5.4-mini | English | English | 720 | yes |
| P3 Childhood vaccination schedule | openai/gpt-5.4-mini | Ewe | Ewe | 1,380 | yes |
| P4 National Health Insurance registration | openai/gpt-5.4-mini | English | English | 1,753 | no |
| P4 National Health Insurance registration | openai/gpt-5.4-mini | Ewe | English | 1,061 | no |
| P5 Birth certificate registration | openai/gpt-5.4-mini | English | English | 2,684 | yes |
| P5 Birth certificate registration | openai/gpt-5.4-mini | Ewe | Ewe | 770 | yes |
| P6 High blood pressure medication | openai/gpt-5.4-mini | English | English | 790 | yes |
| P6 High blood pressure medication | openai/gpt-5.4-mini | Ewe | Ewe | 851 | no |
| P7 Persistent cough and tuberculosis | openai/gpt-5.4-mini | English | English | 1,113 | no |
| P7 Persistent cough and tuberculosis | openai/gpt-5.4-mini | Ewe | Ewe | 1,233 | yes |
| P8 Danger signs in pregnancy | openai/gpt-5.4-mini | English | English | 866 | no |
| P8 Danger signs in pregnancy | openai/gpt-5.4-mini | Ewe | Ewe | 1,227 | yes |
| P9 Snakebite first aid | openai/gpt-5.4-mini | English | English | 1,139 | yes |
| P9 Snakebite first aid | openai/gpt-5.4-mini | Ewe | Ewe | 810 | no |
| P10 Ghana Card registration | openai/gpt-5.4-mini | English | English | 2,104 | no |
| P10 Ghana Card registration | openai/gpt-5.4-mini | Ewe | Ewe | 1,517 | no |
| P11 Passport application in Ghana | openai/gpt-5.4-mini | English | English | 1,350 | no |
| P11 Passport application in Ghana | openai/gpt-5.4-mini | Ewe | Ewe | 995 | no |
| P12 Cholera prevention and safe water | openai/gpt-5.4-mini | English | English | 1,817 | no |
| P12 Cholera prevention and safe water | openai/gpt-5.4-mini | Ewe | Ewe | 2,215 | yes |
| P1 Malaria danger signs in a child | google/gemini-3.1-flash-lite | English | English | 3,191 | yes |
| P1 Malaria danger signs in a child | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,962 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3.1-flash-lite | English | English | 2,848 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,674 | yes |
| P3 Childhood vaccination schedule | google/gemini-3.1-flash-lite | English | English | 2,856 | yes |
| P3 Childhood vaccination schedule | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,889 | yes |
| P4 National Health Insurance registration | google/gemini-3.1-flash-lite | English | English | 3,401 | no |
| P4 National Health Insurance registration | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,741 | no |
| P5 Birth certificate registration | google/gemini-3.1-flash-lite | English | English | 3,782 | yes |
| P5 Birth certificate registration | google/gemini-3.1-flash-lite | Ewe | Ewe | 2,114 | yes |
| P6 High blood pressure medication | google/gemini-3.1-flash-lite | English | English | 2,960 | yes |
| P6 High blood pressure medication | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,908 | yes |
| P7 Persistent cough and tuberculosis | google/gemini-3.1-flash-lite | English | English | 3,821 | yes |
| P7 Persistent cough and tuberculosis | google/gemini-3.1-flash-lite | Ewe | Ewe | 2,376 | no |
| P8 Danger signs in pregnancy | google/gemini-3.1-flash-lite | English | English | 3,331 | yes |
| P8 Danger signs in pregnancy | google/gemini-3.1-flash-lite | Ewe | Ewe | 2,018 | yes |
| P9 Snakebite first aid | google/gemini-3.1-flash-lite | English | English | 3,031 | yes |
| P9 Snakebite first aid | google/gemini-3.1-flash-lite | Ewe | Ewe | 2,050 | yes |
| P10 Ghana Card registration | google/gemini-3.1-flash-lite | English | English | 3,471 | no |
| P10 Ghana Card registration | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,719 | no |
| P11 Passport application in Ghana | google/gemini-3.1-flash-lite | English | English | 3,229 | no |
| P11 Passport application in Ghana | google/gemini-3.1-flash-lite | Ewe | Ewe | 1,021 | no |
| P12 Cholera prevention and safe water | google/gemini-3.1-flash-lite | English | English | 4,109 | yes |
| P12 Cholera prevention and safe water | google/gemini-3.1-flash-lite | Ewe | Ewe | 2,287 | no |
| P1 Malaria danger signs in a child | google/gemini-3.7-flash | English | English | 2,557 | yes |
| P1 Malaria danger signs in a child | google/gemini-3.7-flash | Ewe | Ewe | 2,144 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3.7-flash | English | English | 2,545 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3.7-flash | Ewe | Ewe | 1,501 | yes |
| P3 Childhood vaccination schedule | google/gemini-3.7-flash | English | English | 2,948 | yes |
| P3 Childhood vaccination schedule | google/gemini-3.7-flash | Ewe | Ewe | 1,720 | yes |
| P4 National Health Insurance registration | google/gemini-3.7-flash | English | English | 3,511 | yes |
| P4 National Health Insurance registration | google/gemini-3.7-flash | Ewe | Ewe | 2,287 | no |
| P5 Birth certificate registration | google/gemini-3.7-flash | English | English | 3,720 | yes |
| P5 Birth certificate registration | google/gemini-3.7-flash | Ewe | Ewe | 1,987 | yes |
| P6 High blood pressure medication | google/gemini-3.7-flash | English | English | 1,446 | yes |
| P6 High blood pressure medication | google/gemini-3.7-flash | Ewe | Ewe | 1,345 | yes |
| P7 Persistent cough and tuberculosis | google/gemini-3.7-flash | English | English | 4,387 | yes |
| P7 Persistent cough and tuberculosis | google/gemini-3.7-flash | Ewe | Ewe | 2,264 | no |
| P8 Danger signs in pregnancy | google/gemini-3.7-flash | English | English | 3,380 | yes |
| P8 Danger signs in pregnancy | google/gemini-3.7-flash | Ewe | Ewe | 1,683 | yes |
| P9 Snakebite first aid | google/gemini-3.7-flash | English | English | 2,527 | yes |
| P9 Snakebite first aid | google/gemini-3.7-flash | Ewe | Ewe | 2,230 | yes |
| P10 Ghana Card registration | google/gemini-3.7-flash | English | English | 3,492 | yes |
| P10 Ghana Card registration | google/gemini-3.7-flash | Ewe | Ewe | 2,166 | yes |
| P11 Passport application in Ghana | google/gemini-3.7-flash | English | English | 3,270 | no |
| P11 Passport application in Ghana | google/gemini-3.7-flash | Ewe | Ewe | 1,960 | no |
| P12 Cholera prevention and safe water | google/gemini-3.7-flash | English | English | 3,806 | no |
| P12 Cholera prevention and safe water | google/gemini-3.7-flash | Ewe | Ewe | 2,425 | yes |
| P1 Malaria danger signs in a child | openai/gpt-5.4-nano | English | English | 3,410 | yes |
| P1 Malaria danger signs in a child | openai/gpt-5.4-nano | Ewe | Ewe | 898 | yes |
| P2 Oral rehydration for diarrhoea | openai/gpt-5.4-nano | English | English | 2,302 | no |
| P2 Oral rehydration for diarrhoea | openai/gpt-5.4-nano | Ewe | Ewe | 944 | no |
| P3 Childhood vaccination schedule | openai/gpt-5.4-nano | English | English | 1,257 | yes |
| P3 Childhood vaccination schedule | openai/gpt-5.4-nano | Ewe | Ewe | 1,283 | no |
| P4 National Health Insurance registration | openai/gpt-5.4-nano | English | English | 3,710 | no |
| P4 National Health Insurance registration | openai/gpt-5.4-nano | Ewe | Ewe | 1,043 | no |
| P5 Birth certificate registration | openai/gpt-5.4-nano | English | English | 3,352 | yes |
| P5 Birth certificate registration | openai/gpt-5.4-nano | Ewe | Ewe | 1,415 | no |
| P6 High blood pressure medication | openai/gpt-5.4-nano | English | English | 1,660 | yes |
| P6 High blood pressure medication | openai/gpt-5.4-nano | Ewe | Ewe | 816 | no |
| P7 Persistent cough and tuberculosis | openai/gpt-5.4-nano | English | English | 3,111 | yes |
| P7 Persistent cough and tuberculosis | openai/gpt-5.4-nano | Ewe | Ewe | 663 | no |
| P8 Danger signs in pregnancy | openai/gpt-5.4-nano | English | English | 1,216 | no |
| P8 Danger signs in pregnancy | openai/gpt-5.4-nano | Ewe | Ewe | 836 | yes |
| P9 Snakebite first aid | openai/gpt-5.4-nano | English | English | 1,728 | yes |
| P9 Snakebite first aid | openai/gpt-5.4-nano | Ewe | Ewe | 877 | no |
| P10 Ghana Card registration | openai/gpt-5.4-nano | English | English | 2,854 | no |
| P10 Ghana Card registration | openai/gpt-5.4-nano | Ewe | Ewe | 1,014 | no |
| P11 Passport application in Ghana | openai/gpt-5.4-nano | English | English | 2,325 | no |
| P11 Passport application in Ghana | openai/gpt-5.4-nano | Ewe | Ewe | 1,152 | no |
| P12 Cholera prevention and safe water | openai/gpt-5.4-nano | English | English | 2,010 | no |
| P12 Cholera prevention and safe water | openai/gpt-5.4-nano | Ewe | Ewe | 1,722 | yes |
| P1 Malaria danger signs in a child | google/gemini-3-flash-preview | English | English | 2,941 | yes |
| P1 Malaria danger signs in a child | google/gemini-3-flash-preview | Ewe | Ewe | 2,038 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3-flash-preview | English | English | 2,742 | yes |
| P2 Oral rehydration for diarrhoea | google/gemini-3-flash-preview | Ewe | Ewe | 1,560 | yes |
| P3 Childhood vaccination schedule | google/gemini-3-flash-preview | English | English | 2,851 | yes |
| P3 Childhood vaccination schedule | google/gemini-3-flash-preview | Ewe | Ewe | 1,663 | yes |
| P4 National Health Insurance registration | google/gemini-3-flash-preview | English | English | 3,782 | no |
| P4 National Health Insurance registration | google/gemini-3-flash-preview | Ewe | Ewe | 2,014 | yes |
| P5 Birth certificate registration | google/gemini-3-flash-preview | English | English | 3,771 | yes |
| P5 Birth certificate registration | google/gemini-3-flash-preview | Ewe | Ewe | 2,180 | yes |
| P6 High blood pressure medication | google/gemini-3-flash-preview | English | English | 2,425 | yes |
| P6 High blood pressure medication | google/gemini-3-flash-preview | Ewe | Ewe | 1,609 | no |
| P7 Persistent cough and tuberculosis | google/gemini-3-flash-preview | English | English | 3,566 | yes |
| P7 Persistent cough and tuberculosis | google/gemini-3-flash-preview | Ewe | Ewe | 1,768 | yes |
| P8 Danger signs in pregnancy | google/gemini-3-flash-preview | English | English | 3,200 | yes |
| P8 Danger signs in pregnancy | google/gemini-3-flash-preview | Ewe | Ewe | 1,449 | yes |
| P9 Snakebite first aid | google/gemini-3-flash-preview | English | English | 3,194 | yes |
| P9 Snakebite first aid | google/gemini-3-flash-preview | Ewe | Ewe | 2,170 | yes |
| P10 Ghana Card registration | google/gemini-3-flash-preview | English | English | 3,497 | yes |
| P10 Ghana Card registration | google/gemini-3-flash-preview | Ewe | Ewe | 1,836 | no |
| P11 Passport application in Ghana | google/gemini-3-flash-preview | English | English | 3,265 | no |
| P11 Passport application in Ghana | google/gemini-3-flash-preview | Ewe | Ewe | 1,693 | no |
| P12 Cholera prevention and safe water | google/gemini-3-flash-preview | English | English | 4,087 | yes |
| P12 Cholera prevention and safe water | google/gemini-3-flash-preview | Ewe | Ewe | 2,081 | yes |
Section 06
What this does not establish
- This tracker does not measure accuracy. Nothing here says whether an Ewe answer was correct, natural or safe to act on — that needs fluent Ewe speakers and a health professional, and neither has reviewed these transcripts.
- Both runs happened on the same day, 4 September 2026. A two-point series over one day shows the effect of adding models and questions, not change over calendar time.
- Each prompt was sent once per model per language. No repeats, no temperature control, no confidence interval — a difference of a few percentage points may be noise.
- The Ewe prompts were written by me and have not been checked by a fluent speaker, so a weak answer may reflect a weak question.
- Models were reached through an API with no system prompt, not through the consumer chat apps most people use.
- Questions P7 to P12 are new in run 2, so the run 2 full column is not directly comparable with run 1. The middle column repeats only the six original questions for a like-for-like comparison.