2024 LLM Novel Translation Survey Results
Results are IN !
This is just for translating a page from a Chinese Web novel into English.
In 2023 I did a survey of novel translation quality with various LLMs, you can see the results here: https://www.chaosdrifter.cc/2023-llm-model-translation-survery-results/
I conducted a new one for 2024, see the detailed description of the survey here:
https://www.chaosdrifter.cc/2024-llm-translation-survey/
And now for the results of the 2024 survey. In total 50 netizens participated (Thanks a lot !)
| LLM Model | Approval Rating |
|---|---|
| chatgpt 4o-20240903 | 75.92% |
| Gemini-1.5-pro-exp-0827 | 68.57% |
| Human Translation | 67.35% |
| Claud opus 20240229 | 65.71% |
| LLama-3-70b-instruct | 64.90% |
| Yi-large | 64.49% |
| app.readomni.com | 63.27% |
| Qwen2.5-72b-instruct | 55.92% |
| Google Translate | 37.96% |
Just to Compare, here is the Updated 2023 result:
| LLMs | Approval Ratings |
|---|---|
| Bard | 78.18% |
| llama-2-70b-chat | 76.36% |
| Human Translation | 76.36% |
| Falcon-180B | 65.45% |
| Vicuna-33B | 58.18% |
| Qwen-14B | 54.55% |
| Google Translate | 52.73% |
| GPT4 | 49.09% |
| Claude2 | 32.73% |
here is the link to the whole result google sheet for 2024:
https://docs.google.com/spreadsheets/d/1SVbt0FEjOCH7FtCs09zDT41CL7a5v-94ZptiZ1HZDaw/edit?usp=sharing
Some interesting findings:
LLM seems to have improved a lot in 2024, beating out even Human translations. They all beat google translate in 2024, maybe due to better prompting technique used ? There is hope that with a multi-agent approach + glossaries we'll truly be able to enjoy the Chinese Web novels.
Hopefully someone can take this as an inspiration and do a survey for Korean and Japanese light novels, and maybe translating to other languages like Tamil or Spanish. It is easy to do and completely free on Jotform.
Stay tuned for 2025 Survey with newer models, and I'll also be updating the survey through out the year as models are updated. Please watch for my posts and participate !