A practical result for Korean AI corrections

Which AI should you use to check your Korean?

A test of AI models on 1,418 sentences written by Korean learners: API, Assistants or Local models.

Practical ResultsChoose based on what matters most to you.
  1. Free AI Assistant Option

    DeepSeekFree, web and app

    DeepSeek v4.1 flash placed second in the benchmark and is likely the model powering DeepSeek chat.

  2. Best paid option

    Grok 4.6Paid plan, xAI

    Grok 4.6 clearly outperforms every other option - it is paid only however.

  3. Best offline option

    Muse GlimmerRuns offline on your own hardware

    Meta recently released Muse Glimmer which can comfortably compete with the average paid option. At Q4,K,M it sits at 19GB in size.

Version
2026-10-13
Tested
KoLLA v2 Dataset
Code
GitHub Repository
Updated
October 13, 2026

Overview

The best choice depends on your goal. This benchmark can guide you from privacy and independence with a self hosted Muse Glimmer, to quality-price-complexity tradeoff with the cheap Gemini 3.5 Flash Lite.

Strongest result in this test

Grok 4.6

66.9%F0.5

It performed the best on the whole Dataset, balancing the share of mistakes it caught with the share of useful changes it made. Overall it cost around $8.85 to run for 1000 Sentences.

Cheapest hosted option

Range of Cost to run

$0.06 - $18.34for one full run

The cost of running the benchmark once on a model via the API has a huge range. Google manages to take the bottom and the lead with its very cheap Gemini 3.5 Flash Lite and its very expensive Gemini 3.1 Pro. Cost can scale double with performance: Tokens get more expensive, but the model also produces more reasoning tokens.

The Price-Quality Frontier

10 hosted options

Gemini 3.5 Flash Lite, GPT-6 Luna, Claude Haiku 5.5, Gemini 4 Argon, MiMo V2.6 Pro, GPT-5.6 Terra, Claude Sonnet 5, DeepSeek V4.1 Flash, Grok 4.7, Grok 4.6

Each one gives you more measured quality than every cheaper option tested.

Price vs. correction quality

Each dot is a hosted model. Moving right costs more; moving up means more reliable corrections. Offline models have no API bill and so no place on this axis; the best of them is drawn as a line across the chart.

Correction quality against estimated costA comparison of correction quality and estimated cost per one thousand sentences. The highlighted line connects the best tested quality and price trade-offs. A dashed horizontal line marks the best model you can run on your own computer, which has no price and so no place on a cost axis.Free on your own computer: Muse GlimmerGemini 3.5 Flash LiteClaude Sonnet 5GPT-5.6 TerraDeepSeek V4.1 FlashGrok 4.6GPT-6 LunaGrok 4.7MiMo V2.6 ProClaude Haiku 5.5Gemini 4 ArgonOverall qualityF0.5 scoreEstimated cost for 1,000 sentences

The dashed line is the best local model tested. 11 hosted models you pay for score below it.

The highlighted line connects models that offer more quality than every cheaper model tested. It is a guide to trade-offs, not a universal ranking.

See every model

Compare every completed and unfinished test. Select a column heading to sort the results.

Hosted services

Showing 28 runs

Service
Grok 4.6xAIOpenRouterDefault settings66.9%F0.565.2%74.5%52.8%748 sentences$8.851,4171 failed1,4232,352,541 tokens total
Grok 4.7xAIOpenRouterDefault settings65.1%F0.563%75%50.6%717 sentences$3.811,4162 failed6572,734,245 tokens total
DeepSeek V4.1 FlashDeepSeekOpenRouterDefault settings64.4%F0.562.7%72.2%51.6%728 sentences$2.821,4126 failed2,0282,948,491 tokens total
Muse Spark 1.3MetaOpenRouterDefault settings64.3%F0.562.2%74.5%49.8%706 sentences$3.691,4180 failed8531,281,740 tokens total
Kimi K3Moonshot AIOpenRouterDefault settings63.8%F0.562.1%71.7%52.2%739 sentences$9.241,4171 failed6221,095,091 tokens total
Claude Sonnet 5AnthropicOpenRouterDefault settings63%F0.561.1%72.5%49.4%701 sentences$1.111,4180 failed99224,073 tokens total
Claude Opus 5AnthropicOpenRouterDefault settings60.7%F0.558.1%74.5%43.1%611 sentences$5.741,4180 failed218392,003 tokens total
Qwen3.8 Max 0902Alibaba CloudOpenRouterDefault settings60.4%F0.558.2%71.1%48%668 sentences$11.731,39127 failed9851,478,068 tokens total
GPT-5.6 TerraOpenAIOpenRouterDefault settings60.3%F0.558%71.9%44.9%637 sentences$1.071,4180 failed50123,206 tokens total
MiMo V2.6 ProXiaomiOpenRouterDefault settings60.1%F0.558.3%68.8%46.3%652 sentences$0.591,4099 failed473723,521 tokens total
GLM 5.3 FlashZ.aiOpenRouterDefault settings60%F0.558.2%68.6%46%649 sentences$0.861,4126 failed9951,468,601 tokens total
Gemini 3.1 Pro PreviewGoogleOpenRouterDefault settings59.5%F0.557.2%70.8%47.7%676 sentences$12.941,4171 failed1,0701,553,186 tokens total
Gemini 4 ArgonGoogleOpenRouterDefault settings59.5%F0.556.7%74.3%42%595 sentences$0.481,4180 failed36137,016 tokens total
GLM 5.3Z.aiOpenRouterDefault settings59.4%F0.557.2%70.5%43.4%613 sentences$6.111,4126 failed1,2891,883,143 tokens total
Claude Haiku 5.5AnthropicOpenRouterDefault settings59.4%F0.557.1%71%43.4%616 sentences$0.121,4180 failed232414,961 tokens total
Gemini 3.7 FlashGoogleOpenRouterDefault settings58.8%F0.556.1%73%41%582 sentences$2.161,4180 failed570844,638 tokens total
GPT-6 LunaOpenAIOpenRouterDefault settings58.6%F0.556.5%69%46.3%657 sentences$0.101,4180 failed190321,576 tokens total
MiMo V2.5XiaomiOpenRouterDefault settings58.1%F0.556.7%64.6%45.2%640 sentences$0.111,4171 failed297477,529 tokens total
GPT-6.1 SolOpenAIOpenRouterDefault settings58.1%F0.555.6%71%44.3%628 sentences$1.031,4180 failed95187,882 tokens total
Gemini 3.5 Flash LiteGoogleOpenRouterDefault settings57.6%F0.555.3%68.8%41%582 sentences$0.041,4180 failed1455,906 tokens total
Claude Opus 5.5AnthropicOpenRouterDefault settings57.4%F0.554.5%72.6%39.8%565 sentences$3.671,4180 failed171329,204 tokens total
GPT-6 SolOpenAIOpenRouterDefault settings57%F0.554.6%68.7%45.3%642 sentences$1.451,4180 failed137247,246 tokens total
MiniMax M3MiniMaxOpenRouterDefault settings56.1%F0.554.6%63.5%42.3%596 sentences$0.751,40810 failed5231,008,569 tokens total
GPT-5.6 SolOpenAIOpenRouterDefault settings56.1%F0.553.5%69.9%40.8%579 sentences$1.091,4180 failed101196,316 tokens total
Qwen3.8 FlashAlibaba CloudOpenRouterDefault settings56%F0.554.3%64%44.8%621 sentences$0.971,38632 failed2,0922,990,041 tokens total
MiMo V2.6 FlashXiaomiOpenRouterDefault settings56%F0.554.7%61.9%43.6%617 sentences$0.131,4153 failed339536,855 tokens total
GPT-5.6 LunaOpenAIOpenRouterDefault settings54.5%F0.551.8%69.2%37.9%538 sentences$0.161,4180 failed130237,568 tokens total
Mistral Large 4.0Mistral AIOpenRouterDefault settings52.3%F0.550.1%63.6%35.2%499 sentences$4.791,4180 failed2,1403,079,752 tokens total

Run on your own computer

Showing 18 runs

These have no hosted-service bill, so the price column is replaced by the weights that were actually loaded. A smaller file is quicker and fits more machines; it is also a compressed version of the model, which costs accuracy.

Muse GlimmerMetaBF1630B · 59.6 GB58.5%F0.556.6%67.6%43.6%615 sentences1,4117 failed6741,025,533 tokens total
Muse GlimmerMetaQ4_K_M28B · 18.2 GB57.9%F0.555.9%67.4%41.7%591 sentences1,4180 failed6931,057,607 tokens total
Mi:dm 2.0 Base InstructKTBF1611.5B · 23.1 GB56.4%F0.557.8%51.5%44.2%627 sentences1,4180 failed12766,117 tokens total
Kanana 2 30B A3B InstructKakaoBF1630B-A3B · 61.3 GB53.6%F0.552.2%60.1%40.1%568 sentences1,4180 failed1267,148 tokens total
Gemma 4 26B A4BGoogleQ4_K_M26B-A4B · 18.0 GB52.9%F0.550.4%66.1%37.4%527 sentences1,4108 failed1,0111,482,853 tokens total
Gemma 4 26B A4BGoogleBF1626B-A4B · 51.6 GB52%F0.549.4%65.8%32.2%457 sentences1,4180 failed1584,527 tokens total
Gemma 4 12BGoogleBF1612B · 23.9 GB49.1%F0.546.4%64.1%26.1%370 sentences1,4180 failed1685,725 tokens total
Command A+ 05-2026CohereFP8219B · 225.0 GB46.5%F0.544.4%57.3%30.7%435 sentences1,4153 failed359710,701 tokens total
Qwen3.6 35B A3BAlibaba CloudFP835B-A3B · 37.5 GB45.4%F0.543.4%55.7%31.2%440 sentences1,4126 failed1,6822,430,795 tokens total
Nemotron 3.5 Lightning 30B A3BNVIDIANVFP430B-A3B · 21.6 GB43%F0.541.8%48.7%30.4%428 sentences1,4108 failed1,6762,421,700 tokens total
Gemma 4 E4BGoogleQ4_K_M7.5B · 6.3 GB40.7%F0.538.2%54.8%16.1%229 sentences1,4180 failed81172,332 tokens total
GPT-OSS 20BOpenAIMXFP420B · 12.1 GB39.8%F0.537.8%50.7%20.5%291 sentences1,4180 failed92273,327 tokens total
Kanana 2 30B A3B ThinkingKakaoQ4_K_M30B-A3B · 18.6 GB36.6%F0.534.8%46.6%19.5%276 sentences1,4180 failed9241,364,980 tokens total
Ministral 3 3BMistral AIQ8_03B · 4.5 GB28%F0.526.3%37.9%3.7%52 sentences1,4180 failed1665,908 tokens total
Ministral 3 14B ReasoningMistral AIQ4_K_M14B · 9.1 GB26.5%F0.525.4%31.5%9.7%138 sentences1,4171 failed1564,947 tokens total
Nemotron 3 Nano 4BNVIDIAQ8_04.0B · 4.2 GB23.6%F0.522.6%28.4%8.9%126 sentences1,4162 failed9021,336,980 tokens total
DeepSeek R1 0528 Qwen3 8BDeepSeekQ4_K_M8B · 5.0 GB23.2%F0.521.9%30.3%4.3%60 sentences1,40414 failed382581,964 tokens total
Kanana 2 3B InstructKakaoQ6_K3B · 2.9 GB12.5%F0.512.1%14.1%0.8%12 sentences1,4180 failed1374,979 tokens total

Unfinished tests stay visible for transparency but are left out of the recommendations. Models for your own computer were run through LM Studio or on rented Vast.ai GPUs; their hardware, electricity, and time were not measured.

Output lengthCompletion tokens / sentence is the average generated text per answered sentence. Models used their normal settings, so some spent much more time thinking about the same one-sentence task. That affects both price and speed.

What models actually do

Two models can have similar scores while giving very different advice. Pick a maker and model to see whether it usually gets the sentence right, changes too much, or misses the mistake.

Change nothing at all: 40.5% fully correct (574 of 1,418 sentences)

Fully correctNo FP, no FN
74752.7%

No extra change and no missed mistake

Changed too muchFP only
31021.9%

Added a change that was not needed

Missed a mistakeFN only
735.1%

Left a needed correction undone

Mixed resultFP + FN
28720.2%

Changed too much and still missed something

Could not answerFailed
10.1%

No answer was returned after retries

How the test works

The details for readers who want to check the work: the sentences used, the question each model received, how answers were judged, and what this test cannot tell us.

The sentence corpus

I used KoLLA_multi-refs.m2 from the KoLLA v2 dataset:100 learner essays, split into1,418 sentences. They contain2,828 human corrections and 3,649 individual edits. All but eight sentences have two human references.

Fingerprint
f08943bedab1b149
MD5
9a6f2e3fea1b39bbb7343445db1167f7
License
GNU GPL v3 or later

The question each model received

Every model saw the same one-line instruction:

Correct the Korean sentence. Reply with the corrected sentence only.

Hosted models ran through OpenRouter. Offline models ran through LM Studio. Temporary failures were retried up to three times. Models kept their normal response settings; I did not force the same amount of internal reasoning on every model.

Runner
kolla-benchmark 0.1.0
Run dates
September 10–12, 2026

Scoring whether an answer was right

The scorer compares each answer with both human corrections for that sentence. If either comparison found a valid correction, it keeps the better one. That is the usual approach for sentences with more than one valid answer, though it gives models the benefit of the doubt.

The scorer used here is M2 MaxMatch (Dahlmeier & Ng, 2012). It is checked on every run against the reference Pythonm2scorer.

Useful changes, mistakes caughtPrecision, recall
How often a change was needed, how many mistakes were found, and one combined score that gives extra weight to avoiding bad advice.
Exact sentenceExact match
The output exactly matched one complete human correction.
Fully correctNo FP, no FN
No extra change was made and no needed change was missed.
No changesUnchanged
The answer repeated the learner’s sentence verbatim.
No answerFailed
The model did not return an answer after retries.

On cost and coverage

Hosted costs are the recorded service bill. They reflect exact prices paid on execution, not the monthly price of a consumer subscription.

Offline models have no hosted-service bill, but I did not measure hardware, electricity, or time, so they do not get any cost attribution.

A test is complete when every one of the 1,418 sentences has either an answer or a recorded failure.

This list does not contain every model available, especially not for local runs. The selection for local models was arbitrary, based on what was available to me at the time of testing. The selectino of hosted models was based on the current available services and their popularity.

What the numbers mean

Before comparing models, it helps to know what counts as a good correction. Here is the short version, in plain language.

Sometimes two corrections are both right

Two people corrected every sentence independently. When their answers differed, the corpus accepts either valid reading—because Korean often has more than one natural way to repair a sentence.

What the learner wrote

비행기 음식이 안 막였습니다.

One valid reading: “didn’t eat”

비행기 음식을 안 먹었습니다.

Another valid reading: “didn’t suit me”

비행기 음식이 안 맞았습니다.

One learner sentence, with the two valid repairs used for scoring.

Doing nothing is not a zero

574 of the 1,418 sentences were already correct, so repeating them unchanged is the right answer. That means a model that changes nothing gets 40.5% of the sentences fully right—but it fixes none of the mistakes.

Good feedback catches mistakes without inventing them

Useful changesPrecision
When the model changes something, how often that change was actually needed. If this number is low, the model is more likely to rewrite Korean that was already fine.
Mistakes caughtRecall
Of the mistakes in the sentence, how many the model also found. If this number is low, it is more likely to let errors through.
Overall qualityF0.5 score
Our main score, which rewards both and gives a little extra weight to avoiding bad advice. The technical name is F0.5.

Each model was run once, so nearby results are effectively tied. Use the table to compare broad differences, not to declare a winner by a point or two.

Check the sources and raw data

Feel free to reproduce this benchmark on your own. The code and dataset are open source, and the raw runs results are available for download for reevaluation.

The raw reports contain each sentence, each answer, and the detailed scores.

Read the Personal Version or Try the Result

A benchmark can point you in the right direction. Try Elephant’s sentence analysis on Korean or read my personal telling of how and why I ran this benchmark.

Read the Blog PostTry sentence analysis