The main benchmark uses one short instruction to ask each model to correct Korean sentences. I wanted to see how much that choice affects the results, and whether asking a model to correct its own answer helps. These two experiments use the same learner sentences.
The main benchmark asks each model to “correct the Korean sentence.” For this experiment, I compared that short instruction with three alternatives: one that says to fix only actual errors, a more detailed brief, and an instruction written in Korean.
Each prompt was tested on the same 1,418 learner sentences, once in the message alongside the sentence and once as a system prompt, the separate instruction field used by an API. That makes eight runs per model. Some runs did not finish; their coverage is shown below.
ShortLeaderboard prompt
Correct the Korean sentence. Reply with the corrected sentence only.
Short, real errors only
Correct the following korean sentence. Only correct actual errors. Reply with only the corrected sentence.
Detailed brief
You are correcting Korean sentences written by learners of Korean.
Correct grammatical, lexical, spacing, spelling, and word-order errors while preserving the learner's intended meaning as closely as possible.
Make only the changes necessary to produce a grammatically correct and natural Korean sentence. Do not rewrite the sentence merely to improve style.
If the sentence is already acceptable, return it unchanged.
Return only the corrected Korean sentence, with no explanation or additional text.
Written in Korean
아래 한국어 문장에서 틀린 부분만 수정하고, 교정 문장 한 줄만 출력하라.
단, 문장이 이미 문법적으로 올바르면 원문을 그대로 출력하라.
“Fix only the wrong parts of the Korean sentence below and output the corrected sentence on one line. If it is already grammatically correct, output the original as it is.”
Overall quality by promptEach dot shows one prompt's score. The line connects the lowest and highest scores for that model. Higher is better.
Short
Short, real errors only
Detailed brief
Written in Korean
GPT-5.6 LunaHosted service
+13.4 pts+13.7 ptsbest vs. Short
Gemini 3.5 Flash LiteHosted service
+7.7 pts+7.2 ptsbest vs. Short
Gemma 4 26B A4BOpen weights, rented GPU
+9.6 pts+9.7 ptsbest vs. Short
Muse GlimmerOpen weights, rented GPU88–99% of sentences scored
+7.7 pts+6.3 ptsbest vs. Short
Kanana 2 3B Instruct3B, on your computer
+6.2 pts+19.1 ptsbest vs. Short
Ministral 3 3B3B, on your computer
+1.3 pts+1.2 ptsbest vs. Short
20%30%40%50%60%70%
Show as a table
Model
ShortIn the message
ShortAs system prompt
Short, real errors onlyIn the message
Short, real errors onlyAs system prompt
Detailed briefIn the message
Detailed briefAs system prompt
Written in KoreanIn the message
Written in KoreanAs system prompt
GPT-5.6 Luna
55.8%
54.4%
66.3%
66.3%
64.0%
62.3%
69.2%
68.1%
Gemini 3.5 Flash Lite
55.8%
58.1%
63.5%
64.6%
62.7%
62.0%
63.5%
65.3%
Gemma 4 26B A4B
53.0%
51.8%
62.6%
61.5%
62.2%
61.5%
61.9%
61.1%
Muse Glimmer
56.9%
58.3%
62.0%
64.1%
62.2%
63.1%
64.6%
64.6%
Kanana 2 3B Instruct
27.2%
12.5%
32.7%
18.4%
33.4%
22.3%
30.9%
31.6%
Ministral 3 3B
27.1%
27.8%
28.5%
29.0%
27.7%
27.6%
22.0%
25.3%
Qwen3.8 27Bpartial: 114–521 of 1,418
62.7%
57.2%
61.5%
64.4%
63.7%
64.2%
64.2%
63.8%
Muse Glimmer (88–99% of sentences scored): the missing sentences timed out, came back without an answer, or were never sent before the run stopped. The score covers the rest.
Qwen3.8 27B is only in the table: charts need every run to score at least 85% of the sentences.
The larger models benefited from clearer instructions
The short prompt from the leaderboard was the worst of the four for every larger model. With the instruction in the message, their best scores were+7.7 to +13.4 points higher. Each model did best with either the short “real errors only” prompt or the Korean prompt. The detailed brief did not improve on those simpler alternatives.
The Korean prompt was within 0.7 points of the best on each larger model. That makes it a useful starting point, though this does not isolate the effect of language: its wording differs from the English prompts too.
The two 3B models remained well behind
The smaller models also responded to the wording, but there was no clear best prompt for both. Across all prompts and placements, their highest score was 33.4%, compared with 51.8% for the lowest-scoring run of a larger model. For these two models, changing the prompt was not enough to make them competitive.
What changes when the prompt changes?For the larger models, clearer instructions mainly helped them leave correct sentences alone. The share of mistakes they caught changed less. Select a model to compare both measures, with the instruction in the message.
Correct sentences left aloneamong scored sentences both human annotators left unchanged
Short60.6%
Short, real errors only75.9%
Detailed brief72.7%
Written in Korean77.5%
Mistakes caughtof the edits the human annotators made
Short69.0%
Short, real errors only72.5%
Detailed brief73.6%
Written in Korean73.4%
Three sentences the human annotators left unchanged, and how GPT-5.6 Luna responded
What the learner wrote대학교에서 일이 어렵습니까?
✕ Short prompt대학교에서 공부하는것이어렵습니까?
✓ Every other promptleft it unchanged
What the learner wrote이 사람은 여자 친구입니다.
✕ Short prompt이 사람은 제여자 친구입니다.
✓ Every other promptleft it unchanged
What the learner wrote음식을 만든 후에 드라마를 보고 음식을 먹습니다.
✕ Short prompt음식을 만든 후에 드라마를 보면서먹습니다.
✓ Every other promptleft it unchanged
Does it matter where the instruction goes?The change in overall quality when the instruction moves from the message into the system prompt. Values to the right mean a higher score as a system prompt; values to the left mean a lower one.
GPT-5.6 LunaHosted service
Short−1.4
Short, real errors only+0.0
Detailed brief−1.6
Written in Korean−1.1
Gemini 3.5 Flash LiteHosted service
Short+2.4
Short, real errors only+1.1
Detailed brief−0.6
Written in Korean+1.9
Gemma 4 26B A4BOpen weights, rented GPU
Short−1.2
Short, real errors only−1.1
Detailed brief−0.7
Written in Korean−0.8
Muse GlimmerOpen weights, rented GPU88–99% of sentences scored
Short+1.5
Short, real errors only+2.1
Detailed brief+0.9
Written in Korean+0.0
Kanana 2 3B Instruct3B, on your computer
Short−14.7
Short, real errors only−14.3
Detailed brief−11.1
Written in Korean+0.7
Ministral 3 3B3B, on your computer
Short+0.7
Short, real errors only+0.5
Detailed brief−0.2
Written in Korean+3.2
← better in the messagebetter as system prompt →
For the larger models, the difference was at most 2.4points, with no consistent advantage for either placement. Comparable runs varied by up to 1.8 points, so I would be cautious about reading much into these small differences. Kanana 2 3B Instruct was more sensitive to placement, losing up to14.7 points as a system prompt.
Experiment 2 · Repeated corrections
Does another correction improve the answer?
After a model corrects a sentence, it seems reasonable to ask whether it missed anything. I tested one version of that: sending its answer back as a new sentence to correct. Each request started a fresh chat, so the model only saw the previous answer, not the learner's original sentence.
I repeated this until the answer came back unchanged or the model had received10 requests in total. Every request used the “Short, real errors only” prompt. This tests repeated correction of the output; it does not tell us whether a follow-up question in an ongoing conversation would help.
1Send the learner's sentence
2Model returns a correction
3Send that correction to a fresh chat
4Stop when the answer comes back unchanged, or after 10 requests
Overall quality after each extra requestEach point uses the latest answer for every sentence. Once a model returned a sentence unchanged, that answer was kept for the remaining points.
GPT-5.6 Luna66.9% → 55.2%
Gemini 3.5 Flash Lite63.1% → 55.9%
Gemma 4 26B A4B62.8% → 51.4%
Kanana 2 3B Instruct28.4% → 22.1%
Ministral 3 3B28.6% → 22.7%
20%30%40%50%60%70%
12345678910
Requests per sentence
Every model shown here finished with a lower score than it started with. The largest drop for the larger models came on the second request:GPT-5.6 Luna lost9.6 points at that step. The two 3B models continued to lose quality over later requests.
Left out for scoring fewer than 85% of the sentences: Muse Glimmer (375 of 1,418 sentences), Qwen3.8 27B (86 of 1,418 sentences).
How often did another correction help?The red bars count fully correct first answers that were no longer fully correct at the end. The green bars count the reverse: answers that became fully correct after more requests.
✕ Correct answer lost✓ Wrong answer fixedRatio
GPT-5.6 Luna
171821 : 1
Gemini 3.5 Flash Lite
114167 : 1
Gemma 4 26B A4B
234733 : 1
Kanana 2 3B Instruct
71233 : 1
Ministral 3 3B
68253 : 1
37% → 4%Share of fully correct answers before and after revision, among the 486 sentences that GPT-5.6 Luna changed after its first answer.
31% → 6%Share of fully correct answers before and after revision, among the 402 sentences that Gemini 3.5 Flash Lite changed after its first answer.
37% → 3%Share of fully correct answers before and after revision, among the 666 sentences that Gemma 4 26B A4B changed after its first answer.
How long did the models keep making changes?Share of the 1,418 sentences. The larger models used 2.1 to 2.7 requests on average. For both 3B models, most sentences reached the 10-request limit.
1
2
3–4
5–9
10 (limit)
GPT-5.6 Luna
2.2×requests
Gemini 3.5 Flash Lite
2.1×requests
Gemma 4 26B A4B
2.7×requests
Kanana 2 3B Instruct
8.2×requests
Ministral 3 3B
8.2×requests
Each request means another model call. The two 3B models averaged more than 8 requests per sentence and were still changing 975 (Kanana 2 3B Instruct) and 993 (Ministral 3 3B) sentences at the limit. More calls did not produce better corrections here.
How an answer changes over repeated requests
What the learner wrote시드니에서 서울까지 걸립었습니다.
Human corrections시드니에서 서울까지 걸렸습니다. / 시드니에서 서울까지 아홉 시간 걸렸습니다.
1
시드니에서 서울까지 걸렸습니다.
✓ Matches a human correction
2
시드니에서 서울까지 가는데시간이얼마나걸렸습니다.
✕ No longer matches
3
시드니에서 서울까지 가는 데 시간이 얼마나 걸렸습니까?
✕ No longer matches
4
시드니에서 서울까지 가는 데 시간이 얼마나 걸렸습니까?
Unchanged, loop stops
What the learner wrote아마 장래에 음악 연출가 일 거에요.
Human corrections아마 장래에 음악 연출가 일 거에요. / 아마 장래에 음악 연출가일 거예요.
1
아마 장래에 음악 연출가일거예요.
✓ Matches a human correction
2
아마 장래에는음악 연출가가될거예요.
✕ No longer matches
3
아마 장래에는 음악 연출가가 될 거예요.
Unchanged, loop stops
What the learner wrote저는 심심할 때 자주 그림 그립니다.
Human correction저는 심심할 때 자주 그림을 그립니다.
1
저는 심심할 때 자주 그림을그립니다.
✓ Matches a human correction
2
저는 시크한때 자주 그림을 그립니다.
✕ No longer matches
3
저는 시원한때 자주 그림을 그립니다.
✕ No longer matches
4
저는 추운때 자주 그림을 그립니다.
✕ No longer matches
5
저는 추울때 자주 그림을 그립니다.
✕ No longer matches
6
저는 추워때 자주 그림을 그립니다.
✕ No longer matches
7
저는 추울때 자주 그림을 그립니다.
✕ No longer matches
8
저는 추워서때 자주 그림을 그립니다.
✕ No longer matches
9
저는 추워서 때로자주 그림을 그립니다.
✕ No longer matches
10
저는 추워서 때로는자주 그림을 그립니다.
✕ No longer matches
Practical recommendations
What I would use
For checking a sentence, I would start with either of these two prompts. The Korean version had the highest score on GPT-5.6 Luna and Muse Glimmer and was within 0.7 points of the best on every other larger model shown above. The short English version was at most2.9 points behind it. That makes it a reasonable choice if you prefer to read the instruction in English.
Paste your sentence after the prompt. Both ask for a corrected sentence without an explanation, which is what this benchmark measures. I did not test how well they explain a correction or help you understand it.
Ask the model to correct actual errors.
The Korean prompt also says to leave correct sentences unchanged.
I would avoid repeatedly submitting the answer for another correction; that reduced quality in this test.
For the two 3B models tested here, a different prompt did not close the gap to the larger models.
Written in Korean
아래 한국어 문장에서 틀린 부분만 수정하고, 교정 문장 한 줄만 출력하라.
단, 문장이 이미 문법적으로 올바르면 원문을 그대로 출력하라.
Short, real errors only
Correct the following korean sentence. Only correct actual errors. Reply with only the corrected sentence.
How the test works
The sentences and scoring
I used the same 1,418learner sentences from KoLLA v2 as in the main benchmark, with the same human corrections and M2 scorer. “Overall quality” is the F0.5 score: it balances how many mistakes the model catches with how often its changes are useful, giving more weight to avoiding unnecessary changes.
Seven models, with some incomplete runs
The selection covers two hosted models,three open-weight models on a rented GPU andtwo 3B models on a laptop. The charts require at least 85% of sentences to have a scored answer in each run. This leaves Qwen3.8 27B and Muse Glimmerout of some comparisons; their coverage is listed alongside the results.
Each condition ran once. Comparable runs differed by up to1.8 points, so I would read small gaps as ties. There is no statistical significance analysis here.
What the repeated-correction test covers
Each extra request started a fresh chat with the previous answer as its input. The model could not compare that answer with the learner's original sentence. Asking a follow-up question in the same conversation may behave differently; I did not test that here.
A valid rewrite can still lose points
The scorer checks changes against the human corrections. A different rewrite can be valid Korean and still lose points if those changes are not in the references. For a learner, unnecessary rewrites can make it harder to see which part was actually wrong, but a lower score does not mean every change was bad Korean.
A corrected sentence is useful, but it can leave you wondering why something changed. Elephant's sentence analysis explains corrections word by word to help you work through your own Korean sentences.