AXHUB DEEP READ · 04

We tested 7 translation models
on our own articles

Reading time about 9 minType operating log — a half-day hands-on benchmarkFor teams that have to pick a model

The content factory introduced in Deep Read 01 ran well. Then the next question arrived — "Should we switch translation to a better model? Move to a paid cloud?" Instead of rumors and leaderboards, we measured 7 models directly on our own article sentences. The conclusion first: a free local model came within 0.4 points of the paid flagship, and we decided on local. This is the record of a half-day measurement.

1Someone else's scorecard isn't our answer

There are plenty of model rankings. The trouble is that those rankings don't measure "Korean translation of AX operations articles."

So we made a small test out of our own domain sentences.

4 questions. Korean→English 2 (a sentence and a paragraph), English→Korean 2.
Fixed conditions. Same instruction ("professional translator, output the translation only"), same temperature, same length limit.
4 scoring criteria. Accuracy, naturalness, terminology, completeness — with latency and tokens recorded too.

Seven candidates. From a paid cloud flagship, to a local mid-size model running on our Mac (an MoE structure of 35B total, 3B active), to a 70B and an 8B on a very fast free API, plus the translation-only 1.8B we'd been using in the factory.

2The scorecard — what a 0.4-point gap means

First place was the paid flagship (9.4), as expected. Its Korean prose was the most natural.

The surprise was second place. The local 35B running for free on our Mac scored 9.0 — a 0.4-point gap. It was actually faster, at 1–2 seconds, and the data never leaves the machine.

We measured classification (topic labels) and one-line summaries the same way; on classification, local and flagship were on par (4 of 5), and on summarization local was the best.

We couldn't find a reason to pay monthly for 0.4 points. Bulk work on local, and cloud fallback only for the quality-sensitive cases — that's how we set it.

3The mishaps off the scorecard

The record of mishaps may be more useful than the numbers. These actually happened.

Asked for Korean, got ChineseWhen we called the translation-only 1.8B from the factory in a generic way, it got the target language wrong. This model was stable only under its dedicated calling convention (a target-language tag) — which is why it worked fine in the factory. A small specialist model falls apart outside its prescribed usage.

Repeated "ionization" for 838 tokensThe same 1.8B, after finishing a normal translation, fell into a state of endlessly repeating a single word. It's a known failure mode of micro-models. An output-length cap and repetition detection are essential safeguards.

"Governance" became "rule of the state"The 8B on the free fast API rendered technical terms with a national-governance nuance, and dodged unfamiliar words by transliterating them phonetically ("inference," "private" left as sound-alikes). Everyday translation and domain translation are different tests — which is why you have to measure with your own sentences.

700 tokens of thinking for a one-line translationA reasoning-style cloud model, at default settings, spent 491–695 tokens of thinking on a one-line translation. Adding the option to turn thinking off brought the total to 62–69 tokens — a 10× difference. Some models used up their budget on thinking and returned a blank answer. If you use a reasoning model for machine work, first find the switch to turn thinking off.

Even the top model had a flawThe local 35B rarely leaked a Chinese character here and there. Solved with a one-line post-process that detects Han characters in the output and retries. "A good model + a thin safety net" is faster than "finding the perfect model."

A free API quietly vanishedA free-tier model that had been in our config stopped responding — retired or turned paid at some point. Hang your pipeline on a free cloud and someone else's circumstances become your outage.

4The decision — the math, disclosed

The final routing is three lines.

Bulk, mechanical work (translation, classification, summarization) → local 35B. Free, private, 1–2 seconds.
Small volume where quality is especially sensitive → paid flagship fallback (with thinking off).
Backup where speed is everything → free fast API 70B (aware of the quality trade-off).

The basis for this decision is not a leaderboard but the scorecard and mishap list above. Our sentences, our use, our math — a different team could arrive at a different answer, and that's normal.

5If you want to try it — the half-day benchmark

Make a test out of your own sentences. 4–5 pieces of real work text. Mix short and long.
Fix the conditions. Same instruction, same temperature. Change only the model.
Write the scoring criteria in advance. Ours were four — accuracy, naturalness, terminology, completeness. You can have AI do the scoring, but scan it once with your own eyes.
Record latency and tokens too. When quality is equal, these decide it.
Measure reasoning models with thinking off. Leave it on and the cost comparison is distorted.
Put the result on a single table. That table becomes the permanent answer to "why this model."

All told it was half a day. Shorter than the time we'd spent chasing model rumors.

6The gist

Measure instead of sense it — the third recurrence of this prescription in the series. In productivity (card No.10), in style (Deep Read 02), and this time in model selection.

And this time too, we didn't know until we measured. That free would win.

Where these models run → Deep Read 01 — the 1GB factory · Deep Read contents

Measurement record from June 26, 2026. The sample is a qualitative rating of 4 translation questions + 5 classification items + 1 summary; it's for our own decision-making, not generalization. Model and service names change over time, so they're described by size and type. The original record is in AXHub's operating docs.