How Four Models Draw Chinese Calligraphy

The Setup

Drawing Characters They Cannot Name

In the tldraw calligraphy benchmark, AI models are given a reference image of a Chinese calligraphy character and asked to reproduce it on a digital canvas using structured tool calls. They can create strokes, capture their work, compare to the reference, and iterate. Eventually, they must call save_document() and submit.

But what happens in those final moments? When a model looks at its imperfect calligraphy and decides "this is done" — what is it thinking?

System

# Calligraphy Agent

You reproduce Chinese calligraphy characters on a digital canvas using the canvas tools.

Rough workflow:
- Study the reference image — understand the character structure and stroke order
- Use terminal/file_editor to write code that computes point arrays (bezier curves, pressure gradients)
- Call create_stroke for each stroke, building the character piece by piece
- Call capture_canvas to check your progress visually
- Use update_stroke or delete_strokes to refine individual strokes
- When satisfied, call save_document

BUT, you MUST call save_document before finishing.

## Rules
- Draw the character stroke by stroke, like a calligrapher would
- Use pressure variation (z values) to create thick-thin transitions within strokes

User

🖼 Reference Image: 512x512 binarized PNG

Reproduce the Chinese calligraphy character shown in the reference image by {calligrapher_name}.

The reference image is shown above. Study it carefully before drawing.
When done, call save_document().

No character name, pinyin, or definition is provided. The model must identify the character from pixels alone.

Overview

The treemap shows 45.2 million output tokens from all 352 benchmark traces (4 models × 88 characters). Each rectangle is sized by tokens spent. Click a model to zoom in.

Most tokens are just narration

20–42% of output tokens are progress narration — "Now let me create stroke 3", "Good, continuing." Models spend more tokens talking about what they're doing than actually doing it. Claude and Kimi narrate the most (38–42%). GLM-5 narrates least (21%).

All models refine — but differently

Every model goes back to update strokes after drawing them. In the deep-dive traces, GPT-5.4 drew 10 strokes then updated all 10 in a second pass. Claude rewrites code instead of using update_stroke. Kimi deletes everything and redraws from scratch when it catches errors.

GPT-5.4's categories are structural (tool-based) since its reasoning is encrypted.

GLM routes through code; Claude thinks in text

GLM-5 spends 19% of tokens writing Python bezier generators. Claude spends 3%, preferring to reason about strokes directly. Kimi is in between at 5%. All approaches produce similar quality — the code doesn't buy precision.

Claude is 3x more self-critical than GLM

Claude spends 10% of tokens on self-critique: "The strokes are too thick", "The proportions are wrong." GLM-5 spends 1.5%. Claude sees problems it can't fix through the coordinate API. Self-awareness without self-improvement.

Claude looks at its work 2.5x more than others

Claude spends 10% on visual comparison — examining canvas captures against the reference. Kimi and GLM-5 spend 4%. More looking doesn't produce higher fidelity. The gap is between seeing the error and fixing it through coordinates.

Submission reasoning is minimal

2–5% of tokens go to the save decision. Claude writes a flaw list. GPT-5.4 says one sentence. Kimi declares success on the wrong character. GLM-5 says "the character looks good" after drawing 3 strokes for a 16-stroke character.

19 of 347 traces: wrong character, full confidence

Across the full benchmark, 19 of 347 evaluated outputs (5.5%) drew fewer than 40% of expected strokes — a strong signal the model drew the wrong character entirely. GLM-5 accounts for 16 of these 19 cases (18.4% of its tasks). Claude: zero. Kimi drew 獨 instead of 福. GLM drew 大 (3 strokes) for 鑒 (16 strokes). Both submitted with full confidence.

GLM-5 talks about art, draws the wrong character

GLM-5 discusses calligraphy style 3.2x more than Claude — naming calligraphers, describing brush dynamics. It has the most artistic vocabulary and the highest wrong-character rate (18.4% of tasks). Knowing about art and producing art are different skills.

The full picture

Zoom into any model or task by clicking the treemap. All models mostly draw, but their strategies diverge on refinement, code generation, and self-correction. The models that score best aren't the ones that think most — they're the ones that iterate on their strokes.

	被bdsr_1646	福smh_1157	鑒csl_0944
Claude	"The character looks recognizable and captures the essential structure of 被. The left radical 衤 has the dot, hengpie, vertical, pie, and dot..."	"The character 福 is clearly recognizable with good structure. The left radical 礻 and right component 畐 are well-formed..."	"The character 鉴 looks good and recognizable. The structure matches the reference well with: Top-left 臣-like box component..."
GPT	"I'm saving the completed character now so it's submitted." [908 chars encrypted]	"I'm saving the completed calligraphy document to finalize the work." [2,124 chars encrypted]	"I'm satisfied with this rendition, so I'll save the canvas." [908 chars encrypted]
Kimi	"The character looks very good now! It captures the essence of the reference with the left radical 衤 and right component 皮..."	"I can see the structure of 獨 (du) taking shape with all the key elements." WRONG CHAR	"The character resembles the reference character '鑒' (jian). Let me save the document now."
GLM	"I've captured the essential characteristics of 八大山人's style: Left radical (衤) with small expressive dot at top..."	"The character 沙 is now recognizable." WRONG CHAR	"The character '大' has been successfully created on the canvas with all three strokes." WRONG CHAR (drew 大 not 鑒)

How Four Models Draw Chinese Calligraphy

Drawing Characters They Cannot Name

How Each Model Approaches a Task

被 bdsr_1646 — highest-scoring task

鑒 csl_0944 — hardest task, all models scored lowest

Each Model's Definition of Done

What Each Model Drew — and What It Said