weather-card-benchmark

Methodology

One prompt, every model, the same weather fixture. Here is exactly how the numbers are produced, how community votes are counted, and what a patch is and is not.

1Similarity metrics are measurements, not quality scores. The channels below quantify how alike two cards are — we don't convert that into a judgment of which model is better.

2Community vote standings are a UI preference — which cards people prefer to look at. Vote counts are visible; they are not a model-capability benchmark.

3Everything is deterministic and reproducible. Every batch is anchored to a manifest_hash you can recompute; the measured pixels are the exact pixels the algorithms read.

4Some channels are coarser than their names suggest, and we say so rather than quietly smoothing them. v-palette's Gaussian kernel is inert at the frozen Lab grid (adjacent bin centres sit 25–37 ΔE apart against σ=10), so it behaves as a hard-binned colour cosine; v-color quantizes to 4 levels per channel, where a typical card puts over half its pixels in one bin. Both therefore read closer to "same dominant colour or not" than to a perceptual palette distance. Widening either was tested against the full set and made discrimination worse, so the frozen definitions stand and the limit is documented instead.

The task

Every card on this site was produced from exactly this text — no system prompt, no follow-up, no retry with hints. Both variants are handed to every model identically.

prompt (verbatim)

download

Build a weather card as a single complete HTML file.

Output only the raw HTML file content — no markdown fences, no commentary before or after.

Rules:
- All CSS and JS inline. No external resources; no network requests other than the two API paths below.
- Desktop viewport, 1280x800.
- Read URL query parameters: `lat`, `lon`, `name` (location label), `date` (YYYY-MM-DD). Defaults: 52.52, 13.405, Berlin, today (UTC).
- Fetch weather data from the same-origin API (Open-Meteo compatible, no key):
  - `/api/om/forecast` — parameters and response format identical to api.open-meteo.com/v1/forecast (today and future dates)
  - `/api/om/archive` — identical to archive-api.open-meteo.com/v1/archive (past dates)
- Request with query parameters: `latitude`, `longitude`, `daily=temperature_2m_max,temperature_2m_min,weather_code`, `hourly=temperature_2m`, `timezone=UTC`, `temperature_unit=celsius`, and `start_date`/`end_date` both set to the requested date.
- Display, in Celsius: the location name, the date, that date's maximum and minimum temperature, the weather condition, and the hourly temperatures for that date.
- If the data cannot be loaded, the page must say so.

P-min is the bare task; P-q adds a short quality clause. Each manifest pins the sha256 of the bytes below.sha256 6901ad22fb318de2…

How similarity is measured

Each model draws a single-file HTML weather card from the same prompt against a frozen weather snapshot, rendered in a pinned Chromium at a fixed viewport and frozen virtual clock. Similarity is computed only between valid outputs, per pair, per channel — there is no composite score. The pipeline runs in four layers:

single-card descriptors

Per-card scalars: byte and DOM/CSS/JS structure, visual measures (colorfulness, brightness, contrast, whitespace, frame-change), and a fixed-grid Lab palette. Plus a versioned data-fidelity check against the snapshot.

similarity channels

Visual (phash/dhash/color/palette/layout/edge/ssim), code (shingle/winnow/feature/ast-js/css-prop), extended code-feature (x-color/x-css-values/x-layout/x-naming/x-semantics), and DOM (geom/text/pqgram) channels — 20 formal plus 2 permanent diagnostics, each a distinct, frozen definition; a null pair stays null. The matrix also offers a derived `merged` consensus: because the channels are not on a common scale, each is first stretched over its own p1–p99 across the whole set (a frozen domain, so the value doesn't move when you filter) and only then averaged. It is derived, never measured.

statistics

Descriptive only, N is small (3–5 slots). Means and IQRs over valid pairs, self-consistency, LORO sensitivity, and one randomization paired test for prompt-variant sensitivity. Intervals are descriptive, not inferential.

behavior probes

Contract-level pass/indeterminate/fail on network discipline and error handling, plus out-of-contract observed/not-observed notes (loading states, console errors, no-JS fallback).

Each batch freezes both prompt hashes, the weather-snapshot hash, the city/date set, the render environment, and the pipeline commit into a manifest whose SHA-256 is the manifest_hash. Recompute it to verify — reproducibility is the point.

Patch policy

A broken model card may be hand-fixed so it renders, with an honest, visible note; the original is always preserved and served by default. Four invariants hold, without exception:

A patch NEVER changes slot state, validCount, or any coverage status. A broken slot stays broken in every count; the patch only adds a viewable repaired rendering.

All metrics — similarity, fidelity, shot.webp, telemetry — derive from the ORIGINAL output. The card page states: "metrics refer to the original output".

Arena and every voting / preference surface serve ONLY the original card.html. Vote rows carry patched_served (always 0; an audit column).

The category enum is closed; the hard rule is "a patch may only make the card render, never improve its content or design". The stored unified diff makes this auditable.

Patches applied

Every patch in the data root is listed here — config, category, and a link to the exact original→patched diff — so the whole layer is auditable on one page.

config	slot	category	reason	diff
claude-opus-5-eff-xhigh--cli--claude-code--dev	P-q · #0	render-fix	The card builds its HTML as a leading-`+` string-concatenation chain, and wrote a multi-line ternary inside it — but prefixed the `:` branch line with `+` as well (`(hi ? '<span…>' + hi.int + '…'` / `+ : '<span class="int">–</span>')`). The parser therefore hits `Unexpected token ':'`, the entire inline <script> fails to parse, no API request is ever issued, and even the card's own error handler — which lives in that same script — never comes into existence, so it renders its loading skeleton forever. It happens twice, in the Maximum and Minimum blocks. Fix: delete the two stray '+' characters (replaced by a space, indentation untouched, byte length unchanged) so each ternary parses. Two-character syntax repair — no content, data, or design changed.	diff ↗
glm-5.2--cli--opencode--dev	P-q · #1	render-fix	The card's escapeHtml() lookup object left two values unquoted (`'"':","'":'`), which is a JavaScript SyntaxError ('Unexpected token &'). The whole inline script failed to parse, so the card rendered only its empty skeleton (no weather content). Fix: wrap both entity values in string quotes (`"""`, `"'"`). Pure syntax repair — no content, data, or design changed; the map's intended values are the standard HTML escapes and are unambiguous.	diff ↗
mimo-v2.5-free--zen-free--dev	P-q · #1	render-fix	In the SVG hourly-chart point loop the index bracket and call paren were transposed: `sy(temps[i)]` instead of `sy(temps[i])`, a JavaScript SyntaxError ('Unexpected token )'). The inline script failed to parse and the card stayed stuck on its '⏳ Loading weather…' placeholder. Fix: transpose the two characters back to `temps[i])`. Two-character syntax repair — no content, data, or design changed.	diff ↗
qwen3.7-plus--cli--opencode--dev	P-q · #0	render-fix	The weather-code map WC had a malformed, duplicated '63' entry whose value was written as a bare string instead of a [label, emoji] array (`'63':'Moderate rain','\uD83C\uDF27']`), producing a JavaScript SyntaxError ('Unexpected token ]'). The entire inline script failed to parse and the card rendered fully blank. Fix: insert the single missing '[' so it becomes `'63':['Moderate rain','\uD83C\uDF27']`, matching the model's own correctly-formed '63' entry on the very next line. One-character syntax repair — no content, data, or design changed.	diff ↗

How votes are counted

Community UI preference — which cards people prefer to look at; not a capability measure.

Votes are cast blind: identities never reach the browser before a vote is recorded (opaque single-use tokens, original HTML only, qualified-grade cards only).

Bradley–Terry (a simple iterative MM fit) runs over pair votes only, per angle × variant.
A tie counts one half to each side.
both_bad is excluded from Bradley–Terry (otherwise it becomes a suppression lever) and reported separately as each config's bothBadRate — itself honest community data.
Gate votes (source=gate — the coerced unlock votes) are excluded from the default aggregation as the lowest-quality rows; they remain in the CSV export, labeled. Arena and share votes count.
prefRate carries a Wilson confidence interval; maxSingleVoterShare is shown for concentration transparency (anti-brigade sunlight).
A config is marked insufficient when it has fewer than 20 votes or fewer than 8 distinct voters — such rows carry no numbers and no position on any axis.

Unauthenticated, localStorage-identity web votes — trivially gameable; treat as community signal, not measurement.

download the full votes CSV ↓arena →

Contributing runs — the zero-trust path

You can help fill coverage gaps by supplying an API key for the exact cells you pick. The key handling rules, verbatim:

Your API key stays in memory only — never persisted, never logged.
It is used solely for the runs you selected, then discarded.
The runner is open source; you can read exactly what it does.
Zero-trust alternative: run the batch locally and open a pull request — no key ever leaves your machine.

A contributed run lands in the gallery immediately with an "unreviewed community" badge, and never enters the arena voting pool until a maintainer reviews it. Contributed output is committed to a community/<jobId> branch and never pushed to the default branch without human review.

open the contribute page →

Provenance — measurement sessions

Each batch is one frozen, append-only run against one fixture. Batches are provenance now, not a way to browse the gallery. Similarity numbers are only comparable within a single session — same fixture, same snapshot.

1 measurement session, newest first.

session	date	configs	variants	manifest_hash
2026-07-19--unified	2026-07-19	207	P-minP-q	9f8706c00cc02d971ee764d21a91a78158abe011d2e2b19e6b2c3a5de227bdf7

Full comparison scheme

The frozen technical reference — the deterministic similarity scheme exactly as authored during the trial. The site's presentation follows the positioning above.

The full technical scheme is not bundled in this deployment; the summary above and the layer descriptions are the authoritative site-side account.

A note on language

This site ships in English and 简体中文. Model ids, effort tokens, arm names, grade labels, channel names, and config slugs stay in English on both — they are instrument labels, not prose, and CJK on them would break the mono-instrument read. Everything else is authored per language rather than machine-translated, and numbers are formatted for the active locale.