How to test dictation accuracy beyond word error rate

Use a small set of tasks you actually write: a short reply, a multi-paragraph email, a list of actions, meeting notes, names and numbers, and a sentence you correct while speaking. Keep the audio, a human-checked transcript, and a short brief describing what the finished message should communicate.

Save the first result exactly as it lands in the destination app, before any editing. Keep plain text for word comparisons and a screenshot or rich-text copy for paragraph breaks, lists, and spacing. A well-formatted preview is not enough if pasting into Gmail, Slack, or a document flattens it into one block.

Use the same destination, input, and relevant settings for each candidate. Record whether paragraph and list formatting was explicitly requested or inferred by the app. Test those cases separately: following “new paragraph” and recognizing a natural topic change are different capabilities.

Define “ready to use” before the test: correct facts and intended meaning, complete content, and an appropriate structure for the destination. Ask writers to make the minimum necessary fixes. Log optional personal rewrites separately so one reviewer’s style preferences do not become another product’s error count.

Start the post-paste clock when the first output is fully inserted in the intended field. Stop when the writer accepts it as ready to send or save after review. Keep recognition and insertion latency as separate timings; this measurement is specifically the work left after the paste.

For example, eight word edits in a final 100-word draft means 8 edits per 100 accepted words. If the writer also inserts two paragraph breaks and converts a block into a list, record three formatting repairs. If review takes 20 seconds and corrections take 40 seconds with no idle time, the post-paste total is 60 seconds. These are example calculations, not product scores.

For the word comparison, document tokenization and normalization. An English baseline can split on whitespace, retain case and punctuation attached to words, and exclude layout markup. Keep the original formatting separately. A word diff misses line-break repairs and cannot tell you how many keystrokes, undo operations, or voice-edit attempts the user actually needed.

Do not turn edit distance into a time estimate. One replacement can be a quick typo fix or a slow investigation of a wrong number. For deeper studies, record correction actions with consent and label the reason: recognition, changed meaning, formatting, or optional preference.

NIST’s usability guidance combines task time, errors, completion, and user feedback. The measures here apply that approach to dictation; this particular scorecard is our proposed protocol, not a standardized dictation benchmark.^[2]

Evaluate the unedited first paste. A reader should be able to see where one idea ends, which items belong together, and what needs a response. Formatting belongs in the quality assessment because a word-correct blob can still need manual restructuring.

Both versions can have 0% WER against the same words when the scoring rules ignore whitespace and list markup. That score does not tell you which layout lets a reader find the three requests or the deadline more easily. Test that separately.

The revised Gmail draft below shows the structure to look for: separate paragraphs, meeting times as a list, and a clear sign-off. In your test, count the edits needed to reach a usable layout and time those repairs. Score the original first paste separately from the finished draft.

Choose the applicable checks before seeing the outputs. For each, use 2 = usable as pasted, 1 = understandable but needs a small repair, and 0 = missing or misleading structure that needs reworking. Mark irrelevant checks N/A, such as an email closing in a search query. Report the criteria individually; extra headings or bullets do not automatically make a draft better.

W3C’s page-structure guidance explains how meaningful headings and content organization support navigation and understanding. The general principle informs these checks; our 0–2 rubric is not a W3C rating or accessibility certification.^[3]

A formatting checklist captures structure, but the reader’s task is the useful test. Give representative readers the unedited output in its real destination and ask questions with predefined answers. For the example above: “Which three things must Alex send?” and “When is everything due?”

Record time from showing the text and question to the submitted answer, then score whether the answer is correct and complete. Report time for correct responses alongside wrong-answer and timeout rates. A fast wrong answer is not a readability win. An ease rating can add context, but it should not replace the task result.

To isolate formatting, compare versions with identical words and different layout. To compare whole products, use their actual outputs and report wording or meaning differences too. Hide product names, randomize presentation order, and use different matched passages for repeat tasks so readers do not simply remember an answer they saw a moment earlier.

Keep writer correction effort separate from reader comprehension. The person who dictated a message already knows what it is meant to say; that can hide problems a recipient encounters. Recruit readers who reflect the intended audience, including their usual display and assistive settings.

Label changed names, dates, amounts, negations, and commitments separately. A missing “not” may be only one word error but reverse the instruction. A wrong product spelling may be harmless in a personal note and unacceptable in a customer message. Decide these categories before looking at which app wins.

Treat meaning as an acceptance requirement, not something attractive formatting can compensate for. Check names, dates, amounts, negations, commitments, and invented details against the task brief. A beautifully formatted email with the wrong deadline is not ready to use.

Report empty outputs, wrong insertion targets, retries, and timeouts as failures. A run that never reaches the text field has no post-paste completion time; record that failure rather than giving it zero seconds or silently removing it. If a recording is unusable, apply the same declared exclusion to every tool.

Word error rate is (substitutions + deletions + insertions) divided by the number of words in the reference. State the normalization rules before scoring: punctuation, capitalization, numbers, contractions, and tokenization can all change the result. Use the same rules for every candidate.^[1]

Illustrative example: the reference “send the revised brief tomorrow” has five words. If the output is “send the brief tomorrow,” one word is deleted, so the word error rate is 1 ÷ 5 = 20%. This is an arithmetic example, not a score for VoiceOS or another product.

For a combined result, sum errors and reference words across the test set before dividing. A simple average of sentence percentages gives a two-word command as much weight as a long paragraph. Report the aggregation you use. Keep the per-sample results so an overall number does not conceal a specific failure pattern.

A mixed-language sample might say: “明日のdesign reviewは午後3時です。FigmaのリンクをSlackで送ってください。” That is original sample text, not a tested transcript. It checks Japanese syntax, English product names, a time, and whether the output silently translates or transliterates terms.

Record the intended spelling of names and whether variants are acceptable. Japanese does not use spaces in the same way as English, so a word-level score depends heavily on the tokenizer. State the tokenizer and normalization, or add a character-level result with its own explicit rules. Keep exact-name and meaning checks alongside either metric.

Include several speakers and realistic language switches before making a bilingual accuracy claim. One carefully selected sentence cannot support “best Japanese dictation.” If the system is allowed to use a vocabulary list, give equivalent information to the other candidates or report that configuration difference.

Report the number of writers, readers, samples, and attempts. Break results down by task: short replies, emails, lists, and longer notes. Show the share ready without edits, word edits per 100 accepted words, formatting repairs, and median correction and total post-paste times. Include the spread, and a 90th percentile only when the sample size supports a meaningful estimate.

Time summaries for completed runs must be labeled as such and shown beside failure counts. Keep readability results in a separate view: structure by criterion, correct-answer rate, and time for correct answers. Avoid combining everything into one “accuracy percentage” unless the weighting is declared and justified.

Publish permissioned samples and outputs when possible, and explain who chose them and which product you make or sell. A useful conclusion describes how much work remains and how readable the first paste is, with enough detail for someone else to repeat the test.