Human review study: the protocol

This page documents how we intend to test whether a blind reader can recover both source names from a blend. It is published before any data has been collected, and it currently reports no results, because none exist.

We are saying that plainly rather than filling the page with plausible-looking figures. The whole argument of this site is that the engine measures spelling and cannot judge a name. Inventing a study to support that argument would destroy it.

The question

The blending engine ranks candidates by structure. The open question is whether a person who has never seen a blend before can look at the top-ranked result, say it, and work out which two names it came from. That is what a couple name or a ship name has to do in the wild, and it is the one thing the software cannot check.

Method

• A set of name pairs is submitted, covering common Latin-script first names, pairs where one name is much shorter than the other, and pairs that share letters.
• For each pair the engine produces its ordered candidates. Only the top-ranked candidate is used, so the test measures the ranking rather than the candidate pool.
• Each blend is shown to a reader who does not know the pair, with no explanation and no context.
• The reader is asked to say the blend out loud, and that pronunciation is recorded.
• The reader is then asked to name the two names it was built from. Both guesses are recorded whether right or wrong.
• A blend counts as recovered only when the reader names both sources correctly without prompting. One out of two counts as partial, and is reported separately.

What will be reported

• The number of pairs tested and the number of readers.
• How often both names were recovered, how often one was, and how often neither was.
• How often the reader's pronunciation matched the one intended by the person who submitted the pair.
• Every case where the top-ranked candidate failed, with the pair and the blend named, not summarised away.
• The date, the engine version, and the raw counts rather than percentages alone.

Known weaknesses in this design

Stating these now rather than in a limitations paragraph at the end of a results page.

• Readers recruited through this site are self-selected and more likely to be familiar with fandom naming than the general public, which will make recovery rates look better than they are.
• Familiarity contaminates the result. A reader who already knows Tayvis is not blind to it, and we cannot always tell.
• Latin-script bias. The engine's syllable rule is built on Latin vowels, so any finding here says nothing about names in other scripts. That limit is covered in the limits report.
• Written pronunciation is a lossy record. Asking someone to type how they said a name loses most of what actually happened.

Taking part

Run the blind name test with someone who does not know the pair, then send us the blend, the two source names, what your reader said out loud, and which names they recovered. The contact page has the address. Submissions that report a failure are more useful than ones that report a success, and they are the ones least likely to be sent, so please send them.