Paper 13
Designing distortion experiments: measuring what people cannot fake
A methodology primer on within-subject designs, counterbalancing, cover tasks, and loading your measurement onto errors, using the trigger-encoding experiment as the worked example.
Why memory experiments are unusually easy to fool
Most of the ideas in this corpus are about how memory can be edited: written selectively under arousal, reopened in a labile window after recall, nudged into settling differently. Those are claims about a hidden process. To turn a claim about a hidden process into a number, you have to build an instrument, and memory is an unusually treacherous thing to measure, for one specific reason. The participant is a thinking agent who can guess what you are looking for and, consciously or not, deliver it.
Call anything about the situation that hints at the expected response a demand characteristic (Orne, 1962, established). If a person suspects a study is about memory, they rehearse. If they suspect it is about a particular sound, they attend to it. Then the effect you measure is partly the effect you telegraphed. A weak or badly designed memory study does not fail loudly. It quietly measures compliance and reports it as psychology.
This paper is a primer on the design decisions that make a memory measurement trustworthy. Each idea is motivated in general terms and then shown at work in one concrete instrument: the trigger-encoding experiment documented at docs/trigger-encoding-protocol.md and driven by the /experiment console. The companion paper on arousal and false memory covers what that experiment is trying to find. This paper covers how it is built so that the finding means something.
Within-subject designs: each person is their own baseline
The first decision is who provides the comparison. In a between-subject design, one group of people gets condition A and a different group gets condition B, and you compare the groups. In a within-subject design, every person experiences every condition, and you compare each person to themselves.
For memory, within-subject is almost always the stronger choice. People differ enormously in baseline memory ability, and those differences are usually larger than the effect you are hunting. Comparing groups means fighting through that noise. Comparing a person to themselves cancels it: whatever makes one participant a strong or weak encoder is held constant across their own conditions.
In the trigger-encoding experiment this is explicit. Every participant sees the full ten-event spine, and trigger type, temporal offset, and event centrality all vary within that single session. Each person supplies both their own triggered targets and their own untriggered baselines. The question is never "do triggered people remember differently from untriggered people," which would confound the trigger with who happened to land in which group. It is "does this person remember a triggered event differently from their own untriggered ones."
The cost of within-subject designs is that conditions can contaminate each other. Seeing condition A can change how you respond to condition B through practice, fatigue, or simply learning the game. Managing that contamination is the job of the next idea.
Counterbalancing and Latin squares: separating the effect from the item
Here is a confound that quietly ruins memory studies. Suppose the siren always lands on the moment an actor holds up a lighthouse photo, and that moment turns out to be well remembered. Was it the siren, or is a lighthouse photo just intrinsically more memorable than the other events? With a fixed pairing you cannot tell. The trigger is confounded with the item it sits on.
The fix is counterbalancing: rotate which condition lands on which item across participants, so that over the whole sample every item has served under every condition. Then the item's intrinsic memorability is spread evenly across all conditions and cancels out of the comparison, leaving the trigger effect standing alone.
When you have several conditions and several items and want a compact rotation that pairs each condition with each item exactly once, the tool is a Latin square: an n-by-n grid filled so that every symbol appears once in each row and once in each column. The trigger-encoding experiment uses a 5-by-5 square built by buildPlan(condition) in src/lib/experiment/script.ts. Five target events, five trigger types:
| Target event | Cond 0 | Cond 1 | Cond 2 | Cond 3 | Cond 4 |
|---|---|---|---|---|---|
| book placed | notification | phone ring | doorbell | dog bark | siren |
| stranger passes | phone ring | doorbell | dog bark | siren | notification |
| cup spill | doorbell | dog bark | siren | notification | phone ring |
| phrase spoken | dog bark | siren | notification | phone ring | doorbell |
| photo shown | siren | notification | phone ring | doorbell | dog bark |
Read down any column and you see one participant's assignment. Read across any row and you see one event paired, over five participants, with all five triggers. Run the study in blocks of five and every target has met every trigger equally often. The siren effect, if there is one, can no longer be blamed on any single event being special. The temporal offset (before, during, after) is rotated the same way, so offset is not confounded with position in the sequence either. The takeaway is general: counterbalancing does not remove item differences, it renders them irrelevant to the comparison by balancing them across conditions.
Cover tasks: suppressing strategic encoding
Even a perfectly balanced design fails if participants encode strategically. A person who knows their memory for events will be tested will study the room like a student cramming, and you will measure their effort rather than the automatic, arousal-driven encoding you care about.
The classic remedy is incidental encoding delivered through a cover task: give participants a plausible job that keeps them engaged with the material for a benign reason unrelated to memory. The distinction between incidental encoding (learning as a side effect of doing something else) and intentional encoding (deliberately trying to memorize) is old and well established in memory research; the two produce measurably different memory, and only incidental encoding reflects the everyday case these studies are meant to model.
In the trigger-encoding experiment the cover is that participants believe they are evaluating the comfort and acoustics of the space. That framing does real work. It explains why they are sitting quietly in a furnished room, it explains why sounds occur, and it gives them nothing to strategically memorize. Because the cover involves mild misdirection about the study's purpose, it carries an ethical obligation, met here by a full debrief afterward in which the real manipulation is disclosed. A cover task is a deception about topic, not about risk, and the line matters: participants are told truthfully that loud sounds and lighting changes will occur.
Load the dependent variable onto errors, not hits
This is the central idea of the whole approach, and the one most worth internalizing. A dependent variable is what you measure. The obvious choice for a memory study is the hit rate: how many real events the participant correctly recognizes. It is also the worst choice for demand resistance, because a hit rate is trivially inflatable. A motivated participant who suspects a memory test simply tries harder and says yes more often, and their hit rate climbs for reasons that have nothing to do with your manipulation.
The move that rescues the measurement is to load the dependent variable onto errors that a cooperative participant cannot manufacture. Two kinds do this work.
A binding error is misremembering which things went together, even when each thing is remembered. The trigger-encoding experiment asks, for each target, "which sound was present when this happened?" Getting that wrong is a failure to correctly bind a sound to an event. You cannot fake a binding error into existence by trying hard; if anything, trying hard should reduce it. So if arousal degrades peripheral binding, that shows up as a signal effort cannot wash away.
A gist-lure false alarm is confidently recognizing something that never happened but fits the meaning of the scene: the book was open when it was closed, the coat was blue when it was green. Because the lure is meaning-consistent, a participant reconstructing from gist falls for it precisely to the degree their memory has drifted. This too is unfakeable in the helpful direction: a participant cannot decide to have a confident false memory.
| Measure | What it is | Faked by trying harder? | Role here |
|---|---|---|---|
| Hit rate | Correct yes to real events | Yes, inflates easily | Secondary only |
| Binding accuracy | Correct sound-to-event pairing | No | Primary DV |
| Gist-lure false alarm | Confident yes to plausible non-events | No | Primary DV |
This is why the load-bearing predictions of the trigger-encoding study, that arousal boosts central over peripheral memory and inflates gist-consistent false alarms, are designed to survive a participant who has fully guessed the game. Knowing it is a memory test does not let you fake a binding error or a confident false alarm. Whether arousal actually produces this central-over-peripheral tradeoff in humans is emerging and contested, which is exactly why the design has to be this careful; the point here is that the measurement stays honest either way. The general principle transfers far beyond this study: choose a dependent variable whose informative movement runs opposite to, or orthogonal to, the direction demand would push.
Append-only logging: never lose the ground truth
A subtler failure is not in the room but in the data pipeline. If your recording of what happened is edited, summarized, or overwritten before analysis, you can quietly lose the very co-occurrences you are trying to measure, and you will never know a row went missing.
The discipline that prevents this is append-only logging: every trigger, event, and marker is written as one immutable row, and nothing is deleted or silently rewritten. Corrections are added as new, traceable amendments rather than in-place edits. In the trigger-encoding experiment the Registry (src/lib/experiment/registry.ts) is exactly this, and every row is stamped against a single monotonic session clock (SessionClock, built on performance.now() so a wall-clock adjustment cannot shift it). Live events tapped by an operator get their true time from that clock, and if a tap was late, the correct time is recovered from a timecoded recording and written back through amendLast, which preserves the edit trail rather than erasing the original.
The payoff is that the key independent variable is computed, not judged. Because every trigger and every event carries an exact timestamp, pairEvents in score.ts derives each event's signed offset from its nearest trigger arithmetically:
offset = event.tStart - trigger.tStart
There is no manual coding step where a human decides what counted, and therefore no place for expectation to leak into the data. The log is exported whole, so no summary stands between the raw session and the analysis.
What to take away
- Memory experiments are easy to fool because the participant can guess your hypothesis and unconsciously deliver it. Good design assumes the participant has guessed and stays valid anyway.
- Prefer within-subject designs so each person is their own baseline; this cancels the large individual differences in memory that would otherwise swamp your effect.
- Counterbalance with a Latin square so every condition meets every item equally, which lets you attribute an effect to the manipulation rather than to one item being intrinsically memorable.
- Use a cover task to get incidental encoding, so you measure automatic memory rather than deliberate rehearsal, and debrief fully because the cover is a deception about topic.
- Load your dependent variable onto errors that trying harder cannot fake, such as binding errors and gist-lure false alarms, not onto the easily inflated hit rate. This is what makes a result survive demand.
- Log append-only against a single clock and compute your independent variable arithmetically, so no manual coding step and no silent data loss can let expectation into the numbers.
Sources
- Orne, M. T. (1962). On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications. American Psychologist.
- Roediger, H. L., and McDermott, K. B. (1995). Creating false memories: remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition. The gist-lure paradigm behind the false-alarm probe.
- Fisher, R. A. (1935). The Design of Experiments. The foundational treatment of counterbalancing, blocking, and Latin squares in experimental design.
- Companion whitepaper: Arousal and the false memory (
docs/whitepapers/03-arousal-and-the-false-memory.md), for the theory these methods are built to test. - Project instrument:
docs/trigger-encoding-protocol.mdand the/experimentconsole, with the design implemented acrosssrc/lib/experiment/script.ts(counterbalancing),registry.ts(append-only log), andscore.ts(offset and error scoring).