abxtest - simple ABX double-blind testing script
Copyright (C) 2000-2004 Robert Leslie

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

$Id: abxtest.1,v 1.8 2004/01/23 09:41:31 rob Exp $

abxtest - double-blind ABX comparison testing script abxtest [ -n min ] [ -m max ] [ -g goal ] A-cmd B-cmd abxtest is a tool for conducting listening (or other subjective) tests to determine whether a listener can discern a difference between two subjects under test, denoted A and B .

The listener conducts several trials in which an unknown subject, X , is randomly chosen to be either A or B . The probability that X is A or B is the same. The listener must decide whether X sounds the same as A or the same as B . The listener may repeat A , B , and X as many times as necessary before reaching a verdict.

In order to prove the hypothesis that a difference can be discerned between A and B , the listener must disprove the null hypothesis that A sounds the same as B . This is accomplished by correctly identifying X often enough that randomly guessing correctly the same number of times would be statistically improbable.

The A and B subjects for abxtest are actually arbitrary shell commands. The normal mode of use would be to give commands which play two slightly different sounds. The standard output and standard error from each command are suppressed so as not to reveal the identity of X . The listener may interrupt an executing command (usually by pressing ^C ) without terminating the test.

-n min Set the minimum number of trials. Trials will continue after this number if the statistical probability goal has not yet been met. The default is 10.

-m max Set the maximum number of trials. Trials will end after this number even if the statistical probability goal has not been met. The default is 20.

-g goal Set the minimum statistical probability goal. This is the probability that the number of correct identifications is the same as random guesses. In order to disprove the null hypothesis, the result of the test must be a probability less than or equal to this goal. The default is 0.05.

/dev/random By default this file is read to generate a random sequence of X .

RANDOM_FILE If set, the file named by this variable will be read instead of /dev/random to generate a sequence of X .

abxtest uses the binomial distribution to accurately calculate the probability that the result is the same as random guesses. Robert Leslie <rob@mars.org>