Louis Leon Thurstone in Monte Carlo: creating error bars for the method of paired comparison
Abstract
The method of paired comparison is often used in experiments where perceptual scale values for a collection of stimuli
are desired, such as in experiments analyzing image quality. Thurstone’s Case V of his Law of Comparative Judgments
is often used as the basis for analyzing data produced in paired comparison experiments. However, methods for
determining confidence intervals and critical distances for significant differences based on Thurstone’s Law have been
elusive leading some to abandon the simple analysis provided by Thurstone’s formulation. In order to provide insight
into this problem of determining error, Monte Carlo simulations of paired comparison experiments were performed
based on the assumptions of uniformly normal, independent, and uncorrelated responses from stimulus pair
presentations. The results from these multiple simulations show that the variation in the distribution of experimental
results of paired comparison experiments can be well predicted as a function of stimulus number and the number of
observations. Using these results, confidence intervals and critical values for comparisons can be made using traditional
statistical methods. In addition the results from simulations can be used to analyze goodness-of-fit techniques.