The Continued Prevalence of Dichotomous Inferences at CHI

Made with EulerAPE.

This page is now on OSF.

Researchers

• Lonni Besancon (Linköping University)
• Pierre Dragicevic (Aviz, Inria)

Overview

Dichotomous inference is the classification of statistical evidence as either sufficient or insufficient. It is most commonly done through null hypothesis significance testing (NHST). Although predominant, dichotomous inferences have proven to cause countless problems. Thus, an increasing number of methodologists have been urging researchers to recognize the continuous nature of statistical evidence and to ban dichotomous inferences. We wanted to see whether they have had any influence on CHI. Our analysis of CHI proceedings from the past eight years suggests that they have not.

Publications

Lonni Besançon, Pierre Dragicevic, The Continued Prevalence of Dichotomous Inferences at CHI. CHI '19 - Proceedings of CHI Conference on Human Factors in Computing Systems Extended Abstracts, May 2019, Glasgow, United Kingdom. 2019,

Survey at CHI 2019

During our CHI 2019 presentation of the paper The Continued Prevalence of Dichotomous Inferences at CHI, we asked the audience to fill in a google form to possibly better direct a manual analysis of CHI papers.

Supplementary material

• Data and R analyses (zip). The code for producing the plots and computing Cohen's Kappa, based on results from the coding and parsing process.
• Figures (zip). Our paper figures in PDF format.
• Python parsing scripts (zip). The code for parsing CHI papers. Copyright laws prevent us from re-distributing the text-converted CHI articles needed as input. They are available upon request: please contact Lonni Besançon.
• Presentation alt.chi 2019.

Additional Ressources - How to avoid dichotomous interpretations

Review of the alt.chi submissions highlighted the need for more guidelines/examples on how to avoid dichotomous interpretations.

Examples of studies with nuanced p-value interpretations

This list is not meant to be exhaustive but is taken from Stuart H. Hurlbert & Celia M. Lombardi's Final Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the neoFisherian which is unfortunately paywalled. Most of the papers below are also unfortunately paywalled. Frank Harrell's blog post also suggests guidelines on how to achieve nuanced p-value interpretations.

• Microcosm analysis of salinity effects on coastal lagoon plankton assemblages (1993)
• Salinity and fish effects on Salton Sea microecosystems: benthos (1998)
• Further studies on the saline lakes of the eastern Paroo, inland New South Wales, Australia (1998)
• The benthic invertebrates of the Salton Sea: distribution and seasonal dynamics (2002)
• Organochlorine pesticides, polychlorinated biphenyls, metals, and trace elements in waterbird eggs, Salton Sea, California (2007)
• Coping with multiple stressors: physiological mechanisms and strategies in fishes of the Salton Sea (2007)
• Spatial and temporal patterns of transparency and light attenuation in the Salton Sea, California, 1997–1999 (2007)
• Ovulation Order Mediates a Trade-Off between Pre-Hatching and Post-Hatching Viability in an Altricial Bird (2008)
• Misprescription and misuse of one‐tailed tests (2009)

Examples of studies without p-values and presenting nuanced interpretations

An updated list can be found on aviz.fr/badstats/papers