It is a cliché that the Internet has opened up new ways for HCI researchers to conduct human subjects research. Yet remote, Internet-based experimentation is not yet part of the standard research toolkit because of the concerns about data validity, feasibility of recruiting representative participant samples, and perceived technical barriers. We are developing and validating tools and methods to enable large-scale empirical research with human subjects. Our core goals are to enable a much faster theory-to-experiment cycle, to facilitate access to larger and more diverse participant populations, and to enable larger scale experimentation (in terms of numbers of conditions and experiments) than what is feasible with lab-based methods.

In one project, we developed a system for collecting lab-quality measurements of a person's motor performance by unobtrusively monitoring his or her natural usage of input devices in situ. Our system uses a rare machine learning technique (learning from positive and unlabelled examples) to automatically filter all of user's mouse pointer trajectories for those representing deliberate, targeted movements that can be used to compute standard measures of motor performance. Our results show that, on four distinct metrics, the data collected in-situ and filtered with our system closely matches the results obtained from the same participants in a formal experiment.

In another project, we developed LabintheWild.org — a platform for conducting behavioral experiments with unpaid online volunteers. Volunteers from all over the world to participate in LabintheWild studies in exchange for interesting personalized feedback. Over the past three years, LabintheWild has attracted over 2.7 million distinct visitors from over 200 countries and resulted in approximately 1 million completed experimental sessions. We have validated this platform by demonstrating that results obtained on LabintheWild match those obtained in traditional laboratory settings. LabintheWild has made it possible for us to conduct research that would not have been feasible with traditional methods. I will summarize the findings from several experiments conducted on LabintheWild and I will synthesize the emerging set of best practices for designing studies that attract intrinsically motivated participants and for ensuring validity of the data.