Hostname: page-component-cd9895bd7-gxg78 Total loading time: 0 Render date: 2025-01-05T20:21:07.808Z Has data issue: false hasContentIssue false
  • English
  • Français

Online field experiments: a selective survey of methods

Published online by Cambridge University Press:  01 January 2025

Yan Chen  and
Joseph Konstan
Yan Chen*
Affiliation:
University of Michigan, Ann Arbor, MI, USA
Joseph Konstan
Affiliation:
Department of Computer Science and Engineering, University of Minnesota, 200 Union Street SE, Minneapolis, MN 55455, USA
*
e-mail: yanchen@umich.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

The Internet presents today’s researchers with unprecedented opportunities to conduct field experiments. Using examples from Economics and Computer Science, we present an analysis of the design choices, with particular attention to the underlying technologies, in conducting online field experiments and report on lessons learned.

Keywords

Online field experimentA/B testingInternet

JEL classification

H41: Public Goods C93: Field Experiments
Type
Original Paper
Information
Journal of the Economic Science Association , Volume 1 , Issue 1 , July 2015 , pp. 29 - 42
Copyright
Copyright © Economic Science Association 2015

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bakshy, E., Eckles, D., & Bernstein, M. S. (2014). Designing and deploying online field experiments. In Proceedings of the 23rd International Conference on World Wide Web, WWW ’14 ACM New York, NY, USA, pp. 283–292.CrossRefGoogle Scholar
Bakshy, E., Eckles, D., Yan, R., & Rosenn, I. (2012). Social influence in social advertising: Evidence from field experiments. In Proceedings of the 13th ACM Conference on Electronic Commerce, EC ’12 ACM New York, NY, USA, pp. 146–161.CrossRefGoogle Scholar
Boudreau, K. J., & Lakhani, K. (2011). The confederacy of heterogeneous software organizations and heterogeneous developers: field experimental evidence on sorting and worker effort. doi:10.2139/ssrn.1898277CrossRefGoogle Scholar
Chen, Y., Li, X., & MacKie-Mason, J. (2006). Online fund-raising mechanisms: A field experiment. Contributions to Economic Analysis and Policy, Berkeley Electronic Press, 5(2), Article 4.Google Scholar
Chen, Y. F. M. H., Konstan, J., Li, S. X. (2010). Social comparisons and contributions to online communities: A field experiment on movielens. American Economic Review, 100(4), 1358–1398. 10.1257/aer.100.4.1358CrossRefGoogle Scholar
Chen, Y., Teck-Hua, H., Kim, Y.-M. (2010). Knowledge market design: A field experiment at Google Answers. Journal of Public Economic Theory, 12(4), 641–664. 10.1111/j.1467-9779.2010.01468.xCrossRefGoogle Scholar
Chen, R., Chen, Y., Liu, Y., & Mei, Q. (2015). Does team competition increase pro-social Lending? Evidence from online microfinance. Games and Economic Behavior. doi:10.1016/j.geb.2015.02.001CrossRefGoogle Scholar
Cosley, D., Frankowski, D., Terveen, L., & Riedl, J. (2007). SuggestBot: Using intelligent task routing to help people find work in wikipedia. In Proceedings of the 12th international conference on Intelligent user interfaces, pp. 32–41. Downloaded on February 23, 2003 at http://www.communitytechnology.org/nsf_ci_report/.Google Scholar
Friedman, E. J., Resnick, P. (2001). The social cost of cheap pseudonyms. Journal of Economics and Management Strategy, 10(2), 173–199. 10.1162/105864001300122476Google Scholar
Gee, L. K. (2014). The More You Know: Information Effects in Job Application Rates by Gender in A Large Field Experiment. Tufts University Manuscript.Google Scholar
Gerber, A. S., Green, D. P. (2012). Field experiments: Design, analysis, and interpretation, New York: WW Norton & Company, Inc.Google Scholar
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin and Review, 19(5), 847–857. 10.3758/s13423-012-0296-9CrossRefGoogle ScholarPubMed
Halfaker, A., Song, B., Stuart, D. A., Kittur, A., & Riedl, J. (2011). NICE: Social translucence through UI intervention. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration, WikiSym ’11 ACM New York, NY, USA, pp. 101–104.CrossRefGoogle Scholar
Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008). Predictors of answer quality in online Q&A sites. In CHI ’08: Proceeding of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, ACM New York, NY, pp. 865–874.CrossRefGoogle Scholar
Harrison, G. W., List, J. A. (2004). Field experiments. Journal of Economic Literature, 42(4), 1009–1055. 10.1257/0022051043004577CrossRefGoogle Scholar
Horton, J. J., Rand, D. G., Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. 10.1007/s10683-011-9273-9CrossRefGoogle Scholar
Johnson, G. A., Lewis, R. A., & Reiley, D. (2015). Location, location, location: repetition and proximity increase advertising effectiveness. http://www.davidreiley.com/papers/LocationLocationLocation.pdf.Google Scholar
Karau, S. J., Williams, K. D. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65, 681–706. 10.1037/0022-3514.65.4.681CrossRefGoogle Scholar
Kast, F., Meier, S., & Pomeranz, D. (2011). Under-savers anonymous: Evidence on self-help groups and peer pressure as a savings commitment device. Working Paper, Columbia Business School.Google Scholar
Katok, E., Kwasnica, A. M. (2008). Time is money: The effect of clock speed on seller’s revenue in Dutch auctions. Experimental Economics, 11(4), 344–357. 10.1007/s10683-007-9169-xCrossRefGoogle Scholar
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’13 ACM New York, NY, USA, pp. 1168–1176.CrossRefGoogle Scholar
Lacetera, N., Macis, M., Slonim, R. (2012). Will there be blood? Incentives and displacement effects in pro-social behavior. American Economic Journal: Economic Policy, 4(1), 186–223.Google Scholar
Leider, S., Mobius, M. M., Rosenblat, T., Do, Q.-A. (2009). Directed altruism and enforced reciprocity in social networks: How much is a friend worth? Quarterly Journal of Economics, 124(4), 1815–1851. 10.1162/qjec.2009.124.4.1815CrossRefGoogle Scholar
Lerner, J. S., Gonzalez, R. M., Small, D. A., Fischhoff, B. (2003). Effects of fear and anger on perceived risks of terrorism a national field experiment. Psychological Science, 14(2), 144–150. 10.1111/1467-9280.01433CrossRefGoogle ScholarPubMed
Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., et al. (2005). Using social psychology to motivate contributions to online communities. Journal of Computer-Mediated Communication, 10(4). doi:10.1111/j.1083-6101.2005.tb00273.xCrossRefGoogle Scholar
List, J. A. (2008). Informed consent in social science. Science, 322, 672.Google Scholar
Liu, T. X., Yang, J., Adamic, L. A., Chen, Y. (2014). Crowdsourcing with all-pay auctions: a field experiment on Taskcn. Management Science, 60(8), 2020–2037. 10.1287/mnsc.2013.1845CrossRefGoogle Scholar
Liu, Y., Chen, R., Chen, Y., Mei, Q., & Salib, S. (2012). “I loan because…”: Understanding motivations for pro-social lending. In Proceedings of the fifth ACM international conference on Web search and data mining, WSDM ’12 ACM New York, NY, USA, pp. 503–512.CrossRefGoogle Scholar
Lohr, K. N., Brook, R. H., Kamberg, C. J., Goldberg, G. A., Leibowitz, A., Keesey, J. et al., (1986). Use of medical care in the RAND health insurance experiment: Diagnosis-and service-specific analyses in a randomized controlled trial. Medical Care, 24(9 Suppl), S1–S87.Google ScholarPubMed
Lucking-Reiley, D. (1999). Using field experiments to test equivalence between auction formats: Magic on the internet. American Economic Review, 89(5), 1063–1080. 10.1257/aer.89.5.1063CrossRefGoogle Scholar
Munson, S., Lee, S., & Resnick, P. (2013). Encouraging reading of diverse political viewpoints with a browser widget. In International AAAI Conference on Weblogs and Social Media, ICWSM 2013, Boston, USA.Google Scholar
Pallais, A. (2014). Inefficient hiring in entry-level labor markets. American Economic Review, 104(11), 3565–3599. 10.1257/aer.104.11.3565CrossRefGoogle Scholar
Reiley, D. H., Li, S.-M., & Lewis, R. A. (2010). Northern exposure: A field experiment measuring externalities between search advertisements. In Proceedings of the 11th ACM Conference on Electronic Commerce, EC’10 ACM New York, NY, USA, pp. 297–304.CrossRefGoogle Scholar
Reinecke, K., & Gajos, K. (2015). LabintheWild: Conducting large-scale online experiments with uncompensated samples. In Computer supported cooperative work and social computing (CSCW), Vancouver, BC, Canada.CrossRefGoogle Scholar
Resnick, P., Adar, E., & Lampe, C. What social media data we are missing and how to get it. The Annals of the American Academy of Political and Social Science (forthcoming).Google Scholar
Resnick, P., Zeckhauser, R., Swanson, J., Lockwood, K. (2006). The value of reputation on eBay: A controlled experiment. Experimental Economics, 9(2), 79–101. 10.1007/s10683-006-4309-2CrossRefGoogle Scholar
Sen, S., Lam, S. K., Rashid, A. M., Cosley, D., Frankowski, D., Osterhouse, J., et al. (2006). Tagging, communities, vocabulary, evolution. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, ACM, pp. 181–190.CrossRefGoogle Scholar
Varian, H. R. (2014). Big data: New tricks for econometrics. The Journal of Economic Perspectives, 28(2), 3–27. 10.1257/jep.28.2.3CrossRefGoogle Scholar
Williams, D., Ducheneaut, N., Xiong, L., Zhang, Y., Yee, N., Nickell, E. (2006). From tree house to barracks the social life of guilds in world of warcraft. Games and Culture, 1(4), 338–361. 10.1177/1555412006292616CrossRefGoogle Scholar