Who Participates in Focus Groups? Diagnosing Self-Selection

Abstract

Focus groups have become increasingly popular in political science alongside the growth in field experimental and other causal inference-oriented work in comparative politics. Yet, scholars rarely provide details about recruitment processes and descriptive statistics on focus-group participants. This situation is problematic given the likelihood of self-selection and the fact that scholars often use focus groups to pretest or refine experimental treatments or survey questionnaires. By leveraging a series of focus groups that were recruited from a pool of large-N survey respondents, this article demonstrates a method for assessing which variables drive the decision to participate. I recommend that scholars diagnose self-selection into focus groups whenever possible; that they compare participants to relevant baselines when working with samples of convenience; and that they always provide descriptive statistics and details on how focus-group members were recruited.

Focus groups, once an uncommon method in political science research, have become increasingly popular alongside the growth of causal inference-oriented studies in comparative politics. As summarized in table 1, I identified 36 articles published in the American Political Science Review, American Journal of Political Science, and Journal of Politics from 2013–2022 that convey findings from original focus groups versus only four articles that Cyr (2016) found in the first two journals during the previous decade. The online appendix lists these articles and describes the analysis in more detail. These articles used focus groups as part of a multimethod research design, typically involving field or survey experiments (i.e., 22 of 36 articles); none relied solely on focus groups or combined them primarily with other qualitative methods. The studies were conducted exclusively in countries of the Global South and in research in the subfield of comparative politics or its intersection with international relations. Focus groups were used most commonly to inform a study’s research design—for example, developing or pretesting experimental treatments or survey instruments—and for purposes of triangulation, providing qualitative evidence to bolster quantitative findings or reveal causal mechanisms.

Table 1 Research Using Focus Groups, 2013–2022: Descriptive Statistics

Notes: Figures for Purpose double-count one article that used focus groups for both triangulation and as an outcome measure. Figures for Combined With double-count one article that used observational analysis plus a survey experiment. The field-experiment category includes one lab-in-the-field experiment.

Despite the growing popularity of focus groups in journals and broader research designs with rigorous methodological standards, scholars rarely include much about the methodology used to conduct them. Most articles convey focus-group findings only briefly, often in a mere sentence or two. On average across these studies, the text that conveyed focus-group findings and design details comprised only 1.8% of an article’s overall word count.

Despite the growing popularity of focus groups in journals and broader research designs with rigorous methodological standards, scholars rarely include much about the methodology used to conduct them.

In particular, scholars often include little or nothing about how focus-group participants were recruited or the sample on which their findings are based. Only eight of 36 articles described the focus-group recruitment process in either the main text or an online appendix. Only three articles provided descriptive statistics on focus-group participants, and none compared them to a relevant baseline—such as the sampling frame from which they were drawn or the participants in large-N components of the study. This scant attention to focus-group recruitment contrasts with the extensive detail that authors typically provide when describing how subjects were recruited for original surveys and field experiments.

Leaving focus-group recruitment as a “black box” is problematic because of the potential for participant self-selection. In contrast to answering a survey, which may take 30 minutes and never require leaving one’s house, focus-group participation involves traveling to and from a common gathering location and remaining there for approximately 2 hours; therefore, logistical or time constraints may be a disincentive. Talking about politics with strangers is not everyone’s favorite activity, so people also may opt out due to shyness or a lack of interest in the subject matter.

Given how focus groups are used in political science research, selection bias is a potential concern regardless of whether a researcher samples randomly or purposively. Simple random sampling from a general population is rarely used for focus groups, given the potential for small-N sampling error, the concern for group dynamics that often implies recruiting relatively homogeneous participants, and scholars’ theoretical interest in respondents with particular characteristics (Cyr 2019; Fern 2001; Hennink 2014; Krueger and Casey 2014; Liamputtong 2011; Morgan 2019; Van Ingelgom 2020). However, unless a researcher has a theoretical or methodological reason to recruit participants who are highly interested in the topic, are outspoken or gregarious, and find it convenient to attend, these are not desirable characteristics to have emerge in a focus-group sample.

As a small-N method with nonrandomly selected participants, focus-group findings typically are not generalized to a larger population (Cyr 2019; Hennink 2014; Krueger and Casey 2014; Morgan 2019; Morgan and Scannell 1998; Van Ingelgom 2020). Yet, in political science, focus groups typically are used to inform research on a broader study population, and scholars may seek to generalize those findings beyond the study. For these reasons, undiagnosed self-selection can be particularly problematic. For example, focus groups are routinely used to pilot or refine experimental treatments or survey instruments to ensure that they are relevant and easily understood by study participants. If focus-group members are disproportionately interested in and knowledgeable about politics, their feedback may offer misleading conclusions about how effective particular interventions or measures will be with a broader population.

Self-selection may not always introduce major biases into a focus-group sample; in the following example, I argue that it does not. The problem may be more severe in fields other than political science, where it is more common to recruit vulnerable or hidden populations (e.g., intravenous drug users). However, regardless of their theoretical expectations of self-selection, scholars using focus groups risk ignoring a potentially important source of bias if they fail to diagnose its severity and possible consequences.

Methodological research on focus groups has largely neglected this issue of selection bias. Studies on this method routinely describe the plusses and minuses of different recruitment strategies. However, the major concerns are purely practical: ensuring that enough participants show up and are willing to talk to one another in order to hold a group discussion (Barbour 2018; Cyr 2019; Hennink 2014; Krueger and Casey 2014; Liamputtong 2011; Morgan and Scannell 1998; Van Ingelgom 2020; Wallace, Goodyear-Grant, and Bittner 2021). Some texts refer in passing to possible concerns about self-selection and its implications for sample composition. For example, Stewart and Shamdasani (2015, 66) noted that “growing ‘time poverty’ raises some concerns about the lifestyle representativeness of individuals who do show up for focus groups.” Yet, none of this methodological literature offers practical advice on diagnosing the severity and potential consequences of focus-group self-selection.

The scant attention to sample-selection processes in focus-group research contrasts with the extensive focus on self-selection and nonresponse bias in the survey research methods literature (Berinsky 2007; Groves 2006; Malhotra and Krosnick 2007; Wagner 2012). General survey methods textbooks routinely devote a chapter or more to these topics (Fowler 2013; Groves et al. 2009; Lohr 2022), and specialized books focus on them exclusively (Caughey et al. 2020; Groves and Couper 2012; Sarndal and Lundström 2005).

Selection processes may seem to be a more natural concern in large-N quantitative research, which typically aims to generalize to a broader population. Yet, case-selection strategies and the threat of selection bias also are central topics in the small-N, qualitative methods literature (Collier, Mahoney, and Seawright 2004; Geddes 1990; Gerring and Cojocaru 2016; King, Keohane, and Verba 1994; Seawright and Gerring 2008). Focus-group scholars should give attention to selection processes as well.

ASSESSING THE FOCUS-GROUP SELECTION PROCESS

This section provides an example of how scholars can diagnose what drives the decision to participate in focus-group research. I leveraged a series of focus groups whose members were recruited from a pool of large-N survey respondents, which allowed me to characterize both participants and nonparticipants.

In September–November 2022, I conducted three focus groups as part of a project examining the political attitudes of Brazilian immigrants in the Boston area. The research sought to understand why Boston-area Brazilians were overwhelmingly supportive of right-wing populist Jair Bolsonaro and what role that conservative religion—particularly evangelicalism—played in their attitude formation. The centerpiece of the study was an exit poll of Brazilians who voted in person in their country’s 2022 presidential election at expatriate polling places set up by the Brazilian consulate. Our research team surveyed voters on Election Day for both the first round (October 2) and the runoff (October 30). We also conducted a pretest of the survey at the 2022 Brazilian Independence Day Festival in Boston in early September. The survey took the form of a self-administered Portuguese-language paper questionnaire that respondents filled out and returned to enumerators. The questionnaire asked about their experiences as migrants and their attitudes about Brazilian and American politics.

Respondents were invited to write their name, telephone, and email address on a tear-off sheet to receive a potential invitation to a focus group; 45% of respondents did so. Focus-group participants were offered $50 gift cards as compensation for their time and travel expenses. I used the following recruitment text:

Many thanks! Would you like to participate in a discussion group in Portuguese to talk in greater depth about these issues? We are going to organize groups on the coming weekends. Your participation would last between 1.5 and 2 hours and you would receive a $50 gift card. If you want to receive an invitation to a discussion group, leave your information so we may contact you. This form will be separated from your answers above to maintain anonymity.

Based on previous research, I anticipated—and ultimately found—that being an evangelical Christian was a strong predictor of Brazilian migrants’ support for Bolsonaro. The focus groups sought to understand why evangelicals were such strong Bolsonaro supporters and whether the dynamics were any different for their Catholic counterparts. After each round of the survey, including the pretest at the September festival, a Brazilian-American research assistant invited respondents who were 30 to 70 years old and were churchgoing, Bolsonaro-supporting Christians to participate in a focus group. We invited 22 respondents to the first focus group on September 25, 48 to the second on October 15, and 77 to the third on November 19. Between nine and 11 respondents RSVP’d and six showed up to each focus group. Focus-group discussions were held in meeting rooms of local public libraries on Saturday mornings and Sunday afternoons and ran for 1.5 to 2.5 hours.

Because focus-group participants were recruited from among the survey respondents, I was able to compare them on observable characteristics to those who were invited but did not attend. I also could examine the characteristics of survey respondents who left their contact information to receive a potential focus-group invitation versus those who did not. A table in the online appendix presents descriptive statistics for these four groups, along with p-values corresponding to difference-in-means tests. In addition to variables measured directly in the survey, I calculated the distance from the respondent’s self-reported hometown to the relevant focus-group location.

Descriptive statistics suggest that in the present study, self-selecting into the focus-group–eligible sample and attending after having been invited do not introduce major biases. For two of the three survey rounds, respondents who left their contact information had significantly higher levels of interest in Brazilian politics. However, the difference was substantively small: approximately a third of a standard deviation of the interest variable in each case. Moreover, survey respondents as a whole already were highly interested in Brazilian politics—an average of 3.5 on a 1–4 scale—which is unsurprising because most were surveyed after voting in a home-country election from the United States. Other differences showed up in only one of the three groups.

Table 2 presents results from logistic regressions of the decision to leave contact information and to attend the focus group once invited.¹ These regressions pool data from all three rounds of the survey, with fixed effects for each round. Non-Christians (with Catholics as the baseline category), men, and respondents with more interest in Brazilian politics were more likely to opt into the focus-group–eligible sample by leaving their contact information. Among those invited to the focus group, the only significant predictor of showing up was church attendance. Although I avoided scheduling focus groups on Sunday mornings, some churches hold worship services at other times—and frequent churchgoers also are likely to have other church-related commitments on the weekends. ² Unexpectedly, distance from the focus-group location was not significant in either model. Income also was not a significant predictor, which suggests that compensation was neither coercive nor caused undue influence, per Institutional Review Board (IRB) guidelines. ³

Table 2 Predictors of Boston Brazilians Project Focus-Group Selection

Notes: Entries are logistic regression coefficients with standard errors in parentheses. Group fixed effects are estimated but not reported. *p<0.05; **p<0.01; ***p<0.001.

Observations from the focus groups comport with these quantitative findings. Some participants drove significant distances to attend; two came from neighboring states, an hour or more from Boston. Most participants clearly were interested in politics and enjoyed talking about it. Of course, the quantitative analysis shows that political interest influenced self-selection at the stage of opting into the focus-group–eligible sample but not the decision to show up after being invited. This is because survey respondents as a whole—and especially those who left their contact information—already had high levels of interest in Brazilian politics. One participant remarked that her friend also received an invitation and wanted to attend but had church-related conflicts on Saturday, underscoring the difficulty of recruiting people who are highly active in their congregation. Nevertheless, we did recruit some participants who attend church more than once a week. Gathering a set of respondents that spans the range of relevant variables generally is considered more important in focus-group research than recruiting a representative sample (Barbour 2018; Krueger and Casey 2014).

Based on this analysis, it appears that different factors influenced the decision to opt into a focus-group–eligible sample and to attend once invited. Variables associated with wanting to share one’s opinions in a group setting—gender and interest in the topic being discussed—mattered for opting in. Once those who expressed interest were invited to join a focus group at a particular time and place, practical considerations (e.g., conflicting commitments on the weekend) influenced the decision to participate. In the focus groups conducted for the Boston Brazilians project, concerns about selection bias were relatively minor. However, in analyzing the focus-group data, we might place greater weight on the opinions of those in the highest category of church attendance.

Variables associated with wanting to share one’s opinions in a group setting— gender and interest in the topic being discussed—mattered for opting in. Once those who expressed interest were invited to join a focus group at a particular time and place, practical considerations (e.g., conflicting commitments on the weekend) influenced their decision to participate.

CONCLUSION AND RECOMMENDATIONS

Diagnosing what drives the decision to participate in a focus group should be relatively straightforward for some recruitment methods. Lyall, Zhou, and Imai (2020), for example, recruited focus-group participants from among subjects in a field experiment; therefore, the characteristics of those who opted in or out certainly would be known. When outsourcing recruitment to a firm or collaborating with an organization that maintains lists of potential participants, a researcher may be able to access de-identified data on the sampling frame from which focus-group members were drawn.

In many other instances, focus-group participants constitute—often by necessity—a sample of convenience, such that the sampling frame cannot be characterized. For example, Lindsey (2022) asked village chiefs in the Democratic Republic of the Congo to select local residents for focus groups; drawing up a broader list from which to sample would have impractically lengthened fieldwork. In such cases, scholars should strive to compare the basic demographic and political characteristics of focus-group participants to the broader population of a city, region, or country, drawing on census or survey data. Cyr (2017), for example, noted that her focus-group participants were disproportionately well educated and, in one country, right leaning; these comparisons allowed her to assess any potential biases that might result. Scholars also could compare focus-group members to participants in large-N components of the same study. Even without a formal assessment of the selection process, comparing focus-group participants to a relevant baseline would be a major improvement over current practice; as noted previously, none of the 36 articles examined for this study did so. At the very least, scholars should always report the basic descriptive statistics of focus-group participants. This is quite easily accomplished, requiring nothing more than a short survey at the beginning of each focus-group session; however, it is not yet common practice.

Finally, all scholars using focus groups should describe their recruitment methods in sufficient detail so that readers can understand what was done and could replicate the process if desired. Cyr (2016, 2019) offered a series of valuable recommendations for increasing the transparency and replicability of focus-group research. Key among them was describing the profile and training of the moderator and the questions used to guide the focus-group discussion. How participants were recruited should be added to the list of recommendations.

…scholars should always report the basic descriptive statistics of focus-group participants…all scholars using focus groups should describe their recruitment methods in sufficient detail so that readers can understand what was done and could replicate the process if desired.

Transparency is a broadly supported norm in political science, especially among the editors of leading journals, and it has given rise to an important set of standards intended to facilitate the replicability of research (Bonneau and Kanthak 2015; Lupia and Elman 2014). Some scholars in the interpretivist tradition emphasize an alternative perspective—reflexivity—whereby researchers seek to be honest about their relationship to research subjects and personal role in the production of knowledge (Schwartz-Shea and Yanow 2016; Soedirgo and Glas 2020). Reporting how focus-group participants were recruited, as well as who they are in the aggregate, is in keeping with both goals—which seems particularly appropriate for a method that often bridges the qualitative–quantitative divide.