‘Why Do You Ask?’ The Nature and Impacts of Attitudes towards Public Opinion Surveys in the Arab World

Abstract

For the first time in an Arab country, this article examines attitudes toward public opinion surveys and their effects on survey-taking behavior. The study uses original survey data from Qatar, the diverse population of which permits comparisons across cultural–geographical groupings within a single, non-democratic polity. The authors find that Qatari and expatriate Arabs hold positive views of surveys, both in absolute terms and relative to individuals from non-Arab countries. Factor analysis reveals that the underlying dimensions of survey attitudes in Qatar mostly mirror those identified in Western settings, but a new dimension is discovered that captures the perceived intentions of surveys. Two embedded experiments assess the impact of survey attitudes. The results show that generalized attitudes toward surveys affect respondents’ willingness to participate both alone and in combination with surveys' objective attributes. The study also finds that negative views about survey reliability and intentions increase motivated under-reporting among Arab respondents, whereas non-Arabs are sensitive only to perceived cognitive and time costs. These findings have direct implications for consumers and producers of Arab survey data.

Public opinion data collected in the Arab Middle East and North Africa (MENA) remain scarce relative to the survey data available from most other world regions, and these data are often accompanied by concerns about reliability. A majority of Arab states are non-democratic (Bellin 2004; Bellin 2012), and this closed political environment is a major source of both the obstacles facing survey practitioners working in the region and of reservations among consumers of Arab survey data (Benstead 2018). The authoritarian character of most MENA states limits where scientific surveys can be conducted, who can administer them, what kinds of samples can be drawn, and the types of questions that can be asked (Clark 2006; Pollock 1992; Pollock 2008; Tessler 1987; Tessler 2011). Such limitations have in turn fueled concerns about the representativeness of polls (Farah 1987), about respondents’ ability and willingness to give accurate and truthful answers (Pollock 2008; Tessler and Jamal 2006), and even about the ethical implications of carrying out surveys in settings where research may be monitored (Carapico 2006).

Yet despite the scope and persistence of questions about MENA survey quality, producers and users of Arab opinion data still have a limited understanding of exactly how the region's political climate may influence participation in survey research. Previous studies have sought to detect and account for bias introduced by specific characteristics of surveys, such as the survey sponsor (Corstange 2014; Corstange 2016; Gordoni and Schmidt 2010), the observable attributes of survey enumerators (Benstead 2014a; Benstead 2014b; Blaydes and Gillum 2013) and the presence of third parties during the interview (Diop, Le, and Traugott 2015; Mneimneh et al. 2015). However, theories of survey-taking behavior (for example, Hox, de Leeuw and Vorst 1995; Loosveldt and Storms 2008) suggest a more pervasive possibility: that political conditions in the Arab world may instill negative perceptions of the entire survey research enterprise – whatever the characteristics of an individual survey – and these general attitudes toward surveys may influence participation and response behavior in ways that produce misleading results.

This article extends prior research into the nature and effects of survey attitudes both geographically and substantively to the case of the Arab world. In Europe and North America, general attitudes toward surveys have been found to predict a multitude of respondent behaviors, including non-response and refusal, panel attrition and participation intentions. Moreover, this literature has identified conceptually distinct and empirically separable dimensions of survey attitudes, which allows us to test specific mechanisms linking elements of the MENA survey climate to Arab survey behavior. For instance, if Arabs do indeed tend to view surveys negatively, to what extent is this because survey results are seen as irrelevant to policy making and thus of no societal value? Or is survey research viewed as an unreliable method in light of political censorship and a limited history of scientific polling? Or, yet again, are negative evaluations of survey research due to the association of opinion polls with political surveillance or manipulation?

In what follows, we seek to answer these and other questions using data from a nationally representative face-to-face survey conducted in the Arab Gulf state of Qatar, a highly diverse society that permits comparison of attitudes across cultural–geographical groupings within a single, non-democratic polity. Our analysis proceeds in several steps. We first use factor analysis to assess the extent to which the individual components of survey attitudes in Qatar match those previously identified in Western settings. Next, we test whether citizen and expatriate Arabs in Qatar hold attitudes toward public opinion surveys that differ from those held by non-Arabs in Qatar, and we conduct these tests for each of the attitudinal dimensions we identify. Finally, we report on two novel experiments designed to gauge the impact of survey attitudes on survey participation, as well as cross-group differences in such effects. A conjoint experiment evaluates the impact of objective survey attributes alongside subjective survey attitudes on intentions to participate in a hypothetical survey. A screening experiment estimates a respondent's likelihood of completing our survey and the degree to which this is a function of survey attitudes. To our knowledge, our study represents the first assessment of survey attitudes in an Arab country, and one of the very few such studies in a non-Western setting.¹

Context and Literature Review

Trends and Challenges in the Arab World

Until the 2000s, the investigation of individual attitudes, values and behaviors was the missing dimension in political science research on the Arab world. Such research was limited with respect to the countries where surveys could be conducted, the degree to which representative national samples could be drawn, and the extent to which sensitive questions about society and politics could be asked. Complaints about this situation date to the 1970s.²

Although parts of the Arab world remain inhospitable to survey research, significant changes have occurred in the last two decades. First, there has emerged a number of Arab academic research institutions dedicated to conducting social scientific surveys, such as the Center for Strategic Studies at the University of Jordan, the Palestine Center for Policy and Survey Research, and the Social and Economic Survey Research Institute at Qatar University. Global interest in Arab public opinion has also increased dramatically. This is reflected in such international, multi-country projects as the World Values Survey (WVS) and the Arab Barometer. Until its third wave, in 2000–2004, the WVS had not been carried out in a single Arab country. In its latest (sixth) wave, in 2010–2014, the WVS surveyed citizens in twelve Arab states, more than in all previous waves combined. The Arab Barometer, since its founding in 2006, has conducted thirty-nine surveys in fifteen different countries and interviewed over 45,000 individuals.³ International polling agencies, such as the Pew Research Center and Gallup, have also become more active in the Arab world. During 2009–2011, for example, Gallup conducted two or more polls, and as many as nine, in each of six Arab states (Gallup 2012).

The Arab world today thus looks very different with respect to public opinion research than it did less than twenty years ago. However, not all of the news is good. The trend toward more openness also has created opportunities for those willing to cut corners on methodology and/or shade their findings to support a political agenda. Some opinion polls in the Arab world, like some polls elsewhere, are not transparent about their methods or report details that belie their representativeness.⁴ In addition, data frequently are not made available for replication and secondary analysis, and data falsification in surveys of Arab countries also has drawn attention lately (Bohannon 2016; Kuriakose and Robbins 2016). These issues are not unique to Arab societies, of course. Nor are they limited to research involving surveys and other quantitative methodologies. But they do represent cautions and concerns that have accompanied the expansion of survey research in the MENA region.

Survey Attitudes and the Survey-Taking Climate

This proliferation of surveys – scientific and unscientific – in Arab countries gives new impetus to questions about how ordinary people in the region view survey research, and how these views might impact survey behavior and data reliability. Such questions, generally described as pertaining to the survey climate, are well known to researchers working in the United States and Europe (Kim et al. 2011). Beginning with Sjoberg's (1955) ‘questionnaire on questionnaires’, a large literature has suggested that ‘a positive survey-taking climate in a population’ is an important precondition for effective survey administration (Loosveldt and Storms 2008, 74; Lyberg and Dean 1992). This consensus is rooted in behavioral theories of survey taking and other forms of action that view respondent cooperation as being influenced by generalized norms and attitudes (Ajzen and Fishbein 1980; Stocké and Langfeldt 2004), as well as cognitive limitations (Krosnick 1991; Krosnick 1999), that defy pure rationality. Individual attitudes toward surveys are thus conceived as ‘the expression of the subjective experience of the survey climate’, representing ‘the link between the survey climate at the societal level and the decision to participate [in a survey] at the individual level’ (Loosveldt and Joye 2016, 73).

In Western settings,⁵ survey attitudes have been found to predict a broad range of respondent behaviors, including non-response and refusal (Groves and Couper 1998; Groves, Presser, and Dipko 2004; Groves, Singer, and Corning 2000; Groves et al. 2001; Hox, de Leeuw, and Vorst 1995; Stocké 2006; Weisberg 2005), panel attrition, following survey instructions, timeliness of response and willingness to participate in surveys (Loosveldt and Storms 2008; Rogelberg et al. 2001; Stocké and Langfeldt 2004; Stoop 2005). Studies spanning a multitude of populations and techniques report a consistent conclusion: more positive views of surveys are associated with greater respondent cooperation, and this in turn improves data reliability by reducing error stemming from non-response, satisficing or socially desirable reporting, motivated under-reporting or other behaviors. These findings are generally accepted despite recognition that survey attitude studies are susceptible to selection effects that may bias assessments of surveys in a positive direction (Goldman 1944; Vannieuwenhuyze, Loosveldt and Molenberghs 2012).

Yet the extent to which insights from existing scholarship on survey attitudes can guide understanding of the MENA survey climate is limited by two important factors – one empirical and one theoretical. First, while extant research has broadly differentiated between general attitudes toward surveys and attitudes toward specific questionnaire formats and modes (Hox, de Leeuw and Vorst 1995), studies of survey attitudes in the West have not produced a consensus on the best way to measure and classify survey attitudes. Scholars have commonly used exploratory and sometimes confirmatory factor analysis to map the structure of survey attitudes. But this has not led to agreement about the number and character of individual attitude dimensions. For instance, Goyder (1986) reduces twelve questions about surveys to four underlying attitudinal factors. Rogelberg et al. (2011) employ six items to arrive at two dimensions. Stocké (2001), meanwhile, constructs an eight-item ‘Attitudes towards Surveys Scale’, the components of which load highly on a single factor.

Nor have meta-analyses led to agreement. Loosveldt and Storms (2008) identify from previous studies five different factors that affect individuals’ willingness to participate in a survey. They argue that a respondent's decision about participation:

[W]ill be positive when he or she considers an interview a pleasant activity (survey enjoyment), which produces useful (survey value) and reliable (survey reliability) results and when the perceived cost of cooperation in the interview (time and cognitive efforts; = survey cost) and impact on privacy (survey privacy) are minimal (Loosveldt and Storms 2008, 77).

But more recent work by de Leeuw et al. (2010) retains only three of these factors in their own ‘Survey Attitude Scale’. Thus after six decades of investigation and despite general agreement that more positive attitudes induce greater cooperation and therefore better data, survey researchers working in Western settings with long histories of opinion polling still have not arrived at a common understanding of the elements that constitute survey attitudes.

The second factor limiting the relevance for MENA of existing research on survey attitudes is its theoretical, as opposed to merely geographical, grounding in the Western political context. Indeed, the very first survey on surveys, conducted in 1944, aimed to assess public views about ‘the role of polls in democracy’ (Loosveldt and Joye 2016, 69). This conceptual connection between opinion polling and participatory politics, notably elections, underlies extant studies of survey attitudes, and it begs the question of whether and how the nature and effects of survey attitudes uncovered in the West might differ from those in countries and world regions with very dissimilar political institutions. Additionally, the democratic lens through which scholars have tended to view survey attitudes suggests that previous work may have overlooked other, context-driven concerns of people residing in non-Western and non-democratic settings. These include the use of surveys for purposes of political surveillance and manipulation rather than political participation, and the very political association of survey research with the Western world.

Survey Climate and Political Climate in the MENA Region

It is perhaps natural, then, to ask whether the authoritarian political climate of the Arab world might create an especially inhospitable survey climate. As reported by Sadiki (2009, 252) in his wide-ranging study of autocratic regimes in the Arab world, ‘[i]t is no exaggeration to say that “public opinion” has not had any presence to speak of in the Arabic political vocabulary’. Accordingly, he suggests, the fact that Arab publics are largely excluded from decision making might lead people to value surveys less than do the citizens of more democratic societies in which public opinion has a clearer impact on policy. Other aspects of the Arab world's political environment may also make the region's survey climate inhospitable. A weak capacity for scientific surveys, combined with official restrictions on polling, may make MENA publics more skeptical of survey reliability than publics elsewhere. Privacy concerns surrounding surveys may also be amplified in places, such as the Middle East, where social science research may be surveilled.

In these and other ways, it is possible that the MENA political landscape fosters negative attitudes toward survey research for reasons that are unrelated to its Western connotations. Yet we also theorize that, in the Middle East and perhaps elsewhere, an important and until now neglected dimension of attitudes towards survey research does in fact stem from its multifaceted association with the West. This association has both an epistemological and a political aspect, in addition to the fact that survey research was developed and is most widely used in Western countries. With respect to epistemology, survey research is part of a data collection and analysis toolkit that is centrally concerned with measuring and accounting for variance, particularly at the individual level of analysis. With respect to politics, survey research represents, at least for some in the Arab world, a methodology that has been employed to produce information that can support Western imperial interests in the region.

That such concerns color the MENA survey-taking climate is supported by several recent survey experiments undertaken in Arab countries. These have shown that Arab citizens with higher a priori levels of hostility toward the West are less likely to take part in surveys sponsored by Western governments (Corstange 2014; Corstange 2016), and, similarly, that citizens report more negative views of policies (Bush and Jamal 2015) and political candidates (Corstange and Marinov 2012) when they are endorsed by the United States. We hypothesize and test for a similar but more general mechanism: that viewing surveys as inherently in the service of Western scientific or state interests may dampen survey response and increase the likelihood of early termination. The latter effect may occur if participants form negative judgments about a survey's purpose only after they have started the interview and begun answering questions.

It is also possible that latent attitudes toward survey research may interact with the objective attributes of surveys to produce conditional effects on survey behavior. In particular, worry over the possible misuses of surveys in the Arab political context may generate a kind of mistrust or suspicion, but these attitudes may be activated and impact participation only in combination with specific survey characteristics, such as a survey sponsor or survey topic that Arab respondents are inclined to distrust. This proposition is consistent with experimental results from other authoritarian settings, including Latin America, where respondents have been seen to use the observable attributes of survey enumerators as a heuristic for judging the political intentions of a survey (Bischoping and Schuman 1992). We test for such conditionalities in the analysis to follow.

Data and Case Selection

Our contribution is based on data from the first systematic assessment of survey attitudes in an Arab country. This original and nationally representative survey interviewed 751 citizens and 934 non-citizen residents of the Gulf state of Qatar. Findings from an investigation in any one Arab country cannot, of course, be assumed to characterize all Arab countries, and so caution is necessary when reflecting on the broader applicability of our results. Nevertheless, the case of Qatar offers key methodological and theoretical advantages for a study of survey attitudes in the Arab world.

First, Qatar is a highly diverse society, and its extreme diversity allows comparison across cultural–geographical groupings within a single survey setting. A small, resource-exporting monarchy, Qatar is home to 2.7 million residents, of whom around 300,000 are Qatari citizens (Snoj 2017). The remaining population consists of Arab and non-Arab expatriate workers from around the world. More than fifty countries, including seventeen of the twenty-two member states of the Arab League, are represented in our survey.⁶ Moreover, this expatriate population in Qatar, as elsewhere in the Gulf region, is highly transient. The median length of residence in Qatar among expatriates in our sample is 7 years, meaning that individuals can for the most part be expected to possess the values, traditions and experiences of their home country, including those related to survey research.

Qatar is also a fitting study from a conceptual standpoint, as it is characterized by those country-level features that are commonly cited as barriers to obtaining reliable survey data from the Arab region. These include a lack of democracy and the absence of a survey tradition, coupled with a recent proliferation of surveys. Qatar is a hereditary monarchy rated as having the lowest possible level of democracy for all years since independence according to the widely used Polity IV measure of regime type (Marshall, Jaggers and Gurr 2002). It is also a newcomer to public opinion polling, both in general and relative to most other Arab states. Independent opinion surveys were absent in the country until a decade ago. Qatar therefore fits very well the description of Sadiki quoted earlier: that public opinion traditionally has not had any presence to speak of in the nation's political vocabulary.

However, Qatar has in the last few years moved to the forefront of Arab countries with respect to the quantity and quality of systematic and scientific survey research. In 2008, a survey research institute was established at the national Qatar University and soon began conducting rigorous and nationally representative surveys of both Qatari citizens and the country's expatriate population. Further, given the country's small population, the likelihood of being selected for a survey, or knowing someone who has been selected, is relatively high. Qatar's citizens and resident expatriates have thus had opportunities during the last few years to experience surveys, either directly or indirectly, which has probably led them to form opinions about surveys.

For these reasons, Qatar represents a very appropriate setting in which to observe attitudes toward surveys and the impact of these orientations on survey behavior.

Our survey was conducted face to face in May 2017 by the Social and Economic Survey Research Institute at Qatar University. Households were selected randomly from a comprehensive frame via proportionate stratified sampling, using Qatar's administrative zones for stratification. Individuals were selected through software randomization, with gender pre-specified (Le et al. 2014). The survey was conducted in Arabic or English by bilingual enumerators. The response rate, following the American Association for Public Opinion Research definition RR3, was 35.2 per cent, and the sampling error was 3.1 per cent. Importantly, there was no significant difference in participation rates between Arabs and non-Arabs that might confound analysis of group-based variation in survey attitudes.⁷

Methods

The survey was designed to capture respondent attitudes toward survey research and to offer behavioral measures of the effects of these attitudes. The interview schedule contained items borrowed or adapted from previous investigations, and it also included original questions developed by the authors to explore aspects of surveys that relate to conditions prevailing in the MENA region and thus might not be captured adequately in batteries developed for Western contexts (see Table 1).

Table 1. Items for the six dimensions of survey attitudes in Qatar

Note: E₁, E₂, V₁, V₂, C₁ and C₂ are from the Survey Attitude Scale of de Leeuw et al. (2010); P₁ and R₂ are from Loosveldt and Storms (2008); R₁ and R₃ are similar to other items used by Loosveldt and Storms (2008); and P₂, I₁, I_2, I_3, and I₄ are the authors' creation. Response code 1 corresponds to ‘strongly disagree’ and the maximum category (4/7) to ‘strongly agree.’ Missing values are omitted from the analysis.

In line with previous studies, we utilize factor analysis to explore how items cluster in order to develop a conceptual map of respondent perceptions of surveys. We use exploratory rather than confirmatory factor analysis, both because prior studies have not resulted in an agreed factor structure, and because our theory predicts one or more new attitudinal dimensions connected to the MENA survey climate. Items with high loadings on a common factor after rotation⁸ measure the same underlying concept and, hence, constitute a unidimensional measure. Unidimensionality offers evidence of item reliability, a basis for inferring validity, and justification for combining items into a multi-item index. Due to the number of items in the measurement model and limited subsample sizes, we group all respondents together in the factor analysis. We create variables from predicted factor scores to measure each of the identified dimensions of survey attitudes.⁹ We first use the variables generated from factor analysis as dependent variables to evaluate cultural–geographical group differences in attitudes. Thereafter, we use them as independent variables to investigate the effects of attitudes on survey behavior, conditional on group.

Because we do not observe the survey attitudes of non-participants, we cannot assess the impact of attitudes, either alone or in combination with manipulated survey attributes (cf. Corstange 2014; Corstange 2016), on participation in our survey. Therefore, we embed within our survey two experiments that afford behavioral measures of the effects of the attitude dimensions identified in the factor analysis.

Our first measure comes from a conjoint experiment in which respondents were presented with a description of a hypothetical survey and then asked to rate their likelihood of participating. The conjoint technique simulates the complex nature of the survey participation decision, in which respondents must weigh a combination of pertinent factors. The hypothetical survey was randomly assigned four objective attributes that the literature suggests may affect participation: survey sponsor, topic, mode and length. The survey's sponsor was given as either a Qatari government institution, a university in Qatar, a private company in Qatar or an international agency. The survey's topic was cultural, economic or political. Its mode was either face to face or telephone. Its length was either 10 or 20 minutes for a telephone survey, and 30 or 60 minutes for face-to-face interviews.¹⁰ The selection of treatments was informed both by the relevant scholarly literature, which is based primarily on research in developed democracies, and by our intuition that other, context-driven concerns about surveys might also influence participation in the Arab world. The inclusion of an international sponsor and a political topic reflects the latter consideration. As is typical, respondents were asked to assess three different survey profiles in sequence, in each case rating their intention of taking part as ‘very likely’, ‘somewhat likely’, ‘somewhat unlikely’ or ‘very unlikely’.

Our design is based on the well-known conjoint approach of Hainmueller, Hopkins and Yamamoto (2014) and Hainmueller and Hopkins (2015). Variables were left in their native metric, and the average conditional interactive effects (ACIEs) reported in the following section reflect the effects of treatments in the metric of the dependent variable, conditional on respondent cultural–geographical category. Since all survey characteristics were randomly selected, ACIEs provide an estimate of which attributes have a larger impact than others.

Unlike in most conjoint analyses, however, our interest lies not only in the effects of the experimental treatments, but also in the impact of generalized attitudes that we expect to predict behavior separately from, or in combination with, the treatments. Thus, after considering the attribute-only model of survey participation, we insert the attitude factors as additional independent variables. We also test theoretically motivated interactions between survey attitudes and politically salient survey characteristics.

The second experiment builds on the first by gauging the impacts of survey attitudes on actual respondent behavior within our survey, rather than on hypothetical survey participation. Devised by the authors, the experiment uses a transparent screening question to give respondents an easy option of exiting the interview. Respondents were advised that the final section of the survey, due to its length, would only be completed by half of the respondents, and that the basis for this random selection was their birthday, with only respondents whose birthday fell in the previous six months being asked to continue. Since the distribution of birthdays throughout the year is approximately uniform (McKinney 1966), we interpret deviation from the expected value of 0.5 to be evidence of deliberate falsification, or ‘motivated underreporting’ (Eckman et al. 2014; Tourangeau, Kreuter and Eckman 2012), designed to cut the interview short. We estimate the effects of the survey attitude dimensions on the respondent's likelihood of finishing the survey, as well as differences in these effects across cultural–geographical categories.

In the results sections that follow, we first map the dimensions of survey attitudes among citizens and non-citizen residents of Qatar, and assess how these dimensions correspond to those identified in prior studies of Western populations. We then examine subgroup differences along these dimensions, with respondents divided into five cultural–geographical categories: Qatari (n = 751), Arab non-Qatari (n = 392), South Asian (Indian subcontinent; n = 285), Southeast Asian (Philippines, Malaysia, Indonesia; n = 42) and Western (United States Canada, Europe, Australia; n = 28).¹¹ We next use the results of these mapping operations to determine whether and how survey attitudes influence respondent behavior.

Results: Measuring and Predicting Survey Attitudes

Mapping Survey Attitudes in an Arab Country

We measure survey attitudes in Qatar using fifteen items from the interview schedule. Factor analysis yields a six-factor solution that mostly matches the findings obtained in other contexts. However, the analysis reveals one additional dimension not identified in previous studies. Table 1 presents the components of these six dimensions with means and standard deviations, while Table 2 gives the results of the factor analysis. Between two and four items specified as measures of a particular dimension have high standardized loadings on that dimension. All loadings in the model are significant (p < 0.001), and, except for one dimension, all loadings are above 0.7. This supports the convergent validity of the model. Correlations between the latent factors are low, the highest being between survey enjoyment and survey value (0.56). The highest correlation between any two other factors is 0.27.¹² This indicates that each dimension measures a different aspect of attitudes toward surveys and supports the model's discriminant validity. The values of item reliability (Cronbach's α) are acceptable at 0.67 or above, although some of the composite reliability values are lower.¹³

Table 2. Results of a factor analysis of survey attitudes in Qatar

Note: right-hand column reports Cronbach's α.

As noted, the factors identified in the model largely match those of previous studies conducted in very different settings – most closely those of Loosveldt and Storms (2008), who worked in Belgium. Beyond the notable implication that the dimensions of survey attitudes may not differ much across populations, this result also lends additional confidence to our measurement model. We follow previous work in labeling five of our attitude dimensions: survey enjoyment, survey value, survey reliability, survey costs and survey privacy. A final, sixth factor not previously identified we label as ‘survey intentions’. This dimension is related to the purpose – well intentioned or ill intentioned – of surveys. Taken together, these six factors provide a conceptual map of the survey climate as perceived by individuals in Qatar.

It is instructive that the factor analysis reveals three independent dimensions of attitudes associated with survey burden: one associated with the cognitive or time cost of surveys (survey costs), another related to the invasiveness of surveys (survey privacy), and a final factor related to the possible manipulative or politicized purposes of some surveys (survey intentions). While de Leeuw et al. (2010) have found that concerns about time and privacy load on a single factor, and hence constitute a single dimension of survey burden, respondents in Qatar judge these two separately. One possible reason is the high volume of surveys conducted in Qatar over the previous decade relative to the country's small population, suggesting the proposition that burden associated with time becomes a separate consideration under conditions of high survey exposure.

The survey value and survey reliability factors have also been observed elsewhere and do not require extensive discussion, except to note that they are conceptually distinct from the other dimensions identified. In Qatar, finding surveys burdensome is a separate consideration from questioning either the accuracy or honesty of surveys or the fact that the information they provide can be useful. Similarly, it might have been expected that the skepticism reflected in the survey intentions factor would form a common dimension with negative judgments about the reliability or value of surveys. Somewhat surprisingly, this is not the case.

A final observation concerns the attitudinal dimension exhibited by our respondents in Qatar but not found by studies mapping survey attitudes in Western societies. This is the survey intentions factor, on which the items with high loadings pertain both to foreign methods and interests and to the motivations of those conducting surveys. These items involve a degree of distrust, or at least skepticism, toward surveys, and it is worth pondering the conditions under which attitudes about Western influences and researcher motivation are strongly interrelated and define a dimension of the survey-taking climate that has not been observed in Western contexts.

This finding about the survey intentions factor may stem from Qatar's vast expatriate population, and so apply in countries with similar demographic characteristics. A more likely and also more instructive explanation, however, is that Qatar is an Arab country and the Arab world's relationship with the West has been complex and frequently problematic, and this has often given rise to suspicions about Western and especially American activities in the MENA region (for example, Blaydes and Linzer 2012; Jamal 2012; Katzenstein and Keohane 2007). Previous surveys conducted in Qatar and other Gulf states have revealed citizen worries over Western interference in domestic affairs (Gengler 2012; Gengler 2017). If these are indeed relevant scope conditions, similar findings should be expected in other Arab countries and perhaps other societies with similar attitudes toward the West.

Cross-Cultural Differences in Survey Attitudes

We now consider the extent to which these six dimensions of survey attitudes vary across the distinct cultural–geographical groupings represented in Qatar. We employ an ordinary least squares (OLS) method to estimate differences according to group while controlling for respondent gender, age and education.¹⁴ The baseline subgroup category is set to Arab expatriates, so that coefficients in Table 3 report differences in attitudes relative to the average of a diverse cross-section of the Arab world. This test matches our theoretical interest in understanding whether Arabs as a broad category of respondents possess systematically different orientations toward surveys than other publics. Here and elsewhere, we treat Qatari citizens separately both because they constitute a distinct population that is sampled separately from expatriates, and to capture possible differences in attitudes based on citizenship status or on Qataris' particular social and political experiences.

Table 3. Survey attitudes of cultural-geographical groups in Qatar, relative to Arab non-Qatari baseline

Notes: estimates by OLS regression; p-values in parentheses; ⁺ p < 0.10, * p < 0.05, ** p < 0.01, *** p < 0.001; results of demographic controls and ‘Other’ group category not reported; sampling probability weights utilized.

The results show that, far from being more negatively oriented toward surveys, both citizen and expatriate Arabs in Qatar hold substantially more positive views of surveys than do individuals from Western, South Asian and Southeast Asian nations. This applies to five of the six attitude dimensions. In the cases of survey value and enjoyment, there is little subgroup variation, although respondents from Western countries seem to enjoy surveys less compared to Arabs (p = 0.07). By contrast, regarding survey reliability, all subgroups except for Qataris report more negative evaluations than Arab respondents. Similarly, for the three dimensions of survey burden, Arab respondents perceive surveys as being less burdensome than do Western and Asian residents of Qatar. Western, Southeast Asian and even Qatari respondents report more negative attitudes than Arab expats regarding the time and cognitive costs required by survey participation; the two Asian subgroups express greater concern than Arabs over the privacy implications of surveys; and South Asian, Southeast Asian and – ironically – Western respondents are more likely than Arabs to view surveys as serving partisan and foreign interests.

Table 3 shows how the survey attitudes of Arabs differ from those of other respondent types. A related but separate question is whether the attitudes of Arabs and the other cultural–geographical groupings are substantively positive or negative across the six attitudinal dimensions. That is, the findings reported in Table 3 demonstrate that Arab attitudes toward surveys do indeed differ from those of most other groups in Qatar – albeit not in the direction one might have predicted – but the question remains whether one should characterize Arabs or others as being positively or negatively oriented toward surveys. The findings on this question are presented in Table 4, which reports post-regression predicted values by subgroup for all six survey attitude dimensions. Since the factor variables that represent the dimensions are normalized with a mean of 0 and standard deviation of 1, a predicted value different from 0 can be interpreted as representing positive/high or negative/low survey attitudes, depending on its sign. Predicted values are directly interpretable in standard deviation terms, such that a value of 0.5 corresponds to attitudes that are half a standard deviation more positive than average, for example.

Table 4. Predicted values on survey attitude dimensions, by cultural-geographical group

Notes: columns report post-OLS predicted values; ⁺ p < 0.10, * p < 0.05, ** p < 0.01, *** p < 0.001; sampling probability weights utilized.

Table 4 shows that the Arab residents of Qatar are not only more positively inclined toward surveys compared to other cultural–geographical groups; they also report qualitatively positive attitudes on each survey dimension. Meanwhile, the attitudes of Qatari citizens are substantively neutral on all but one dimension. Notably, this factor is survey intentions, about which Qataris report more positive views than average.

Conversely, the survey attitudes of Westerners in Qatar are negative across all dimensions except survey value, although only one of these results – survey costs – is significant at the standard level of statistical confidence. This is due, at least in part, to the limited size of the Western subsample. South Asian respondents report positive enjoyment of surveys and more positive views than average about the time and effort required of survey participants; yet they hold negative perceptions about survey reliability, privacy and intentions. Finally, members of the Southeast Asian subgroup, like other respondents, appear to attach positive value to surveys, yet they may consider them unreliable (these first two results are not statistically significant), and they find surveys burdensome across all three measured dimensions. In short, Arab residents of Qatar have broadly positive views of surveys, Qatari citizens are mostly neutral, South Asian respondents are mixed, and individuals from Southeast Asian and, to a lesser extent, Western countries have generally negative orientations toward surveys.

Results: The Behavioral Impacts of Survey Attitudes

Survey Attitudes, Attributes and Participation

But how do these attitudes influence Arabs and non-Arabs in Qatar when they are asked to participate in surveys? Here we estimate the behavioral impacts of survey attitudes via our two embedded experiments. We begin with a conjoint model of survey participation that includes only objective survey characteristics: survey mode, length, topic and sponsor. We estimate treatment effects conditional on cultural–geographical category, with all non-Arab respondents grouped together due to insufficient subsample sizes for some groups. We next add to the model the six survey attitude dimensions to examine the extent to which variance in participation is explained by generalized attitudes about surveys compared to the specific attributes of the hypothetical surveys in our experiment. We conclude this analysis by considering how the politically salient survey attributes manipulated in the experiment – topic and sponsor – interact with survey attitudes to affect participation. In particular, we are interested in understanding the possible conditionalities associated with the attitude dimension that has not been identified in previous studies but is an important consideration of survey takers in Qatar, namely survey intentions.

The findings of the attribute-only model, visualized in Figure 1, reveal robust treatment effects across all four survey attributes considered, as well as notable similarities and differences across respondent types.¹⁵ All subgroups prefer shorter surveys. Arab residents and Qatari citizens prefer telephone surveys to face-to-face interviews, whereas the estimated effect of face-to-face mode on participation among non-Arab respondents is also negative but not statistically significant. Compared to the baseline of an economically focused survey, all population groups are substantially less likely to participate if the survey topic is political in nature. Yet the size of this effect is twice as great among Arab and non-Arab expatriates, among whom it is indeed the strongest of any treatment effect. Perhaps unexpectedly, then, citizens in Qatar appear to be less disinclined than Arab non-citizens to participate in political surveys (p = 0.065). This suggests that the negative treatment effects witnessed among non-citizen respondents may be at least partially due to a lack of interest or engagement in local politics, rather than to apprehensions over divulging political opinions in a non-democratic environment.

Figure 1. Effects of survey attributes on participation intentions

A more qualitative disparity emerges in the case of survey sponsor. Non-Arab respondents have no perceptible preferences about who sponsors a survey: they are equally willing to participate if a survey is commissioned by a university, state institution, local private company or international organization. Conversely, relative to the baseline of a university sponsor, both Qatari and non-Qatari Arabs report a much lower willingness to participate in surveys for private companies, and an international sponsor also dampens participation among Qatari respondents. Finally, Qataris are no less likely to participate when a survey is sponsored by a government entity, while Arab expatriates are potentially more likely to participate in such surveys (p = 0.109). Thus there is no general fear among Qatar's citizens or residents of participating in a survey connected to the state.¹⁶ Rather, it is surveys commissioned by companies and/or foreign entities that are associated with a lower willingness to participate among Arab populations in Qatar.

However, the decision to take part in a survey is not an isolated response to the parameters of a given participation request; it is also shaped by overall feelings about surveys. For one who dislikes pizza, as it were, the choice of toppings is irrelevant. To better understand the relative importance of survey attitudes and objective survey attributes in influencing participation, we now add to the conjoint model of participation our six survey attitude dimensions. These are introduced into the model sequentially to demonstrate how coefficient and error estimates change with additional regressors. As before, the effects are estimated conditional on respondent type. Table 5 reports the results of this analysis in the form of post-estimation marginal effects.¹⁷ To ensure the effects are meaningful, columns report estimated changes in participation resulting from a change in survey attitudes from substantively negative (−1 standard deviation) to positive (+1 standard deviation) on that dimension.

Table 5. Marginal effects of survey attitudes on participation, by respondent group

Notes: p-values in parentheses; ⁺ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001; columns report estimated marginal effects on participation evaluated at −1 and +1 standard deviations of the respective survey attitude dimension; standard errors clustered by respondent; sampling weights utilized.

The findings are notable in several respects. First, the effects of the attitude dimensions on participation are on par with those of the most impactful experimental treatments manipulating survey topic and sponsor. Secondly, the inclusion of survey attitudes in the model of participation significantly improves overall fit in the Arab and especially Qatari models, but not in the model of participation for non-Arabs. The full specification that includes all survey attributes and attitudes achieves an adjusted R ² of 0.132 for Qataris, for instance, compared to 0.054 in the attribute-only model. Among non-Arabs, the difference in fit is a mere 0.007.

This disparity highlights another key result: like objective survey attributes, the effects of survey attitudes on participation vary across the three cultural–geographical groupings. Most notably, whereas the perceived reliability of surveys strongly predicts participation for both Arab subgroups irrespective of survey attributes, it exerts no effect among non-Arabs. Participation is also more likely for Qatari citizens and Arab expats when they hold more positive views about the time and cognitive costs of surveys, while again this consideration does not impact non-Arabs independently of the survey attribute treatments. Compared to Arabs, participation by non-Arabs is more contingent upon the perceived privacy implications of surveys. Indeed, the direction and magnitude of the coefficient estimates suggest only two consistent effects on participation: a positive impact of survey enjoyment, and a null effect of perceived survey value.

Finally, the survey intentions dimension is unique in being a negative predictor of participation among citizens and, with a lower degree of statistical confidence, Arab expats (p = 0.106) in Qatar. That is, more positive views about the intentions of surveys are associated with a reduced likelihood of participation. To help elucidate this result, and more generally to understand the possible conditionalities associated with our new attitude dimension not found in previous studies, we investigate the interaction between perceived survey intentions and survey attribute treatments that could signal a negative political purpose. Figures 2 and 3 give the effects of the survey intentions factor conditional on survey sponsor and topic, respectively. In each case, the findings reveal that the intentions dimension depresses participation among Arabs only for surveys whose topics (politics) and sponsors (polling companies and foreign organizations) they tend to disfavor, whereas the intentions dimension never affects participation among non-Arabs.

Figure 2. The impact of survey intentions, by survey sponsor and respondent group

Figure 3. The impact of survey intentions, by survey topic and respondent group

The generally favorable attitudes toward surveys observed among Arabs in Qatar thus turn to unfavorable attitudes under certain conditions, namely when the survey topic or sponsor makes people suspicious of its purpose. If views of survey intentions are negative to begin with, suspicious sponsorship is not required to make Arabs distrustful and dampen participation. But if one is predisposed to be positive, as the majority of men and women in our sample are, that changes only when the sponsor or topic is distrusted, reflecting a sort of dissonance between preconceived attitudes and contradictory feelings prompted by a specific request for survey participation.¹⁸ That it is sponsorship by private polling firms and international entities, rather than the government, that signals negative intentions and so contributes to lowering participation in Qatar accords with the negative effects of these two treatments among Qatari and non-Qatari Arabs observed in Figure 1.

Survey Attitudes and Early Termination

We conclude our study by assessing the effects of survey attitudes on actual rather than hypothetical respondent behavior. Since our design does not allow us to estimate the impact of survey attitudes on participation in the survey for lack of data on non-participants, we instead examine respondent willingness to complete the entire interview schedule when offered an easy exit via a simple filter question. The introduction to the final section of the interview informed respondents that, in order to shorten the survey administration time, half of them would be randomly selected into the final module: those whose birthday fell in the past six months would be asked to continue. For others, the survey would conclude. Having clearly spelled out the purpose and implications of answering this screening question, we expected that respondents whose birthday occurred within the past six months but who wished to terminate the survey might lie about when they were born, thereby ending the survey. The survey did not collect information about the respondent's birth date, so respondents did not need to fear being caught in a lie.

Since birthdays tend to be randomly distributed, it was expected that, absent falsification, approximately 50 per cent of respondents would continue into the final survey section. However, only 39 per cent of respondents reported a birthday in the past six months and thus completed the full schedule. This proportion varies by respondent type, with 35 per cent of Qataris, 39 per cent of Arab expats and 45 per cent of non-Arabs finishing the entire survey. We take this result as an indication that some respondents lied in order to exit the survey. To understand the determinants of such motivated under-reporting, we consider the extent to which survey attitudes predict the likelihood of drop-out.

Our general hypothesis was that respondents with more negative survey attitudes would be more likely to terminate the interview. In fact, three attitude dimensions are unrelated to willingness to continue the survey. These are survey enjoyment, survey value and survey privacy. As shown in Table 6, none is associated with continuation (birthday) for any respondent group. The three remaining factors – survey reliability, survey costs and survey intentions – do have significant predictive power, but only among particular respondent groups.¹⁹

Table 6. Marginal effects of survey attitudes on continuation (birthday), by respondent group

Notes: columns report post-logit marginal effects as percentage changes in the predicted probability of continuation (a birthday in the past six months) evaluated at −1 and +1 standard deviations of the respective survey attitude dimension; * p < 0.05, ** p < 0.01, *** p < 0.001; sampling probability weights utilized.

Among Qatari respondents, survey reliability is the only dimension related to survey completion: Qataris are 30 per cent more likely to enter the final module if they view surveys as being reliable. Survey reliability also predicts the likelihood of continuation among non-Qatari Arabs, who are a striking 75 per cent more likely to finish the interview if they possess high rather than low values on this dimension. Survey intentions also has predictive power among non-Qatari Arab expats, who are 42 per cent more likely to continue if they have positive views of survey intentions. These Arab expatriates are the only respondent category for which more than one survey attitude is related to the willingness to complete the survey.²⁰ Finally, among non-Arab respondents, only the survey costs dimension predicts motivated under-reporting. For this group, the likelihood of continuation increases by 51 per cent when views of the time and cognitive burden of surveys are positive rather than negative. We had expected that any respondent who views surveys as burdensome would be more likely to seek to shorten the survey duration, but in fact this is not the case among the Arab citizens and residents of Qatar.

The effect of attitudes about survey reliability on continuation, conditional on respondent social category, is visualized in Figure 4. Beyond depicting the disparate impacts of this attitude dimension across respondent groups, the illustration clearly shows that the higher drop-out rates observed among the two Arab categories reflect the decisive impact of their survey attitudes on actual survey behavior, rather than any propensity toward falsification. Whereas Qatari and non-Qatari Arab respondents are only around 25 per cent likely to finish the survey when they possess negative views of survey reliability (that is, evaluated at −1), non-Arab respondents with negative views are nearly twice as likely to do so, at an estimated 45 per cent. As indicated by the flat line denoting non-Arab respondents, survey continuation is unrelated to perceptions of survey reliability for this group (p = 0.597). Meanwhile, for both categories of Arab respondents who view surveys as highly reliable, the estimated probability of continuation is indistinguishable from the corresponding likelihood among non-Arabs.

Figure 4. The effect of survey reliability on continuation (birthday), by respondent group

The opposite result obtains in the case of survey costs. Pessimistic views of survey time and cognitive burden do not depress the likelihood of continuation for Qataris or other Arabs in Qatar, yet they strongly predict interview completion among non-Arabs. Non-Arab respondents are an estimated 54 per cent likely to finish the survey if they perceive survey costs positively, compared to 36 per cent likely if they perceive them negatively. The latter probability is statistically identical to the probability of continuation among Qataris and Arab non-Qataris, irrespective of where they fall on the costs dimension. (Visualization not shown.)

Conclusion

The expansion of survey research in the Arab world has brought with it questions about the region's survey-taking climate. Do authoritarian political institutions and publics lacking experience with opinion polls undermine the validity of the growing number of surveys being conducted in MENA countries? Findings from this study suggest that these apprehensions are misplaced. In Qatar, a country that typifies these environmental conditions, both citizens and expatriates from a disparate set of Arab countries are actually more positively oriented toward surveys than people from other cultural–geographical regions.

Nevertheless, experimental evidence suggests that ensuring data quality in Arab contexts may entail some challenges. Despite being more positively oriented toward surveys on average, our results show that Arab survey takers are disproportionately sensitive to their subjective impressions about the reliability and intentions of surveys, whereas the survey behavior of non-Arab respondents depends on their enjoyment of survey taking and a survey's perceived cognitive and time burden. Notable, too, is the partial discrepancy between the impacts of survey attitudes on hypothetical willingness to participate in a survey, as measured in the conjoint experiment, and on actual survey behavior, as measured in the birthday experiment. This suggests a promising and potentially very productive avenue for future research: that different considerations may influence the decision to participate and the decision to continue a survey to completion. With respect to our findings from Qatar, it may be that respondents form and update opinions about the reliability, intentions and burden of a survey after it has begun, so these factors predict both participation and drop-out, whereas survey enjoyment and privacy concerns are less connected to a particular survey and so are more closely related to participation.

That Qatari citizens are less sensitive to political topics than non-citizen respondents, and also are not averse to government sponsors, even on political topics, is also noteworthy. For citizens and resident Arabs, the least acceptable survey sponsors are private and international organizations. Our findings on interaction effects suggest that these sponsors may signal perceived negative intentions, and, if so, hesitation to participate may stem from worries that survey results will be manipulated or used for nefarious purposes, rather than from the type of sponsor or topic per se. This helps explain why the effects of survey sponsorship differ by cultural–geographical category, which suggests that they likely also differ across countries.

Finally, these results highlight the importance of measuring and understanding the impact of attitudes about the perceived purpose of surveys, distinct from their value or reliability. To the authors' knowledge, this dimension has not been identified, or even searched for, in previous quantitative studies of survey attitudes. Of course, public skepticism about the purposes of surveys is not limited to the MENA region, and indeed our findings show that respondents from various other parts of the world tend to hold more negative views along this dimension. Nevertheless, especially given the challenges posed by declining interest in survey participation across many contexts, it is worth further studying the effects of this and other survey attitudes.