Hostname: page-component-78c5997874-g7gxr Total loading time: 0 Render date: 2024-11-13T03:45:43.391Z Has data issue: false hasContentIssue false

Does ideology influence hiring in China? evidence from two randomized experiments

Published online by Cambridge University Press:  03 February 2022

Jennifer Pan
Affiliation:
Department of Communication, Stanford University, Stanford, CA, USA
Tongtong Zhang*
Affiliation:
Department of Political Science, Stanford University, Stanford, CA, USA
*
Corresponding author. Email: ttzhang7@stanford.edu
Rights & Permissions [Opens in a new window]

Abstract

China after Mao is typically characterized as a country where economic opportunities are based on merit instead of ideological conformity. However, the salience of ideology has grown under the rule of Xi Jinping. Using a large-scale resume audit experiment and a conjoint survey experiment of hiring managers in China, we find that firms in China do not reward job candidates for expressing conformity to the ideology of the regime, but job candidates who express support for Western democracy are less employable. Results suggest that firms in innovative industries designated as strategically important by the Chinese regime (e.g., artificial intelligence) penalize support for Western democracy by the largest magnitude while the remaining firms in innovative industries do not penalize political non-conformity.

Type
Original Article
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press on behalf of the European Political Science Association

1. Introduction

China is one of the largest economies in the world. It has a robust and diverse business sector, and most scholars describe its labor market as one that rewards human capital.Footnote 1 While adherence to the political ideology of the Chinese Communist Party (CCP) and outward expressions of ideological “correctness” (表现) determined access to economic opportunities in the Maoist era,Footnote 2 since the launch of economic reform and opening in the late 1970s, China is typically characterized as a country that has moved away from ideology in favor of economic pragmatism, where economic opportunities are based on merit instead of ideological conformity (Cai et al., Reference Cai, Park and Zhao2008; Gallagher, Reference Gallagher2011; Landry et al., Reference Landry, Lü and Duan2018).

However, under the leadership of Xi Jinping, ideology has made a comeback. Xi has launched numerous ideological campaigns since taking power.Footnote 3 “Xi Jinping Thought on Socialism with Chinese Characteristics for a New Era” was added to the preamble of the country's constitution and widely promoted. The CCP has explicitly rejected multiparty democracy, free media, and other “Western” ideology as antithetical to China's official ideology, publicly censoring and penalizing those who support them (Ministry of Education, 2003; Xinhua, Reference Xinhua2015). Strikingly, for the first time since the 1970s, employees of public and private enterprises are regularly required to study political ideology.Footnote 4

What, if any, effects does this resurgence in ideology have on economic opportunity? In this paper, we examine whether displaying ideological conformity to the CCP and expressing ideas that deviate from CCP ideology influence the employability of job seekers in China. Are there labor market returns to expressing conformity to CCP ideology? Does expressing support for Western-style democracy make job seekers less employable?

We answer these questions by conducting a resume audit experiment to measure the behavior of firms in China's labor market and by conducting a conjoint survey experiment of Chinese hiring managers to uncover their explanations about the role of ideology in hiring. In the resume audit study, we submitted three resumes—one signaling ideological conformity, one signaling ideological non-conformity, and one that does not signal any political viewpoints—to each of 6,407 job vacancies found on one national-level Chinese job posting site. Our outcome of interest is callbacks, personalized phone or email contact requesting the applicant to advance to the next stage of the application process. In the conjoint survey experiment, we asked 506 hiring managers across China to evaluate resumes with the same three political signals and explain why they would or would not advance the hypothetical candidate to the next stage of the hiring process.

The resume audit experiment shows that firms in China do not reward job candidates for expressing conformity to the ideology of the CCP. Firms are as likely to hire a job candidate who expresses no political affinity as someone who professes adherence to CCP ideology. However, job candidates who express politically non-conformist views supporting Western democratic institutions are less employable in China's labor market. The conjoint survey experiment shows that hiring managers do not prioritize politically conformist candidates because they see no upside in hiring such workers; namely, hiring such candidates would not help the firm win any benefits from the regime and hence would add little value to the firm's operations. Hiring managers avoid non-conformist candidates to minimize the political risk such candidates are thought to generate. Penalties imposed on expressing politically non-conformist views do not vary by firm ownership but do vary by industry sector. Specifically, firms in innovative industries prioritized by the Chinese government as strategically important, such as artificial intelligence and nanotechnology, are most likely to reject applicants supporting Western democracy while the remaining firms in innovative industries do not penalize political non-conformity.

Our study establishes a causal link between expressing political ideology and labor market outcomes. On the one hand, while Xi has revived many Maoist practices in firms, such as self-criticism and regular political study, firms have not gone so far, as in the Maoist period, as to reward political conformists in hiring. On the other hand, the penalty for supporting Western democracy suggests that China's ideological efforts are manifesting in labor market outcomes. The heterogeneous effects by industry reinforce this view. Existing research predicts that firms that rely on innovation would be less likely to discriminate against job candidates based on political views (Lazear, Reference Lazear1999; Yam et al., Reference Yam, Guan, Pun and Tang2004; Ernst and Naughton, Reference Ernst and Naughton2008; Page, Reference Page2008), but we show that firms operating in innovative fields prioritized by the CCP regime impose the highest penalties on supporting Western democracy.

The paper proceeds in four sections. Section 2 describes the design of the experiments. Section 3 shows the main results of the experiments. Section 4 presents the heterogeneous effects and explores their theoretical implications, and Section 5 concludes by discussing the findings.

2. Experimental design

To measure the effects of job seeker ideology on employability, we conducted two main experiments: (1) a resume audit experiment, and (2) an online, conjoint experiment. The same sets of treatments—resumes that vary in political orientation they signal—were used for both experiments. All elements of this research were approved by our university IRB (see Appendix for detailed discussion of ethics).

2.1. Treatments

To construct fictitious, but realistic, resumes with high construct and external validity, we analyzed 30,409 job postings and nearly 100 actual student resumes to determine what features (e.g., name, university, internship, extracurricular experience) resumes should include so we could construct a pool of elements for each feature to randomly sample from (see Appendix for details). We then interviewed 40 students and 12 employers to identify attributes that can potentially signal political orientations on a resume. This produced three options: the applicant's major, courses highlighted in the resume, and extracurricular study group. To select which of these three attributes produced the highest construct validity, we conducted an online conjoint survey experiment (Hainmeuller et al., Reference Hainmeuller, Hopkins and Yamamoto2014).Footnote 5 Extracurricular study group emerged as the attribute with the strongest impact on assessments of political ideology in the expected direction (treatment is effective) and with the smallest impact on merit and connections (treatment is clean). For details, see Appendix Tables A1 and A2.

Participation in a “Socialism with Chinese Characteristics Study Group” was used to signal conformity to China's ruling ideology, participation in a “Western Political Philosophy Study Group” was used to signal non-conformity, and participation in a “Comic Book Club” was the control that did not signal any political orientation. As Table 1 shows, each extracurricular is described with up to three bullet points. The first bullet highlights leadership capabilities, the second emphasizes organizational capabilities, and the third describes what the applicant has learned from the activity, which is an explicit statement of the applicant's political position.Footnote 6 These bullet points were randomly selected to be included in the submitted resumes, making it less likely that particularities of the activity description, rather than political orientation, would drive employer callbacks.

Table 1. Treatments

Since any treatment construct used in an audit experiment will be multi-dimensional, we conducted a second conjoint experiment to test whether the extracurricular study group treatments signal other characteristics besides political ideology.Footnote 7 We recruited 320 online respondents with work experience in mainland China, asking them to assume the role of HR manager and evaluate two job candidate profiles side by side. Respondents were then asked to rate each candidate on political conformity as well as 17 other characteristics that hypothetically could be associated with our treatment and employer callbacks.

The results of this manipulation check (see Figure 1) show that our treatments first and foremost signal political conformity and political non-conformity in the expected direction. The y-axis of Figure 1 lists all 18 characteristics, starting with political conformity to the CCP. The x-axis shows the difference in ratings between applicants in the Socialism with Chinese Characteristics treatment group and applicants in the apolitical control (denoted by ), as well as the difference in ratings between applicants in the Western Political Philosophy treatment group and those in the apolitical control (denoted by ). The estimate for an applicant characteristic will appear to the right of the zero vertical line if a treatment increases rating on that characteristic compared to the control group and to the left of the zero line if a treatment decreases ratings comparatively.

Figure 1. Ratings of applicants who participate in the Socialism with Chinese Characteristics Study Group and Western Political Philosophy Study Group compared with the rating of applicants who participate in the Comic Book Club on political loyalty as well as 17 other characteristics that may affect hiring decisions; coefficient estimate as well as 95 percent confidence interval for each rating comparison is shown.

Figure 1 shows that the political conformity and non-conformity treatments change ratings of political loyalty by similar magnitudes (in opposite directions) compared to the apolitical control.Footnote 8 Figure 1 also shows that these treatments are much stronger signals of political ideology than of other applicant characteristics. Our treatments for political conformity and/or non-conformity do not differ significantly from the apolitical control on 11 applicant characteristics: academic merit, leadership, work efficiency, social connections, problem solving ability, strategic behavior, independence, dedication, obedience, patriotism, and attitude toward risk. There are three characteristics where ratings for both treatments differ from the control—political activism, collaborative ability, and bookishness—but ratings between the two treatment groups are not statistically distinguishable. For the final three characteristics—creativity, conservatism, and critical thinking—ratings for both treatments differ significantly from the control and differ from each other. The Western Political Philosophy Study Group is associated with more creativity, less conservatism, and more critical thinking than the Socialism with Chinese Characteristics Study Group. However, the effect of our treatment constructs on ratings of these characteristics is much smaller in magnitude than the effect of the constructs on ratings of political orientation.

2.2. Resume audit experiment

After developing the treatments, we submitted three resumes, each signaling a different political orientation, to each of 6,407 job vacancies found on one national-level job posting site between November 27, 2016 and February 24, 2017 (19,221 unique resumes in total).Footnote 9 The three resumes submitted to each job vacancy had different names, different universities, different internship experiences, different extracurricular activities, different certificates, and different visual formating because we randomly selected these elements from respective pools of options without replacement.Footnote 10

The job vacancies we applied to were focused on jobs for recent college graduates that covered a diverse range of geographies, ownership structures (including public institutions, state-owned enterprises or SOEs, private firms, and foreign/joint venture firms), industries, and types of job positions (see Table A3 in Appendix). To ensure fit between resume and job vacancy, we block-randomized majors and internships by job vacancy to align the major and internship experiences of our submitted resume with the requirements of the job posting (see Appendix for details). Three resumes containing different treatment conditions were submitted in randomized order to each job vacancy on different days within one week of the job vacancy being posted.

The outcome we measure is a “callback” from the firm. Specifically, we count as callbacks any personalized phone or email contact by a potential employer, usually requesting that the applicant advance to the next stage of the application process (for the detailed protocol on recording callbacks, see Appendix).

2.3. Conjoint survey experiment with hiring managers

To gain a better understanding of the rationale behind hiring decisions, we conducted a conjoint survey experiment with 506 hiring managers across China between December 14, 2018 and January 6, 2019. Participants were recruited nationally through a domestic survey firm in China. All respondents had at least one year of experience in hiring, and we sampled respondents working in the same provinces and in the same ownership sectors as the job vacancies in our resume audit experiment (for details of the survey sample, see Appendix Table A17).Footnote 11

We showed three resumes randomly selected from the resumes we used in the audit experiment, each signaling a different political orientation, in randomized order to each of the 506 hiring managers. After reviewing each resume, hiring managers were asked whether or not they would advance the applicant to the next stage of the hiring process.Footnote 12 Subsequently, hiring managers were specifically asked how the extracurricular study group in the resume affected their callback decision in an open-ended response.

If we were using the hiring manager conjoint experiment to estimate the prevalence of behaviors, we would be concerned that the hiring managers would not be entirely truthful in their evaluation of the political treatments because of preference falsification or because of the cost of “advancing” candidates is much less tangible in a survey than in real hiring practice. However, we do not use the conjoint experiment for this purpose but instead use it to elucidate the rationale behind hiring practices. Using the conjoint experiment to shed light on rationale is valid as long as there is variation in hiring managers’ reported “callbacks.”

3. No reward for conformity, penalty for non-conformity

Resume characteristics—such as level of academic achievement, gender, CCP membership, and quality of university—were balanced across treatment groups (see Appendix Table A4). Callbacks also varied in expected ways: all other characteristics being equal, applicants from top-tier universities got more callbacks than applicants from lower-tier universities; applicants who were geographically more proximate to the job received more callbacks than those who lived further away; men received more callbacks than women, which we expected based on previous studies of gender discrimination in China's labor market (Kuhn and Shen, Reference Kuhn and Shen2009; Magnani and Zhu, Reference Magnani and Zhu2012).Footnote 13

Table 2 shows the summary statistics for the audit experiment. The overall callback rate, based on the 19,221 resumes submitted, was 19.1 percent. The callback rate for resumes that did not signal any political orientation was 19.7 percent, and 19.5 percent for resumes signaling conformity to CCP ideology and 18.0 percent for resumes signaling support for Western democracy.

Table 2. Summary statistics for callbacks in audit experiment

Table 3 shows the main result of the audit experiment. The dependent variable is a dummy variable for whether a resume received a callback. Participation in the Comic Book Club (apolitical signal) is the comparison category. Results are based on logistic regression, but remain substantively unchanged if other parametric models are used (see Appendix Table A6). The two main findings are: (1) expressing conformity to CCP ideology has no positive or significant effect on callbacks when compared to resumes that do not signal any political orientation; and (2) expressing views that deviate from CCP ideology decreases callbacks across all model specifications. In the first column of Table 3, the only variables are dummy variables for political conformity and non-conformity. Column (2) adds vacancy fixed effects, and column (3) includes demeaned covariates and their interactions with the treatment dummies (Lin, Reference Lin2013).Footnote 14 Expressing political non-conformity on the resume decreases callback rates by 9 percent ($1.7$ percentage points) when compared to the apolitical condition and decreases callback rates by 8 percent ($1.5$ percentage points) when compared to the political conformity treatment.

Table 3. Effect of conformity and non-conformity on callback rates

Notes: The dependent variable is a dummy for whether the employer called back. All columns use logistic regression. Coefficients are average marginal effects on the probability of callback. $^{\ast }$p<0.1; $^{\ast \ast }$p<0.05; $^{\ast \ast \ast }$p<0.01.

3.1. Rationale

In the conjoint survey experiment of hiring managers, 431 hiring managers (85 percent) reported they would call back candidates who signaled support for Socialism with Chinese Characteristics and 402 (79 percent) for applicants who signaled support for Western democracy. Overall, hiring managers were 7 percent less likely to report they would call back politically non-conformist candidates than politically conformist ones, which means the size of the penalty on support for Western-style democracy, comparing callbacks for candidates who express political non-conformist versus conformist views, is similar between the 2016 audit experiment (8 percent) and the 2018/2019 hiring manager conjoint experiment (7 percent).

We use open-ended responses that followed the conjoint treatment to gather information on why hiring managers do not reward conformity and why they penalize non-conformity. Hiring managers recognized participation in a voluntary study group related to Socialism with Chinese Characteristics as a signal of sincere interest in and support for CCP ideology and believed such applicants could help the firm demonstrate more loyalty to the regime. However, hiring managers do not prioritize such candidates because they see little value in political loyalty for a firm's operations. One hiring manager from a private, manufacturing firm said: “While it's good that the applicant aligns with the government's ideology, frankly speaking it has little value for our firm.” Hiring managers from the public sector and SOEs echoed these sentiments. One hiring manager at an SOE said,

“Given the current societal environment, it's good to move closer to the party ideologically. Also, four years’ participation in the socialism study group suggests his support for the party is not cheap talk. But such political belief does not have any positive or negative effect on the work in our firm.”

While hiring manager did not see any business upside in hiring political conformists, they did associate political non-conformity with business risk. Hiring managers penalize political non-conformity because they worry that hiring those who support Western democracy invites penalties or investigations from the regime and such penalties/investigations would negatively impact firm operations and profits. One hiring manager from an organization doing social work expressed concern that political non-conformists could “affect the political thoughts of our work unit and exert a negative influence on the ideological work the government assigns to state-owned enterprises like us.” Hiring managers from private and foreign sectors echoed this desire to avoid government sanctions in rejecting applicants who signal support for Western democracy, even though they were aware that the non-conformists they rejected could be productive workers. Below are quotes from two different hiring managers, the first from a private education company and the second from a foreign information technology firm:

“Close contact with Western political culture means the applicant is open-minded, which is something our firm likes. But unfortunately, these values do not fit with China's circumstances, making it more likely the student will come into conflict with what the firm requires for business success in China, and create unseen dangers down the road.”

“Actually, knowledge of political thought beyond China is a valuable skill for our firm. However, in the current political system of China, [this candidate] expresses political thoughts that contradict those of the regime, which makes us worry that she can easily cause our firm to be seen as making a political mistake.”

The open-ended responses suggest that penalties on political non-conformity do not derive from taste-based discrimination.Footnote 15 No hiring managers reported rejecting candidates who supported Western democracy because such political beliefs diverged from those of the business’ leaders or the hiring managers themselves. Similarly, when hiring managers evaluated resumes that signaled support for Socialism with Chinese Characteristics, none said that advancing such a candidate aligned with the political thoughts of the firm leadership or the hiring managers.

Overall, the open-ended responses from the conjoint survey show that hiring managers do not reward political conformity because they see little reward from the regime in hiring such candidates. However, hiring managers are less likely to move forward political non-conformists out of fear that such candidates will spell trouble for the firm from the CCP.

4. Heterogeneity in penalty on non-conformity

In this section, we explore two potential sources of heterogeneity: (1) by ownership structure and (2) by industry sector.Footnote 16

4.1. Ownership structure

In the study of firms in China, the incentives and hence behavior of SOEs and government-affiliated public institutions (e.g., public hospitals and public schools) are often thought to be different from those of private and foreign firms; however, the expected implications of those differences for penalizing political non-conformity are mixed.

Some research argues that SOEs and public institutions are more likely to conform to government positions because the government is better able to make demands of such firms. Top appointments in SOEs and public institutions are controlled through the CCP nomenklatura system (Landry, Reference Landry2008) and these firms are often given political tasks by the government. In the Xi Jinping era, SOEs and public institutions were the first to reinstate the Maoist practice of self-criticism and political study sessions (Wu, Reference Wu2017). However, other studies argue that SOEs and public institutions are less likely to conform to government demands because of their insider status, direct channels of access to policymakers, and control of industries related to national security, all of which increase their bargaining power vis-a-vis the regime (Dickson, Reference Dickson2003; Tsai, Reference Tsai2007; Truex, Reference Truex2014; Shi, Reference Shi2015; Naoi et al., Reference Naoi, Shi and Zhu2017). In this view, private firms, which have less bargaining power against the government, are more likely to conform to the demands of the regime than SOEs and public institutions. On foreign firms, some believe they, like private firms, would conform to government demands out of fear of losing access to the lucrative Chinese market (Child and Tse, Reference Child and Tse2001) while others argue foreign firms have greater bargaining power against the regime because they can credibly threaten to exit (Gallagher, Reference Gallagher2011; Wang, Reference Wang2015).

When we analyze the heterogeneous effects of political non-conformity by ownership sectors, we find no statistically significant difference in the penalty on support for Western-style democracy across firms with different ownership structures. Figure 2 shows the heterogeneous effects by ownership sector, with 95 percent confidence intervals (see regression table in Appendix). For each sector, the comparison group is all other ownership sectors. For example, the top estimate in Figure 2 represents the difference in the size of effect of political non-conformity, as compared to political conformity and not expressing any political viewpoints, on callback rate between public institutions and firms of all other ownership types. The penalty on non-conformity is not significantly different in statistical terms for public institutions, state-owned enterprises, private Chinese firms, or foreign/joint venture firms.

Figure 2. Heterogeneous effects on callbacks for non-conformity versus apolitical and conformity by ownership sector, with 95 percent confidence intervals.

4.2. Differences by industry sector

The incentives, and hence behavior, of firms often differ by industry sector. However, there are numerous ways of grouping firms—for example, by level of investment in R&D, by degree of reliance on government licences, by importance to national security or national strategic interests—so we begin our exploration of heterogeneous effects by industry sector without imposing assumptions on how firms might divide into groups. We use unsupervised machine learning—specifically a structural topic model (STM) (Roberts et al., Reference Roberts, Stewart, Tingley, Lucas, Leder-Luis, Gadarian, Albertson and Rand2014)—to see how firms cluster into industry groupings (“topics”). We use the names and descriptions of firms as the textual input into the STM. This unsupervised analysis clustered the 6,407 job vacancies in our audit experiment into 12 industry groupings (“topics”).Footnote 17 We then hand-label each topic by examining the words that are most strongly associated with the topic (See Table A9 in Appendix).

The output of the STM alllows us to explore the relationship between these 12 industry groupings and our covariate of interest—whether or not expressing political non-conformity is penalized by a firm. We define a firm as penalizing non-conformity if the firm does not advance the political non-conformist to the next stage of recruitment but the firm advances either the apolitical applicant or the political conformist. Figure 3 shows the correlation between penalizing non-conformity and topic proportion for each of the 12 industry groupings. The point estimate in Figure 3 is the difference in topic proportionsFootnote 18 between firms that penalize non-conformity and firms that do not penalize, and the lines are 95 percent confidence intervals. Estimates to the right of the zero vertical line are industry groupings that are more likely to penalize political non-conformity. Estimates to the left of the zero line are industry groupings that are less likely to penalize non-conformity.

Figure 3. Correlation between penalizing political non-conformity and topic proportion for industry groupings.

In Figure 3, the Intelligent Software grouping includes firms that work in artificial intelligence and cloud computing. The Biotech, Education grouping consists of firms working on genetics, bioengineering, and private education institutes such as after-school programs. Media, Culture, and Sports includes news and publishing firms, movie/drama companies, and firms that organize sporting events. Financial Service Outsourcing includes firms that provide service to banks and other financial institutions—for example, stock market analysis, debt restructuring, and customer support. The IT and Electronics R&D grouping includes electronics firms, e-commerce firms, and computer hardware firms. Real Estate refers to real estate developers. The Materials, Metallurgical grouping encompasses firms that focus on material manufacturing and mineral processing. Construction includes companies that build roads, bridges, and buildings. University Research Lab includes science, social science, and humanities research teams in universities. Investment Management includes firms that do investment consulting and management in different asset classes such as equities, commodities, and hedge funds. The Pharmaceutical, Hospital grouping includes public and private hospitals as well as pharmaceutical companies. Finally, Securities includes firms that issue and trade bonds and securities.

Figure 3 reveals two possible dimensions of heterogeneous effects by industry sector: (1) CCP strategic priority and (2) innovation. Some industry groupings that are associated with innovation are less likely to penalize non-conformity (e.g., Pharmaceutical, Hospital and University Research Lab), which aligns with existing research. However, other industry groupings associated with innovation are more likely to penalize non-conformity (e.g., Intelligent Software and Biotech). Interestingly, the innovative industries that penalize non-conformity, as well as other industry groups likely to penalize non-conformity (e.g., education), are fields that the Chinese government emphasized as strategic priorities for China's economic development during the time of the study (National People's Congress, 2016; State Council, 2016b). Other types of industry groupings, such as by whether firms operate in fields that require specialized permits of the regime (e.g., real estate development, banking, natural resources extraction), do not appear related to penalties on political non-conformity (Shih, Reference Shih2004; Brehm, Reference Brehm2008).Footnote 19

To assess these patterns more systematically, we hand-code each firm into whether the firm operates in an industry that the CCP has identified as a strategic priority for economic development and whether the firm works in a field characterized by innovation. We identify industries that the CCP regards as strategic priorities based on the 13th Five-Year Plan of China (2016–2020) (National People's Congress, 2016). We identify innovative fields by drawing on prior research, categorizing firms and institutes operating in academic research, information technology, chemical production, pharmaceutical manufacturing, instrument manufacturing, machine manufacturing, metallurgy, and material manufacturing as innovative fields (Prat, Reference Prat2002; Yam et al., Reference Yam, Guan, Pun and Tang2004; Ernst and Naughton, Reference Ernst and Naughton2008).

Our hand-coding places firms into the four quadrants shown in Figure 4. Firms in the top right quadrant fall into the six strategic emerging industries (战略性新兴产业) identified in China's 13th Five-Year Plan. These industries—next-generation information technology, biotechnology, alternative energy, new materials, geospatial technology, and alternative energy vehicles—were identified by the CCP regime as priorities in fostering technological upgrading of domestic economy (National People's Congress, 2016). Firms in the top left quadrant require innovation but are not considered strategic by the CCP. Examples include traditional pharmaceutical companies, information technologies that are not “next generation” (e.g., 4G), and machine manufacturing. Firms in the bottom right quadrant are those outside of innovative fields that are nonetheless prioritized by the CCP in the 13th Five-Year plan, including public and private education, media, and culture. Finally, firms in the bottom left quadrant are not characterized by innovation and not prioritized by the CCP in the study time period--for example, consumer products, construction firms, finance, and legal services.

Figure 4. Hand-annotation of firms.

We estimate heterogeneous effects by these four groups of firms in Figure 5 (see regression table in Appendix). For each group, the comparison group is all other firms that do not have the characteristics of that group. For example, the top estimate in Figure 5 is the difference in the size of effect of the political non-conformity treatment, as compared to the political conformity and non-political treatment groups, on callback rate between firms that are in innovative industries prioritized by the CCP compared to all other firms.

Figure 5. Heterogeneous effects on callbacks for non-conformity versus apolitical and conformity by firm characteristics, with 95 percent confidence intervals.

Figure 5 shows that firms in innovative fields the CCP identified as strategic priorities are 2.7 percentage points more likely to penalize political non-conformity compared to other firms, and this difference in effects is statistically significant. Breaking down this difference, on average, firms in CCP priority and innovative industries penalize non-conformity by 3.5 percentage points (see Table A11 in Appendix), while all other firms penalize non-conformity by 0.8 percentage points. In contrast, innovative firms that are not prioritized by CCP as strategic emerging industries are 2.4 percentage points less likely to penalize non-conformity compared to other firms. Breaking down this difference, on average, firms that are innovative but not prioritized by the CCP are more likely to call back those who support Western democracy by 0.5 percentage points while all other firms penalize non-conformity by 1.9 percentage points. Also from Figure 5, firms that are in sectors the CCP has emphasized as strategic priorities but that do not emphasize innovation are slightly more likely to penalize political non-conformity compared to other firms, but the difference is small and not statistically significant. Firms that are not prioritized by the CCP and do not emphasize innovation are less likely to penalize political non-conformity than the rest firms, but the difference is not statistically significant.

4.3. Theoretical implications

These heterogeneous effects suggest that ownership structure does not predict how firms respond to job seekers that express politically non-conformist views, but industry sector does. Here, we discuss the theoretical implications of these exploratory results.

If we begin with the assumption that firms want to maximize profit by increasing revenue and decreasing costs, then all else being equal, firms want to hire employees who will help the company increase revenue without increasing costs. If we take into account the rationale hiring managers provided in avoiding applicants who express politically non-conformist views—namely, that such job seekers would increase the political risk faced by the firm, then there are several potential reasons why firms falling into innovative fields identified by the CCP as strategic priorities would be more likely to penalize support for Western democracy while firms in other innovative industries would be less likely to penalize this form of political non-conformity.

The expectations of prior research on innovation predict that the more important innovation is to a firm's business model, the more the firm will base its hiring decisions on worker productivity rather than other factors (Prat, Reference Prat2002; Page, Reference Page2008). In addition, such firms will aim to increase diversity among employees because diversity of background, experiences, and perspectives is critically important to innovation (Lazear, Reference Lazear1999; Page, Reference Page2008). Our analysis of heterogeneous effects by industry sector aligns, in part, with this theoretical expectation. If we exclude firms in the CCP's strategic industries, then firms that are in fields requiring higher levels of innovation are indeed less likely to penalize job seekers who express views antithetical to the CCP's ideology. In fact, such firms are as likely to call back a job seeker that expresses politically non-conformist views as they are to call back someone who expresses politically conformist views or someone who does not express any political orientation. In other words, firms in innovative fields not prioritized by the CCP exhibit no discrimination based on political ideology.Footnote 20

Why is the behavior of firms in innovative industries that are CCP priorities different? We start with three stylized facts. First, industries identified as strategic priorities of the CCP have special access to governmental resources such as funding, tax incentives, and government contracts (Yang, Reference Yang2014; State Council, 2016a). Second, firms in these strategically prioritized industries are subject to more monitoring, oversight, and enforcement than firms in other industries (Lauer and Liefner, Reference Lauer and Liefner2019). Local officials and special government committees may keep a close eye on such firms and be more willing to enforce state rules because the performance of such firms is embedded in the metrics used to assess officials' performance and prospects for career advancement. For example, the “National Innovation-Driven Development Strategy (国家创新驱动发展战略纲要)” published by the State Council of China explicitly states that “growth in the strategic emerging industries will become an important criterion to assess the performance of local officials” (State Council, 2016b). Third, the innovative industries identified as strategic priorities of the CCP are often new industries where the dynamics of competition are not yet established (National People's Congress, 2016; Arenal et al., Reference Arenal, Armuña, Feijoo, Ramos, Xu and Moreno2020; Li et al., Reference Li, Ku, Liu and Zhou2020). In these industries, firms often compete in fragmented markets where there is no dominant market leader.

Because firms are competing for market position and access to government resources while being subjected to high levels of oversight, firms have little bargaining power vis-a-vis the state. As such, the risk to firms of hiring job applicants with politically non-conformist views may outweigh the productivity gains such employees might bring. Politically non-conformist employees may create costly distractions. While it is unlikely firms would be blamed or punished for the political actions of a low-level subordinate, dissenting behaviors of employees would be more visible to the regime for firms operating in fields that the regime prioritizes. Hence, dissenting behaviors of employees in the CCP's strategic emerging industries are more likely to draw the attention of the party and state investigators. Cooperating with such investigations, which are usually tedious and time-consuming, will cost the firm substantial time and resources (He, Reference He2020; Feng, Reference Feng2021). This would disrupt normal firm operations and possibly demoralize other employees. In the competitive emerging industries where the firm is operating, these disruptions may give competitors an edge in the market.

5. Discussion

The results of the audit experiment and conjoint experiment among hiring managers contribute to our understanding of the role of political ideology. There is a long-standing view that political loyalty is rewarded and political non-conformity is penalized in the labor market under authoritarian rule. Existing studies on political selection show that authoritarian governments are more likely to hire and promote politically loyal subordinates, even if they are incompetent (Wintrobe, Reference Wintrobe1998; Egorov and Sonin, Reference Egorov and Sonin2011). In Maoist China, outward expression of conformity to the CCP was an important determinant of labor market outcomes (Walder, Reference Walder1988). In the post-Mao era, human capital has become increasingly important in firms’ hiring calculus. Yet many studies contend that all else being equal, firms in public and private sectors still prefer to hire those who show higher ideological loyalty to the CCP (Walder, Reference Walder1995; Dickson, Reference Dickson2014; Yan, Reference Yan2014). In recent years, as Xi Jinping has revived Maoist practices such as regular political study meetings in firms, some argue that the expression of political conformity (表现) is again an important factor in determining economic opportunities (Bloomberg, 2019; Liu, Reference Liu2019). In contrast to these views, our experimental results show that while job seekers do diminish their job opportunities by expressing support for Western political institutions, it is not the case that job seekers who express politically non-conformist views can never be employed, at least in the early stage of recruitment, and ideology is not so important that demonstrating ideological conformity will improve a person's chance of being hired in China's labor market.

The exploratory analysis of heterogeneous effects by industry sector has implications for our understanding of state-led development and state-led innovation in China. Over the past decade, China has become a global, economic powerhouse with a dynamic business sector. There are two main groups of explanations for China's overall economic success, which both emphasize the role of the state. The first argues that China's growth was driven by top-down leadership and guidance from the center (Huang, Reference Huang1999; Wong, Reference Wong2004; Beeson, Reference Beeson2009; Lin, Reference Lin2011). The second argues that growth occurred through gradual, bottom-up adaptation by local governments (Montinola et al., Reference Montinola, Qian and Weingast1995; Oi, Reference Oi1999; Tsai, Reference Tsai2006; Ang, Reference Ang2016). With respect to innovation, some scholars have argued that innovation in China has also benefited from state leadership—specifically that the central government's use of direct intervention and policy instruments have successfully stimulated technological upgrading in the economy (Liu et al., Reference Liu, Simon, Sun and Cao2011; Lauer and Liefner, Reference Lauer and Liefner2019). Our results suggest that there are shortcomings to state-led innovation as firms designated by the regime as strategically important for innovation may not be hiring talent in ways that maximize the firms’ innovative potential. This, in turn, may help explain why state-led innovation in China has not always been accompanied by substantial gains in innovation (Appelbaum et al., Reference Appelbaum, Parker and Cao2011; Yang, Reference Yang2014; Ang et al., Reference Ang, Jia and Huang2020).

Finally, this research provides a new approach for developing treatments with high construct validity in audit experiments. Most audit experiments rely on ad hoc approaches such as focus groups or an experimenter's personal knowledge to construct treatment conditions. This experiment uses analysis of large-scale observational data, interviews, and conjoint experiments to design treatments with strong construct validity, allowing us to quantify the measurement bias of potential proxies for the treatment and to measure the extent to which research participants will interpret treatment proxies as experimenters intend.

Supplementary material

The supplementary material for this article can be found at https://doi.org/10.1017/psrm.2021.77. To obtain replication material for this article, please visit https://doi.org/10.7910/DVN/CLZZTW.

Acknowledgments

Our thanks to Haotian Bu, Luchi He, Shixian Li, Lin Liu, Yongbo Liu, Beibei Tan, Xuewei Tian, Tianyi Wang, Yue Wang, Qiushi Yin, Ni Zeng, Kan Zhang, and many others for superb research assistance; David Broockman, Daniel Butler, Alex Coppock, Charles Crabtree, Ana De La O, Donald Green, Jens Hainmueller, Gregory Huber, Shanto Iyengar, Holger Kern, Gary King, Peter Lorentzen, Eddy Malesky, Neil Malhotra, Kristin Michelitch, Brendan Nyhan, Jean Oi, Elizabeth Perry, Jonathan Rodden, Weiyi Shi, David Szakonyi, Georg Vanberg, Andrew Walder, Chenggang Xu, Yiqing Xu, Boliang Zhu for many helpful comments and suggestions; and to the Freeman Spogli Institute China Fund and Asia-Pacific Scholars Fund for research support.

Footnotes

1 There is debate over whether China is a full-blown labor market (Chan et al., Reference Chan, Madsen and Unger1992; Nee, Reference Nee1992; Meng et al., Reference Meng, Shen and Xue2013) or whether political factors retain primacy (Oi, Reference Oi1992; Walder, Reference Walder1995; Walder et al., Reference Walder, Li and Treiman2000; Liu, Reference Liu2012). However, the literature generally agrees that human capital is an important factor in hiring.

2 See Shirk (Reference Shirk1982) and Walder (Reference Walder1988).

3 Examples include promotion of the “China Dream” and circulation of “Communiqué on the Current State of the Ideological Sphere (Document No. 9)” that calls for eradicating “Western values” (e.g., Western constitutional democracy) in Chinese society.

5 This conjoint experiment served to refine the design of treatment in our main experiments. It is distinct from the conjoint experiment of hiring managers. In this conjoint experiment for design, we asked 121 respondents to review job applicant profiles side by side and assess the applicant's political conformity to the CCP, academic merit, and political connections (a total of 1,210 resumes were assessed).

6 The third bullet point was always included in the resume audit study because it boosted the construct validity of the treatment. We removed this restriction in the hiring manager conjoint survey described in Section 2.3, where we fully randomized bullet points in each treatment. The main results remain unchanged (see Section 3.1).

7 This conjoint experiment also served to refine the design of treatment in our main experiments. It is also distinct from the conjoint experiment of hiring managers.

8 This assuages concerns that expressing non-conformity is more credible and stronger than expressing conformity.

9 See Appendix for more information about this job website.

10 For the remaining components on the resume (e.g., gender and CCP membership), we selected an element from the corresponding pool with replacement. See Appendix for details.

11 Results from the hiring manager survey are weighted such that the distribution of ownership sectors is same between the survey sample and the audit experiment sample.

12 We asked the hiring managers to assume the candidate was applying for an entry-level position that matched his/her major to increase the level of comparability with the resume audit experiment.

13 See Table A12 in Appendix.

14 Covariates include resume characteristics (gender, prestige of university, geographic region of university, CCP membership, major, other skills and activities, and hobbies), the ownership sector of the job vacancy, and the applicant's geographic proximity to the job, which equals 1 if the candidate's university is in the same province as the job vacancy, and equals 0 otherwise. Robust standard errors are clustered at the vacancy level.

15 Existing studies also find political discrimination in hiring in democratic countries, but such discrimination is often taste-based where employers favor like-minded job candidates (Gift and Gift, Reference Gift and Gift2015).

16 The analysis of heterogeneity by ownership sector is detailed in our pre-analysis plan while heterogeneity by industry sector is listed as an exploratory analysis in our pre-analysis plan.

17 The STM method assumes a user-specified number of topics. So we first compare the performance of models under a range of topic numbers between 5 and 50, and choose K$=$12 for the preferred specification. This specification performs relatively well on a series of empirical tests (residuals fit, held-out likelihood, semantic coherence, and exclusivity of topics) that are conventionally used to assess the relative performance of models (Mimno and Lee, Reference Mimno and Lee2014; Roberts et al., Reference Roberts, Stewart and Tingley2019).

18 Topic proportion means the proportion of words in a firm's name and description that belong to an industry grouping.

19 We systematically test whether firms requiring government specialized permits exhibit heterogeneous effects through hand-coding (see Appendix Table A8), but do not find any statistically significant effects.

20 See Table A11 in Appendix.

References

Ang, YY (2016) How China Escaped the Poverty Trap. Ithaca: Cornell University Press.CrossRefGoogle Scholar
Ang, YY, Jia, N and Huang, KG (2020) The limits of state-led innovation: evidence from Chinese patents. Working Paper.Google Scholar
Appelbaum, RP, Parker, R and Cao, C (2011) Developmental state and innovation: nanotechnology in China. Global Networks 11, 298314.CrossRefGoogle Scholar
Arenal, A, Armuña, C, Feijoo, C, Ramos, S, Xu, Z and Moreno, A (2020) Innovation ecosystems theory revisited: the case of artificial intelligence in China. Telecommunications Policy 44, 101960.Google Scholar
Beeson, M (2009) Developmental states in East Asia: a comparison of the Japanese and Chinese experiences. Asian Perspective 33, 539.CrossRefGoogle Scholar
Bloomberg (2019) Party loyalty in China helps private companies get cheaper loans. https://bloom.bg/3d1bUpv.Google Scholar
Brehm, S (2008) Risk management in China's state banks-international best practice and the political economy of regulation. Business and Politics 10, 129.Google Scholar
Cai, F, Park, A and Zhao, Y (2008) The Chinese labor market in the reform era. China's Great Economic Transformation 26, 167214.CrossRefGoogle Scholar
Chan, A, Madsen, R and Unger, J (1992) Chen Village under Mao and Deng. Berkeley: University of California Press.Google Scholar
Child, J and Tse, D (2001) China's transition and its implications for international business. Journal of International Business Studies 32, 521.CrossRefGoogle Scholar
Dickson, B (2003) Red Capitalists in China: The Party, Private Entrepreneurs, and Prospects for Political Change. New York: Cambridge University Press.CrossRefGoogle Scholar
Dickson, BJ (2014) Who wants to be a communist? Career incentives and mobilized loyalty in China. The China Quarterly 217, 4268.Google Scholar
Egorov, G and Sonin, K (2011) Dictators and their viziers: endogenizing the loyalty–competence trade-off. Journal of the European Economic Association 9, 903930.Google Scholar
Ernst, D and Naughton, BJ(2008) China's emergent political economy–insights from the it industry. China's Emergent Political Economy–Capitalism in the Dragon's Lair Routledge’ Routledge.Google Scholar
Feng, E(2021, March) How China's massive corruption crackdown snares entrepreneurs across the country. https://n.pr/3vd15rs (Accessed April 22, 2021).Google Scholar
Gallagher, ME (2011) Contagious Capitalism: Globalization and the Politics of Labor in China. Princeton, N.J.: Princeton University Press.Google Scholar
Gift, K and Gift, T (2015) Does politics influence hiring? Evidence from a randomized experiment. Political Behavior 37, 653675.Google Scholar
Hainmeuller, J, Hopkins, D and Yamamoto, T (2014) Causal inference in conjoint analysis: understanding multidimensional choices via stated preference experiments. Political Analysis 22, 130.CrossRefGoogle Scholar
He, L(2020, September) Xi jinping wants China's private companies to fight alongside the communist party. https://cnn.it/3ndzaF0 (Accessed April 22, 2021).Google Scholar
Huang, Y (1999) Inflation and Investment Controls in China: The Political Economy of Central-Local Relations During the Reform Era. New York: Cambridge University Press.Google Scholar
Kuhn, P and Shen, K Employers’ preferences for gender, age, height and beauty: direct evidence. Technical report, National Bureau of Economic Research. http://www.nber.org/papers/w15564.Google Scholar
Landry, P (2008) Decentralized Authoritarianism in China: The Communist Party's Control of Local Elites in the Post-Mao Era. New York: Cambridge University Press.Google Scholar
Landry, PF, , X and Duan, H (2018) Does performance matter? Evaluating political selection along the chinese administrative ladder. Comparative Political Studies 51, 10741105.Google Scholar
Lauer, J and Liefner, I (2019) State-led innovation at the city level: policy measures to promote new energy vehicles in Shenzhen, China. Geographical Review 109, 436456.CrossRefGoogle Scholar
Lazear, EP (1999) Globalisation and the market for team-mates. The Economic Journal 109, 1540.CrossRefGoogle Scholar
Li, J, Ku, Y, Liu, C and Zhou, Y (2020) Dual credit policy: promoting new energy vehicles with battery recycling in a competitive environment?. Journal of Cleaner Production 243, 118456.CrossRefGoogle Scholar
Lin, N (2011) Capitalism in China: a centrally managed capitalism (CMC) and its future. Management and Organization Review 7, 6396.Google Scholar
Lin, W (2013) Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique. The Annals of Applied Statistics 7, 295318.Google Scholar
Liu, Q (2012) Unemployment and labor force participation in urban China. China Economic Review 23, 1833.CrossRefGoogle Scholar
Liu, H (2019) The logic of authoritarian political selection: evidence from a conjoint experiment in China. Political Science Research and Methods 7, 853870.CrossRefGoogle Scholar
Liu, F-c., Simon, DF, Sun, Y-t. and Cao, C (2011) China's innovation policies: evolution, institutional structure, and trajectory. Research Policy 40, 917931.CrossRefGoogle Scholar
Magnani, E and Zhu, R (2012) Gender wage differentials among rural–urban migrants in China. Regional Science and Urban Economics 42, 779793.CrossRefGoogle Scholar
Meng, X, Shen, K and Xue, S (2013) Economic reform, education expansion, and earnings inequality for urban males in China, 1988–2009. Journal of Comparative Economics 41, 227244.CrossRefGoogle Scholar
Mimno, D and Lee, M) 2014) Low-dimensional embeddings for interpretable anchor-based topic inference. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1319–1328.Google Scholar
Ministry of Education (2003) Si Xiang Zheng Zhi [Ideology and Politics]. Beijing: People's education press.Google Scholar
Montinola, G, Qian, Y and Weingast, BR (1995) Federalism, Chinese style: the political basis for economic success in China. World Politics 48, 5081.Google Scholar
Naoi, M, Shi, W and Zhu, B (2017) ‘Yes-man’ firms: government campaigns and policy positioning of businesses in China. working paper.Google Scholar
National People's Congress (2016, March) The 13th five-year plan for the national economic and social development of the People's Republic of China (中华人民共和国国民经济和社会发展第十三个五年规划纲要). https://bit.ly/3dGOhn4 (Accessed April 5, 2021).Google Scholar
Nee, V (1992) Organizational dynamics of market transition: hybrid forms, property rights, and mixed economy in China. Administrative Science Quarterly 37, 127.Google Scholar
Oi, J (1992) Fiscal reform and the economic foundations of local state corporatism in China. World Politics 45, 99126.Google Scholar
Oi, JC (1999) Rural China Takes Off: Institutional Foundations of Economic Reform. Berkeley: University of California Press.Google Scholar
Page, SE (2008) The Difference: How The Power of Diversity Creates Better Groups, Firms, Schools, and Societies – New Edition. Princeton, N.J.: Princeton University Press.CrossRefGoogle Scholar
Prat, A (2002) Should a team be homogeneous?. European Economic Review 46, 11871207.CrossRefGoogle Scholar
Roberts, ME, Stewart, BM, Tingley, D, Lucas, C, Leder-Luis, J, Gadarian, SK, Albertson, B and Rand, DG (2014) Structural topic models for open-ended survey responses. American Journal of Political Science 58, 10641082.Google Scholar
Roberts, ME, Stewart, BM and Tingley, D (2019) Stm: an R package for structural topic models. Journal of Statistical Software 91, 140.Google Scholar
Shi, W (2015) The Political Economy of China's Outward Direct Investments. Ph. D. thesis, UC San Diego.Google Scholar
Shih, V (2004) Dealing with non-performing loans: political constraints and financial policies in China. The China Quarterly 180, 922944.CrossRefGoogle Scholar
Shirk, SL (1982) Competitive Comrades: Career Incentives and Student Strategies in China. Berkeley: University of California Press.CrossRefGoogle Scholar
State Council T (2016a, July) The national informatization development strategy (国家信息化发展战略纲要). https://bit.ly/3sMgSfe (Accessed April 5, 2021).Google Scholar
State Council T (2016b, May) The national innovation-driven development strategy (国家创新驱动发展战略纲要). https://bit.ly/3v7ZCT4 (Accessed April 5, 2021).Google Scholar
Truex, R (2014) The returns to office in a “rubber stamp” parliament. American Political Science Review 108, 235251.CrossRefGoogle Scholar
Tsai, KS (2006) Adaptive informal institutions and endogenous institutional change in China. World Politics 59, 116141.Google Scholar
Tsai, K (2007) Capitalism without Democracy: The Private Sector in Contemporary China. Ithaca: Cornell University Press.Google Scholar
Walder, AG (1988) Communist Neo-traditionalism: Work and Authority in Chinese Industry. Berkeley: University of California Press.CrossRefGoogle Scholar
Walder, AG (1995) Career mobility and the communist political order. American Sociological Review 60, 309328.CrossRefGoogle Scholar
Walder, AG, Li, B and Treiman, DJ (2000) Politics and life chances in a state socialist regime: dual career paths into the urban chinese elite, 1949 to 1996. American Sociological Review 65, 191209.CrossRefGoogle Scholar
Wang, Y (2015) Tying the Autocrat's Hands. New York: Cambridge University Press.Google Scholar
Wintrobe, R (1998) The Political Economy of Dictatorship. New York: Cambridge University Press.CrossRefGoogle Scholar
Wong, J (2004) The adaptive developmental state in East Asia. Journal of East Asian Studies 4, 345362.CrossRefGoogle Scholar
Wu, W (2017, June 17) How the communist party controls China's state-owned industrial titans. South China Morning Post 65, 191209.Google Scholar
Xinhua, N (2015, August) Zhun Que Ren Shi Xi Fang Zheng Zhi Min Zhu Zhi Du [understand the western political democracy correctly]. https://bit.ly/3xlhHiH (Accessed July 10, 2016).Google Scholar
Yam, RC, Guan, JC, Pun, KF and Tang, EP (2004) An audit of technological innovation capabilities in chinese firms: some empirical findings in beijing, china. Research Policy 33, 11231140.CrossRefGoogle Scholar
Yan, X (2014) Engineering stability: authoritarian political control over university students in post-Deng China. The China Quarterly 218, 493513.CrossRefGoogle Scholar
Yang, C (2014) State-led technological innovation of domestic firms in Shenzhen, China: evidence from liquid crystal display (lcd) industry. Cities (London, England) 38, 110.Google Scholar
Figure 0

Table 1. Treatments

Figure 1

Figure 1. Ratings of applicants who participate in the Socialism with Chinese Characteristics Study Group and Western Political Philosophy Study Group compared with the rating of applicants who participate in the Comic Book Club on political loyalty as well as 17 other characteristics that may affect hiring decisions; coefficient estimate as well as 95 percent confidence interval for each rating comparison is shown.

Figure 2

Table 2. Summary statistics for callbacks in audit experiment

Figure 3

Table 3. Effect of conformity and non-conformity on callback rates

Figure 4

Figure 2. Heterogeneous effects on callbacks for non-conformity versus apolitical and conformity by ownership sector, with 95 percent confidence intervals.

Figure 5

Figure 3. Correlation between penalizing political non-conformity and topic proportion for industry groupings.

Figure 6

Figure 4. Hand-annotation of firms.

Figure 7

Figure 5. Heterogeneous effects on callbacks for non-conformity versus apolitical and conformity by firm characteristics, with 95 percent confidence intervals.

Supplementary material: Link

Pan and Zhang Dataset

Link
Supplementary material: PDF

Pan and Zhang supplementary material

Pan and Zhang supplementary material

Download Pan and Zhang supplementary material(PDF)
PDF 374 KB