Evaluation of the psychometrics of the Social Impact Scale and its association with depression among asymptomatic COVID-19 carriers

Abstract

Background

COVID-19 carriers experience psychological stresses and mental health issues such as varying degrees of stigma. The Social Impact Scale (SIS) can be used to measure the stigmatisation of COVID-19 carriers who experience such problems.

Aims

To evaluate the reliability and validity of the Chinese version of the SIS, and the association between stigma and depression among asymptomatic COVID-19 carriers in Shanghai, China.

Method

A total of 1283 asymptomatic COVID-19 carriers from Shanghai Ruijin Jiahe Fangcang Shelter Hospital were recruited, with a mean age of 39.64 ± 11.14 years (59.6% male). Participants completed questionnaires, including baseline information and psychological measurements, the SIS and Self-Rating Depression Scale. The psychometrics of the SIS and its association with depression were examined through exploratory factor analysis, confirmatory factor analysis and receiver operating characteristic analysis.

Results

The average participant SIS score was 42.66 ± 14.61 (range: 24–96) years. Analyses suggested the model had four factors: social rejection, financial insecurity, internalised shame and social isolation. The model fit statistics of the four-factor SIS were 0.913 for the comparative fit index, 0.902 for the Tucker–Lewis index and 0.088 for root-mean-square error of approximation. Standard estimated factor loadings ranged from 0.509 to 0.836. After controlling for demographic characteristics, the total score of the 23-item SIS predicted depression (odds ratio: 1.087, 95% CI 1.061–1.115; area under the curve: 0.84, 95% CI 0.788–0.892).

Conclusions

The Chinese version of the SIS showed good psychometric properties and can be used to assess the level of perceived stigma experienced by asymptomatic COVID-19 carriers.

Since first reported in Wuhan, Hubei Province, China at the end of 2019, COVID-19 subsequently spread rapidly all over the world, causing great concern.^1,2 COVID-19-infected pneumonia was included as a category B infectious disease, and the Chinese National Health Commission adopted the control strategies for category A infectious diseases in January 2020.³ In March, COVID-19 was declared to be a pandemic by the World Health Organization.⁴ Approximately 773 million individuals have been diagnosed with COVID-19 globally, resulting in more than 7 million deaths, as of the end of December 2023.⁵

Stigma in asymptomatic COVID-19 carriers

In addition to physical symptoms, patients with COVID-19 and asymptomatic carriers experience many psychological stresses and other mental health problems, such as depression, anxiety, stigmatisation, insomnia, symptoms of post-traumatic stress disorder, social isolation and loneliness.^6–9 Patients, carriers and healthcare workers of COVID-19 have experienced varying degrees of stigma. In Mexico, people have reported being physically assaulted and denied access to public transportation and jobs.¹⁰ Some companies in China have even added ‘never infected by COVID-19’ in recruitment requirements. The stigma associated with COVID-19 has mainly been in the forms of labelling and discrimination against groups from infected areas, stigmatisation and exclusion of carriers and close contacts, and internalised stigma in patients.¹¹ A systematic review and meta-analysis on 50 studies showed that the estimated prevalence of stigma during the COVID-19 pandemic was 35% (95% CI 26–44%), and the prevalence in China ranged from 44 to 62%.¹² As the pandemic progressed and more people contracted COVID-19, the public perception of COVID-19 has undergone a significant shift. However, it is important to acknowledge that stigma related to COVID-19 still exists and needs to be addressed because of its impact on mental health.^6–9 Also, the emergence of new variants, rebound infection and ongoing waves of the virus can create uncertainties and potentially lead to the resurgence of stigmatising attitudes. Moreover, stigma related to COVID-19 can intersect with existing social determinants of health, such as race, ethnicity and socioeconomic status. There are still subpopulations (e.g. patients with HIV) that face disproportionate stigmatisation and its associated consequences, and have limited or no access to prevention, care or treatment.¹³ Stigma may not only increase illness burden, mental stress and negative health outcomes for the people involved, but also further threaten the control of the COVID-19 pandemic, as well as the development of the economy and society. Asymptomatic COVID-19 carriers, who made up the majority of carriers of COVID-19 at the primary stage of the pandemic in various cities in China, may experience stigma similar to patients because of the highly infectious nature of COVID-19. It is imperative to explore levels of stigma in asymptomatic infected individuals.

Definition and measurements of stigma

Stigma was first examined by sociologist Goffman in 1963,¹⁴ and this work was further interpreted by subsequent researchers as the cooccurrence of labelling, stereotyping, separation, status loss and discrimination in the context of power being exercised.^15,16 Stigma associated with infectious diseases like AIDS and leprosy has received significant attention globally because of its adverse impact on patients’ health, disease management and public health.¹⁷ The Social Impact Scale (SIS) is a measurement tool for stigmatisation that was originally developed by Fife and Wright,¹⁸ and is based on modified labelling theory to measure stigma in patients with cancer and AIDS in 2000. They examined four dimensions of perceived stigma: social rejection, internalised shame, social isolation and financial insecurity. In 2007, Pan translated the SIS into Chinese and discovered that it had acceptable psychometric qualities concerning internal consistency, item validity, person validity, sensitivity and concurrent validity when applied to patients with schizophrenia, depression and HIV/AIDS in Taiwan.¹⁹ Some studies divided stigma into enacted stigma and anticipated stigma according to different modes of action. Enacted stigma refers to negative actions against people with diseases, such as individual discrimination and devaluation, and anticipated stigma reflects the anticipation of being discriminated against and stigmatised,^12,20 whereas others hold a different opinion.^21,22 A cross-sectional study in the early stage of the pandemic revealed that patients with COVID-19 had a SIS score of 57.37 ± 9.99, indicating a moderate stigma level.²³

Relationship between stigma and depression

Strong positive association between stigma and depression has been found in previous studies.²⁴ Mechanisms of stigma that patients with AIDS may experience include enacted stigma, anticipated stigma and internalised stigma, all of which are associated with depression.^25–27 Also, internalised HIV-related stigma is associated with higher levels of depression.²⁸ A study focusing on people recovering from COVID-19 in Santa Marta, Colombia, reported that depression was significantly associated with perceived stigmatisation (odds ratio 3.79, 95% CI 2.28–6.31) after adjusting for age, gender and income.²⁹ In the first year of the COVID-19 pandemic, global prevalence of depression increased by 25%.³⁰ According to a meta-analysis of 51 studies, the prevalence of depression in patients with COVID-19 was about 45%.³¹ Studies focusing on mental health-related problems in COVID-19 suggested that stigma is one of the main causes of negative thoughts and adverse mental health outcomes among patients with COVID-19.^32,33 It is important to explore the association between stigma and depression, with a view to reducing the incidence of depression in COVID-19 carriers and improving the recovery of patients with COVID-19.

Aims

The aim of this study was to test the reliability and validity of the Chinese version of the SIS among asymptomatic COVID-19 carriers. To accomplish this, a series of exploratory factor analysis, confirmatory factor analysis and receiver operating characteristic analysis were adopted to evaluate the psychometrics of the SIS, and its association with depression among asymptomatic COVID-19 carriers.

Method

Participants

Participant inclusion criteria were individuals who (a) were aged 18 years and above; (b) tested positive via SARS-CoV-2 nucleic acid test in respiratory specimens, and diagnosed as asymptomatic COVID-19 carriers and (c) were able to read the questionnaire. People who met the criteria and were willing to attend the study were recruited.

Procedure

A cross-sectional study was conducted from March to April 2022, in the Ruijin Jiahe Fangcang Shelter Hospital, Shanghai. Convenience sampling was used in this study. Trained healthcare professionals informed the asymptomatic COVID-19 carriers admitted into this hospital about this research and the potential benefits and risks of participating. The fieldworkers were responsible for recruiting participants, aiming to cover 80% of the patients admitted. After providing informed consent, participants were sent the online questionnaire link via WeChat Official Accounts (Tencent, Changsha, Hunan, China; see https://wx.qq.com/mp/), and completed the questionnaire anonymously on their smartphone. The questionnaire was released through the online survey platform ‘Questionnaire Star’ (Changsha Ranxing Science and Technology, Shanghai, China; see https://www.wjx.cn), which is a professional survey platform used by many Chinese researchers.

Ethical approval

The study was conducted according to the guidelines of the Helsinki Declaration of 1975, as revised in 2008, and approved by the Ethics Committee of Shanghai Jiao Tong University School of Medicine affiliated Ruijin Hospital (approval number LL202070). Written informed consent was provided by all participants.

Instruments and measures

Data on participants’ age, gender, marital status, education level, vaccination situation and whether a family member accompanied them in the hospital was collected, and the following self-report tools were applied.

Chinese version of the SIS

The 24-item SIS, published by Fife and Wright¹⁸ and translated by Pan, with separation reliability reaching 0.99,¹⁹ was developed to assess the level of stigmatisation for people with HIV/AID or cancer. Previous studies showed that the SIS examines four dimensions of perceived stigma: nine items on social rejection, three items on financial insecurity, five items on internalised shame and seven items on social isolation. Each item is rated from 1 (strongly disagree) to 4 (strongly agree). Total scores range from 24 to 96, and higher scores indicate more severe stigma.

Chinese version of the Self-Rating Depression Scale

The 20-item Self-Rating Depression Scale (SDS) is a widely used tool for assessing depression in general population.³⁴ Past studies have supported the scale's reliability and validity among various participants.³⁵ The response is rated on a four-point Likert scale (1 indicating never or little, four indicating most of the time or all the time). With half items reverse-scored, total scores range from 20 to 80. Higher scores reflect more serious depression, and a score of 53 or more on the SDS categorises individuals as having depression.³⁶

Data analysis

The primary aim of this study was to examine the psychometric structure of the SIS among people with COVID-19. First, the Shapiro–Wilk test and Spearman's rank test were applied to test whether the data were normally distributed and the correlation of the variables before factor analysis. Second, all participants were randomly separated into two samples at a 1:1 ratio, to conduct exploratory factor analysis and confirmatory factor analysis separately. Third, a parallel analysis was conducted to determine the number of factors that best portray the structure of the SIS in sample 1 after the Bartlett's test of sphericity. Then, exploratory factor analysis in sample 1 was conducted to explore the factor structure. Robust maximum likelihood estimator and oblique rotation was applied. Fourth, the factor structure of the SIS was confirmed by confirmatory factor analysis in sample 2, and the composite reliability was calculated. Then, multivariable logistic regression controlling for age, gender, marital status, education level, presence of a family member and vaccination status was adopted, and adjusted odds ratios and corresponding 95% confidence intervals were presented. Last, receiver operating characteristic analysis was applied to assess the accuracy of the SIS total scores in predicting depression. Analysis process was performed with the software R (R Foundation for Statistical Computing, Vienna, Austria; see https://www.R-project.org) version 4.2.1, on a Windows operating system.

As indices of model fit, the chi-squared test, comparative fit index (CFI), Tucker–Lewis index (TLI), root-mean-square error of approximation (RMSEA) and root-mean-square residual (SRMR) were reported. CFI and TLI values >0.90 and RMSEA and SRMR values <0.08 were identified as acceptable model fit metrics in this study.³⁷

Result

Description of the study sample

Of the 1750 patients admitted to the hospital, 1425 participants completed the online questionnaire, giving a response rate of 81.4%. A total of 142 participants failed to meet the inclusion criteria and were excluded. As shown in Table 1, the mean age of the 1283 participants was 39.64 ± 11.14 years, 59.6% were men and 73.9% were married. The average SIS score was 42.66 ± 14.61, and 2.7% of participants had an SDS score of ≥53, as shown in Table 1. There were no statistically significant differences on any of the demographics, SDS score or SIS score between sample 1 and sample 2. Intercorrelations of the SIS items for all participants are presented in Fig. 1, with coefficients ranging from 0.367 to 0.866 significantly. The results of the Shapiro–Wilk test (P < 0.001 for 24 SIS items) indicated that the responses were non-normally distributed. Therefore, Spearman correlation was applied and all items were significantly related.

Table 1 Demographic characteristics of the study sample (N = 1283)

SIS, Social Impact Scale; SDS, Self-Rating Depression Scale.

Fig. 1 Intercorrelations among Social Impact Scale items (N = 1283).

Construct validity

Exploratory factor analysis

The Kaiser-Meyer-Olkin measure of sampling adequacy was 0.968 and the P-value of Bartlett's test of sphericity was <0.05, indicating that the construct validity of the scale is adequate to conduct factor analysis.

To determine the number of factors that would best describe the Chinese version of the SDS, parallel analysis was conducted in sample 1, and the result suggested the number of factors was four (see Fig. 2). The SRMR of the four-factor model was 0.02, supporting the retention of the four-factor model.

Fig. 2 Parallel analysis scree plot.

The factor loadings of each item are presented in Table 2. Except for item 13 (‘I feel others think I am to blame for my illness’), all items measured the intended factor. Items 1–9 were loaded onto social rejection, items 10–12 measured financial insecurity, items 14–17 were loaded onto internalised shame and items 18–24 measured social isolation. Item 13, which measured internalised shame, had a larger loading value of social rejection and social isolation than that of internalised shame. The exclusion of item 13 improved the model fit (i.e. drop in χ ² of 27.08).

Table 2 Standardised loadings based on the correlation matrix in sample 1 (N = 643)

Bold indicates significance.

Confirmatory factor analysis

Fit statistics of the four-factor model in sample 2 indicated acceptable results. The CFI and TLI was 0.913 and 0.902, respectively, exceeding the cut-off score (i.e. 0.90). Also, the RMSEA was 0.088, which is slightly higher than 0.08 but still acceptable, and the SRMR was 0.056.^37,38 Estimated factor loadings for the four-factor model are displayed in Table 3. All items were loaded onto anticipated latent factors as exploratory factor analysis suggested significance, with factor loadings ranging from 0.509 to 0.836.

Table 3 Model estimated factor loadings, covariances, P-values and item-total correlations in sample 2 (N = 640)

* Significant at 0.05.

Reliability

Cronbach's alpha of the Chinese version of the 23-item SIS was 0.966 for the whole scale, 0.942 for social rejection, 0.893 for financial insecurity, 0.871 for internalised shame and 0.934 for social isolation, with satisfactory internal consistency. The composite reliability of each latent factor met the acceptable cut-off of 0.7, as shown in Table 3.³⁹

Criterion validity

The mean score of the SIS for participants with an SDS score <53 was 42.17 ± 14.43, whereas the mean score of the SIS for participants with depression was 60.11 ± 9.68. Logistic regression analysis (forward: LR method) indicated that the total score of the Chinese version of the 23-item SIS predicted depression, with an odds ratio of 1.087 (95% CI 1.061–1.115) after controlling for age, gender, marital status, education level, presence of a family member and vaccination status. For every score increment in the 23-item SIS, the risk of depression increases by 8%. A total of 2.7% of the participants’ SDS scores reached 53 (i.e. depression), and using the summed 23-item SIS scores to assess depression produced an area under the receive operating characteristic curve of 0.84 (95% CI 0.788–0.892) (Fig. 3), indicating moderate to good accuracy. Meanwhile, the cut-off value of the SIS to predict depression was 50, which means that individuals whose SIS scores exceed 50 may be at higher risk of depression. Based on this, a 2×2 table of the SIS and the SDS is shown in Supplementary Table 1 available at https://doi.org/10.1192/bjo.2023.651, and indicates that there were statistically significant differences (P < 0.001) in SDS scores between participants with high and low scores on the SIS.

Fig. 3 Receiver operator characteristic (ROC) curve for the Social Impact Scale as a predictor of depression (N = 1283). AUC, area under the ROC curve.

Discussion

The present study aimed to validate the Chinese version of the SIS among a group of asymptomatic COVID-19 carriers who experienced various discrimination after a positive SARS-CoV-2 nucleic acid test in respiratory specimens. As Yuan et al claimed, infectious disease-related stigma not only burdens the mental health of those who experience discrimination and hinders their social adaptation after recovery, but it is also detrimental to the control of a pandemic like COVID-19.¹² Thus, a reliable and valid instrument to assess the perceived stigma experienced by those who experience infectious disease is important for similar studies.

The average total scores of the SIS among asymptomatic COVID-19 carriers from Shanghai in this study was lower than that found by Lin et al in a study of patients with COVID-19 from Wuhan, China (57.37 ± 9.99 points)m²³ as well as that found in Yuan et al's study of COVID-19 survivors from Chongqing, China (70.2 ± 12.9).⁹ Considering that the participants in those studies were patients with COVID-19 patients in 2020, little was known about COVID-19 except that it was highly infectious at the beginning of the outbreak. People were so afraid of the disease that the related persons experienced various forms of discrimination and harassment at that time. With the pandemic progressing and the development of COVID-19 vaccines, more knowledge about COVID-19 became available, people learned more about the infection and became less afraid of the disease than they were initially. Meanwhile, lived experiences (e.g. infection with COVID-19, knowing others who have COVID-19, receiving the COVID-19 vaccine) may have an even stronger impact on stigma. Evidence has shown that social contact was the most effective type of intervention to improve stigma-related knowledge and attitudes in the short term, and contact-based interventions can reduce mental health-related stigma.^40,41 The reduced level of stigma among asymptomatic COVID-19 carriers in Shanghai may be a testament to the effectiveness of contact intervention in alleviating stigma.

As the parallel analysis indicated, the four-factor structure of the Chinese version of the SIS among asymptomatic COVID-19 carriers was consistent with previous studies in groups of individuals diagnosed with major depression, schizophrenia, HIV/AIDS and cancer.^18,19 The exploratory factor analysis showed that 24 items of the SIS were loaded onto the intended factor, except for item 13 (‘I feel others think I am to blame for my illness’), and the exclusion of item 13 improved the model fit. The SIS was first developed to measure stigma and compare the impact of stigma on the self of persons with HIV/AIDS and cancer.¹⁸ COVID-19 shared similarities and differences with these illnesses. One explanation is that pandemic-related infectious diseases like COVID-19 are mainly caused by external factors, and broke out so rapidly that it was difficult to blame someone for being infected. COVID-19-affected persons may not blame themselves entirely for the experienced discrimination and rejection, but rather COVID-19 itself. Another explanation is that there exists a discrepancy between people's understanding of this item in Chinese and the meaning expressed in the original English version of the item. In addition, a pool of 30 items was identified during the construction of the original scale by Fife and Wright.¹⁸ Six of the items did not fall into any theoretical dimensions being examined in further analysis and were therefore excluded. Item 13 may be not appropriate to evaluate the internalised shame dimension of stigma perceived by asymptomatic COVID-19 carriers, and can be excluded. The confirmatory factor analysis indicated acceptable model fit of a four-factor model of the 23-item SIS, and confirmed that social rejection, financial insecurity, internalised shame and social isolation were distinct but corelated positively with each other. Furthermore, the 23-item SIS showed good internal consistency and composite reliability, consistent with Pan et al’ study using the Rasch measurement model.¹⁹ As Fife and Wright suggested, these aspects of stigma as measured by the SIS seem to describe a common experience of being stigmatised, although the level of social impact of stigmatisation can be inspected from different aspects.¹⁸ The exclusion of item 13 shed light on future studies applying the SIS to measure stigma in COVID-19-affected populations.

The theory that depression is a response to stigmatisation has been tested through both theoretical and empirical research.^42,43 The results of the present study are concordant with the notion that people with higher SIS scores may report more depressive symptoms before and after controlling for demographic characteristics. Further, the result of the receiver operating characteristic analysis indicated that the 23-item SIS has good criterion validity. Individuals whose SIS scores exceed 50 may be at higher risk of depression, suggesting possible comorbidity or causal relationships that need to be further explored by prospective study design. On the other hand, the SIS quantifies the level of stigma among asymptomatic COVID-19 carriers, and therefore allows for an assessment of the effectiveness of interventions to reduce social discrimination and exclusion (and thereby stigma) of COVID-19-affected individuals, and comparison of the level of stigma during and after the pandemic to explore which factors have a greater impact on individuals’ stigma. Despite the positive changes in public perception of COVID-19, stigma related to COVID-19 still exists and needs to be addressed. Understanding and addressing COVID-19-related stigma is crucial to promoting public health and well-being. As long as there is a risk of contracting the virus, there is a need to educate and raise awareness to prevent the re-emergence of stigma and its negative consequences, both on individuals and on efforts in controlling the pandemic. Further studies could explore the evolving public perception of COVID-19 and how it affects individuals’ experiences, during acute phases of the pandemic and in its aftermath. Understanding the factors influencing shifting attitudes toward COVID-19, including the role of scientific evidence, public health messaging and personal experiences, can contribute to more targeted and effective interventions to reduce stigma, and promote a supportive and inclusive response to the ongoing challenges posed by the virus.

To the best of our knowledge, this is the first study to validate the Chinese version of the SIS that has used factor analysis among asymptomatic COVID-19 carriers, although several studies have applied the SIS to measure stigma related to COVID-19 among COVID-19 patients and survivors.^9,23 Notwithstanding some of the strengths of this study, some limitations exist in this study. First, this is a cross sectional study and a causation relationship cannot be derived. Second, participants’ other psychological states, such as loneliness and anxiety, were not assessed, nor were they re-tested with the SIS after a period of time. Therefore, an evaluation of the convergent validity, discriminant validity and retest validity of the SIS cannot be achieved in this study. Future research could confirm criterion validity and the relationship with more adverse mental health outcomes – for example, assessing anxiety by using the Generalized Anxiety Disorder-7 (GAD-7), and well-being by using the World Health Organization Five Well-Being Index (WHO-5) – or test–retest reliability through prospective research. Finally, the current sample had a greater proportion of male participants and people with a high school education or less compared with general residents in Shanghai, which limits its representation and the generalisation of the findings.

In conclusion, the Chinese version of the SIS shows good psychometric properties and can be used to assess the level of perceived stigma experienced by asymptomatic COVID-19 carriers during the outbreak. This study provides a reference for future studies focusing on the quantitation of stigma in COVID-19-affected populations.