The societal impact of individual placement and support implementation on employment outcomes for young adults receiving temporary health-related welfare benefits: a difference-in-differences study

Abstract

Background

Individual placement and support (IPS) is an evidence-based practice that helps individuals with mental illness gain and retain employment. IPS was implemented for young adults at a municipality level through a cross-sectoral collaboration between specialist mental healthcare, primary mental healthcare, and the government funded employment service (NAV). We investigated whether IPS implementation had a causal effect on employment outcomes for all young adults in receipt of a temporary health-related rehabilitation (work assessment allowance, WAA) welfare benefit, measured at the societal level compared to municipalities that did not implement IPS.

Method

We used a difference in differences design to estimate the effects of IPS implementation on the outcome of workdays per year using longitudinal registry data. We estimate the average effect of being exposed to IPS implementation during four-years of implementation compared to ten control municipalities without IPS for all WAA recipients.

Results

We found a significant, positive, causal effect on societal level employment outcomes of 5.6 (p = 0.001, 95% CI 2.7–8.4) increased workdays per year per individual, equivalent to 12.7 years of increased work in the municipality where IPS was implemented compared to municipalities without IPS. Three years after initial exposure to IPS implementation individuals worked, on average, 10.5 more days per year equating to 23.8 years of increased work.

Conclusions

Implementing IPS as a cross sectoral collaboration at a municipality level has a significant, positive, causal, societal impact on employment outcomes for all young adults in receipt of a temporary health-related rehabilitation welfare benefit.

Introduction

Individual placement and support (IPS) is an evidence-based practice that helps individuals with mental illness gain and retains employment (Bond, 2004). It is a form of supported employment that is integrated with mental health services to provide comprehensive multidisciplinary support. IPS is manualized (Becker & Drake, 2003) with a fidelity scale (Bond, Peterson, Becker, & Drake, 2012) which assesses whether it is being implemented as intended. IPS has been shown to be both the most effective and cost-effective way of supporting individuals with mental illness into employment with over 27 randomized controlled trials (RCTs) finding employment rates to be more than doubled in IPS compared to other vocational approaches (Brinchmann et al., 2020; Park et al., 2022). Internationally, observational studies demonstrate IPS can be implemented into routine clinical practice to good fidelity with local contextual adaptations (Bond, Lockett, & van Weeghel, 2020; Richter & Hoffmann, 2019). Based on the effectiveness for individuals with mental illness, IPS is expanding, with positive emerging findings, to serve health conditions beyond mental illness (Bond, Drake, & Pogue, 2019) including young adults at risk of early work disability (Sveinsdottir et al., 2020). At a macro-economic level, IPS effectiveness is found to be independent of gross domestic product, unemployment rates, generosity of welfare benefits, or type of integration policies (Brinchmann et al., 2020).

For people with mental illness there is good RCT evidence for IPS at the individual level (de Winter et al,. 2022) and emerging positive RCT evidence for other health conditions (Probyn et al., 2021). However, there is a lack of evidence for a societal impact (Boardman & Rinaldi, 2013) and a need for a higher order test beyond individual level efficacy and effectiveness RCTs. This study reports on the implementation of IPS as a cross sectoral collaboration at a municipality level for young adults with mental illness and in receipt of a temporary health-related rehabilitation welfare benefit. An assumption was made that by implementing IPS as a cross sectoral collaboration it would influence employment outcomes that extend beyond the target group as the implementation of IPS would impact on the ways of working across both specialist and primary mental healthcare, and the government funded employment service. It is important to test this hypothesis because of the potential population health and economic benefits as well as implications for societal well-being. Considering this, the aim of this study is to test whether IPS implementation within a municipality area has an effect on employment outcomes for all young adults in receipt of a temporary health-related rehabilitation welfare benefit, measured at the societal level compared to municipalities that did not implement IPS.

Methods

Setting

The intervention municipality was Bodø, which is the second largest city in northern Norway and the capital in Nordland County. The municipality has approximately 50 000 inhabitants and a population density of 39.3/km². The specialist mental health services in Bodø provide both inpatient and outpatient care and in primary care there is a mental health outreach service for people with mental illness who need longer-term support based on the nature, duration, and complexity of their needs. The city has a government funded employment service (NAV) which provides all employment and welfare services.

IPS implementation

IPS was implemented at a municipality level through a cross sectoral collaboration, led by specialist mental health services with the primary care outreach service and NAV. An implementation support team included a ‘change agent’ within each sector responsible for the preparation and implementation of IPS. Throughout the implementation, clinicians, NAV frontline staff and leaders were frequently brought together for education, training, and guidance about IPS and associated ways of working to counteract the traditional silos between services. To understand the impact of this, repeated testing of NAV staff attitudes towards IPS happened in 2013 and 2017 (Brinchmann et al., 2022).

Two implementation frameworks were used during the preparation and implementation stages: The New Hampshire-Dartmouth Research Center Toolkit (Swanson, Becker, Drake, & Merrens, 2008) with the IPS fidelity scale and, the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Aarons, Hurlburt, & Horwitz, 2011) to understand the inner and outer contexts within the implementation and the interplay between them. For a review of the outer context see (Moe et al., 2021).

IPS implementation occurred in three stages: a preparation stage (2010–2012), an implementation stage (2013–2016) and a sustainability stage (2017–2019). Table 1 shows the preparation and implementation stage factors, timeline, implementation context and process outcome data including independently assessed fidelity scores.

Table 1. Preparation and implementation stage factors, timeline, implementation context, and process outcome data

¹ Inner context is understood as micro- and meso-level influences, whereas the outer context refers to macro-level influences.

Target population for IPS

The target population for IPS were young adults (18–40 years) receiving support from a multidisciplinary psychosis team within specialist mental health services, those receiving support from the primary care mental health outreach service and, receiving the work assessment allowance (WAA) welfare benefit. Clinicians were instructed that individuals they considered being unable to pursue life goals such as employment could be included. The WAA is the only temporary health-related rehabilitation benefit in Norway and is available to individuals assessed as having at least a 50% reduced work capacity due to a medical condition (National Insurance Act, 2017).

Study population

Norwegian inhabitants aged 18–40 with an ongoing WAA in Bodø municipality or ten comparable control municipalities without IPS were our study population. Control municipalities were selected a priori based on KOmmune STat RApportering (KOSTRA) reporting from Statistics Norway (SSB). The KOSTRA report classifies Norwegian municipalities into “population size, economic workload, and economic capacity. Economic workload and capacity measures are estimated by the local government spending behavior model and depend on local government income, socio-demographic factors and geographic variables” (Kringlebotten & Langørgen, 2020). Control municipalities were Kongsberg, Lier, Røyken, Horten, Tønsberg, Larvik, Faerder, Porsgrunn, Grimstad, and Steinkjær.

Study data source

We used high quality longitudinal registry data collected and linked by NAV. Demographics, contractual man-days (defined as “the number of days a person has agreed to work for his employer in a given period, adjusted for fraction of employment, weekends and public holidays.” (Statistikk sentralbyrå (Statistics Norway), 2000), WAA, and diagnoses were included in the dataset. WAA was originally recorded with exact start and stop dates. WAA main diagnoses were registered using either International Classification of Diseases (ICD-9 or 10) or International Classification of Primary Care (ICPC, ICPC-1, ICPC-2). Before 2015, workdays were reported quarterly; after 2015, monthly. Workdays per month/quarter were merged into ‘workdays per year’ for comparison across the study period.

Longitudinal data from 2010–2019 enables us to follow individuals. Deaths and migrations are included for the time they were present. To avoid selection bias, first-time WAA exposure in the intervention group (Bodø), where IPS was implemented, is compared to first time WAA exposure controls. Thus, both controls and intervention groups had WAA-triggering health conditions the same year.

Study design

Registry data allowed us to use a longitudinal interrupted time series quasi-experimental design, one of the strongest non-experimental difference-in-differences (DID) estimate methods that facilitates causal inference when randomization is not possible (Leatherdale, 2019). We used a DID to estimate the effects of IPS implementation on workdays per year. DID estimates the average treatment effect on the treated group (ATET). We estimate the ATET of being exposed to IPS implementation in Bodø during four-years of implementation (2013–2016). IPS-exposure is estimated for all Bodø WAA recipients.

The DID framework is based on two differences: the difference in outcome before and after treatment for both controls and treatment groups and, the difference in mean outcome between the two groups. This second difference, given some restrictions, provides unbiased estimates of the effect of interest.

Given the longitudinal format and repeated observations on each individual, we specify a fixed effects panel data model for the DID analyses.

(1.1)$$Y_{ict} = \alpha _i + \Upsilon _t + z_{ict}\beta + {\rm D}_{ct}\delta + \varepsilon _{ict}$$

Here, y_ict represents the dependent variable ‘work-days-per-year’ for individual i at time t which ranges from year 1 to 7, where 4 is the intervention year. Thus, we follow individuals for three years before and after intervention. The group-level variable c denotes city of residence. α _i are the individual fixed effects and ϒ_t are time fixed effects. z _ict are time-varying covariates depicting marital status and children, and e _is is the error term. D _ct denotes IPS-exposure that varies over time and municipality-level. IPS was implemented in Bodø in 2013–2016, and the DID model in 1.1 is estimated for the four years combined, thus providing an overall effect of the program.

The fixed-effects procedure has great strengths. It allows the control for effects of measured and unmeasured time-constant variables and unmeasured variables need not be independent from the measured (Petersen, 2004). Unfortunately, these advantages only allow estimation of time-varying variables. The fixed effects estimator uses the within-individual-level deviation from the mean of each variable across time; it is not possible to estimate the effects of time-constant background variables. Thus, variables such as gender, country of birth, and family background can only be controlled for, but not estimated directly in the fixed-effects model.

Unbiased estimates rely on two assumptions. Firstly, there are parallel trends between controls and treatment before the intervention to assure the effects are not driven by trends not related to the intervention. Second, the parallel development would have been the same without the intervention. Only the first assumption is testable.

Figure 1 shows trend plots used to assess the parallel-trends assumption. The left-hand plot depicts the mean outcome over time for treatment and control groups. The right-hand plot incorporates interactions of time with a treatment indicator into our DID model and calculates predicted values of our augmented model for both groups. The vertical lines indicate one year before treatment. Additional F-tests on the trajectories of the mean number of workdays confirms the null-hypothesis of parallel trends cannot be rejected.

Figure 1. Parallel trend plots to assess the parallel-trends assumption².

²Internal validity of DID models rely on the parallel trends assumption: That there are parallel trends between controls and treatment before the intervention to ensure the effects are not driven by trends unrelated to the intervention.

Post treatment effects over time

Rather than assuming a single treatment-effect estimate is constant, we examined ATET changes over time. We fitted a DID model that included lags and leads of an indicator at the time of IPS initiation. Lag coefficients were used to evaluate any changes in ATET during the post treatment era. Granger plots (Fig. 2, online Supplementary Appendix Figure 2) illustrate pre- and post-intervention treatment effects of IPS implementation in Bodø.

Figure 2. Granger plot – post treatment effects over time⁷.

⁷Granger plots show time-specific treatment effects. Time 1–3 represents the pre-exposure period, the three years before an individual received WAA. Time 4 represents the year an individual started receiving the WAA for the first time (and was thus exposed to IPS implementation). Time 5–7 corresponds to the post-exposure period, i.e. the three years following initial WAA receipt/exposure to IPS implementation.

Testing if the IPS effect is dependent on diagnosis by triple difference estimation

We used a triple difference method (DiDiD) (Olden & Møen, 2022), an extension of the DiD method, to delve deeper into the impact of IPS implementation across four diagnostic subgroups: (1) all non-organic mental disorders, (2) severe mental illness (SMI), (3) non-severe, non-organic mental disorders, and (4) somatic disorders. The DiDiD method enables a more nuanced causal inference by introducing a third layer of comparison (in this case, diagnostic subgroups). By doing this, we aimed to isolate and estimate the causal effects of IPS exposure within each diagnostic category while controlling for potential biases due to time trends, geographic variations, and other unobserved heterogeneities. The DiDiD approach can estimate if the causal impact of the IPS intervention varied systematically across different diagnostic groups, thus providing a more comprehensive and detailed understanding of the intervention's effectiveness and applicability across diverse patient groups in the context of workdays per year.

The DiDiD estimator is computed as the difference between two difference-in-difference estimators. In our case, the differences between the broad group of WAA participants in Bodø and controls as well as the difference between the diagnostic subgroups in Bodø and controls. The triple difference estimator does not require two parallel trend assumptions for a causal interpretation (Olden & Møen, 2022).

The fixed effects triple-difference model is given by

(1.2)$${\rm y}_{icst} = \alpha _i + \Upsilon _t + \Upsilon _t\Upsilon _c + \Upsilon _t\Upsilon _s + z_{ict}\beta + {\rm D}_{ct}\delta + \varepsilon _{icst}$$

In addition to the elements in 1.1, the triple-difference model in 1.2 incorporates the interactions of the group level variables and time. Thus, the city of residence c is interacted with time t, as well as the diagnostic group-variables with time t.

Ethics and consent statement

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. All procedures involving human subjects/patients were approved by The Regional Committee for Medical and Health Research Ethics Region North, Norway, approval number: 2012/2239. The ethics committee waived the need for individual consent for this study, given that the register data used are in an anonymized and in a de-identified format.

Results

Descriptives

Bodø and controls were comparable across most demographic variables (Table 2). Our sample is fairly homogeneous, made up of individuals who are on average in their late-20s. While women are generally overrepresented, Bodø had 5.6% (p = 0.01) more females than the control group. Bodø residents were also significantly less likely than controls to be married/de-facto (p = 0.003) although their average number of children was similar. Bodø had a slightly lower proportion of individuals with SMI and other non-organic mental disorders and a slightly higher proportion with somatic disorders compared to controls.

Table 2. Demographics and diagnostic distribution³

³ Measured at first time of WWA recipiency.

⁴ Note that percentages may not add up to 100 due to rounding up.

⁵ The ̈Missing̈ category indicates that this data was missing from the register.

⁶ Severe mental illness as defined in the Norwegian Opptrappingsplan for psykisk helse (Escalation plan for mental health) (2023–2033): Substance use disorders, severe bipolar disorders, major depressive disorder, schizophrenia, and personality disorders.

Causal effects of IPS exposure: difference-in-differences

Our analyses using the DiD method found that exposure to IPS implementation has a significant, positive, effect on workdays per year at a societal level. The ATET of IPS implementation was 5.6 (p = 0.001) increased workdays per year per individual. This is equivalent to 3141.6 increased workdays per year for the total Bodø sample (n = 561). In Norway, there are 248 workdays per year, an ATET of 5.6 workdays per year corresponds to 12.7 (3141.6/248) increased years of work for the whole group exposed to IPS implementation.

The associated granger plot (Fig. 2) indicates the treatment effect improves over time. The coefficients on leads for the first three years (time 1–3) are close to 0, indicating no anticipatory effects prior to IPS-exposure. However, following initial exposure (time 4), treatment effects increased steadily throughout the post-exposure period (time 5–7). Three years after initial exposure to IPS implementation (time 7), Bodø residents worked, on average, around 10.5 more days per year equating to 23.8 years of increased work.

Causal effects of IPS exposure: triple difference

While all our analyses using the DiDiD method were statistically insignificant (Table 3), they do suggest that exposure to IPS implementation was more effective in the context of workdays per year for individuals who receive the WAA due to mental disorders than it is for those who receive the WAA recipients due to somatic disorders.

Table 3. Triple difference results

Sensitivity check

Our results came from Bodø or control municipalities residents with valid-observation years across the observation period. Thus, contributing to the estimates for the years they were present in the municipality. This is comparable to an ‘intention to treat’ RCT design.

A design could include only those who are resident in the municipalities over the full observation period which would be comparable to an RCT design including only the treated. Excluding the possibility of selection effects driving our results, we ran analyses excluding those who died, moved, or migrated. 980 individuals were lost to follow-up. 682 moved to another municipality, 31 died, 13 migrated, and 254 were unknown.

Descriptive statistics for this second analytical sample (online Supplementary Appendix Table 1) are markedly like the first analytical sample (Table 1). The most notable difference is within both groups there was a slightly lower proportion granted WAA due to non-organic mental disorders and a slightly higher proportion with somatic disorders (Table 1, online Supplementary Appendix Table 1).

Parallel trends plots (online Supplementary Appendix Figure 1) and F-tests confirmed the parallel trends assumption was fulfilled for this narrower study population and DID analysis again found a significant positive result in favor of Bodø, with ATET of 5.9 (p = 0.002) workdays per year, corresponding to a societal impact of 11.0 increased years of work for the whole treatment group. Furthermore, the associated granger plot (online Supplementary Appendix Figure 2) shows the effect of IPS exposure improved over time, after three years the ATET was around 9.5 workdays per year equating to 17.8 increased years of work for the treatment group.

DiDiD estimates, excluding those lost to follow-up, were all statistically insignificant (online Supplementary Appendix Table 2).

Discussion

We tested the bold assumption that implementing IPS as a collaborative partnership within a municipality would have a societal impact on the employment outcomes for young adults who received WAA. We found a significant, positive, effect on societal level employment outcomes corresponding to 5.6 (p = 0.001) increased workdays per year per individual which is equivalent to 12.7 years of increased work where IPS was implemented, compared to municipalities without IPS. The effect found is measured for a large population, all WAA recipients, not just those who received IPS employment support, or individuals with mental illness. Additionally, the effect improves over time, three years after initial exposure to IPS implementation individuals worked, on average, around 10.5 more days per year equating to 23.8 years of increased work. When carefully conducted, quasi-experimental designs can be a robust alternative to RCTs (Kontopantelis, Doran, Springate, Buchan, & Reeves, 2015). Assuming one accepts the premises of the statistical model and that the assumptions have been satisfied; longitudinal interrupted time series quasi-experimental design models provide unbiased estimates. However, and given the design of the study, the analytical approach does not allow the direct identification of the mechanism mediating the effect. In our case the effects can be the result of two separate mechanisms or the combination of them. Thus, the estimated effects can be a direct cause of IPS participation for the approximately 200 IPS participants, or it can be a spill-over effect stemming from the larger WAA population of Bodø. Thirdly, and most likely, the estimated effect from IPS can be a combination of direct and spill-over effects.

As far as we know, this is the first study to investigate a societal impact of IPS implementation on employment outcomes. To date, RCTs demonstrate the effectiveness of IPS for individuals with mental illness (de Winter et al., 2022) with emerging evidence for other populations (Bond et al., 2019; Probyn et al., 2021; Sveinsdottir et al., 2020). The majority of IPS implementation studies demonstrate effectiveness at the individual level with only one study demonstrating a population level impact on the employment rates of individuals using specialist mental health services (Rinaldi, Montibeller, & Perkins, 2011).

From an implementation perspective, the estimated direct and spill-over effects found have several possible explanations. IPS implementation was a purposeful collaborative partnership between specialist and primary mental healthcare services, and NAV with the aim to implement the values, principles, and practice of IPS across each organization. It is therefore unsurprising to find that exposure to IPS implementation was more effective for individuals with mental illness than it was for those with somatic disorders. NAV frontline staff and primary and specialist mental healthcare professionals received extensive IPS training and technical assistance before and during implementation. The change agents actively used the inner context implementation outcome data to enhance implementation efforts and improve the quality of services. Whilst health professionals’ attitudes to individuals with mental illness gaining employment are well documented (Finne & Holt, 2023; Lettieri, Soto-Pérez, Díez, Bernate-Navarro, & Franco-Martín, 2022) it was important for the implementation team to understand the attitudes of NAV frontline staff as they are pivotal in the assessment, decision-making and trajectories of all WAA and Disability Pensions claimants. NAV frontline staff in Bodø were consistently more positive towards the evidence-based principles of IPS and associated ways of working compared to municipalities where IPS was not implemented (Brinchmann et al., 2022). Media (newspapers and social media) were actively used to frame the unemployment of individuals with mental illness as a community challenge. The collaborative partnership ensured IPS was embedded within each organization's broader strategies whilst the employment specialists and the implementation team worked horizontally and vertically across the organizations to bridge the silos between specialist and primary mental healthcare and NAV. Frequent collaborative meetings brought together leaders, clinicians, employment specialists and frontline NAV staff which we believe provided an implementation mechanism to help to counteract the traditional silos of services, supported the diffusion and spread of IPS, challenged stigma and discrimination for individuals with mental illness whether or not they received IPS and provided better continuity of support for individuals across the organizations.

The IPS service received ‘good’ ratings from independent fidelity reviews. Though, short-term annual project funding caused a high turnover of employment specialists which appears to be a common phenomenon (Butenko et al., 2022). However, all employment specialists who left their IPS roles continued to support unemployed individuals with mental illness or somatic disorders to gain and retain employment within Bodø. They left to work in NAV, health services or private vocational rehabilitation agencies which may have further supported the spill-over effect found.

Regardless of the merits of IPS as an intervention, how such interventions are implemented within and across systems matters. In most countries, health services and government funded employment services operate independently of each other, with different aims and objectives along with different approaches and are often organized under different government departments. Since 1997, Norwegian health policy has prioritized the employment of individuals with mental illness (Ministry of Health & Care Services, 1997) and in 2007, the Ministry of Labour and Social Inclusion and, Ministry of Health and Care Services jointly published a national strategic plan for work and mental health (Ministry of Health & Care Services & Ministry of Labour & Social Inclusion, 2007). This policy framework highlighted IPS and recognized the need for coordinated support from health and social services and the Labour and Welfare Administration to support individuals with mental illness to be able to work. The effects found in our study support the use of multisectoral and collaborative approaches to the implementation of IPS. Individuals exposed to IPS implementation had a shorter duration on WAA before returning to employment suggesting they received an early vocational intervention with support that was personalized and addressed their needs.

There are several strengths to our study. Control municipalities were selected a priori, and registry data was used for the main outcome measure. Before being approved for research, registry data is subjected to rigorous quality controls. The study is well powered, and causality is assured as the parallel trend assumptions for a DID were met. The NAV interventions in the control municipalities were also available in Bodø. Finally, author SW under the supervision of author TL, neither involved in the IPS implementation, performed the statistical analysis. There are several limitations. Whilst well powered, this is an n = 1 study and our findings warrant replication. There could be a bias to something else occurring however, to the best of our knowledge we are unaware of other initiatives occurring in the control municipalities and, NAV financial allocations are per capita. Unemployment rates across all the municipalities ranged from a 1% decrease to a 1.8% increase during the study period; however, IPS effectiveness is not moderated by unemployment rates. This study addressed societal employment outcomes and the impact on welfare benefits is unknown but will be addressed through a future publication. Finally, we do not know whether the higher employment outcomes come at the expense of lower hourly wages though, IPS is typically associated with higher wages earned (Bejerholm, Areberg, Hofgren, Sandlund, & Rinaldi, 2015; Burns et al., 2007; Drake et al., 1999).

This study is the first in the IPS literature to move from RCTs or observational studies at the individual level to showing the relationship between IPS implementation, a societal impact on employment outcomes for individuals on temporary health-related welfare benefits and a policy effect. The findings have implications for population health and economic benefits as well as implications for societal well-being. The traditional separation of health services from employment and education services typically results in those individuals with the greatest need not receiving effective approaches or support to enable them to achieve their goals. This separation can, in part, be driven by attitudes but also by siloed government funding. Instead, by integrating services through multisectoral and collaborative approaches, there is an impact that is larger than the sum of its parts.