Entry Assessment of Student Learning Preconditions in Higher Education: Implications for the Transition from Secondary to Tertiary Education in Germany

Abstract

The acquisition of domain-specific knowledge and interdisciplinary skills such as critical thinking is increasingly gaining significance as key learning outcomes in higher education that are crucial for all professionals and engaged citizens and that enable lifelong learning. Despite this socio-political consensus, up until the last decade there have only been a few evidence-based insights into the competencies of higher education students. Therefore, the Germany-wide research program Modelling and Measuring Competencies in Higher Education (KoKoHs) was established in 2011 by the Federal Ministry of Education and Research. In the 85 projects, theoretical-conceptual competence models and corresponding assessments were developed for selected large study domains (e.g. economics) to reliably measure the students’ competencies in different phases of higher education (entering, undergraduate, graduate). More than 100 technology-based assessments of both discipline-specific competencies and generic skills were validated across Germany at over 350 universities with over 75,000 students. This article presents findings from the Germany-wide entry diagnostics in the one KoKoHs project (WiWiKom II) with beginning students in business, economic and social sciences that provide evidence-based insights into students’ learning preconditions and their impact on domain-specific knowledge acquisition in bachelor’s degree courses. The results lead to far-reaching practical implications for successful transitions between secondary and tertiary education, including recommendations for the development of mechanisms to support access to tertiary education and to prevent high dropout rates.

The beginning of this century was marked by the introduction of the Bologna reforms (Adelman 2009). In this context, ‘Diplom’ degree courses were restructured into bachelor’s and master’s degree courses. With this reform, the European Union (EU) aimed to make its higher education system more competitive compared with the well-established Anglo-Saxon system and to simplify cross-national transitions within the EU (European Commission/EACEA/Eurydice 2018). The uniform system was meant to contribute to more transparency and flexibility for students across the EU and other countries and thus to the globalization of the EU higher education space and greater student mobility. Expanded student exchange programmes as well as increasing migration (including refugee migration) have contributed both to higher numbers of prospective students in the EU as well as to an extreme heterogeneity when it comes to the students’ preconditions when entering university (Zlatkin-Troitschanskaia et al. 2018a; OECD 2018).

In the past decade, higher education stakeholders as well as practitioners have increasingly criticized that the Bologna reforms have hardly contributed to the transparency and comparability of degrees (Söll 2017; Crosier and Parveva 2013). On the contrary, degrees and grades are hardly comparable both nationally and internationally. Studies by the Organization for Economic Co-Operation and Development (OECD) assessing the knowledge and skills of students in different countries (e.g., AHELO, OECD 2012, 2013a, 2013b) confirm the observed deficits. Particularly in mass study programmes such as Business and Economics (B&E), practitioners criticize the extreme heterogeneity of students’ preconditions upon entering higher education (Klaus et al. 2016). Their lack of previous knowledge relevant to the respective degree courses and generic skills such as language skills and self-regulation are considered a particular challenge (Zlatkin-Troitschanskaia et al. 2017; Berthold et al. 2009; KMK 2009). As a consequence, many degree programmes are characterized by extremely high dropout rates (Heublein et al. 2010, Heublein 2014; Federal Statistical Office 2017; OECD 2010, 2018).

Due to the practical challenges, there have been several efforts in the past decade to develop instruments for the valid and reliable assessment of the entry requirements of students (for an overview, see Zlatkin-Troitschanskaia et al. 2017; Zlatkin-Troitschanskaia et al. 2018b). Additionally, numerous support programmes for beginning students have been implemented at German universities (Klaus et al. 2016). However, both practical experience and numerous studies show that such programmes have not been sufficiently effective so far and that a stronger internal differentiation with regard to the individual needs of the students is necessary (Zlatkin-Troitschanskaia et al. 2017; Jaeger et al. 2014). An elaborate diagnosis of learning preconditions at the beginning of a course of study is a necessary prerequisite for providing effective and individualized offers to the students on this basis (Hell et al. 2008; Heinze 2018).

The approach of Goldman and Pellegrino (2015) describes the connected Triad of Curriculum-Instruction-Assessment and states that curricular and instructional measures are to be developed on the basis of valid assessments that reliably measure study results (in the form of student learning outcomes), and curricula and instruction should then be aligned accordingly. However, evidence suggests that this alignment is currently deficient in educational practice. A particular challenge in higher education as described by Shepard and colleagues (2018) is that the programme designs, instructional activities, and assessment strategies commonly used in higher education do not lead to the desired student learning outcomes. This was one of the focal points of the Germany-wide large-scale research program Modelling and Measuring Competencies in Higher Education (KoKoHs), which was established by the German Federal Ministry of Education and Research in 2011 (for an overview, see Zlatkin-Troitschanskaia et al. 2017; Zlatkin-Troitschanskaia et al. 2020). In this research programme, the focus has been on the challenge of developing valid assessments for student learning outcomes for the past 10 years, and on the following superordinate research questions (RQs):

1. (1) What levels of study-related knowledge and skills do students have when they enter higher education? (RQ 1)

2. (2) How do a student’s domain-specific knowledge and skills change over the course of studies? (RQ 2)

3. (3) Which factors influence the students’ learning and their competence development in higher education? (RQ 3)

In this contribution, an overview of the projects and research work conducted in the KoKoHs programme will be presented with a particular focus on the study entry diagnostics and therefore on RQ 1. In addition, we present findings regarding RQ 2, which also show to what extent the acquisition of knowledge and skills over the first year of study is influenced by students’ preconditions upon entry into higher education (RQ 3). The presented findings from the Germany-wide entry diagnostics provide important evidence-based insights into students’ preconditions and lead to far-reaching practical implications for successful transitions between secondary and tertiary education. This includes recommendations for the development of mechanisms to support access to tertiary education and to prevent high dropout rates.

Entry Assessments in Higher Education in the Context of the KoKoHs Programme

To draw conclusions about the development of students’ competencies over the course of higher education studies, their level of study-related knowledge and skills must be validly assessed upon entry into higher education. Many national and international studies have shown that grades and degrees in secondary school education do not provide valid and reliable information about the knowledge and skills students actually acquire (e.g. OECD 2012, 2013a, 2013b, 2016). As a consequence, stakeholders in higher education do not have sufficient, reliable information about the students’ study-related preconditions (Berthold et al. 2009; KMK 2009). To allow for a valid and reliable measurement of students’ competencies and skills from the beginning to the end of their studies, theoretical–conceptual models and corresponding test instruments are necessary to describe the knowledge and skills of students with adequate precision and complexity, and to measure them reliably and validly (Mislevy 2018).

For this reason, in 2011, the German Federal Ministry of Education and Research launched a new KoKoHs research programme to develop, validate, and test newly developed models and instruments in practice. The KoKoHs projects follow an evidence-based assessment design (Riconscente et al. 2015; Mislevy 2018). In the first phase of the programme, which ran from 2011 to 2015, models to describe and operationalize these competencies were developed. The modelling included the (e.g. multidimensional, content-related, cognitive) structure, taxonomy (e.g. difficulty levels), and development of (generic or domain-specific) competencies (for examples, see Kuhn et al. 2016; Förster et al. 2015a) that

1. (1) are based on cognitive psychology, teaching–learning theory, domain-specificity and didactics, and

2. (2) serve as a basis for the development of suitable measurement instruments for valid competency assessment.

Based on these elaborated competency models, the KoKoHs projects then developed corresponding test instruments for the valid and reliable assessment of different facets and levels of students’ competencies and tested them comprehensively using suitable psychometric models (for examples, see Zlatkin-Troitschanskaia et al. 2017, 2020).

In summary, in the 85 subordinate KoKoHs projects, more than 80 theoretical competency models were developed to describe student learning preconditions and outcomes, which were then validated comprehensively in accordance with the standards for psychological and educational testing (AERA, APA and NCME 2014). All of these different models follow the assumption that students’ competencies are complex, multifaceted constructs (for a definition, see Weinert 2001). Depending on which competence facets and levels the individual KoKoHs projects focused on, for example, domain-specific knowledge or generic skills of beginning or advanced students, different appropriate measurement approaches were developed and tested in practice.

Following the model developed by Blömeke et al. (2015), students’ competencies can be described across the continuum: dispositions – skills – (observed) performance. Cognitive or affective-motivational dispositions (e.g. previous domain-specific knowledge or academic self-efficiency), which are considered the latent traits of students, are often measured using multiple-choice tasks and items or self-report scales and are modelled in accordance with Item Response Theory. Situation-specific skills (e.g. teachers’ instructional skills) are typically measured using a situative approach (social-contextual roots) and, for instance, problem-based scenarios (e.g. assessments with video vignettes; see, for example, Kuhn et al. 2018). Finally, performance is usually assessed using concrete, realistic, hands-on tasks, sampled from criterion-related situations and analysed using holistic approaches such as generalizability theory models (e.g. Shavelson et al. 2019).

In general, all KoKoHs projects follow the Evidence-Centred Model for Instantiating Assessment Arguments by Riconscente and colleagues (2015). According to this model, a valid assessment allows for a psychometric connection between the construct that is to be measured (e.g. previous knowledge) and the developed tasks (e.g. subject-specific multiple-choice items). When assessing constructs such as students’ competencies, the focus lies on the question of which capabilities the KoKoHs projects want to assess. Developing an appropriate task follows the question: in what kinds of situations a person would demonstrate these capabilities. Each KoKoHs project focuses on a different construct, i.e. a different facet of student competencies and learning outcomes (e.g. domain-specific problem-solving, self-regulation, etc.) and develops tasks that are suitable for the valid and reliable assessment of this facet (for an overview, see Zlatkin-Troitschanskaia et al. 2017, 2020). The instruments developed in KoKoHs are supposed to assess various competence facets at different levels, including domain-specific and generic knowledge and skills as well as student performance, for example, during teaching internships. In total, more than 200 test instruments of different assessment formats (e.g. paper–pencil, computer-based) were developed, and a few already available international tests were adapted for use in Germany.

Additionally, some of the test instruments have already been translated and adapted for use in other countries (more than 20 countries across four continents) and have already been used in many international studies (e.g. Yang et al. 2019; Brückner et al. 2015; Krüger et al. 2020).

Even though the KoKoHs research programme is a large-scale collaboration, it was not possible to develop instruments for all of the various domains and disciplines in higher education. Therefore, the KoKoHs projects focused on four popular study domains and one domain-independent cluster: B&E, medicine, sociology, teacher education, and interdisciplinary generic skills. Overall, the teams of the KoKoHs projects assessed more than 85,000 students at over 350 higher education institutions across Germany; approximately one third of the test takers were beginning students in the study domains of business, economic and social sciences (Zlatkin-Troitschanskaia et al. 2020).

Study Entry Diagnostics

An Example from the Study Domains of Economics and Social Sciences

To demonstrate the potentials of entrance testing of student learning preconditions in higher education, the RQs are examined on the basis of one of the KoKoHs projects: the WiWiKom II study (for an overview of all assessments conducted in this project, see Zlatkin-Troitschanskaia et al. 2019a). In this Germany-wide longitudinal study (after the pre-tests), a test to assess previous subject-related knowledge in economics and social sciences was used at the beginning of the winter term 2016/2017 (T1) at more than 50 universities across Germany, assessing more than 9000 students of B&E (N = 7679) and social sciences (N = 1347). The assessment took place during preparatory courses or introductory lectures in the first weeks of the term. The sample is on average 20.53 (± 2.753) years old, the gender distribution is approximately equal (52.28% male), about 30% have at least one parent who was born in a country other than Germany, and only 3.19% stated that they prefer a communication language other than German. The average school leaving grade was 2.35 (± 0.577). About one third of the sample had attended an advanced course in B&E at school. Exchange students were excluded because of deficient comparability in terms of education background.

The students were surveyed using a paper–pencil test of approximately 30 minutes, which comprised not only questions about sociodemographic and study-related variables but also a validated domain-specific knowledge test consisting of 15 items on (socio-)economic basic knowledge (adapted from the fourth version of the Test of Economic literacy, TEL IV; Walstad et al. 2013; for the adaptation process, see Förster et al. 2015b) and ten items on deeper economic content (adapted from the fourth version of the Test of Understanding in College Economics, TUCE IV; Walstad et al. 2007; for the adaptation process, see Zlatkin-Troitschanskaia et al. 2014) as well as a scale of fluid intelligence (extracted from the Berlin test of fluid and crystallized intelligence for grades 11+, BEFKI; Schipolowski et al. 2017). Missing answers in the knowledge test were interpreted as a lack of knowledge and coded as wrong answers, with more than 12 missing answers being assumed to indicate insufficient test motivation. The latter were not included in the analyses.

At the second measurement point (T2), one year later, 1867 students of B&E and 356 students of social sciences at 27 universities were assessed using the domain-specific knowledge test in courses that are scheduled in the curricula for third-semester students. Seven-hundred and fourteen of the students of business and economics and 155 of the social sciences students could be identified via a pseudonymized panel code as repeatedly assessed in both measurements (T1 and T2).

The following results are based on analyses using Stata 15 (StataCorp 2017). The metric level of measurement invariance (χ ²(1169) = 1331.98, p = 0.0006; RMSEA = 0.014, WRMR = 0.998, CFI = 0.976, TLI = 0.975) was investigated using MPlus (Muthén and Muthén 1998–2011).

Results

With regard to the level of study-related knowledge at the beginning of a bachelor’s degree course (RQ 1), only the first-semester students were examined (N = 7986, N _B&E = 6870, N _S = 1116). On average, the students solved 13.35 of 25 tasks (SD = 4.329) correctly, which showed a middle level of prior knowledge at the beginning of their bachelor’s studies. Out of the basic knowledge questions (from the TEL), an average of 9.21 items (SD = 2.992; 61.40%) were solved correctly, whereas out of the TUCE items, an average of 4.08 items (SD = 1.975; 40.8%) were solved correctly (after metric adjustments, t(7985) = 96.492, p>0.001). The results did not differ significantly between the study courses (e.g. for overall sum score: t(7927) = –1.016, p = 0.310, see Figure 1), so that a significant pre-selection effect cannot be assumed. However, this means that a significant part of the beginning students does not have sufficient previous knowledge of (socio-)economics fundamentals; 26.80% of the beginning students solved exactly or fewer than 10 tasks (40% of the test) correctly overall (for histogram, see Figure 2).

Notes. TEL part up to 15 correctly solved items; TUCE part up to 10 correctly solved items possible.

Figure 1. Average results from entry diagnostics.

Figure 2. Histogram of sum score.

Regarding RQ 2, in a first step, we analysed the students’ level of domain-specific knowledge at the end of their first year of study. Therefore, we only considered students in their third semester (N _overall = 1956, N _B&E = 1646, N _S = 310). The participants of the cross-section after one academic year solved an average of 13.81 (SD = 4.93) items correctly, i.e. 9.38 (SD = 3.175, 62.53%) items on the TEL part and 4.41 (SD = 2.221, 44.1%) items on the TUCE part. While the degree courses B&E and social sciences do not differ in the overall sum score (t(1949) = 0.031, p = 0.975) and not with regard to the basic content TEL part (t(1954) = 1.225, p = 0.221), the students of B&E in the third-semester cross-section show significantly better values with regard to the more in-depth contents in the TUCE part (t(1954) = –1.989, p = 0.05). This finding was in line with our expectations due to differences in the curricula for the first year of study in economics and social sciences studies.

To draw conclusions about the development of knowledge, we analysed the longitudinal sample. To examine how economics knowledge changes over the course of the first year of studies, we matched the two samples T1 and T2. Based on this matched subsample, a significant but very slight change after the first year of studies was identified. On closer examination, the significant change relates primarily to the TUCE part of the assessment. As expected, this development pattern is only evident in the group of B&E students, and not in the group of social sciences students (Table 1 and Figure 3).

Table 1. Changes in knowledge scores in the longitudinal sample.

Note. Difference between T1 and T2 in students of social sciences is not significant.

Figure 3. Average change in sum scores between T1 and T2.

Finally, when focusing on RQ 3, we investigate the influencing factors on the knowledge level and its development. In particular, we explore whether there are any differences in economics knowledge at the beginning and at the end of the first year of studies that may be influenced by factors related to secondary school. In this study with a focus on the transition between secondary to tertiary education, indicators of the students’ secondary education, such as their school leaving grade were included. In addition, it was taken into account whether or not the students had attended advanced B&E courses in secondary school. Regression analyses show that these two indicators have a significant influence on the level of economics knowledge that beginning students have in higher education (T1, see Table 2, multi-level modelling (MLM) due to a multi-level structure regardless of a low ICC of 0.047). This applies both to the group as a whole (as illustrated in Table 2) and to the subgroups of B&E students and social sciences students.

Table 2. MLM with knowledge test score at T2 as dependent variable (group variable: 54 universities).

Notes. N = 7,692; UEQ = university entrance qualification; in Germany, lower grades indicate better performance, * indicates significance on a 5% level.

When analysing what influences changes in the level of economics knowledge over the first study year in the longitudinal sample, the school leaving grade from secondary education as well as the attendance of an advanced course in B&E in school show a significant impact. This finding was observed in both subgroups.

We also analysed other personal and structural factors in MLM (ICC = 0.081) that might influence student learning over the course of the first year of studies, such as overload due to media consumption and level of fluid intelligence (Table 3). Although other explanatory variables are included, the average school leaving grade continues to be a significant predictor of domain-specific knowledge even after one year in tertiary education. Attending a business-related advanced course is no longer significant, which is probably due to the inclusion of the test score for T1, which can already reflect the influence of school knowledge. The degree of fluid intelligence as well as media overload also contribute significantly to variance clarification (for more information on media use, see Maurer et al. 2020; Jitomirski et al. 2020).

Table 3. MLM in the longitudinal sample with knowledge test score at T2 as the dependent variable (group variable: 27 universities).

Note. N = 851; ¹ UEQ = university entrance qualification; in Germany, lower grades indicate better performance; ² lower numbers indicate more overload; * indicates significance at a 5% level.

For students of the social sciences neither of the new explanatory variables was significant in the prediction, for students of B&E the overload has no significant part in the explanation of variance, but the level of fluid intelligence does. The test value for T1 is the most significant predictor of the test value for T2 in all samples.

In this context, our study also produced an interesting additional finding: in an online survey we asked the students about the sources of learning they use when preparing for their lectures and for exams, including traditional learning media such as textbooks and lecture notes, scientific sources as well as mass and social media. This survey, differentiating between 14 different sources, showed that a significant proportion of students use online sources, such as Wikipedia, more often than textbooks when preparing for exams. Overall, it became evident that students use a huge variety of different media and sources that may comprise very diverse or even contradictory information (for details, see Maurer et al. 2020; Jitomirski et al. 2020).

Discussion and Conclusion

As an example of the investigation of the transition from secondary to tertiary education within the framework of the KoKoHs research programme, findings from the WiWiKom II project regarding students of B&E and social sciences were presented, indicating the high heterogeneity in students’ study-related preconditions. In particular, secondary school graduates arrive at the transition to higher education with a very heterogeneous level of domain-specific knowledge, even when looking at more homogeneous subgroups (after formal selection processes when entering university) (RQ 1).

The development of domain-specific knowledge over the course of study in the domains examined here is lower than can be expected after the first year of study. Particularly in basic knowledge, no significant progress seems to have been made even though the content measured by the test items is part of the first-year curricula (RQ 2). This can partly be explained by the fact that both study domains are known for being mass study courses. Previous studies reveal only a low and often non-significant role of large-scale (introductory) lectures (Schlax et al. 2020). Alternative learning analysis approaches based on the decomposition of the overall pre-post-test score allow deeper insights into the changes in knowledge levels (Walstad and Wagner 2016; Schmidt et al. 2020; Schlax et al. 2019).

With regard to influencing factors (RQ 3), school entry preconditions are significant in predicting the level of knowledge at the beginning of higher education and in predicting the level of knowledge after one academic year. Consequently, previous secondary education is not only significantly relevant for the formal transition to higher education but also has an impact on domain-specific student learning and study progress.

In this context, there is also evidence of influences from other generic interdisciplinary skills such as critical dealing with learning information. In particular, the results on the influence of overload due to media use over the course of higher education studies stress the relevance of carefully filtering out and critically dealing with different kinds of information, which are considered crucial generic skills in the twenty-first century. As many current studies indicate, university students have great deficits when it comes to critically dealing with this multitude of media and online information (Hahnel et al. 2019a; Hahnel et al. 2019b; McGrew et al. 2018; Zlatkin-Troitschanskaia et al. 2019b). This is evidently the case for beginning students when entering higher education but also for advanced bachelor’s and even master’s students (Shavelson et al. 2019; Nagel et al. 2020).

The findings from these studies show that even though they are studying in bachelor’s or master’s degree courses, the students have great deficits when it comes to critically dealing with (online) information. The university students are much less capable of critically dealing with online information than one would expect considering their good performance in their course of studies, as evidenced by their results in the subject-specific knowledge assessment and their achieved grades. For instance, one of the studies, iPAL, focuses on measuring students’ critical handling of (online) information on a proximal basis, using a performance assessment (for details, see Zlatkin-Troitschanskaia et al. 2019b; Shavelson et al. 2019; Nagel et al. 2020).

In the other study, CORA (Critical Online Reasoning Assessment), an assessment framework developed by Wineburg was adapted and used to measure students’ online reasoning skills, i.e. how well the students are able to assess the trustworthiness of information sources on controversial societal issues by asking them to evaluate real websites in an open web search (Molerov et al. 2019). Although the ability to critically deal with online information is required both in curricula and as learning outcomes in higher education, as confirmed by a curricula study in 2018 (Zlatkin-Troitschanskaia et al. 2019b) the bachelor’s and master’s students seem to have been insufficiently or ineffectively educated in this skill so far in undergraduate studies.

Limitations

Empirical longitudinal research in higher education is characterized by very specific challenges, such as natural panel mortality, in which a significant proportion of students drop out of their studies. The current dropout rates in the assessed study domains are approximately 30% (Federal Statistical Office 2017). While the sample drawn randomly in T1 is a representative sample, the sample in T2 as well as the matched sample do not rule out different selection effects and bias. This must be critically considered especially in the results for RQ 2.

While the domain-specific knowledge in T1 and T2 was assessed using the valid test, all other variables, such as school grades, are based on the self-reported information of test takers and are therefore not objective data. Due to data protection regulations, it was not possible to include school grades that had actually been achieved. Also, in the domain-specific test, bias – for instance, due to test motivation – cannot be ruled out since this was a low-stake test (e.g. Biasi et al. 2018).

Moreover, only a selection of indicators on student learning outcomes and their predicting variables were considered in the study due to the limited testing time. Even if all the variables examined here are included in the modelling, only about a quarter of the variance of domain-specific knowledge in T1 and T2 is explained. Further influencing variables and indicators of learning in higher education must therefore be sought out (e.g. Shavelson et al. 2018).

Implications

The findings we have presented in this paper allow for the following theses as input for further research and for considerations in practice.

1. (1) When transitioning from secondary education to higher education, students’ learning preconditions significantly influence their acquisition of knowledge and skills in higher education.

2. (2) Students’ unfavourable preconditions are often not offset in higher education and may, for instance, lead to low knowledge growth.

3. (3) Teaching–learning settings in higher education are significantly less formalized than in secondary education and seem to have only little influence on students’ learning.

4. (4) Many students have severe deficits in terms of both domain-related basic knowledge and generic skills upon entering higher education; this is particularly evident with regard to critically evaluating information and online reasoning, as further studies indicate. These deficits are also rarely offset over the course of studies in higher education.

This poses great challenges for universities if they want to maximize the achievement of knowledge acquisition and professional preparation to remain competitive in the EU context. Especially in mass study subjects such as B&E, individual support in particular is extremely challenging. In this context, initial assessments are required to objectively and reliably identify students’ learning potentials and deficits, and, based on these results, to optimize the allocation to needs-based preparatory courses and tutorials.

Considering the domain-specific and generic student learning outcomes that are expected in higher education, for instance in economics, only a few of these facets seem to be explicitly fostered in higher education. As our findings indicate, we cannot expect that deficits at the end of upper secondary school are offset over the course of higher education studies. The large-scale studies indicate a great variance in the skill and knowledge levels of students entering higher education. The results stress the importance of valid entry diagnostics at the stage of transition between secondary and higher education.