No CrossRef data available.
Article contents
How Did English Schools and Pupils Really Perform in the 1995 International Comparisons in Mathematics ?
Published online by Cambridge University Press: 26 March 2020
Abstract
The recently published results of mathematics tests set to representative samples of pupils in over forty countries provide an important opportunity to re-assess priorities for reforms in English schooling. Five Western European countries—Austria, Belgium, France, Netherlands, Switzerland—are suggested in this critical study as providing appropriate standards for England's immediate aims. Attainments there are shown to be about a year ahead of England for average pupils at age 14; the gap is larger for those of below-average attainment, suggesting some structural bias in English schooling. The gap is particularly evident in those arithmetical fundamentals which need to be mastered by all school-leavers (rather than in advanced aspects suitable only for mathematical specialists); and that gap has its clear origins at the primary level of schooling.
- Type
- Articles
- Information
- Copyright
- Copyright © 1997 National Institute of Economic and Social Research
Footnotes
∗
This Note has been prepared as part of the National Institute's research programme into international comparisons of educational attainment, carried out in association with inspectors and teachers in the London Borough of Barking and Dagenham. I am grateful to Roger Luxton and Graham Last (Principal and Senior Inspectors in that Borough), Professor Geoffrey Howson (University of Southampton), Dr John Marks (School Curriculum and Assessment Authority), Jason Tarsh (Department for Education and Employment) and to my colleagues at the National Institute for helpful discussion and comments on an earlier draft. For help in interpreting the results for Germany, I am particularly indebted to Professors Aurin (Freiburg) and W. Blum (Kassel) and to Helvia Bierhoff (previously researcher at the National Institute, now in Germany). The work was carried out with the financial support of the Gatsby Charitable Foundation. Responsibility for errors of fact and judgement remains with the author.
References
Notes
(1) IEA = International Association for the Evaluation of Educational Achievement (established in 1959 as an international co-ordinating research centre; funded for the 1995 TIMSS centre by the US, with each participating country funding its own national study). TIMSS = Third International Mathematics and Science Study (previous studies in mathematics were carried out in 1964 and 1981; and in science in 1971 and 1984).
(2) W. Keys, S. Harris, C. Fernandes, Third International Mathematics and Science Study: First National Report Part 1 and Appendices (NFER, Slough, November 1996). A.E. Beaton et al, Mathematics Achievement in the Middle School Years: IEA's Third International Mathematics and Science Study (TIMSS International Study Center, Boston College, Chestnut Hill, MA, USA (November 1996); do for Science. Supplementary lists of questions in mathematics and science were released by the international centre in March 1997, but without the percentage of pupils giving correct answers in each country (requests for percentages correct, even for only selected questions, have gone unanswered). A parallel international enquiry into the attainments of 9 year-olds was undertaken by about half the countries involved in the 13-14 year- olds enquiry, including England but few European countries; certain of those findings are relevant in tracing the age at which England's deficiencies become evident (I.VS. Mullis et al, Mathematics Achievement-the Primary School Years, TIMSS Center, June 1997).
(3) For convenience of the English reader, the English class-nomenclature is used throughout this Note rather than the ‘inter national nomenclature’ used in the IEA report (which is one year younger; eg pupils are aged 14 in the middle of English Year 9, but are in ‘international class 8’ in other countries where formal schooling begins one year later).
(4) TIMSS, Mathematics, p. 28, n. 8. Unfortunately the TIMSS report did not attach this caveat to the relevant tables, nor did it explicitly say in the text (aside from that text-footnote) that the very low calculated rise in average scores for Belgium between the two successive years is fundamentally misleading. Separate returns were published by TIMSS for the Flemish and French speaking parts of Belgium; for our purposes a simple average of the two parts is adequate, and has been quoted here throughout for that country.
(5) The contrast between the ‘modern mathematics' type of curriculum introduced in Nordrhein-Westfalen, in which 'set language and arrows abound’, and the more traditional emphasis of Baden-Württemberg on ‘the acquisition of compe tence in more limited areas’, is outlined in G. Howson's National Curricula in Mathematics (Mathematical Association, Leicester, 1991), esp. pp. 94, 97.
(6) J. Baumert, R. Lehmann, et al, TIMSS: Mathematik-Naturwissenschaftlicher Unterricht im internationalen Vergleich, Max-Planck-Institut, Berlin, 1997; esp. pp. 118 and 125, table D2 and figure D3. This gives scores for three groups of Länder according to their proportions of comprehensive schools; average scores for two particular Länder were subse quently reported in the general press (on the measures reported in table 1 below: Bavaria 534, Nordrhein-Westfalen 493). Limitations of space prevent further discussion here; a fuller (duplicated) comment by the present writer is available from the National Institute's Publications Department.
(7) Unfortunately for the reader, the Land Baden-Württemburg did not participate in TIMSS; there were also serious prob lems with the age-ranges of the classes covered (pupils were eight months older on average in Germany than in England).
(8) The sampling error of the rise in attainments between one year and the next can be expected to be quite small, since the basic approach of TIMSS was that the same schools provided the corresponding upper and lower grades in each country (TIMSS Mathematics, Appendix A, p. A-11); in other words, the sampling error of the average rise can be calculated on the well-known basis of ‘paired comparisons’. Unfortunately this was not noticed by the TIMSS statisticians, and they quoted (in their table 1.3) very high sampling errors, calculated as if the successive classes in each country were independ ent samples.
(9) Ibid, Appendix table D3. More strictly: this additional test was carried out only in the German-speaking parts of Switzer land; but since that part accounts for about two-thirds of the whole country, the comparison cannot be far out. The choice of Switzerland as an exemplar for reforms in mathematical education in the London Borough of Barking and Dagenham, carried out jointly with National Institute researchers, thus seems to be supported by these new results from TIMSS. (Incidentally, the description in TIMSS Appendix table D3 of Swiss classes aged 15.1 as being in the ‘eighth grade’ must be regarded as an error for the ‘ninth grade’: otherwise serious inconsistencies would occur with the main maths results reported for the seventh and eight grades.) The Flemish-speaking part of Belgium showed significantly higher scores than the French part (average scores for Year 9 of 565 and 525); it would be worth looking more closely at these results once the full national reports for that country become available (till then, given the sampling problems that arose in that country in respect of Year 8, as mentioned in n. 4 above, some caution is in order).
(10) I am indebted to Dr John Marks for these calculations based on the SATs results; see also his studies for the Social Market Foundation: Standards of English and Maths in Primary Schools for 1995; Social Market Foundation memorandum no. 24, 1996; Standards of Reading, Spelling and Maths for 7-year Olds in Primary Schools for 1995, Social Market Foundation memorandum no. 25, 1997.
(11) This was an ‘open’ question (ie not multiple choice). For further comment see my Productivity, Education and Training (CUP, 1995), pp. 85 and 127, n. 22.
(12) See Helvia Bierhoff, Laying the foundations of numeracy: a comparison of primary school textbooks in Britain, Germany and Switzerland, Teaching Mathematics and its Applications, December 1996.
(13) See TIMSS, p. 62 for the subtraction question, and p. 58 for the ‘percent correct’. Scotland's total scores in mathematics were 463 and 498, at ages 13 and 14 respectively, compared with England's 476 and 506 (ibid, pp. 22, 26); Scotland was thus behind England by about a term's work. Scotland's current concern with its low attainments in basic arithmetic thus seem well borne out by the TIMSS results (Fourth Survey of Mathematics 1994, Education and Industry Department, Scottish Office, 1996), p. 7.
(14) TIMSS, pp. 94 and 97.
(15) TIMSS report on primary mathematics, op cit p. 68, table 3.3. Response rates to the primary survey were less satisfactory than for the secondary survey; where replacement schools were introduced to compensate for non-participation (28 per cent of the final sample in England, 52 per cent in the Netherlands, 23 per cent in Austria), it is not clear that any stratification by attainment-range of schools was attempted (eg in England using SATs results at Key Stages 1 and 2).
(16) Ibid, pp. 58, 62. We would need to examine closely the translated wording of that question (in all three Swiss languages) to understand why no more than 83 per cent of Swiss pupils answered it correctly.
(17) Whether English pupils' mathematical attainments have fallen (or perhaps risen?) since the previous IEA survey of 1981 is a matter of obvious interest and importance; a certain number of questions from that previous survey were therefore included in TIMSS (‘the anchor items’). But so far there has been no comparative analysis of the two surveys, neither at the international level (by IEA) nor for England by (NFER), though this was done on the previous occasion (see, for example, the NFER study The Second International Mathematics Study in England and Wales, by M. Cresswell and J. Gubb, NFER-Nelson, 1987, pp. 56-66) Could a lack of funding for this important aspect reflect official worries as to what the results may show?
(18) One of the questions—on which English pupils did well—in the content-area of ‘data representation, etc.’ required the representation in a pictograph of 55 students, in which one symbol (a head, shown as a circle in which there are two dots for eyes, and a curved line for a mouth) ‘represents ten students’. Should the respondent ‘round down’ and give the answer as five symbols; or ‘round up’ and provide six symbols? Or should he provide 5½ symbols? According to the report (example 19, p. 83), the last is the only correct answer; this may well be in accord with conventions as taught in some countries, but it is not obvious to the present writer that other conventions are not equally acceptable as correct (it seems more of an exercise in what may be called ‘inverse palaeography’—an inference from digital arithmetic to primitive symbolism—which is surely of doubtful value to a pupil at that stage). While this content-area is at present an explicit part of the English National Curriculum, it should be noticed that English pupils did relatively well only when compared to the average of all 41 other countries; compared with our Western European neighbours, English pupils did not do as well— though the areas of ‘data representation, etc.’ do not feature explicitly on their curricula but arise at this stage merely as common-sense applications of basic arithmetic.
(19) For an earlier comparison of mathematical attainments at ages 13-14 indicating that English Grammar School pupils were also considerably ahead of German Gymnasien pupils, see Appendix B on the 1964 IEA survey in my (1985) paper with K. Wagner (op cit), final para.
(20) A.G. Howson, chairman of a joint committee of leading mathematical institutions, Tackling the Mathematics Problem, London Mathematical Society, 1995).
(21) It would go beyond the proper ambit of this paper to do more than footnote a few factors frequently mentioned as inhibiting the progress of top pupils: reduced possibilities today for early entry of bright children to secondary school (at age 10, rather than 11 for the majority); substitution of the GCSE broader curriculum for the previous narrower but deeper O-level examination; too much time spent under National Curriculum requirements on ‘investigations’ …
(22) Scores for percentiles quoted in this section have been derived by graphical interpolation (using probability graph paper) from the tabulated 5th, 25th and 50th percentiles in tables E1 and E2 of TIMSS.
(23) For example, participating substitute schools but had GCSE results at ages 16+ in the same attainment bracket, but may have been stronger in attainments at the survey ages 13-14 than those schools who refused to participate. Published response rates by English schools' characteristics suggest that Metropolitan schools (including ‘inner city’ schools) were slightly under-represented even after ‘replacement schools’ were added (NFER, Appendix Volume p. 40). In addition, some 7 per cent of English schools were excluded because they were taking part in National Curriculum trials (ibid, p.39), and this accounts for the greater part of the special reservation attached by TIMSS to England's results (‘only England exceeded the 10% limit’ of exclusions; TIMSS Report, p. A 11); since those excluded schools were presumably chosen in a representative manner, that reservation is probably of no substantial significance (the critical reader may nevertheless wonder why they were not excluded from the sampling frame in the first place). The detailed international response rates are to be found in TIMSS, Appendix A, table A 4 for 14 year-olds (referred to in the text above); for 13 year-olds table A6 shows slightly lower response rates. In the IEA's primary school survey, 63 per cent of English schools responded on first approach, and an additional 25 per cent from a second replacement list; but, in contrast to arrangements for the secondary school survey, there was no provision for choosing replacement schools from the same attainment stratum (eg by relying on results from the SAT tests); the reliability of the primary survey is thus more questionable.
(24) TIMSS Mathematics, p. A 14. The proportions for 13 year-olds lead to much the same conclusions and, for simplicity of exposition, are not reproduced in the text above.
(25) Jahrbuch, op cit, calculated from pp. 66-82; Education Statistics for the United Kingdom, 1995 Edition (HMSO, 1996), table 12b.
(26) The problem posed by absentees in interpreting the results of the TIMSS tests was not adequately addressed in the international report. We must however note that absence rates above 10 per cent were recorded for Germany, Hungary and Scotland; 9 per cent for the United States; and even Japan recorded 5.5 per cent. It is not clear how absence rates as recorded for TIMSS tests compared with absence rates in the rest of the year. A more precise comparison of the length of the tail of under-achievement seems difficult in such circumstances (better sampling design, employing the familiar device of asking teachers to provide advance estimates of pupils' performance, would have permitted the calculation of response rates by attainment-stratum, and hence to less-biased estimates for the population).
(27) Questions were asked about the extent of pupils' television watching (TIMSS, p, 116), but are difficult to interpret. The survey shows that pupils who spend more time watching have lower scores in mathematics; but causation could be in the other direction—pupils who are badly taught, tend to become bored with the subject, and with homework on it, and fill in their evenings with more television watching. Many questions were asked—with equally inconclusive results—on pupils' attitudes to, and teachers' perceptions about, mathematics: 93 per cent of English pupils—with their nationally low attainments—agreed that they ‘usually do well in mathematics’, while only 44 per cent of Japanese pupils—with their nationally high attainments—agreed that they ‘usually do well’ (TIMSS p. 118). Does it follow that English teachers are better at spreading complacency amongst pupils? And does it follow that they are pedagogically counter-productive in doing so?
(28) A.E. Lapointe et al, Learning Mathematics, International Assessment of Educational Progress (IAEP), Princeton, NJ, 1992, p. 49.
(29) TIMSS, p. 154.
(30) TIMSS, p. 146; IAEP, p. 59. On the basis of direct observations in German-speaking Switzerland I suspect that the higher IAEP figure is correct. The employment of generalist teachers for middle and lower streams is normal in Switzerland (as the TIMSS report correctly noted); it would have been of some interest to be more precise, and to tabulate the contrasting use of generalist/specialist teachers according to the level of the mathematics attainment-group-but this was not done.
(31) IMSS, p. 157.
(32) Ibid, pp. 163-4, 166.
(33) TIMSS, Primary mathematics, op cit, table 5.14, p. 176 (based on teachers' reports; the associated table 5.16, based on pupils' reports, gives a broader and less helpful classification). The issue has attained further policy prominence (at the time of writing) with the commissioning by the new Labour Government of a fresh enquiry into the use of calculators at primary schools. A previous National Institute study (Bierhoff, op cit, pp. 152-3) emphasized the importance of training in mental agility in primary mathematics, and the negative role of calculators in that respect. A subsequent official (and anonymous!) Discussion Paper from SCAA defended the use of calculators; while discussing some of the considerations raised in the National Institute's study, the issue of mental agility was not given adequate attention (The Use of Calculators at Key Stages 1-3, Discussion Paper no. 9, SCAA, March 1997).
(34) Ibid, pp. 163, 167-8. There seems to have been some ambiguity in the term ‘computers' in that survey, since an astonishing 89 per cent of English sample reported having ‘a computer in the home’ (p. 163; exceeded only by Scotland's 90 per cent— in contrast to poor Switzerland's 66 per cent). But Britain's General Household Survey reported only 48 per cent of households with dependent children as owning a computer (CSO, Social Trends, HMSO, 1996, p. 119). It seems more than possible that computerised games—many of which, though intellectually challenging, have little to do with math ematical attainments as understood for the present purposes—were included by respondents to TIMSS on questions relating to the ownership of computers.
(35) The associated issues are clearly relevant to the proposed revision of the National Curriculum. Limitations of space prevent a fuller discussion here; a supplementary note is available in duplicated form from the National Institute's Publi cations Department.
(36) See, for example, R.G. Luxton and G. Last, ‘Under-achievement and Pedagogy: Experimental Reforms in the Teaching of Mathematics Using Continental Approaches in Schools in the London Borough of Barking and Dagenham’, National Institute Discussion Paper no. 112, February 1997.