Hostname: page-component-cd9895bd7-gvvz8 Total loading time: 0 Render date: 2024-12-29T08:27:48.209Z Has data issue: false hasContentIssue false
  • English
  • Français

Measurement validation: lessons from the use and misuse of UN General Assembly roll-call votes

Published online by Cambridge University Press:  22 May 2009

Brian W. Tomlin
Get access

Extract

An oft-claimed advantage of scientific studies of international politics is the intersubjectivity of such inquiries. Although the ultimate promise of scientific knowledge is the understanding it imparts to patterns of association among classes of events, much contemporary research falls considerably short of this goal. As a result, the principal, immediate value of such research lies in its adherence to scientific practices that ensure that findings are not wholly dependent on the methods of measurement and analysis adopted by a particular researcher. More than a decade ago James Caporaso demonstrated the utility of measurement validation as a specific application of the general scientific mandate that inquiry be intersubjective. 1 Yet despite widespread positive reaction to Caporaso's examination of alternative measures of the concept “integration,” his study stands out as a rare example of the type of systematic investigation of measurement validity that must be undertaken in order to fulfill this scientific mandate. 2

Type
Research Note
Information
International Organization , Volume 39 , Issue 1 , Winter 1985 , pp. 189 - 206
Copyright
Copyright © The IO Foundation 1985

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1. Caporaso, J., “Theory and Method in the Study of International Integration,” International Organization 25 (1971), pp. 228–53.CrossRefGoogle Scholar

2. Caporaso's study does not stand alone, however. In particular, users of international event data displayed a welcome sensitivity to potential validity problems. See Hopmann, P. and Hughes, B., “The Use of Events Data for the Measurement of Cohesion in International Political Coalitions,” in Azar, E. and Ben-Dak, J., eds., Theory and Practice of Events Research (New York: Gordon & Breach, 1975)Google Scholar; Hughes, and Schwarz, J., “Dimensions of Political Integration and the Experience of the European Community,” International Studies Quarterly 16 (1972), pp. 263–94CrossRefGoogle Scholar; and Munton, D., Measuring International Behaviour (Halifax, N.S.: Centre for Foreign Policy Studies, Dalhousie University, 1978).Google Scholar

3. SSIP is the acronym employed Zinnes, D., Contemporary Research in International Politics (New York: Free Press, 1976)Google Scholar, for scientific studies of international politics.

4. They continue to be used. Since 1975 no fewer than 20 articles using roll-call-based indicators have appeared in the following leading journals: American Journal of Political Science, American Political Science Review, International Interactions, International Organization, International Studies Quarterly, Journal of Conflict Resolution, Journal of Peace Research, and World Politics.

5. MacRae, D., Issues and Parties in Legislative Voting: Methods of Statistical Analysis (New York: Harper & Row, 1970).Google Scholar

6. According to Ibid., p. 175, a faction is any group of interacting members of a legislative body who may be expected to vote alike on the basis of their interaction. Thus the term may refer to formally organized groups, such as legislative parties, as well as to the more informal blocs typically addressed in studies of the General Assembly.

7. See Ibid., pp. 41, 74, 175, 209; also Anderson, L., Watts, M., and Wilcox, A., Legislative Roll-Call Analysis (Evanston: Northwestern University Press, 1966), p. 74.Google Scholar

8. Gurr, T., Politimetrics: An Introduction to Quantitative Macropolitics (Englewood Cliffs, N.J.: Prentice-Hall, 1972).Google Scholar

9. Phillips, B., Social Research: Strategy and Tactics (New York: Macmillan, 1966).Google Scholar

10. Inasmuch as measures are selected because they are expected to reflect the same concept, a substantial degree of convergence should be established as a threshold for validation. Translating this into a (necessarily arbitrary) decision rule, association in the 0.7 range provides a criterion for assessment in the tests that follow.

11. Russett, B., International Regions and the International System (Chicago: Rand McNally, 1967), p. 60.Google Scholar

12. On the importance attached to the United Nations by African states see Hovet, T., Africa in the United Nations (Evanston: Northwestern University Press, 1963)Google Scholar; Kay, D., The New Nations in the United Nations (New York: Columbia University Press, 1970Google Scholar; and Mazrui, A., “The United Nations and Some African Political Attitudes,” International Organization 18 (1964), pp. 499–520.CrossRefGoogle Scholar

13. The African Foreign Relations and International Conflict Analysis (AFRICA) project draws on African sources to describe the external behavior of 32 Black African states in the period 1964–66. See McGowan, P., “A Manual and Codebook for the Identification, Abstraction, and Coding of Foreign Policy Acts,” AFRICA Project Report 1 (Syracuse, N.Y.: Syracuse University, 1972).Google Scholar

14. The use of this single year as a basis for comparison will not seriously affect the results in light of the stability of voting patterns established by previous analyses. See Alker, H. and Russett, B., World Politics in the General Assembly (New Haven: Yale University Press, 1965)Google Scholar; Russett, , International RegionsGoogle Scholar. In the selection and coding of votes the procedures employed by Russett, , International Regions, p. 62Google Scholar, were adopted, resulting in the elimination of 4 African states (Botswana, Congo, B., Gambia, and Lesotho) that were absent for more than 40% of the roll calls.

15. Alker, and Russett, , World PoliticsGoogle Scholar; MacRae, , Issues and Parties.Google Scholar

16. Product-moment coefficients were used. Factor analysis used the principal component method and orthogonal rotation according to the varimax criterion, with unities inserted in the principal diagonal of the matrix.

17. Each factor accounted for 24.7% of the total variance.

18. On grouping analysis see Veldman, D., Fortran Programming for the Behavioral Sciences (New York: Holt, Rinehart & Winston, 1967)Google Scholar. The technique was employed to ensure that the groups are as inclusive as possible, while minimizing the increase in error resulting from successive combinations of groups.

19. The groups conform to those presented in previous analyses to the extent that some of the former members of the Brazzaville caucusing group are found clustered in the lower left quadrant and some of those formerly of the Casablanca group are in the lower right quadrant. Malawi and Gabon are not part of any cluster.

20. See McGowan, , “A Manual and Codebook.”Google Scholar The project provides information on 14,669 foreign-policy acts involving 32 sub-Saharan African states as actors, 28 of which are included in the present analysis. The data were made available by the Inter-University Consortium for Political Research. Neither the original collector of the data nor the consortium bears any responsibility for the analyses or interpretations presented here. In his analysis of these data McGowan, P., “Dimensions of African Foreign Policy Behaviour: In Search of Dependence” (Paper presented at the Annual Meeting of the Canadian Association of African Studies,Ottawa,February 1973), pp. 32–33Google Scholar, found that behavioral patterns were stable irrespective of whether events were aggregated on a one- or three-year basis. In light of this finding the three-year period was adopted in order to maximize the number of behavioral exchanges upon which to base analysis.

21. Lijphart, A., “The Analysis of Bloc Voting in the General Assembly,” American Political Science Review 57 (1963), pp. 902–17CrossRefGoogle Scholar. The degree of agreement between pairs is computed as follows:

where f is the number of votes on which two states agree completely; g is the number of votes on which two states partially agree (one in favor or against, the other abstaining); and t is the total number of votes in which both states participate.

22. Events in the AFRICA project are classified according to a revised version of the action typology employed in the CREON project, in which actions are first identified as cooperative, conflictful, or neutral. See Hermann, C. et al. , CREON: A Foreign Events Data Set (Beverly Hills: Sage, 1973)Google Scholar. The coefficient of imbalance was employed to accommodate the inclusion of the neutral category. The coefficient is computed as follows:

where X is the number of cooperative actions, Y the number of conflictful actions, and Z the number of neutral actions exchanged between two states. See Janis, I. and Fadner, R., “The Coefficient of Imbalance,” in Lasswell, H. and Leites, N., eds., Language of Politics (Cambridge: MIT Press, 1965).Google Scholar

23. Only those dyads engaging in some exchange of behavior in the subsystem were included in the analysis.

24. A similar lack of correspondence emerged in additional analyses of these same data in which states were grouped on the basis of their event-interaction behavior. The more direct test of the correspondence between dyadic interaction and voting was selected for inclusion here. The relationship between Assembly voting and the behavior of Black African states toward a shared object is explored below in the analysis of issues.

25. Portions of the analysis that follows are drawn from Tomlin, B., “Events vs. Votes as Indicators of Nation Behaviour in the United Nations,” in Munton, D., ed., Measuring International Behaviour: Public Sources, Events and Validity (Halifax, N.S.: Centre for Foreign Policy Studies, Dalhousie University, 1978)Google Scholar, which addresses the validity of event data.

26. These crises were selected for analyses since both voting and interaction were focused on a single, overarching issue.

27. McGowan, , “A Manual and Codebook.”Google Scholar

28. France served only infrequently as a target of behavior in 1956.

29. Twenty-eight states in 1956, and 64 in 1967.

30. Using the principal component method and orthogonal rotation according to the varimax criterion, with unities inserted in the principal diagonal of the matrix. This method was also used for factor analyses reported below.

31. The latter two because of the central roles they have played in the region and on the issue.

32. See note 22.

33. Signs of the coefficients have been adjusted, where necessary, to reflect the direction of voting on the issue.

34. Russett, , International Regions, p. 66.Google Scholar

35. Clark, J., O'Leary, M., and Wittkopf, E., “National Attributes Associated with Dimensions of Support for the UN,” International Organization 25 (Winter 1971), pp. 1–25.CrossRefGoogle Scholar

36. Alker, H., “Supranationalism in the United Nations,” in Rosenau, J., ed., International Politics and Foreign Policy (New York: Free Press, 1969).Google Scholar

37. A proper test of convergent validation would employ procedures independent of voting to identify supranational roll calls. The same effect is achieved in the comparative analysis of the supranationalist content of issues, decribed below.

38. This same factor was also used by Moore, D., “Governmental and Societal Influences on Foreign Policy in Open and Closed Nations,” in Rosenau, J., ed., Comparing Foreign Policies: Theories, Findings, and Methods (Beverly Hills: Sage, 1974)Google Scholar, and Rosen, D., “Leadership Change and Foreign Policy” (Paper presented at the Annual Meeting of the American Political Science Association,Chicago,1974)Google Scholar, to construct an indicator of foreign-policy positions on the issue of supranationalism.

39. The third (Tunisia), fourth (financial crisis), and fifth (South West Africa) special sessions, and the first (Suez), second (Hungary), fourth (Congo), and fifth (Six-Day War) emergency special sessions were included.

40. Two votes on Jerusalem in the fifth emergency special session loaded on a fifth factor, which is excluded from the present analysis.

41. The effect of this procedure is to apportion the supranationalist content of each resolution among the four issues according to the interpretable variance in the roll call that is accounted for by each factor.

42. Alker, , “Supranationalism,” p. 709.Google Scholar

43. The data were made available by the Inter-University Consortium for Political Research. Neither the original collector of the data nor the consortium bears any responsibility for the analyses or interpretations presented here.

44. The high supranationalist content of the southern Africa issue is only tangentially significant in the context of the present analysis. This finding, however, suggests that resolution content may, in part, reflect efforts on the part of sponsors to legitimize resolutions through appeals to symbols of the organization.

45. The results of this analysis also suggest that the cold war dimension of Assembly voting be isolated for this purpose. Studies employing roll calls on which the United States and USSR adopt opposing positions to develop measures of alignment or compliance combine the cold war and southern Africa issues, and only the former was found to be related to system interaction. See Richardson, N. and Kegley, C., “Trade Dependence and Foreign Policy Compliance: A Longitudinal Analysis,” International Studies Quarterly 24 (1980), pp. 191–222.CrossRefGoogle Scholar

46. See Ibid., p. 201.