Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-fscjk Total loading time: 0 Render date: 2024-12-28T15:23:15.320Z Has data issue: false hasContentIssue false

4 - Domain Definition Inference for a Virtual Interactive Aviation English Test (VIAET) for Military Air Traffic Controllers

from Part II - Investigating Score Interpretations

Published online by Cambridge University Press:  14 January 2021

By
Moonyoung Park
Edited by
Carol A. Chapelle  and
Erik Voss
Carol A. Chapelle
Affiliation:
Iowa State University
Erik Voss
Affiliation:
Teachers College, Columbia University
Get access

Summary

This chapter reports creation of a prototype assessment using virtual simulation for a test intended to assist in decision-making for non-native English-speaking military air traffic controllers’ placement into appropriate work assignments and to provide them with diagnostic feedback. The research and development focused on the domain definition inference, whose warrant was that observations of performance on the virtual interactive tasks for aviation English assessment are representative of the knowledge, skills and abilities required of air traffic controllers in an Asian country. Evidence Centered Design (ECD) guided the domain analysis and methods from Task-Based Language Assessment (TBLA) were also used, including eliciting opinions of two domain experts, document analysis, and a survey of 71 participants in the domain. Results indicated good backing for the first two assumptions: (1) critical aviation English skills, knowledge, and processes needed for aviation English communication in the military ATC context can be identified and (2) assessment tasks that are representative of the English for Specific Purposes (ESP) domain of ATC can be modeled as assessment tasks. The third assumption, (3) assessment tasks that require important knowledge and skills for communication in aviation English can be simulated in Second Life, was only partially supported.

Keywords

aviation English communicationmilitary air traffic controllersdomain analysisassessment tasksvirtual environment
Type
Chapter
Information
Validity Argument in Language Testing
Case Studies of Validation Research
 , pp. 73 - 95
Publisher: Cambridge University Press
Print publication year: 2021

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

AAS. (2012). Air traffic control. Army Aviation School.Google Scholar
Alderson, J. C. (2006). Final report on a survey of aviation English tests: Lancaster University and the European Organisation for the Safety of Air Navigation (Eurocontrol).Google Scholar
Alderson, J. C. (2009). Air safety, language assessment policy, and policy implementation: The case of Aviation English. Annual Review of Applied Linguistics, 29, 168–187. https://doi.org/10.1017/s0267190509090138CrossRefGoogle Scholar
Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.Google Scholar
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests (Vol. 1). Oxford: Oxford University Press.Google Scholar
Bachman, L. F., & Palmer, A. (2010). Language assessment in practice: Developing language assessments and justifying their use in the real world. Oxford: Oxford University Press.Google Scholar
Bowen, G. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9, 27–40.Google Scholar
Brown, J. D. (2004). Performance assessment: Existing literature and directions for research. Second Language Studies, 22(2), 91–139.Google Scholar
CAA. (2014). Manual on the English Language Proficiency Assessment (ICAO language proficiency requirements). Republic of Moldova: Civil Aviation Authority.Google Scholar
Chapelle, C. A. (1998). Construct definition and validity inquiry in SLA research. In Bachman, L. F. & Cohen, A. D. (Eds.), Interfaces between second language acquisition and language resting research (pp. 32–70). New York: Cambridge University Press.Google Scholar
Chapelle, C. A. (2008). The TOEFL validity argument. In Chapelle, C. A., Enright, M. E., & Jamieson, J. (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 319–350). New York: Routledge.Google Scholar
Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple. Language Testing, 29(1), 19–27.Google Scholar
Chapelle, C. A., Enright, M. K., & Jamieson, J. (Eds.). (2008). Building a validity argument for the test of English as a foreign language. New York: Routledge.Google Scholar
Chapelle, C. A., Schmidgall, J., Lopez, A., Blood, I., Wain, J., Cho, Y., Hutchison, A., Lee, H.-W., & Dursun, A. (2018). Designing a prototype tablet-based learning-oriented assessment for middle school English learners: An evidence-centered design approach. ETS Research Report Series, 2018(1), 1–55.CrossRefGoogle Scholar
Corbin, J., & Strauss, A. (2008). Basics of qualitative research (3rd ed.). Thousand Oaks, CA: Sage Publications.Google Scholar
Cushing, S. (1994). Fatal words: Communication clashes and aircraft crashes. Chicago: University of Chicago Press.Google Scholar
Douglas, D. (2000). Assessing languages for specific purposes. Cambridge: Cambridge University Press.Google Scholar
Emery, H. J. (2014). Developments in LSP testing 30 years on? The case of aviation English. Language Assessment Quarterly, 11(2), 198–215.Google Scholar
Hamp-Lyons, L., & Lumley, T. (2001). Assessing language for specific purposes. Thousand Oaks, CA: Sage Publications.CrossRefGoogle Scholar
Hines, S. (2010). Evidence-centered design: The TOEIC speaking and writing tests. The Research Foundation for TOEIC: A Compendium of Studies, 7.1.Google Scholar
Kane, M. (2013). Articulating a validity argument. In The Routledge handbook of language testing (pp. 48–61). London: Routledge.Google Scholar
Krosnick, J. A., Narayan, S. S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, 29–44.CrossRefGoogle Scholar
Long, M. H., & Norris, J. M. (2000). Task-based teaching and assessment. Encyclopedia of Language Teaching, 597–603.Google Scholar
McNamara, T. F. (1996). Measuring second language performance. Harlow: Longman.Google Scholar
Messick, S. (1989). Validity. In Linn, R. L. (Ed.), Educational measurement (3rd ed., pp. 60–68). New York: Macmillan.Google Scholar
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.CrossRefGoogle Scholar
Mislevy, R. J. (2006). Cognitive psychology and educational assessment. In Brennan, R. L. (Ed.), Educational measurement (4th ed., pp. 257–305). Westport, CT: American Council on Education/Praeger Publishers.Google Scholar
Mislevy, R. J. (2013). Evidence-centered design for simulation-based assessment. Military Medicine, 178(Suppl_10), 107–114.CrossRefGoogle ScholarPubMed
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29.Google Scholar
Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence‐centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6–20.CrossRefGoogle Scholar
Mislevy, R. J., & Steinberg, L. S. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3–62.Google Scholar
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). Design and analysis in task-based language assessment. Language Testing, 19(4), 477–496.CrossRefGoogle Scholar
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67.Google Scholar
Mislevy, R. J., Steinberg, L., Almond, R. G., & Lucas, J. F. (2006). Concepts, terminology, and basic models of evidence-centered design. In Williamson, D. M., Mislevy, R. J., & Bejar, I. I. (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 49–82). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Norris, J. M. (2001). Identifying rating criteria for task-based EAP assessment. In Hudson, T. D. & Brown, J. D. (Eds.), A focus on language test development: Expanding the language proficiency construct across a variety of tests (pp. 163–204). Honolulu: University of Hawai’i Press.Google Scholar
Norris, J. M. (2014). How do we assess task-based performance? Invited LARC/CALPER testing and assessment webinar.Google Scholar
Norris, J., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Technical report 18, University of Hawaii, Honolulu.Google Scholar
Park, M. (2015). Development and validation of virtual interactive tasks for an aviation English assessment. Doctoral dissertation, Iowa State University, Ames, IA.Google Scholar
Pearlman, M. (2008). Finalizing the test blueprint. In Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (Eds.), Building a validity argument for the Test of English as a Foreign Language™ (pp. 227–258). Philadelphia, PA: Routledge.Google Scholar
Zokić, M., Boras, D., & Lazić, N. (2012). Computer-aided Aviation English testing on example of RELTA test. Paper presented at the 2012 Proceedings of the 35th International Convention MIPRO.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×