Hostname: page-component-cd9895bd7-gvvz8 Total loading time: 0 Render date: 2024-12-26T09:10:52.710Z Has data issue: false hasContentIssue false
  • English
  • Français

Interlingual annotation of parallel text corpora: a new framework for annotation and evaluation

Published online by Cambridge University Press:  15 June 2010

BONNIE J. DORR ,
REBECCA J. PASSONNEAU ,
DAVID FARWELL ,
REBECCA GREEN ,
NIZAR HABASH ,
STEPHEN HELMREICH ,
EDUARD HOVY ,
LORI LEVIN ,
KEITH J. MILLER  and
TERUKO MITAMURA
...Show all authors
BONNIE J. DORR
Affiliation:
Institute for Advanced Computer Studies, University of Maryland, AVW Williams Building 3153, College Park, MD 20742, USA e-mail: bonnie@umiacs.umd.edu
REBECCA J. PASSONNEAU
Affiliation:
Center for Computational Learning Systems, Columbia University, 475 Riverside Drive MC 7717, New York, NY 10115, USA e-mails: becky@cs.columbia.edu, habash@cs.columbia.edu, rambow@cs.columbia.edu
DAVID FARWELL
Affiliation:
Computing Research Laboratory, New Mexico State University, Las Cruces, NM 88001, USA e-mails: david@crl.nmsu.edu, shelmrei@crl.nmsu.edu
REBECCA GREEN
Affiliation:
OCLC Online Computer Library Center, Inc., 6565 Kilgour Place, Dublin, OH 43017-3395, USA e-mail: greenre@oclc.org
NIZAR HABASH
Affiliation:
Center for Computational Learning Systems, Columbia University, 475 Riverside Drive MC 7717, New York, NY 10115, USA e-mails: becky@cs.columbia.edu, habash@cs.columbia.edu, rambow@cs.columbia.edu
STEPHEN HELMREICH
Affiliation:
Computing Research Laboratory, New Mexico State University, Las Cruces, NM 88001, USA e-mails: david@crl.nmsu.edu, shelmrei@crl.nmsu.edu
EDUARD HOVY
Affiliation:
Information Sciences Institute, University of Southern California, Marina del Rey, CA 90292, USA e-mail: hovy@isi.edu
LORI LEVIN
Affiliation:
Language Technologies Institute, Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh, PA 15213-3890, USA e-mails: lsl@cs.cmu.edu, teruko@cs.cmu.edu
KEITH J. MILLER
Affiliation:
The MITRE Corporation, 7515 Colshire Drive, Mc Lean, VA 22102-7539, USA e-mail: freeder@mitre.org, keith@mitre.org
TERUKO MITAMURA
Affiliation:
Language Technologies Institute, Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh, PA 15213-3890, USA e-mails: lsl@cs.cmu.edu, teruko@cs.cmu.edu
OWEN RAMBOW
Affiliation:
Center for Computational Learning Systems, Columbia University, 475 Riverside Drive MC 7717, New York, NY 10115, USA e-mails: becky@cs.columbia.edu, habash@cs.columbia.edu, rambow@cs.columbia.edu
ADVAITH SIDDHARTHAN
Affiliation:
Department of Computing Science, University of Aberdeen, Aberdeen, AB24 3UE, Scotland, UK e-mail: advaith@abdn.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper focuses on an important step in the creation of a system of meaning representation and the development of semantically annotated parallel corpora, for use in applications such as machine translation, question answering, text summarization, and information retrieval. The work described below constitutes the first effort of any kind to annotate multiple translations of foreign-language texts with interlingual content. Three levels of representation are introduced: deep syntactic dependencies (IL0), intermediate semantic representations (IL1), and a normalized representation that unifies conversives, nonliteral language, and paraphrase (IL2). The resulting annotated, multilingually induced, parallel corpora will be useful as an empirical basis for a wide range of research, including the development and evaluation of interlingual NLP systems and paraphrase-extraction systems as well as a host of other research and development efforts in theoretical and applied linguistics, foreign language pedagogy, translation studies, and other related disciplines.

Type
Papers
Information
Natural Language Engineering , Volume 16 , Issue 3 , July 2010 , pp. 197 - 243
Copyright
Copyright © Cambridge University Press 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Artstein, R., and Poesio, M. 2005a. Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005, Edinburgh, UK, pp. 141–150.Google Scholar
Artstein, R., and Poesio, M. 2005b. Kappa Cubed = Alpha (or Beta). Technical Report NLE Technote 2005-01, University of Essex.Google Scholar
Artstein, R., and Poesio, M. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34: 555–596.CrossRefGoogle Scholar
Baker, C. F., Fillmore, C. J. and Lowe, J. B. 1998. The Berkeley FrameNet project. In Boitet, C., and Whitelock, P. (eds.), Proceedings of the Thirty-Sixth Annual Meeting of the Association for Computational Linguistics and Seventeenth International Conference on Computational Linguistics, pp. 86–90. San Francisco, CA: Morgan Kaufmann Publishers.Google Scholar
Baker, Kathryn, Bloodgood, Michael, Dorr, Bonnie J., Filardo, Nathaniel W., Levin, L., and Piatko, C. 2010. A modality lexicon and its use in automatic tagging. In Seventh Language Resources and Evaluation Conference (LREC-2010). University of Malta, Malta.Google Scholar
Baker, K., Bethard, S., Bloodgood, M., Brown, R., Callison-Burch, C., Coppersmith, G., Dorr, B., Filardo, W., Giles, K., Irvine, , Ann, K., Mike, L., Lori, M., Justin, M., Jim, M., Scott, P., Aaron, P. A., Piatko, C., Schwartz, L., and Zajic, D 2009. Semantically informed machine translation. Technical Report 002, Human Language Technology Center of Excellence, Summer Camp for Applied Language Exploration, Johns Hopkins University, Baltimore, MD.Google Scholar
Bannard, C., and Callison-Burch, C. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), Ann Arbor, MI, pp. 597–604.CrossRefGoogle Scholar
Barzilay, R., and Lee, L. 2003. Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In Proceedings of HLT-NAACL, Edmonton, Canada, pp. 16–23.Google Scholar
Bateman, J. A., Kasper, R. T, Moore, J. D., and Whitney, R. A. 1989. A general organization of knowledge for natural language processing: The Penman upper model. Technical Report Unpublished research report, USC/Information Sciences Institute, Marina del Rey. ISI-TR-85-029.Google Scholar
Böhmová, A., Hajič, J., Hajičová, E., and Hladká, B. 2003. The prague dependency treebank: three-level annotation scenario. In Abeillé, A. (ed.), Treebanks: Building and Using Syntactically Annotated Corpora, pp. 103–128. Dordrecht, The Netherlands: Kluwer Academic Publishers.CrossRefGoogle Scholar
Callison-Burch, C., Koehn, P., and Osborne, M. 2006. Improved statistical machine translation using paraphrases. In Proceedings of HLT-NAACL, New York, pp. 17–24.Google Scholar
Cerrato, L. 2004. A coding scheme for annotation of feedback phenomena in conversational speech. In Proceedings of the LREC Workshop on Models of Human Behaviour for the Specification and Evaluation of Multimodal Input and Output Interfaces, Lisbon, Portugal, pp. 25–28.Google Scholar
Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1): 37–46.Google Scholar
Cohen, J. 1968. Weighted Kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70: 213–220.CrossRefGoogle ScholarPubMed
di Eugenio, B., and Glass, M. 2004. The Kappa statistic: A second look. Computational Linguistics 30 (1): 95–101.Google Scholar
Dice, J. L. R. 1945. Measures of the amount of ecologic association between species. Ecology 26: 297–302.Google Scholar
Dolan, W., Quirk, C., and Brockett, C. 2004. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. In Proceedings of COLING 2004. Geneva, Switzerland.Google Scholar
Dorr, B. J. 1993. Machine Translation: A View from the Lexicon. Cambridge, MA: The MIT Press.CrossRefGoogle Scholar
Dorr, B. J., Green, R., Levin, L., Rambow, O., Farwell, D., Habash, N., Helmreich, S., Hovy, E., Miller, K. J., Mitamura, T., Reeder, F., and Siddharthan, A. 2004. Semantic annotation and lexico-syntactic paraphrase. In Proceedings of the Workshop on Building Lexical Resources from Semantically Annotated Corpora (LREC-2004). Portugal.Google Scholar
Dorr, B. J., Olsen, M., Habash, N., and Thomas, S. 2001. LCS verb database. Technical Report Online software database, University of Maryland, College Park, MD. http://www.umiacs.umd.edu/~bonnie/LCS_Database_Documentation.html [2010, March 29].Google Scholar
Farwell, D. and Helmreich, S. 1999. Pragmatics and translation. Procesamiento de Lenguaje Natural 24: 19–36.Google Scholar
Farwell, D., Helmreich, S., Reeder, F., Miller, K., Dorr, B., Habash, N., Hovy, E., Levin, L., Mitamura, T., Rambow, O., and Siddharthan, A. 2004. Interlingual annotation of multilingual text corpus. In Proceedings of the Workshop on Frontiers in Corpus Annotation. Workshop at the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), Boston, MA, pp. 55–62.Google Scholar
Fellbaum, C. (ed.) 1998. WordNet: An Electronic Lexical Database. Cambridge, MA: The MIT Press. http://wordnet.princeton.edu/ [2010, March 29].Google Scholar
Fellbaum, C., Grabowski, J., and Landes, S. 1998. Performance and confidence in a semantic annotation task. In Fellbaum, C. (ed.), WordNet: An Electronic Lexical Database, pp. 217–239. Cambridge, MA: MIT Press. http://wordnet.princeton.edu/ [2010, March 29].CrossRefGoogle Scholar
Fellbaum, C., Palmer, M., Dang, H. T., Delfs, L., and Wolf, S. 2001. Manual and automatic semantic annotation with wordnet. In Proceedings of the Workshop on WordNet and Other Lexical Resources. Pittsburgh, PA.Google Scholar
Ferro, L., Mani, I., Sundheim, B., and Wilson, G. 2001. TIDES temporal annotation guidelines, Version 1.0.2. Technical Report MTR 01W0000041, Mitre, McLean, VA.Google Scholar
Fillmore, C. 1968. The case for case. In Bach, E., and Harms, R. (eds.), Universals in Linguistic Theory, pp. 1–88. New York: Holt, Rinehart and Winston.Google Scholar
Fillmore, C., Johnson, C., and Petruck, M. 2003. Background to FrameNet. International Journal of Lexicography 16 (3): 235–250.CrossRefGoogle Scholar
Fleischman, M., Echihabi, A., and Hovy, E. H. 2003. Offline strategies for online question answering: answering questions before they are asked. In Proceedings of the ACL Conference. Sapporo, Japan.Google Scholar
Francis, W. N., and Kucera, H. 1982. Frequency Analysis of English Usage. Boston, MA: Houghton Mifflin.Google Scholar
Funaki, S. 1993. Multi-lingual machine translation (mmt) project. In Proceedings of the MT Summit IV. Washington, DC.Google Scholar
Garside, R., Leech, G., and McEnery, A. M. 1997. Corpus Annotation: Linguistic Information from Computer Text Corpora. London: Addison Wesley Longman.Google Scholar
Gut, U., and Bayerl, P. S. 2004. Measuring the reliability of manual annotations of speech corpora. In Proceedings of Speech Prosody, Nara, Japan, pp. 565–568.Google Scholar
Habash, N., and Dorr, B. J. 2003. Interlingua annotation experiment results. In Proceedings of AMTA-2002 Interlingua Reliability Workshop. Tiburon, CA.Google Scholar
Habash, N., Dorr, B., and Monz, C. 2009 Symbolic-to-statistical hybridization: extending generation-heavy machine yranslation. Machine Translation 23 (1): 23–63.CrossRefGoogle Scholar
Habash, N., Dorr, B. J., and Traum, D. 2003. Hybrid natural language generation from lexical conceptual structures. Machine Translation 18 (2): 81–128.CrossRefGoogle Scholar
Hajič, J., Vidová-Hladká, B., and Pajas, P. 2001. The prague dependency treebank: annotation structure and support. In Proceedings of the IRCS Workshop on Linguistic Databases, pp. 105–114. University of Pennsylvania, Philadelphia, PA.Google Scholar
Helmreich, S., and Farwell, D. 1998. Translation differences and pragmatics-based MT. Machine Translation 13 (1): 17–39.Google Scholar
Hirst, G. 2003. Paraphrasing paraphrased. In Keynote address for The Second International Workshop on Paraphrasing: Paraphrase Acquisition and Applications. Association for Computational Linguistics ACL 2003, Sapporo, Japan. http://ftp.cs.toronto.edu/pub/gh/Hirst-IWP-talk.pdfGoogle Scholar
Hovy, E. H., Marcus, M., Palmer, M., Pradhan, S., Ramshaw, L., and Weischedel, R. 2006. OntoNotes: the 90% solution. In Proceedings of the Human Language Technology/North American Association of Computational Linguistics conference (HLT-NAACL 2006), New York.Google Scholar
Hovy, E., Marcus, M., and Weischedel, R. 2003a. OntoBank. In Presentation at Darpa PI Meeting. Arden House, Harriman, New York.Google Scholar
Hovy, E. H., Philpot, A., Ambite, J. L., Arens, Y., Klavans, J., Bourne, W., and Saroz, D. 2003c. Data acquisition and integration in the DGRC's energy data collection project. In Proceedings of the NSF's dg.o 2001 Conference. Los Angeles, CA.Google Scholar
Hovy, E., Philpot, A., Klavans, J. L., Germann, U., and Davis, P. T. 2003b. Extending metadata definitions by automatically extracting and organizing glossary definitions. In Proceedings of the National Conference on Digital Government Research. Boston, MA.Google Scholar
Jaccard, P. 1908. Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44: 223–70.Google Scholar
Jackendoff, R. 1972. Grammatical relations and functional structure. In Semantic Interpretation in Generative Grammar. Cambridge, MA: The MIT Press.Google Scholar
Kingsbury, P., and Palmer, M. 2002. From treebank to PropBank. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC-2002). Las Palmas, Spain.Google Scholar
Kingsbury, P., Snyder, B., Xue, N., and Palmer, M. 2003. PropBank as a bootstrap for Richer annotation schemes. In Sixth Workshop on Interlinguas: Annotations and Translations, MT Summit IX. New Orleans, LA.Google Scholar
Kipper, K., Palmer, M., and Rambow, O. 2002. Extending PropBank with VerbNet semantic predicates. In Workshop on Applied Interlinguas (AMTA-2002). Tiburon, CA.Google Scholar
Knight, K., and Luk, S. K. 1994. Building a large-scale knowledge base for machine translation. In Proceedings of AAAI. Seattle, WA.Google Scholar
Kozlowski, R., McCoy, K. F., and Vijay-Shanker, K. 2003. Generation of single-sentence paraphrases from predicate/argument structure using lexico-grammatical resources. In Proceedings of the Second International Workshop on Paraphrasing: Paraphrase Acquisition and Applications (IWP2003), Sapporo, Japan, pp. 1–8. ACL 2003.Google Scholar
Krippendorff, K. 1980. Content Analysis: An Introduction to Its Methodology. Beverly Hills, CA: Sage Publications.Google Scholar
Krippendorff, K. 2007. Computing Krippendorff's alpha-reliability. http://www.asc.upenn.edu/usr/krippendorff/webreliability.doc [2010, March 29].Google Scholar
Levin, B., and Rappaport-Hovav, M. 1998. From lexical semantics to argument realization. In Borer, H. (ed.), Handbook of Morphosyntax and Argument Structure. Dordrecht: Kluwer Academic Publishers.Google Scholar
Madnani, N., Ayan, N. F., Resnik, P., and Dorr, B. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the ACL Workshop on Statistical Machine Translation. Prague, Czech Republic.Google Scholar
Mahesh, K., and Nirenburg, S. 1995. A situated ontology for practical NLP. In Proceedings of the Workshop on Basic Ontological Issues in Knowledge Sharing, International Joint Conference on Artificial Intelligence (IJCAI-95). Montreal, Canada.Google Scholar
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. 1994. Building a large annotated corpus of english: the Penn treebank. Computational Linguistics, 19 (2): 313–330.Google Scholar
Martins, T., Rino, L. H. Machado, Nunes, M. G. Volpe, Montilha, G., and Novais, O. O. 2000. An interlingua aiming at communication on the web: how language-independent can it be? In Proceedings of Workshop on Applied Interlinguas: Practical Applications of Interlingual Approaches to NLP, ANLP-NAACL. Seattle, WA.Google Scholar
Mel'čuk, I. A. 1988. Dependency Syntax: Theory and Practice. New York: State University of New York Press.Google Scholar
Mitamura, T., Miller, K. J., Dorr, B. J., Farwell, D., Habash, N., Levin, L., Helmreich, S., Hovy, E., Levin, L., Rambow, O., Reeder, F., and Siddharthan, A. 2004. Semantic Annotation of Multilingual Text Corpora. Portugal.Google Scholar
Miyoshi, H., Sugiyama, K., Kobayashi, M., and Ogino, T. 1996. An overview of the edr electronic dictionary and the current status of its utilization. In Proceedings of the 16th conference on Computational Linguistics, Copenhagen, Denmark, pp. 1090–1093.Google Scholar
Moore, R. C. 1994. Semantic evaluation for spoken-language systems. In Proceedings of the 1994 ARPA Human Language Technology Workshop. Princeton, NJ.Google Scholar
Palmer, M., Dang, H. T., and Fellbaum, C. 2005a. Making fine-grained and coarse-grained sense distinctions. Journal of Natural Language Engineering 13: 137–163.Google Scholar
Palmer, M., Gildea, D., and Kingsbury, P. 2005b. The proposition bank: a corpus annotated with semantic roles. Computational Linguistics 31 (1): 71–106.CrossRefGoogle Scholar
Pang, B., Knight, K., and Marcu, D. 2003. Syntax-based alignment of multiple translations: extracting paraphrases and generating new sentences. In Proceedings of HLT-NAACL. Edmonton, Canada.Google Scholar
Passonneau, R. 2004. Computing reliability for coreference annotation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC). Lisbon, Portugal.Google Scholar
Passonneau, R. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC). Genoa, Italy.Google Scholar
Passonneau, R. J. 2010. Formal and functional assessment of the pyramid method for summary content evaluation. Natural Language Engineering 16: 107–131.Google Scholar
Passonneau, R., Habash, N., and Rambow, O. 2006. Inter-annotator agreement on a multilingual semantic annotation task. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC). Genoa, Italy.Google Scholar
Passonneau, R., Nenkova, A., McKeown, K., and Sigelman, S. 2005. Applying the pyramid method in DUC 2005. In Proceedings of the Document Understanding Conference (DUC) Workshop. Vancouver, Canada.Google Scholar
Passonneau, R. J., Salleb-Aouissi, A., and Ide, N. 2009. Making sense of word sense variation. In Proceedings of the NAACL-HLT 2009 Workshop on Semantic Evalutions: Recent Achievements and Future Directions (SEW-2009), Boulder, CO, pp. 2–9.Google Scholar
Philpot, A., Fleischman, M., and Hovy, E. H. 2003. Semi-automatic construction of a general purpose ontology. In Proceedings of the International Lisp Conference. New York.Google Scholar
Philpot, A., Hovy, E., and Pantel, P. 2005. The omega ontology. In Proceedings of IJCAI. Edinburgh, Scotland.Google Scholar
Pradhan, S., Hovy, E. H., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R. 2007. OntoNotes: a unified relational semantic representation. In Proceedings of the First IEEE International Conference on Semantic Computing (ICSC-07), Irvine, CA, pp. 517–524.Google Scholar
Rambow, O., Dorr, B., Farwell, D., Green, R., Habash, N., Helmreich, S., Hovy, E., Levin, L., Miller, K. J., Mitamura, T., Reeder, F., and Advaith, S. 2006. Parallel syntactic annotation of multiple languages. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC2006). Genoa, Italy.Google Scholar
Reeder, F., Dorr, B., Farwell, D., Habash, N., Helmreich, S., Hovy, E., Levin, L., Mitamura, T., Miller, K., Rambow, O., and Siddharthan, A. 2004. Interlingual Annotation for MT Development. Georgetown University, Washington, DC.Google Scholar
Reidsma, D., and Carletta, J. 2008. Reliability measurement without limits. Computational Linguistics 34: 319–326.CrossRefGoogle Scholar
Rinaldi, F., Dowdall, J., Kaljurand, K., Hess, M., and Moll, D. 2003. Exploiting paraphrases in a question-answering system. In Proceedings of the Second International Workshop on Paraphrasing: Paraphrase Acquisition and Applications (IWP2003), Edmonton, Canada, pp. 25–32. ACL 2003.Google Scholar
Scott, W. 1955. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly 17: 321–325.CrossRefGoogle Scholar
Siegel, S., and Castellan, N. J. 1988. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill.Google Scholar
Stowell, T. 1981. Origins of Phrase Structure. PhD thesis, MIT.Google Scholar
Tapanainen, P., and Jarvinen, T. 1997. A non-projective dependency parser. In Proceedings of the Fifth Conference on Applied Natural Language Processing and Association for Computational Linguistics. Washington Marriott Hotel, Washington, DC.Google Scholar
Véronis, J. 2000. From the Rosetta stone to the information society: a survey of parallel text processing. In Véronis, J. (ed.), Parallel Text Processing: Alignment and Use of Translation Corpora, pp. 1–24. London: Kluwer Academic Publishers.CrossRefGoogle Scholar
Walker, K., Bamba, M., Miller, D., Ma, X., Cieri, C., and Doddington, G. 2003. Multiple-translation arabic corpus, Part 1. Technical Report catalog number LDC2003T18 and ISBN 1-58563-276-7, Linguistic Data Consortium (LDC).Google Scholar
White, J., and O'Connell, T. 1994 The ARPA MT evaluation methodologies: evolution, lessons, and future approaches. In Proceedings of the Conference of the Association for Machine Translation in the Americas. Columbia, MD.Google Scholar