Hostname: page-component-78c5997874-8bhkd Total loading time: 0 Render date: 2024-11-10T15:08:32.783Z Has data issue: false hasContentIssue false
  • English
  • Français

A scaffolding approach to coreference resolution integrating statistical and rule-based models

Published online by Cambridge University Press:  21 March 2017

HEEYOUNG LEE ,
MIHAI SURDEANU  and
DAN JURAFSKY
HEEYOUNG LEE
Affiliation:
Stanford University, Stanford, CA, USA e-mails: heeyoung@cs.stanford.edu, jurafsky@stanford.edu
MIHAI SURDEANU
Affiliation:
University of Arizona, Tucson, AZ, USA e-mail: msurdeanu@email.arizona.edu
DAN JURAFSKY
Affiliation:
Stanford University, Stanford, CA, USA e-mails: heeyoung@cs.stanford.edu, jurafsky@stanford.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

We describe a scaffolding approach to the task of coreference resolution that incrementally combines statistical classifiers, each designed for a particular mention type, with rule-based models (for sub-tasks well-matched to determinism). We motivate our design by an oracle-based analysis of errors in a rule-based coreference resolution system, showing that rule-based approaches are poorly suited to tasks that require a large lexical feature space, such as resolving pronominal and common-noun mentions. Our approach combines many advantages: it incrementally builds clusters integrating joint information about entities, uses rules for deterministic phenomena, and integrates rich lexical, syntactic, and semantic features with random forest classifiers well-suited to modeling the complex feature interactions that are known to characterize the coreference task. We demonstrate that all these decisions are important. The resulting system achieves 63.2 F1 on the CoNLL-2012 shared task dataset, outperforming the rule-based starting point by over seven F1 points. Similarly, our system outperforms an equivalent sieve-based approach that relies on logistic regression classifiers instead of random forests by over four F1 points. Lastly, we show that by changing the coreference resolution system from relying on constituent-based syntax to using dependency syntax, which can be generated in linear time, we achieve a runtime speedup of 550 per cent without considerable loss of accuracy.

Type
Articles
Information
Natural Language Engineering , Volume 23 , Issue 5 , September 2017 , pp. 733 - 762
Copyright
Copyright © Cambridge University Press 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bagga, A., and Baldwin, B., 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC 1998 Workshop on Linguistic Coreference, Granada, Spain, pp. 563–6.Google Scholar
BBN Technologies 2006. Coreference Guidelines for English OntoNotes – Version 6.0. https://catalog.ldc.upenn.edu/docs/LDC2007T21/.Google Scholar
Bengtson, E., and Roth, D., 2008. Understanding the value of features for coreference resolution. In Proceedings of EMNLP 2008, Honolulu, Hawaii, pp. 294–303.Google Scholar
Björkelund, A., and Kuhn, J., 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of ACL 2014, Baltimore, Maryland, pp. 47–57.Google Scholar
Boyd, A., Gegg-Harrison, W., and Byron, D., 2005. Identifying non-referential it: a machine learning approach incorporating linguistically motivated features. In Proceedings of the ACL Workshop on Feature Engineering for Machine Learning in NLP, Ann Arbor, Michigan, pp. 40–7.Google Scholar
Breiman, L., 2001. Random forests. Machine Learning 45 (1): 5–32.Google Scholar
Burges, C. J., 1998. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2: 121–67.Google Scholar
Cer, D. M., Marneffe, M.-C. D., Jurafsky, D., and Manning, C. D., 2010. Parsing to Stanford dependencies: trade-offs between speed and accuracy. In Proceedings of LREC 2010, Valletta, Malta, pp. 1628–32.Google Scholar
Chen, C., and Ng, V., 2012. Combining the best of two worlds: a hybrid approach to multilingual coreference resolution. In Proceedings of EMNLP-CoNLL 2012, Jeju Island, Korea, pp. 56–63.Google Scholar
Chen, D., and Manning, C. D., 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP 2014, Doha, Qatar, pp. 740–50.Google Scholar
Clark, K., and Manning, C. D., 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL, Beijing, China, pp. 1405–15.Google Scholar
Clark, K., and Manning, C. D., 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, pp. 2256–62.Google Scholar
Collins, M. 1999. Head-Driven Statistical Models for Natural Language Parsing. PhD Thesis, University of Pennsylvania, Philadelphia, PA.Google Scholar
Connolly, D., Burger, J. D., and Day, D. S., 1994. A machine learning approach to anaphoric reference. In Proceedings of the International Conference on New Methods in Language Processing (NeMLaP-1), UMIST, Manchester, pp. 255–61.Google Scholar
Daumé III, H., and Marcu, D., 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In HLT-EMNLP 2005, Vancouver, B.C. Canada, pp. 97–104.Google Scholar
de Marneffe, M.-C., and Manning, C. D., 2008. The Stanford typed dependencies representation. In Proceedings of COLING Workshop on Cross-framework and Cross-domain Parser Evaluation, Manchester, UK, pp. 1–8.Google Scholar
Denis, P., and Baldridge, J., 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of NAACL-HLT 2007, Rochester, NY, pp. 236–43.Google Scholar
Denis, P., and Baldridge, J., 2008. Specialized models and ranking for coreference resolution. In Proceedings of EMNLP 2008, Honolulu, HI, pp. 660–9.Google Scholar
Durrett, G., and Klein, D. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of EMNLP-2013, Seattle, Washington.Google Scholar
Durrett, G., and Klein, D., 2014. A joint model for entity analysis: coreference, typing, and linking. TACL 2: 477–90.Google Scholar
Fernandes, E. R., dos Santos, C. N., and Milidiú, R. L., 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In EMNLP-CoNLL, Jeju, Republic of Korea, pp. 41–8.Google Scholar
Gabbard, R., Freedman, M., and Weischedel, R., 2011. Coreference for learning to extract relations: yes Virginia, coreference matters. In ACL 2011, Portland, Oregon, pp. 288–93.Google Scholar
Gilbert, N., and Riloff, E., 2013. Domain-specific coreference resolution with lexicalized features. In Proceedings of ACL 2013, Sofia, Bulgaria, pp. 81–6.Google Scholar
Haghighi, A., and Klein, D., 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of EMNLP 2009, Suntec, Singapore, pp. 1152–61.Google Scholar
Haghighi, A., and Klein, D., 2010. Coreference resolution in a modular, entity-centered model. In Proceedings of HLT-NAACL 2010, Los Angeles, CA, pp. 385–93.Google Scholar
Hajishirzi, H., Zilles, L., Weld, D. S., and Zettlemoyer, L. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In Proceedings of EMNLP 2013, Seattle, Washington.Google Scholar
Hobbs, J. R., 1978. Resolving pronoun references. Lingua 44 (4): 311–38.Google Scholar
Jindal, P., and Roth, D., 2013. Using domain knowledge and domain-inspired discourse model for coreference resolution for clinical narratives. Journal of the American Medical Informatics Association (JAMIA) 20 (2): 356–62.Google Scholar
Kehler, A., 1997. Probabilistic coreference in information extraction. In Proceedings of EMNLP 1997, Providence, Rhode Island, pp. 163–73.Google Scholar
Kilicoglu, H., Fiszman, M., and Demner-Fushman, D., 2013. Interpreting consumer health questions: the role of anaphora and ellipsis. In Proceedings of the 2013 Workshop on Biomedical Natural Language Processing, Sofia, Bulgaria, pp. 54–62.Google Scholar
Klein, D., and Manning, C. D., 2003. Accurate unlexicalized parsing. In Proceedings of ACL 2003, Sapporo, Japan, pp. 423–30.Google Scholar
Kummerfeld, J. K., and Klein, D., 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of EMNLP 2013, Seattle, Washington, pp. 265–77.Google Scholar
Lappin, S., and Leass, H. J., 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics 20 (4): 535–61.Google Scholar
Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., and Jurafsky, D., 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics 39 (4): 885–916.Google Scholar
Lee, H., Peirsman, Y., Chang, A., Chambers, N., Surdeanu, M., and Jurafsky, D., 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of CoNLL 2011: Shared Task, Portland, Oregon, pp. 28–34.Google Scholar
Levy, R., and Andrew, G., 2006. Tregex and tsurgeon: tools for querying and manipulating tree data structures. In Proceedings of LREC 2006, Genoa, Italy, pp. 2231–4.Google Scholar
Luo, X., 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP 2005, Vancouver, B.C., Canada, pp. 25–32.Google Scholar
Luo, X., Ittycheriah, A., Jing, H., Kambhatla, N., and Roukos, S. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In Proceedings of ACL 2004, Barcelona, pp. 21–6.Google Scholar
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., and McClosky, D., 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of ACL 2014, Baltimore, Maryland, pp. 55–60.Google Scholar
Mccallum, A., and Wellner, B. 2004. Conditional models of identity uncertainty with application to noun coreference. In Proceedings of NIPS 2004, Vancouver, British Columbia, Canada.Google Scholar
McCarthy, J. F., and Lehnert, W. G. 1995. Using decision trees for coreference resolution. In Proceedings of IJCAI 1995, Montréal, pp. 1050–5.Google Scholar
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J., 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS’13), Lake Tahoe, Nevada, USA, pp. 3111–9.Google Scholar
Mitkov, R., 2002. Anaphora Resolution. London: Longman.Google Scholar
Mitkov, R., Evans, R., Orăsan, C., Dornescu, I., and Rios, M. 2012. Coreference resolution: To what extent does it help nlp applications? In International Conference on Text, Speech and Dialogue, Berlin, Heidelberg: Springer, pp. 16–27.Google Scholar
Mitkov, R., Evans, R., Orasan, C., Ha, L. A., and Pekar, V. 2007. Anaphora resolution: to what extent does it help NLP applications? In Branco, A. (ed.), Proceedings of DAARC 2007, LNAI, vol. 4410, pp. 179–90. Berlin/Heidelberg: Springer-Verlag.Google Scholar
Müller, C., 2006. Automatic detection of nonreferential it in spoken multi-party dialog. In Proceedings of EACL 2006, Trento, Italy, pp. 49–56.Google Scholar
Ng, V., 2010. Supervised noun phrase coreference research: the first fifteen years. In Proceedings of ACL, Uppsala, Sweden, pp. 1396–411.Google Scholar
Ng, V., and Cardie, C. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of ACL 2002, Philadelphia, pp. 104–11.Google Scholar
Petrov, S., and Klein, D., 2007. Improved inference for unlexicalized parsing. In Proceedings of HLT-NAACL 2007, Rochester, New York, pp. 404–11.Google Scholar
Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. 2012. CoNLL-2012 shared task: modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the 16th Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea.Google Scholar
Pradhan, S. S., Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R., 2007. OntoNotes: A unified relational semantic representation. In Proceedings of ICSC, Irvine, CA, USA, pp. 517–26.Google Scholar
Raghunathan, K., Lee, H., Rangarajan, S., Chambers, N., Surdeanu, M., Jurafsky, D., and Manning, C., 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cambridge, Massachusetts, pp. 492–501.Google Scholar
Rahman, A., and Ng, V., 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), Suntec, Singapore, pp. 968–77.Google Scholar
Ratinov, L., and Roth, D., 2012. Learning-based multi-sieve co-reference resolution with knowledge. In Proceedings of EMNLP-CoNLL 2012, Jeju Island, Korea, pp. 1234–44.Google Scholar
Recasens, M., Can, M., and Jurafsky, D., 2013. Same referent, different words: unsupervised mining of opaque coreferent mentions. In Proceedings of NAACL 2013, Atlanta, Georgia, pp. 897–906.Google Scholar
Roark, B., and Hollingshead, K. 2008. Classifying chart cells for quadratic complexity context-free inference. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING), Manchester, United Kingdom.Google Scholar
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., and Young, M. 2014. Machine learning: The high interest credit card of technical debt. In SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop), Montreal, Canada.Google Scholar
Soon, W. M., Ng, H. T., and Lim, D. C. Y., 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics 27 (4): 521–44.Google Scholar
Steinberger, J., Poesio, M., Kabadjov, M. A., and Jezek, K., 2007. Two uses of anaphora resolution in summarization. Information Processing and Management 43 (6): 1663–80.Google Scholar
Stoyanov, V., Gilbert, N., Cardie, C., and Riloff, E., 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP 2009, Suntec, Singapore, pp. 656–64.Google Scholar
Stuckardt, R. 2002. Machine-learning-based vs. manually designed approaches to anaphor resolution: the best of two worlds. In Proceedings of the 4th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC2002), University of Lisbon, pp. 211–6.Google Scholar
Stuckardt, R. 2005. A machine learning approach to preference strategies for anaphor resolution. In Branco, A., McEnery, A., and Mitkov, R. (eds.), Anaphora Processing: Linguistic, Cognitive and Computational Modeling, pp. 47–72. John Benjamins, Amsterdam/Philadelphia.CrossRefGoogle Scholar
Surdeanu, M., Hicks, T., and Valenzuela-Escárcega, M. A. 2015. Two practical rhetorical structure theory parsers. In Proceedings of NAACL-HLT 2015, Denver, Colorado, USA.Google Scholar
Vilain, M., Burger, J., Aberdeen, J., Connolly, D., and Hirschman, L., 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, Columbia, Maryland, pp. 45–52.Google Scholar
Wiseman, S., Rush, A. M., and Shieber, S. M., 2016. Learning global features for coreference resolution. In Proceedings of NAACL 2016, San Diego, CA, pp. 994–1004.Google Scholar
Wiseman, S., Rush, A. M., Shieber, S. M., and Weston, J., 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of ACL-IJCNLP 2015, Beijing, China, pp. 1416–26.Google Scholar
Yang, X., Su, J., Lang, J., Tan, C. L., Liu, T., and Li, S., 2008. An entity-mention model for coreference resolution with inductive logic programming. In Proceedings of ACL-HLT 2008, Columbus, Ohio, pp. 843–51.Google Scholar
Yi, Y., Lai, C.-Y., Petrov, S., and Keutzer, K., 2011. Efficient parallel CKY parsing on GPUs. In Proceedings of the 2011 Conference on Parsing Technologies, Dublin, Ireland, pp. 175–85.Google Scholar
Yuan, B., Chen, Q., Xiang, Y., Wang, X., Ge, L., Liu, Z., Liao, M., and Si, X., 2012. A mixed deterministic model for coreference resolution. In Proceedings of EMNLP-CoNLL 2012, Jeju, Republic of Korea, pp. 76–82.Google Scholar
Zhang, X., Wu, C., and Zhao, H., 2012. Chinese coreference resolution via ordered filtering. In Proceedings of EMNLP-CoNLL 2012, CoNLL’12, Jeju, Republic of Korea, pp. 95–9.Google Scholar
Zhou, G., and Su, J., 2004. A high-performance coreference resolution system using a constraint-based multi-agent strategy. In Proceedings of COLING 2004, Geneva, Switzerland, pp. 522–9.Google Scholar