Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2024-12-26T09:17:31.280Z Has data issue: false hasContentIssue false
  • English
  • Français

Definitional, personal, and mechanical constraints on part of speech annotation performance

Published online by Cambridge University Press:  06 December 2005

ANNA BABARCZY ,
JOHN CARROLL  and
GEOFFREY SAMPSON
ANNA BABARCZY
Affiliation:
Department of Informatics, University of Sussex, Falmer, Brighton BN1 9QH, UK Present Address: Budapest University of Technology and Economics.
JOHN CARROLL
Affiliation:
Department of Informatics, University of Sussex, Falmer, Brighton BN1 9QH, UK
GEOFFREY SAMPSON
Affiliation:
Department of Informatics, University of Sussex, Falmer, Brighton BN1 9QH, UK
Get access Rights & Permissions [Opens in a new window]

Abstract

For one aspect of grammatical annotation, part-of-speech tagging, we investigate experimentally whether the ceiling on accuracy stems from limits to the precision of tag definition or limits to analysts' ability to apply precise definitions, and we examine how analysts' performance is affected by alternative types of semi-automatic support. We find that, even for analysts very well-versed in a part-of-speech tagging scheme, human ability to conform to the scheme is a more serious constraint than precision of scheme definition. We also find that although semi-automatic techniques can greatly increase speed relative to manual tagging, they have little effect on accuracy, either positively (by suggesting valid candidate tags) or negatively (by lending an appearance of authority to incorrect tag assignments). On the other hand, it emerges that there are large differences between individual analysts with respect to usability of particular types of semi-automatic support.

Type
Papers
Information
Natural Language Engineering , Volume 12 , Issue 1 , March 2006 , pp. 77 - 90
Copyright
2005 Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

A version of this paper was presented orally at the workshop “Empirical methods in the new millennium: Linguistically Interpreted Corpora” (LINC-01), at the 34th Meeting of the Societas Linguistica Europaea, Leuven, Belgium, 28 Aug–1 Sep 2001. The research was supported by the Economic and Social Research Council (UK) under award no. R00023 8146.