Hostname: page-component-cd9895bd7-lnqnp Total loading time: 0 Render date: 2024-12-26T06:19:28.587Z Has data issue: false hasContentIssue false
  • English
  • Français

Exploration of Domain Relevance by Legal Professionals in Information Retrieval Systems

Published online by Cambridge University Press:  27 April 2022

Gineke Wiggers ,
Suzan Verberne ,
Gerrit-Jan Zwenne  and
Wouter Van Loon
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper, written by Gineke Wiggers, Suzan Verberne, Gerrit-Jan Zwenne and Wouter Van Loon, addresses the concept of ‘relevance’ in relation to legal information retrieval (IR). They investigate whether the conceptual framework of relevance in legal IR, as described by Van Opijnen and Santos in their paper published in 2017, can be confirmed in practice.1 The research is conducted with a user questionnaire in which users of a legal IR system had to choose which of two results they would like to see ranked higher for a query and were asked to provide a reason for their choice. To avoid questions with an obvious answer and extract as much information as possible about the reasoning process, the search results were chosen to differ on relevance factors from the literature, where one result scores high on one factor, and the other on another factor. The questionnaire had eleven pairs of search results. A total of 43 legal professionals participated consisting of 14 legal information specialists, 6 legal scholars and 23 legal practitioners. The results confirmed the existence of domain relevance as described in the theoretical framework by Van Opijnen and Santos as published in 2017.2 Based on the factors mentioned by the respondents, the authors of this paper concluded that document type, recency, level of depth, legal hierarchy, authority, usability and whether a document is annotated are factors of domain relevance that are largely independent of the task context. The authors also investigated whether different sub-groups of users of legal IR systems (legal information specialists who are searching for others, legal scholars and also for legal practitioners) differ in terms of the factors they consider in judging the relevance of legal documents outside of a task context. Using a PERMANOVA there was found to be no significant difference in the factors reported by these groups. At this moment there is no reason to treat these sub-groups differently in legal IR systems.

Keywords

legal informationlegal professionalsinformation retrievallegal informatics
Type
Legal Informatics
Information
Legal Information Management , Volume 22 , Issue 1 , March 2022 , pp. 49 - 67
Copyright
Copyright © The Author(s), 2022. Published by British and Irish Association of Law Librarians

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Footnotes

1 Van Opijnen, M and Santos, C, ‘On the Concept of Relevance in Legal Information Retrieval’ (2017) Artificial Intelligence and Law 25, 65–87CrossRefGoogle Scholar

2 M Van Opijnen and C Santos (n 1).

3 Oxford English Dictionary (Oxford University Press, 2019) https://www.oed.com/view/Entry/161891?redirectedFrom=relevance#eid

4 T Saracevic, ‘Relevance Reconsidered, Information Science: Integration in perspectives’ in (1996) Proceedings of the Second Conference on Conceptions of Library and Information Science, 203.

5 M Van Opijnen and C Santos (n 1).

6 As discussed by Mart in 2017, the algorithms of commercial legal information retrieval systems are trade secrets, but her work and information obtained from LexisNexis (2013) and the system used in our research, Legal Intelligence, indicate that algorithmic and topical relevance are still the main focus. S N Mart, ‘The Algorithm as a Human Artifact: Implications for Legal [Re]Search’ (2017) Law Library Journal 109, 387.

7 Barry, C L, ‘User-Defined Relevance Criteria: An Exploratory Study’ (1994) Journal of the American Society for Information Science 45, 3, 149–1593.0.CO;2-J>CrossRefGoogle Scholar.

8 Park, T K, ‘The Nature of Relevance in Information Retrieval: an Empirical Study’ (1993) Library Quarterly 63, 3, 318–351Google Scholar. R Core Team. 2017. ‘R: A Language and Environment for Statistical Computing’ https://www.R-project.org/

9 Van Opijnen, M and Santos, C, ‘On the Concept of Relevance in Legal Information Retrieval’ (2017) Artificial Intelligence and Law 25, 71CrossRefGoogle Scholar.

10 Cosijn, E and Ingwersen, P, ‘Dimensions of Relevance’ (2000) Information Processing and Management 36, 541CrossRefGoogle Scholar.

11 T K Park (n 8).

12 Van Opijnen, M and Santos, C, ‘On the Concept of Relevance in Legal Information Retrieval’ (2017) Artificial Intelligence and Law 25, 65–87CrossRefGoogle Scholar.

13 E Cosijn and P Ingwersen (n 10) 541.

14 T Saracevic (n 4).

15 Cool, C, Belkin, N, Frieder, O and Kantor, P., ‘Characteristics of Text Affecting Relevance Judgments’ (1993) National Online Meeting 14, 77Google Scholar.

16 Verberne, S, He, J, Kruschwitz, U, Wiggers, G, Larsen, B, Russell-Rose, T and de Vries, A P, ‘First International Workshop on Professional Search’ (2019) ACM SIGIR Forum 52, 2, 153–162CrossRefGoogle Scholar.

17 T Saracevic, ‘Relevance: A Review of and Framework for the Thinking on the Notion in Information Science’ (1975) Journal of the American Society for Information Science, 341.

18 https://www.legalintelligence.com

19 Park, T K, ‘The Nature of Relevance in Information Retrieval: an Empirical Study’ (1993) Library Quarterly 63, 3, 322Google Scholar.

20 M Van Opijnen and C Santos (n 1).

21 T Saracevic (1975 and 1996) (n 17 and n 4).

22 T Saracevic, ‘Relevance Reconsidered, Information Science: Integration in perspectives’ in (1996) Proceedings of the Second Conference on Conceptions of Library and Information Science, 203, 12

23 T Saracevic (n 4).

24 M Van Opijnen and C Santos (n 1).

25 M Van Opijnen and C Santos (n 1), 70.

26 Bruza, P D and Huibers, T W, ‘A Study of Aboutness in Information Retrieval’ (1996) Artificial Intelligence Review 10, 5–6, 381–407CrossRefGoogle Scholar.

27 M Van Opijnen and C Santos (n 1).

28 M Van Opijnen and C Santo (n 1) 70–71 and 76–77.

29 E Cosijn and P Ingwersen (n 10), 533–550

30 E Cosijn and P Ingwersen (n 10), 541.

31 M Van Opijnen and C Santos, (n 1), 71.

32 M Van Opijnen and C Santos, (n 1), 73.

33 Schamber, L, Eisenberg, M B and Nilan, M S, ‘A Re-examination of Relevance: Toward a Dynamic, Situational Definition’ (1990) Information Processing and Management 26, 6, 760CrossRefGoogle Scholar.

34 T Saracevic, ‘Relevance: A Review of and Framework for the Thinking on the Notion in Information Science’ (1975) Journal of the American Society for Information Science, 335; Wilson, P., ‘Situational Relevance’ (1973) Information Storage and Retrieval 9, 8, 457–471CrossRefGoogle Scholar.

35 P Ingwersen and K Järvelin. ‘Information Retrieval in Context: IriX’ (2005) ACM SIGIR Forum 39, 2, 31–39.

36 T Saracevic (n 22)

37 C Cool, N Belkin, O Frieder and P Kantor, ‘Characteristics of Text Affecting Relevance Judgments’ (1993) National Online Meeting 14, 77.

38 E G Toms, H L O'Brien, R Kopak and L Freund, ‘Searching for Relevance in the Relevance of Search’ in (2005) International Conference on Conceptions of Library and Information Sciences. (Berlin, Heidelberg: Springer, 2005), 59–78.

39 T Saracevic (n 17), 340.

40 C L Barry (n 7), 152 uses the term ‘predicted relevance’.

41 Y Rieh and N J Belkin, ‘Understanding Judgment of Information Quality and Cognitive Authority in the WWW’ in (1998) Proceedings of the 61st Annual Meeting of the American Society for Information Science, 279–289.

42 R Savolainen and J Kari, ‘User-defined Relevance Criteria in Web Searching’ (2006) Journal of Documentation 62, 6, 685–707.

43 A Taylor. ‘User Relevance Criteria Choices and the Information Search Process’ (2012) Information Processing and Management 48, 136–153; A Taylor, ‘Examination of Work Task and Criteria Choices for the Relevance Judgment Process’ (2103) Journal of Documentation 69, 4, 523–544; A R Taylor, C Cool, N J Belkin and W J Amadio, ‘Relationships Between Categories of Relevance Criteria and Stage in Task Completion’ (2007) Information Processing and Management 43, 1071–1084.

44 C A Cuadra and R V Katter,. Experimental studies of relevance judgments: Final report. Vol. I: Project summary (NSF Report No. TM-3520/001/00). (Santa Monica, CA, US: System Development Corp, 1967). LexisNexis. 2013. LexisNexisLawSchools. Understanding the technology and search algorithm behind Lexis Advance. https://www.youtube.com/watch? v=bxJzfYLwXYQ&feature=youtu.be

45 A M Rees and D G Schultz. A field experimental approach to the study of relevance assessments in relation to document searching. Vol. I. Final report (NSF Contract No. C-423). (Cleveland, OH, US: Case Western Reserve University, 1967).

46 C L Barry, ‘User-Defined Relevance Criteria: An Exploratory Study’ (1994) Journal of the American Society for Information Science 45, 3, 149–159.

47 Thereby excluding known item retrieval.

48 C L Barry (n 46).

49 C L Barry and L Schamber, ‘Users’ Criteria for Relevance Evaluation: a Cross-situational Comparison’ (1998) Information Processing and Management 34, 2/3, 219–236.

50 C L Barry (n 46), P Ingwersen and K Järvelin (n 35), Y Rieh and N J Belkin (n 41), R Savolainen and J Kari (n 42).

51 T K Park (n. 19).

52 T Saracevic (n 17).

53 L Schamber et al. (n 33).

54 C L Barry and L Schamber (n 49).

55 T K Park (n 19).

56 T Saracevic (n 22).

57 Translated the examples read: ‘This could be because the title or summary seems more relevant, the result comes from an authoritative source, the publication date of the document, or because it is a document type where you expect to find the answer to the query.’

58 C L Barry (n 7), 156.

59 M Van Opijnen and C Santos (n 1), 80.

60 C L Barry and L Schamber (n 49).

61 M Van Opijnen and C Santos (n 1), 68.

62 Mentioned by users to Legal Intelligence employees.

63 C L Barry and L Schamber (n 49).

64 Also described as source quality; C L Barry (n7) 156.

65 Also described as relationship with author and source reputation/visibility; C L Barry (n 7), 155–156.

66 M Van Opijnen and C Santos (n 1), 71

67 M Van Opijnen and C Santos (n 1), 68 mentions the large diversity in document types in legal information retrieval.

68 C L Barry and L Schamber, (n 49).

69 Translated from: ‘optie 1 omdat dit hogere rechtspraak is. Bij optie 2 is de noot interessant.’

70 Of the 31 times authority was mentioned, 15 mentions appear to be related to source authority, 9 mentions appear to be related to the authority of the author, and 7 mentions provide no context as to whether the authority refers to the source or author.

71 C L Barry and L Schamber (n 49).

72 C L Barry (n 7).

73 C L Barry and L Schamber (n 49).

74 A R Taylor, C Cool, N J Belkin and W J Amadio, ‘Relationships Between Categories of Relevance Criteria and Stage in Task Completion’ (2007) Information Processing and Management 43, 1071–1084.

75 Most notable is the absence of a group for factors of algorithmic or topical relevance. The research of Schamber, as described in C L Barry and L Schamber in 1998 (n 49), extracted relevance judgments from documents that were pursued after viewing the search result, meaning the algorithmic and topical relevance can be inferred (otherwise the respondent would not have pursued the document). In the research of C L Barry in 1994 (n 7) however, responses were also noted for documents that were crossed out as ‘would not pursue’. But, also, in this research the focus was on the identification of relevance ‘criteria beyond topical appropriateness’ (C L Barry, (n 7) 149).

76 M Van Opijnen and C Santos (n 1), 70.

77 M Van Opijnen and C Santos (n 1), 70.

78 M Van Opijnen and C Santos (n 1), 71

79 M Van Opijnen and C Santos (n 1).

80 M Van Opijnen and C Santos (n 1), 71.

81 M Van Opijnen and C Santos (n 1), 71.

82 M Van Opijnen and C Santos (n 1), 71.

83 M J Anderson, ‘A New Method for Non-Parametric Multivariate Analysis of Variance’ (2001) Austral Ecology 26, 1, 32–46; M J Anderson, ‘Permutational Multivariate Analysis of Variance (PERMANOVA)’ (2017) 1–15. https://doi.org/10.1002/9781118445112.stat07841

84 M J Anderson (2017) (n 83).

85 M J Anderson (2017) (n 83).

86 A Singhal, ‘Modern information retrieval: A brief overview’ (2001) IEEE Data Eng. Bull. 24, 4, 35–43.

87 PERMANOVA was performed in R R Core Team (2017) (n 8), using package vegan: Jari Oksanen, F Guillaume Blanchet, Michael Friendly, Roeland Kindt, Pierre Legendre, Dan McGlinn, Peter R Minchin, R B O'Hara, Gavin L Simpson, Peter Solymos, M Henry, H Stevens, Eduard Szoecs and Helene Wagner, (Vegan: Community Ecology Package, 2019). https://CRAN.R-project.org/ package=vegan R package version 2.5-5.

88 ‘Regelgeving, bijvoorbeeld een Staatscourant.’ translated: ‘Regulation, for example the Government Gazette’

89 ‘Naar rechtspraak over de vernietiging van een overeenkomst met terugwerkende kracht’ translated: ‘case law on the cancellation of an agreement with retroactive effect’

90 C L Barry (n 7).

91 C L Barry and L Schamber (n 49).

92 C L Barry and L Schamber (n 49).

93 C L Barry (n 7).

94 C L Barry (n 7).

95 C L Barry and L Schamber (n 49).

96 M Van Opijnen and C Santos (n 1),71.

97 C L Barry and L Schamber (n 49)

98 P K Park (n 19); T Saracevic, ‘Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part III: Behaviour and Effects of Relevance’ (2007) Journal of the American Society for Information Science and Technology 58, 13, 2126–2144.

99 S Mi and J Jiang, ‘Understanding the Interpretability of Search Result Summaries’ in (2109) Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 989–992.

100 M Van Opijnen and C Santos (n 1), 71.

101 M Van Opijnen and C Santos (n 1), 71.

102 M Van Opijnen and C Santos (n 1), 71.

103 Translated: ‘The second option because it is civil law’, ‘Civil, not fiscal’.

104 M Van Opijnen and C Santos (n 1), 70.

105 M Van Opijnen and C Santos (n 1), 71.

106 M Van Opijnen and C Santos (n 1).

107 M Van Opijnen and C Santos (n 1), 71.

108 M Van Opijnen and C Santos (n 1), 71.

109 M Van Opijnen and C Santos, (n 1), 71.

110 Translated: ‘reports are often very long’, original:’zijn rapporten vaak erg lang’.

111 M J Anderson, ‘Permutational Multivariate Analysis of Variance (PERMANOVA)’ (2017) 1–15. https://doi.org/10.1002/9781118445112.stat07841; Jari Oksanen et al (n 87).

112 Jari Oksanen et al (n 87).

113 It appears that for legal hierarchy, the relevance of the court is not only determined by the level of hierarchy, but also by the question at which court the case for which the research is being done will be decided, related to Schamber's geographic proximity factor (Barry and Schamber (n 49)). Aspect like this context of legal hierarchy are not made visible by this research.

114 M Van Opijnen and C Santos (n 1).