3 - Data Collection Tools

Summary

Introduction

Like the library in Ranganathan's (1931) five laws of library science, the web is a growing organism, and the changing nature of the web is one of the themes that will be developed throughout this book. As this chapter will show, the technologies being utilised are changing, as are the sites and services available to investigate the web. Librarians need to understand the changing nature of such sites and services if they are to make use of web metrics for either relational or evaluative purposes.

Since the mid-1990s the web has become a ubiquitous part of the modern world, and the development of associated tools and technologies has had a big impact on the nature of the web metric investigations that have taken place. During this period search engines have indexed large amounts of the web, and provided advanced search functionality that has enabled increasingly complex investigations of web content and hyperlink networks. The Web 2.0 revolution saw the major search engines and social media services provide application programming interfaces (APIs) so that developers could automatically interact with a website's content, enabling larger-scale investigations to take place. The establishing of semantic web standards promised a web where many of people's mundane computing tasks could be completed automatically by computer programs (Berners-Lee, Hendler and Lassila, 2001), while the development of cookies has enabled organisations to track users both within and across websites (Turow, 2011). However, technological progress has not been smooth and uninterrupted. Search engine and API functionality has in many instances been retracted, a semantic web has not emerged in the way some people expected and restrictions have been placed on cookies through data protection legislation, most notably General Data Protection Regulation (GDPR; Information Commissioner's Office, n.d.). All these changes have had an effect on the types of web metrics that can be developed: those metrics that were available yesterday may not be available tomorrow, but new ones will emerge. Today we are seemingly at a pivotal point in the emergence of artificial intelligence. Understanding how the tools have changed in the past can help us to understand how they may change in the future and help us to recognise potential areas of opportunity.