BACKGROUND
The aim of health technology assessment (HTA) is the comprehensive evaluation of healthcare technologies. This includes a thorough reflection on ethical issues associated with the use of a particular health technology (10;16) or on values involved in the assessment process. In this context, ethics embraces, for example, issues of autonomy, beneficence, non-maleficence, and justice (Reference Beauchamp2;17;Reference Vaughn23). In addition, ethics is also concerned with the decision as to why a particular assessment is performed, with the involved experts as well as manufacturers’ interests. Moreover, balancing benefit and harm, issues regarding the selection of endpoints, or morally relevant issues regarding the selection of studies and meta-analyses to be included in the HTA etc. are of interest in reviewing ethical issues (11).
Although reflections on ethical issues are an integral part of comprehensive HTAs, in practice only few reports specifically address them (e.g., Reference Droste, Gerhardus and Kollek8;Reference Hofmann15;Reference Lehoux and Williams-Jones20). The number of reports also considering ethical questions has increased in recent years, but is still generally low. Information retrieval processes for effectiveness assessments are subject to only identify by chance publications on ethical issues related to health technologies or publications including discussions of ethical issues. Systematic reviews on ethical issues, therefore, require separate, individually adapted information retrieval processes.
The retrieval of information on ethical issues related to health technologies requires a specific methodological approach, as—due to these multifaceted issues—common information retrieval methods used in effectiveness assessments and standard search filters do not achieve sufficient sensitivity and precision in retrieval results. Publications describing in detail the adapted process for the required methodological approach are so far lacking.
OBJECTIVE
Our objective is to develop a methodological approach to the retrieval of information on ethical issues related to health technologies, which should provide a basis for reviewing ethical issues systematically. Depending on the chosen methodological approach on considering ethical issues the central ethical dimensions then should be answerable (11;Reference Lampe, Mäkelä and Garrido19).
METHODS
We, first, updated a comprehensive literature search to ensure that no similar methodological approach had been published elsewhere. Except for Droste (Reference Droste9), this update retrieved no hits. We then developed a workflow based on the commonly used step-by-step procedure of information retrieval for effectiveness assessments. Each step is adapted to the specific needs of the retrieval of information on ethical issues. The method proposed is applied to different topics (including various ethical issues) and validated by comparing the results with published search strategies in HTAs including ethical issues.
Objective of Information Retrieval
To fulfill the objective of completely and comprehensively evaluating ethical issues related to health technologies, information retrieval calls for the highest possible sensitivity in high precision results. To reach this goal, one must identify all relevant publications (maximum sensitivity), while at the same time trying to achieve the highest precision possible.
Sensitivity (= recall) quantifies the “fraction of known relevant documents which were effectively retrieved” while precision quantifies the “fraction of retrieved documents which are known to be relevant” (Reference Baeza-Yates and Ribeiro-Neto1).
Workflow of Information Retrieval
The proposed workflow of information retrieval processes comprises eight steps. These are as follows: Step 1: Translation of the research question into a search question; Step 2: Concept building by modeling search components; Step 3: Identification of synonyms; Step 4: Selection of relevant information sources; Step 5: Designing search strategies; Step 6: Executing the search; Step 7: Saving of retrieval results and standardized documentation and presentation; and Step 8: Final quality check and calculation of precision and recall.
We are going to present the workflow using the example of autologous stem cell transplantation in metastatic breast cancer (the complete reporting on this example is presented in Supplementary Figure 1, which can be viewed online at www.journals.cambridge.org/thc2010029).
Step 1: Translation of the Research Question into a Search Question. The starting point of the information retrieval process is the conceptualization of the search question using the PICO scheme (5) and additional components. The aim of this step is to define the search components relevant to the research question. These search components are not identical to the predefined inclusion criteria of the assessment. The definitions of the search components should be broader and more general than in the information retrieval process in effectiveness assessments.
The PICO scheme
P = Patient or problem (patient: e.g., age group; problem: e.g., disease)
I = Intervention (technology to be assessed)
C = Comparator: (if relevant: e.g., gold standard)
O = Outcomes (patient-relevant outcomes of interest)
Adjuncts:
E = Ethics (e.g., patient autonomy)
S = Healthcare system / spatial setting
The PICO scheme supports the translation of the research question: not all components of the scheme are necessarily to be completed, and, if required, additional components may be defined (cf. our example in Table 1). It might be reasonable not to translate all parts of the research question into search components.
Table 1. Definition of Search Components for the Example “Ethical Issues of Autologous Stem Cell Transplantation in Metastatic Breast Cancer”
Step 2: Concept Building by Modeling Search Components. When the relevant search components are defined, it is crucial to build concepts by modeling and linking these components. The components are often linked by adding the search component for ethical issues by means of the Boolean “AND” (see Figure 1).
Figure 1. Venn diagram to illustrate search concept.
To combine search times, most of the bibliographic databases use the Boolean operators “AND,” “OR,” “NOT.” For instance, the query “information” AND “ethics” searches for records containing both terms, the query “information” OR “ethics” searches for records including either term, and “information” NOT “ethics” searches for records containing the first term but not the second. “OR” is commonly used to combine terms within a search component, and “AND” is commonly used to combine one search component with another. The use of “NOT” should, if possible, be avoided due to the risk of the unintended exclusion of relevant records.
The sample space and its left (Patient / Problem ∩ Ethical issues) and right intersections (Intervention ∩ Ethical issues) are of interest.
The sample of interest is defined by the intersection of all search components, by the intersections of the search components “ethical issues” and “intervention,” and in some instances also by the intersection of the search components “ethical issues” and “problem.” More than one model may be appropriate and the model introduced may be more complex. When the model is more complex or comprises more than four search components, figures derived from Info-Crystals or Unified Modeling Language (UML) should be preferred to present the developed concept of information retrieval.
Info-Crystals are enhanced Venn diagrams. Each intersection or part of it can be addressed separately. Info-Crystals are suitable for presenting complex search models, such as those on ethical issues. Unified Modeling Language (UML) derived presentations are particularly suitable for search models with more than four search components.
Step 3: Identification of Synonyms. Depending on the definitions of search components, synonyms have to be identified component by component. These synonyms should be collected to achieve a high degree of completeness, as the recall of the retrieval result will depend on this factor. The main language of synonyms is English, but all terms in other relevant languages should be collected. Different spelling, inflected forms, and definitions of terms specific to countries should be considered. Sources for identifying synonyms include encyclopedias, thesauri, lexica, textbooks, systematic reviews, and HTAs. Text analyses and snowballing in bibliographic databases (e.g., PubMed®) may also be helpful. Some database providers support synonym-finding by specific routines. Potential ethical issues and related synonyms can be derived from reflective methods and instruments on ethical issues in HTA such as Hofmann's question list (Reference Hofmann14) (cf. our example in Supplemental Table 1, which can be viewed online at www.journals.cambridge.org/thc2010029).
Snowballing is a kind of non-systematic information retrieval that can be done in addition to systematic searching to identify records in the vicinity of a known relevant record or author or institution. The hierarchy of the listing of the snowballed records depends on the similarity to the starting record. In PubMed® snowballing is done by using the “Related Articles” feature. The more extensive the starting record, the better the tracked records fit. Because ethical issues are multifaceted, not only one but several starting points should be used.
Step 4: Selection of Relevant Information Sources. Numerous and diverse sources are available to identify information on ethical issues related to health technologies (see Supplemental Table 2, which can be viewed online at www.journals.cambridge.org/thc2010029). They include the following: Bibliographic and full text databases, Journals, Ethics institutes, and Experts.
BIBLIOGRAPHIC AND FULL TEXT DATABASES
Information on ethical issues related to the use of health technologies is covered by many databases from the disciplines of HTA, biomedicine, nursing, social sciences, psychology, and ethics. Biomedical databases are important sources, particularly MEDLINE®, EMBASE®, and the Science Citation Index® (Reference Fangerau12). In addition to these international databases, some national ones are available for all disciplines, which may contain additional relevant information. In detail, the following databases may be considered:
HTA DATABASES
The Cochrane Library®. (including NHS CRD databases): Medline Subject Headings (MeSH)® can be used. In some records only the publication titles are indexed (searchable). Consideration or discussion of ethical issues in full text is often not identifiable by means of bibliographic data.
BIOMEDICAL DATABASES
MEDLINE®. (includes all journals formerly indexed in BIOETHICSLINE®): Some ethics terms are included in the MeSH® thesaurus (see relevant terms in Table 2), and MEDLINE® offers the subsets “bioethics” (PubMed®, OVID) and “bioethics journals” (OVID). Additional free text queries are always necessary. Major free text terms are: “ethic*,” “moral*,” “justice,” “fairness,” “equity,” “access*,” “harm,” “patient autonomy,” and “conflict of interest”. Valuable guidance is published by the Kennedy Institute of Ethics (Reference Kahn and Ninomiya18).
Table 2. List of Thesaurus Terms / Controlled Terms, Relevant to Ethical Issues in Health Technologies in MEDLINE®, EMBASE® and PsycInfo®
EMBASE®. Some ethics-related terms are listed in the EMTREE® thesaurus (see relevant terms in Table 2), but not all their synonyms are specific to ethics. This may result in a reduced precision of retrieval results. Again, additional free text queries are necessary (see MEDLINE®).
Science Citation Index® (ISI Web of Knowledge). No thesaurus or subset exists, but “Topics” do. “Ethics” is a “Topic” and can be added to some free text queries (see MEDLINE®).
Publisher databases. Some publisher databases, for example, ScienceDirect® or Springer, provide additional publications also including non-indexed journals.
LocatorPlus®. (all monographs formerly indexed in BIOETHICSLINE® are included): The publications collected by the National Library of Medicine are searchable with MeSH® terms. Additional free text queries are necessary (see MEDLINE®).
Non-English language databases. In addition to the large biomedical databases, many national and non-English language databases exist that are relevant to the retrieval of information on ethical issues, for example, CCMed® (Germany), CISMef® (France), IME® (Spain), Italian Union Catalogue of Serials® (Italy), SveMed+® (Sweden, Scandinavia), or UDB-MED® (Russia).
SOCIAL SCIENCES AND PSYCHOLOGY DATABASES
Depending on the research question, complementing the database selection with databases for social sciences and psychology is recommended, for example, Social Science Citation Index®, PsycInfo® or Psyndex®. PsycInfo® is supported by a thesaurus comprising ethics-related terms (see relevant terms in Table 2). Additional free text queries are necessary as listed above (see MEDLINE®). All other databases are searchable solely by free text queries.
ETHICS DATABASES
Some databases with a major focus on ethics exist. As ethics depends on social and cultural context and values, national and non-English language ethics databases are important and should always be considered additionally. Important ethics databases are as follows:
Bioethics Research Library at Georgetown University. (continues ETHXWeb®): This database comprises journal articles as well as books, book chapters and reports and is supported by the Bioethics thesaurus (Kennedy Institute of Ethics) (see 21).
Euroethics®. (European Database Network on Ethics in Medicine): This database comprises several databases: BioGea® (Italy), CENDIBEM® (Spain), CRIB® (Belgium), ETHINSERM® (France), ETHMED® (Austria, Germany, Switzerland), EUROETHIK® (Germany), MIKS® (Sweden), and SveMed+® (Sweden). Euroethics® is searchable by means of free text queries and is supported by TELS® (Thesaurus Ethics in the Life Sciences) and MeSH® (see additional guidance Reference Gavel, Andersson and Knutssøn13).
BELIT®. (Bioethics Literature Database by the German Reference Centre for Ethics in the Life Sciences): This database is an integrative database comprising the DRZE catalogue (Germany), CDE® (France), ETHMED® (Germany), LEWI® (Germany), and the Bioethics Research Library at Georgetown University (United States of America). BELIT® is searchable by means of free text queries and is supported by TELS® (Thesaurus Ethics in the Life Sciences).
SIBIL®. This database is an online information system on bioethics and comprises Italian bioethics literature from 1995 to date. SIBIL® is searchable by means of free text queries and is supported by TIB® (Italian Bioethics Thesaurus) (see additional guidance Reference Dracos7).
ALL DISCIPLINES
Catalogues of monographs/gray literature: WorldCat®, The European Library®, Karlsruhe Virtual Catalogue®, OpenSigle® etc. can be addressed. These sources are also helpful in finding gray literature. Search queries using Boolean operators are applicable, but they should be rather simply constructed, using a small number of terms with few truncations, and should not contain nested parentheses. Monographs are often indexed without abstracts; see additional guidance on finding gray literature (3).
WEB SEARCH ENGINES
Many Web search engines are available. Some support specific HTA topics such as Health Technology Assessment international (HTAi) Vortal® featured by Google® custom search. Full texts, including annual reports are searched. For instance, entering the terms “societal values AND stem cell transplantation” resulted in forty-one hits. Non-HTA specific search engines such as Google Scholar® may be helpful for some topics. In the case of new and emerging technologies or if only few publications are available in biomedical and ethics databases, there is a reasonable chance of finding additional relevant publications. The availability of online books, book chapters and further publication types other than journals is also an advantage. The disadvantages of Web search engines are less frequently peer-reviewed information, less frequently disclosed conflict of interests or industrial funding, and more time-consuming retrieval. General Web search engines are at large less efficient in identifying additional relevant information on ethical issues.
JOURNALS
Articles on ethical issues related to health technologies are published in a large variety of journals and are not concentrated in a few “key journals.” Hand-searching is recommended for journals focusing on ethics that are not or not completely indexed by the selected databases. Journals that are not indexed in MEDLINE® are, for example, Chisholm Health Ethics Bulletin, Clinical Ethics, and Poiesis & Praxis. Nowadays most of the journal publishers offer online access to tables of contents and also sometimes to abstracts. Hand-searching can, therefore, mostly be conducted online. Supplements are partially inaccessible online.
INSTITUTIONS
A large number of international and national institutes, societies or working groups are engaged in ethics. Some of them have published articles on health technologies. It is not possible to list the most important institutions at this stage, but they can be found in some databases, including the International Bioethics Organizations Database® (http://bioethics.georgetown.edu/databases/organizations/index.html), the Global Ethics Observatory (GEObs, UNESCO) (http://www.unesco.org/shs/ethics/geo/user/?action=select&lng=en&db=), or the Bioethics Communication and Information System® (http://bekis.drze.de/). Further institutions may be of interest, for example, the EGE European Group on Ethics in Science and New Technologies (http://ec.europa.eu/european_group_ethics/sitemap_en.htm), and the WHO Ethics and Health Initiative (http://www.who.int/ethics/topics/en/).
Some HTA institutions give consideration to ethical issues related to health technologies. The methods used and the extent of their involvement differ widely. Expert statements are most frequently used, followed by literature reviews applying the principles approach. Interdisciplinary approaches are uncommon. Canadian and Scandinavian HTA institutions are pioneers in considering ethical issues (e.g., Reference DeJean, Giacomini, Schwartz and Miller6;11). The INAHTA and EUnetHTA have published some survey data and guidance on methods used when addressing ethical issues (11;17).
EXPERTS
When preparing a review on ethical issues related to health technologies, for some research questions it may be meaningful to consult an ethics expert. Experts may be identified by means of institutional databases (see above), bibliographic databases or with the help of a Web search engine (e.g., Google scholar®). For selected topics, this may be time-saving and more efficient.
To avoid bias in the retrieval of ethical issues, it is recommended to select several databases. International and additional national databases need to be searched to achieve sufficient sensitivity of retrieval results.
Step 5: Designing Search Strategies. To conduct an appropriate search in the selected databases, it is necessary to design search strategies. For each of the defined search components (see step 1) a search strategy consists of the identified synonyms (see step 3). Step 5 requires identifying thesaurus terms related to the synonyms and complementing them by free text queries. The Boolean operator “OR” joins them to an aggregate for each component, a procedure that is often but not always feasible (see Table 2 for available thesaurus terms on ethical issues). The aggregates of each component then have to be joined by the Boolean operator “AND.” This procedure is the same as in the information retrieval process for effectiveness assessments, with the exception that more intersections of components might be of interest to the retrieval results (c.f. step 5 to step 7 reporting in Supplementary Figure 1).
Step 6: Executing the Search. The next step in the workflow is to execute the composed search strategies. Wild cards and truncations (syntax- and provider-specific rules) should be taken into account, and when indicated, proximity operators, translations, and syntax rules. Considerable hand-searching is part of the search, as ethics journals are more incompletely indexed in bibliographic databases and are on average indexed later than medical journals. Subsequently, more free text queries and more simple strategies are necessary to search the selected relevant databases. Because functionalities such as export options are often restricted in these databases, some pre-screening must be performed while retrieval. This step 6 is accompanied by the continuous quality assurance of the retrieved results. If precision is expected to be too low, inclusion of some additional terms may avoid this problem.
Special characters are used for wildcards and truncations. Truncations are commonly used to search word stems. Most providers apply “*” for none, one or more unknown characters at the end of a term. Few providers offer truncations at the beginning of a word. Wildcards are placed in the middle of a term, for example, to mask spelling variations (colo?r or wom#n).
Proximity operators are features for defining distances between two or more terms. These operators are commonly named “ADJACENT,” “NEAR,” “NEXT,” etc.
Step 7: Saving of Retrieval Results and Standardized Documentation and Presentation. A reference management tool is essential for traditional information retrieval as well for the retrieval of ethical issues. All identified records including their complete references are saved in the reference management tool. Only duplicates are deleted. Duplicate elimination is more laborious for retrieval results on ethical issues as the varying quality of bibliographic datasets result in greater language diversity, more variations in syntax, and a higher error rate.
The reporting and presentation of the retrieval process in an appropriate manner requires the application of recommendations on the reporting of information retrieval processes and results (cf. Reference Niederstadt and Droste22 and steps 5–7 reporting in Supplementary Figure 1).
Step 8: Final Quality Check and Calculation of Precision and Recall. Formulas to measure recall and precision can be calculated from the two-by-two table (Figure 2) (Reference Baeza-Yates and Ribeiro-Neto1). However, the recall of retrieval results is de facto hardly measurable, as the universe remains unknown. The selection of many different and multidisciplinary information sources and comprehensive open-angle searching, including additional hand-searching, are good preconditions for an adequate recall. Again, reflective instruments such as Hofmann's question list, as well as the ethical dimensions, are valuable resources. The most important objective is to uncover the potentially relevant ethical aspects by means of identifying arguments and to find answers to the ethically relevant questions and central ethical dimensions.
Figure 2. Two-by-two table for calculating recall and precision of information retrieval results.
Compared with recall, precision can be calculated easily. The overall number of retrieved records (without duplicates) is known and the publications included in an assessment are also known. The number of the identified and relevant publications is related to the overall number of identified records. The result is a rate named precision, commonly expressed as a percentage.
Depending on the chosen method of assessing the potentially relevant ethical issues, the designated recall and precision of the information retrieval may be different. Therefore, it is not possible to declare general trade-offs in precision. When precision is low (perhaps due to institutional demands and context terms, as well as resources available), the consecutive workload will increase (screening of more irrelevant records that would be rejected in retrievals with higher precision). However, high recall tends to result in low precision and vice versa. Thus, the most important challenge in high-quality information retrieval in the compilation of systematic reviews is to ensure the highest possible precision.
After the retrieval process, a final quality check is recommended. The Peer Review of Electronic Search Strategies (PRESS) checklist® is a helpful tool for reviewing the reported search strategies with regard to accuracy and completeness. This checklist is validated, published and freely available from the Canadian Agency for Drugs and Technologies in Health (4, Appendix G). The checklist queries, among other things, for insufficient translation of the search question, missing thesaurus terms, misused Boolean operators, spelling variants, and typing errors.
The screening process may be then started following the predefined inclusion criteria. When the selection process is completed, flow charts of this process should be compiled. This means reporting the overall number of retrieved records, the number of included and excluded records in the first (title / abstract) screening, the number of documents included and excluded by full text screening, as well as the number of documents analyzed and finally included.
Barriers and Pitfalls in Practice
The use of standard search filters or inductive retrieval processing is not recommended and should be avoided to obtain retrieval results of high sensitivity and precision. Ethical issues may be so different between topics and so diverse within the same topic that they cannot be mapped by standard terms or similarity searching.
Not all ethical issues related to health technologies can be identified by means of the approach described. For example, publications referring to alternative technologies with a more favorable balance of benefit and harm would be identified by means of background information searching, not by means of specific information retrieval of ethical issues. It should be noted that alleged innovations cannot be identified as such by the latter methodological approach, either.
Quality-of-life studies often contain arguments relevant to ethical issues. Except for descriptive pre–post studies, they can be used as indicators of benefit and harm. Additional patient-relevant outcomes may also be identified.
It is quite helpful to know in more detail the ethical issues relevant to the health technology investigated before starting the systematic information retrieval. The compilation of synonyms and the selection of databases will be much easier.
Some ethical issues cannot be found with systematic approaches; additional non-systematic searching may be useful. Book reviews, the search for other indications, and the use of broader wording may also help to identify further issues of interest. If, for example, the technology assessed could potentially have a different meaning to different social or religious groups (which could have an impact on the use of the technology) specific searches for such different meanings might be useful.
If there is no information on ethical issues related to the use of a specific health technology, information on other indications or alternative technologies in the same population may be a valid source if similar or comparable ethical issues are touched. In particular, results from publications on comparable target populations or technologies may be applied or adapted to the actual research question. However, possible analogies have to be thoroughly scrutinized.
If initial retrieval results are too low in precision, it may be necessary to further narrow the search queries. For example, in PubMed®, MeSH® terms can be restricted to MajorMeSH indexing, while in OVID, adjacency conditions can be introduced or defined more restrictively, or more “AND” operators and additional wording can be introduced.
If sensitivity is low, it is recommended to review the relevant ethical issues and their definitions, the synonym lists, and spelling variations. Definitions of the search components may need to be broadened or further sources searched to achieve higher sensitivity.
Screening, Extraction, and Synthesis of Arguments
The screening of the records retrieved may be performed in two steps: title-abstract screening and full text screening. Frequently the steps cannot be separated accurately, as bibliographic records on ethical issues tend to be less complete than those on conventional effectiveness assessments. Full texts are analyzed according to relevant arguments by means of extraction or paraphrasing. This can be done with the assistance of text analysis tools. The arguments identified are assigned to the central ethics dimensions. These represent the medical, economic, psychological, social, and cultural consequences of applying the health technology to the individuals, stakeholders, and groups involved directly or indirectly in this application. In addition to patients, these are their relatives, physicians, nursing staff, social groups, as well as society as a whole (see 11). Finally, the arguments are synthesized and presented. This can be achieved in two ways: the assessment of ethical issues is either included in the effectiveness assessment or published as a separate literature review.
CONCLUSIONS
A methodological approach specifically adapted to ethical issues can be performed following the common retrieval workflow of effectiveness assessments. However, this approach should be undertaken separately using the adapted procedure described.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2
Supplementary Figure 1
CONTACT INFORMATION
Sigrid Droste, Dipl.-Geogr. (sigrid.droste@iqwig.de), Research Fellow, Quality in Health Care, Charalabos-Markos Dintsios, MPharm, MA, MPH (charalabos-markos.dintsios@iqwig.de), Deputy Head of Department, Department of Health Economics, Andreas Gerber, MD, PhD, MA, MSc (andreas.gerber@iqwig.de), Head of Department, Department of Health Economics, Institute for Quality and Efficiency in Health Care (IQWiG), Dillenburger Strasse 27, Cologne, Germany, 51107
CONFLICT OF INTEREST
All authors report having no potential conflict of interests.
