The number needed to retrieve: a practically useful measure of information retrieval?

Abstract

Introduction At an early point in the development of evidencebased health care, its advocates came to realize that the way evidence, particularly quantitative data, is presented has a major bearing on how it is interpreted and implemented. Thus, Fahey and colleagues sent 182 health-care executives and nonexecutives details of a hypothetical mammography programme and a cardiac rehabilitation programme.' They presented the same results of research evidence in four different ways. Decisions on which programmes to fund were significantly influenced by the way in which data were presented. This and similar research has helped to stimulate enthusiasm for the 'number needed to treat' as an intuitive and clinically meaningful measurement of treatment benefit.2 Does this original experience have implications within evidence-based library and information practice? Would librarians and information officers make different decisions about everyday practices if presented with data in a way that is different than that to which they are accustomed? Desperately seeking evidence? Systematic reviews increasingly feature on the agenda of the health librarian.3 Large numbers of information specialists are employed as members of multi-disciplinary research teams to support systematic review activities. At the same time, health librarians in general are expected to advise on whether a published systematic review has taken reasonable steps to identify all relevant evidence. Of course, 'reasonable' is a subjective concept that is difficult to quantify. Certainly, early held preconceptions suggested that such searches should be exhaustive, if not in the strict sense of interrogating multiple data sources, then in the resultant physical and mental state of the hard-working information specialist! More recent developments within health technology assessment have led to recognition that it is not always possible (or indeed desirable) to expend considerable resources in the pursuit of diminishing returns from the evidence+ Time and funding for systematic searching is usually finite. In many cases, 'good enough' is regarded as an acceptable substitute for the ideal. Here again, 'good enough' is both subjective and elusive. Current controversies Not everyone subscribes to the notion of 'good enough'. The well-publici sed incident at John Hopkins University is used by some to emphasize the importance of exhaustive searching.' Just as some clinicians refuse to acknowledge the concept of 'medical futility' (i.e. a point at which therapy should not be performed because available data show that it will not improve the patient's medical conditionj.e so some librarians will not recognize 'bibliographic futility' (i.e. a point at which further literature searching should not be performed because available data show that it will not affect the overall result of retrieval). There is also a need for agreement on what is a worthwhile outcome. Librarians tend to focus on 'numbers of items retrieved', with the implicit assumption that, if citations identified are topically relevant, then they are worth finding. More important for the review team, is the concept of 'appropriateness', that is that the retrieved references are not simply on the right topic but that they are also eligible for inclusion in the review. Of course, such a viewpoint is not only fundamentally pragmatic, but can only be evaluated in retrospect-'was this item of evidence included in the final review?' An even less forgiving verdict would be 'did this item