Comment 2.1

Summary

This chapter provides an excellent overview of how to think about evaluating public sector knowledge transfer activities. It provides both a conceptual framework for doing so, as well as potential metrics. And it also includes a nice review of the now large body of economic and policy literature on these topics that has been developed over the past two decades.

Background

This chapter provides an excellent overview of how to think about evaluating public sector knowledge transfer activities. It provides both a conceptual framework for doing so, as well as potential metrics. And it also includes a nice review of the now large body of economic and policy literature on these topics that has been developed over the past two decades.

Overall, the conceptual framework seems complete. Unlike much previous work in this area, it emphasizes that firms benefit from academic research not only through what the authors call formal channels (patenting and licensing) but also through more informal channels, often associated with so-called open science. And that there may be tensions, as well as complementarities, between the two channels.

Here, I offer a few additional thoughts on the conceptual framework and the indicators, and also on public policy and evaluation going forward.

Conceptual Framework

As I mentioned, the conceptual framework seems fairly comprehensive. There are, however, three things that I think are missing from the potential benefits side of the equation.

First, while the authors mentioned financial benefits, it is important to emphasize that these revenues are not “profits” for public research organizations, but rather are typically used to fund future research. That is, the potential benefit is more funding for science and technology, which may be particularly important in resource-constrained environments. I am not necessarily endorsing this rationale: as the authors point out, the financial benefits for many organizations may be small, and there are costs as well, But I think it is an important one to keep in mind since it is often a major part of the justification for formal involvement in knowledge transfer activities.

A second motivation for knowledge transfer organizations, and taking out patents and licenses in particular, is to create a way to incentivize inventor involvement and commercialization. I did not see much about this in the chapter. This may be particularly important in countries and contexts where academic involvement has previously been limited or where there are strong cultural norms militating against it. And it is most important for “embryonic” inventions needing further development, where the inventor possesses specialized tacit knowledge. However, in this context, it should be emphasized, at least in countries where inventors rather than the public research organizations previously held title to patents, that it is unclear that shifting toward ownership by the research organizations increases inventor incentives, and could in fact blunt them. Hvide and Jones’s (2018) paper in the American Economic Review provides one example. Specifically, in Norway, university researchers used to have rights to their own inventions, under the so-called “professor’s privilege.” After this was changed to be more like the US model, in which universities took rights, entrepreneurship and patenting rates by academic researchers decreased. But in general, the conceptual framework might also consider the effects of these organizations, and of patent rights, on incentivizing inventor involvement in the commercialization process.

Third, another potential benefit public sector ownership is the ability to harness this ownership to influence downstream outcomes, such as prices, access, or availability. This is mentioned in passing in the discussion of patents and access to medicines in developing countries. But it might be brought into the conceptual framework as well. That said, as far as I know, this potential role for public sector ownership has been used only sparingly.

Another observation is less about the conceptual framework than about its application. I would like to see more recognition in academic knowledge and knowledge transfer in general about what is one of the most robust empirical findings from economics over the past half-century: Patents matter more for research incentives in some fields than others. In particular, in drugs and chemical-based industries, patents are more important for appropriating returns from R&D than in other sectors (Cohen et al. 2000). Although there is no direct evidence on this in the context of university or public sector knowledge transfer (at least as far as I can recall), it would stand to reason that patents (and the prospect of exclusive licenses) are more important as commercialization incentives in some fields than others. (Drugs and biotech inventions seem like the strongest case.) In some industries, academic patents and KTOs might simply get in the way of transfer or commercialization (although they may help achieve other objectives noted here and in the chapter, such as financial returns or upstream control of particular technologies). I suspect that the costs and benefits of different channels of knowledge and knowledge transfer presented in the conceptual framework will vary sharply by field, a fact that should be considered in its application.

Indicators

The list of indicators provided is quite comprehensive. One thing I will add is that at least some of these indicators could be manipulated. For example, it is possible for an organization to increase invention disclosures and patent applications without really increasing the underlying construct of interest, namely, the extent of knowledge or knowledge transfer. Often the policy discussion ends up focusing on the indicator rather than the underlying construct. The fact that there are multiple indicators, not all of which are so easily manipulable, does help here.

But this leads to my second point, one that the authors acknowledge but is important enough to restate. It is much easier to measure the more formal activities than to measure the informal ones. If one accepts that the informal ones are important (maybe even more important based on the qualitative and historical analyses cited in the chapter), this presents a big problem. Specifically, it is possible that KTO activities could be nominally increasing some of the formal indicators but having a detrimental effect on knowledge transfer using informal channels. But evaluators are not really seeing it since we cannot measure the latter well. Even worse, and this is a theme emphasized in personnel economics, if performance is multidimensional but we only have good performance measures for some dimensions and reward based on those, this could distort incentives (for organizations, researchers) toward the better measurable but less important dimensions. I am not sure what to do about this – perhaps better bibliometric measures of more informal contributions would help (see, e.g., Bryan et al. 2019) – but policymakers, in particular, should keep this in mind. Mission statements acknowledging that traditional channels of knowledge dissemination are also important to the organization may also be helpful in setting norms.

Beyond Benchmarking: Better Evidence for Policy

Let’s step back a bit. The academic knowledge transfer movement started to accelerate in the United States of America in the 1970s and was codified by the Bayh-Dole Act of 1980. I and others have argued that Bayh-Dole was passed based on questionable evidence that the lack of patents and exclusive licenses on academic research had previously limited social returns from public research in any serious way, with the possible exception of some pharmaceuticals requiring significant investment in clinical trials (Eisenberg 1996; Mowery et al. 2004). Bayh-Dole ignored technology and knowledge transfer through the informal channels, and differences across fields in the importance of patents. And the specific indicators measuring how well the formal channels were (or were not) working were problematic (Eisenberg 1996). Similarly, other countries emulating Bayh-Dole have drawn largely on aggregate evidence of patenting and licensing (and perhaps revenues and startups) to make the case that this policy was a success, with a lack of attention to (a) the extent to which these indicators actually capture knowledge transfer and (b) potential negative effects on informal channels (Mowery and Sampat 2004).

This has been an active debate for several decades, and need not be rehashed here. However, to avoid having this same debate again several decades from now, it might be useful to implement new KTOs and patent policies in a way that facilitates evaluation going forward. That is, drawing on the conceptual framework presented in this chapter, it would be useful to prespecify outcomes and indicators of interest (including effects on formal and informal knowledge transfer), and to be clear about what would constitute evidence that the policies and institutions are working or not. Since prepost analyses can be hard to interpret, some experimentation may also help, for example rolling out policies across institutions or regions or campuses in a way that facilitates quasi-experimental evaluation. The questions raised in this chapter about what works, and potential tradeoffs, are hard ones, and in addition to collecting better indicators, policymakers might implement new laws in a way that helps us learn from new experiences in a more structured way than was possible with Bayh-Dole and its early counterparts in other OECD countries. This approach will also force organizations to be transparent and precise about the objectives they hope to achieve.

One type of experimentation that might be particularly fruitful is on licensing practices. As this chapter points out, to the extent that the goals of KTO activities are more than simply financial, patents and exclusive licenses are really only needed only for a subset of research outputs. Codifying this idea in KTO policies and missions, and making better efforts to gauge the need for an exclusive license, could also be useful (Ayres and Ouellette 2016). Building a rebuttable presumption of low-cost non-exclusive licensing into KTO patent policies and practices might be one way to do this. It may work better in some fields and countries than others, but could also create an additional layer of bureaucracy that impedes knowledge transfer, or be subject to gaming. It is quite hard to know theoretically. This is exactly why more experimentation – with a commitment to later evaluation, based on prespecified indicators and hypotheses, drawing on the framework presented in this chapter – could be extremely valuable.