Friday, 20 June 2014

How much can we rely on forward patent citations?

This blogger recently summarized, here, the findings of Christopher Cotropia, Mark Lemley and Bhaven Sampat in their article, “Do applicant patent citations matter” (Research Policy 42 (2013) 844-854). The gist of the article is that US patent examiners tend to pay scant attention to applicant-submitted prior art, preferring to rely on the examiner’s own patent searches. An interesting by-product of their findings is considered at the end of their article. Based on their findings that “[m]ost of the art that appears on the face of the patent played no substantive role in the examination of that patent”, the authors go on to consider what they describe as “[i]mplications for use of patent citations as economic indicators”.

The background to the authors’ question is the cottage industry that has grown up around research by economists that focuses on the analysis of patent citations. Generally speaking, researchers have used the number of citations that a patent receives in subsequent patents, which they call “forward citations” and which serve as a proxy measure for the “private” value of the patents for the owners. Elaborating on this notion of the private value of patents, researchers have taken patent citation counts as indicating the “quality”, “importance” and “impact” of these patents (though there does not appear to be any single agreed-upon understanding of what they these terms mean). Lurking behind this research is the assumption that there “ought to be a relationship between citations and value.” As stated by a leading figure in this field, Professor Manuel Trajtenberg, here, in a seminal 1990 paper on the topic, “[t]he presumption that citation counts are potentially informative of something like the technological importance of patents is thus well grounded.”

The authors question whether this presumption has in fact been shown by focusing on the assumption that patent examiners are in fact taking into account all (or at least a critical mass) of the submitted citations. Thus, as Campbell and Nieves concluded in a pioneering paper in 1979, patent citations are a better proxy for indicating patent quality that are citations in scientific publications because the examination citation is “evidence that the particular piece of prior art was examined as a possible reason for rejecting the patent applications in the first place and yet the patent application was accepted.” But how much of this assertion stands if there is reason to believe that applicant-submitted citations may not in fact to large extent by the examiner.

Because of this, the authors are of the view that there is more “noise” in citations than the leaders of citation analysis appear to have recognized (although what precisely the authors mean by “noise” is not further stated in the article). They then go on conclude that “while our results call into question of the assumptions in patent bibliometrics—that citations are carefully vetted—it is not obvious whether violation of this assumption would bias common citations based measures, or simply add noise.” It is regretted that the authors do not elaborate on their observations, especially given the popularity of patent bibliometrics in academic research, particularly in IP management programs.

This is even more so given, as the authors note in a footnote,
“[a]s a practical matter, most validation studies find a positive association between forward citations and measures of value”. Although, in the battle of footnotes, the authors also observe that “[r]ecent work has also raised questions about the use of citations as indicator, because many of those patents turn out to be invalid and the process of adding those citations is quite complex” (citations omitted). 
 All in all, given that size and momentum enjoyed by the patent citation cottage industry, it seems that it will take a lot to undermine its legitimacy.


Unknown said...

Firstly, I need to declare that I am a vendor of citation analysis products.

With that out of the way, I thank the author for his carefully considered review of the field.

Neil does make a point that citation data can be noisy. I would fully agree, but have two further points in response:

1) Other patent analysis techniques are also noisy. For example, have any of the readers ever done a conventional patent analysis or search, say based on keywords or class codes, and found that many if not the majority of the patents found are not relevant to their objective? You too? Experience to date suggests that citation data can be somewhat less noisy than alternative techniques

2) There are techniques out there to reduce the noise, for example techniques developed by the company I work with, Ambercite.

I apologize if this look like an advertisement, but having seen the issue of citation noise raised, it is only fair that this issue be put into perspective and the readers informed that solutions to this issue are available.

New Paltzer said...

On particular bias in forward citations that is difficult to resolve is how to count citations by an assignee or inventor to their own prior art. From a patentability perspective their own prior art may not be the most relevant, but an attorney or agent working on the case will probably feel obligated to cite the potentially "material" prior art. So a prolific inventor, or a business that generates many patents will tend to have many forward references (including their own). But it would not be appropriate to completely eliminate those citations from the data set - because they may truly indicate the relevance of the prior art.