How much can we rely on forward patent citations?

This blogger recently summarized, here, the findings of Christopher Cotropia, Mark Lemley and Bhaven Sampat in their article, “Do applicant patent citations matter” (Research Policy 42 (2013) 844-854). The gist of the article is that US patent examiners tend to pay scant attention to applicant-submitted prior art, preferring to rely on the examiner’s own patent searches. An interesting by-product of their findings is considered at the end of their article. Based on their findings that “[m]ost of the art that appears on the face of the patent played no substantive role in the examination of that patent”, the authors go on to consider what they describe as “[i]mplications for use of patent citations as economic indicators”.

The background to the authors’ question is the cottage industry that has grown up around research by economists that focuses on the analysis of patent citations. Generally speaking, researchers have used the number of citations that a patent receives in subsequent patents, which they call “forward citations” and which serve as a proxy measure for the “private” value of the patents for the owners. Elaborating on this notion of the private value of patents, researchers have taken patent citation counts as indicating the “quality”, “importance” and “impact” of these patents (though there does not appear to be any single agreed-upon understanding of what they these terms mean). Lurking behind this research is the assumption that there “ought to be a relationship between citations and value.” As stated by a leading figure in this field, Professor Manuel Trajtenberg, here, in a seminal 1990 paper on the topic, “[t]he presumption that citation counts are potentially informative of something like the technological importance of patents is thus well grounded.”

The authors question whether this presumption has in fact been shown by focusing on the assumption that patent examiners are in fact taking into account all (or at least a critical mass) of the submitted citations. Thus, as Campbell and Nieves concluded in a pioneering paper in 1979, patent citations are a better proxy for indicating patent quality that are citations in scientific publications because the examination citation is “evidence that the particular piece of prior art was examined as a possible reason for rejecting the patent applications in the first place and yet the patent application was accepted.” But how much of this assertion stands if there is reason to believe that applicant-submitted citations may not in fact to large extent by the examiner.

Because of this, the authors are of the view that there is more “noise” in citations than the leaders of citation analysis appear to have recognized (although what precisely the authors mean by “noise” is not further stated in the article). They then go on conclude that “while our results call into question of the assumptions in patent bibliometrics—that citations are carefully vetted—it is not obvious whether violation of this assumption would bias common citations based measures, or simply add noise.” It is regretted that the authors do not elaborate on their observations, especially given the popularity of patent bibliometrics in academic research, particularly in IP management programs.

This is even more so given, as the authors note in a footnote,

“[a]s a practical matter, most validation studies find a positive association between forward citations and measures of value”. Although, in the battle of footnotes, the authors also observe that “[r]ecent work has also raised questions about the use of citations as indicator, because many of those patents turn out to be invalid and the process of adding those citations is quite complex” (citations omitted). 

 All in all, given that size and momentum enjoyed by the patent citation cottage industry, it seems that it will take a lot to undermine its legitimacy.