There’s been enormous debate in the US over the years about whether Google and other search engines violate copyright laws by indexing content of various sorts. The Google book scanning litigation was a copyright lawsuit. And the newspaper industry has repeatedly accused Google of building its news site on the back of their copyrighted material. News organizations AP and AFP both sued Google several years ago for copyright violations. (The deal that settled the AP case is now up for renewal.)
Now, according to PaidContent, a proposed amendment to a pending UK law (“Digital Economy Bill“) would permit search engines to index any or all of the content on a “publicly accessible website” through a “presumed . . . standing and non-exclusive license.” Accordingly, indexing of third party content, however extensive, would not be liable for copyright infringement. This amendment (whose adoption is uncertain) is from self-described Libertarian-Conservative Lord Lucas. But it’s a fairly radical provision.
It’s unclear how using robots.txt would play here in terms of burdening the rights conferred by the amendment, which are much broader than “fair use” in US copyright law.
If the law and the Lucas amendment were to be adopted, imagine a scenario where a raft of new “search engines” appeared in the UK and simply cloned large portions of “publicly accessible” content sites or copied articles wholesale. This might create havoc for publishers, legitimate search engines and end users. It’s unlikely to come to that but such scenarios are clearly implicated by the wide-ranging immunity of the amendment.