After five clicks--you pay...Google and news publishers

More details about the new relationship between Google and some publishers.

"Placating Publishers by Limiting Links: A Google 5-Click FAQ"

by

Eliot Van Buskirk, Ryan Singel, and John C Abell

December 2nd, 2009

Wired

Google’s been taking it on the chin from traditional publishers (i.e., News Corp. chairman Rupert Murdoch) a lot lately. So it should come as no surprise that the search giant has come up with a new way for media sites to throw up a digital checkpoint where money or credentials can be demanded from readers who got there in a Google search.

It’s probably also no accident that the latest initiative came less than 24 hours after Murdoch railed yet again against Google and its ilk. That rant continues no matter how many times Google tells content owners how to keep their content out of Google searches.

It remains to be seen if anyone will take the bait. Few have employed the blunt-force padlock that is robots.txt, a piece of two-line code that essentially bars the door from search bots. Google says any disgruntled publisher can start using it yesterday, but publishers don’t want to be left alone — they really want to be paid.

But it’s a complicated issue, and Wired.com is here to help.

Google requires that sites show a visiting user the same thing that it shows to Google’s indexing robot. If your site requires a subscription or registration to see pages of the site, those locked-down pages won’t show up in search results. News sites have dealt with this by taking advantage of Google’s First Click Free program, where any single page of a website can be seen, as long as the visitor shows up through a link from Google News or Google web search. If a user then clicks on another link to the site from within the site or from Google, the website can push the user to register, sign in or subscribe.

The idea is to allow searchers to find and read content on pay sites like the Wall Street Journal or mandatory-registration sites like the New York Times, while simultaneously letting those sites encourage readers to sign up. Being included in the Google index is important for news sites, because Google search traffic can make up a majority of traffic to news sites. Google News, the automated newspaper created through aggregated links, drives comparatively little traffic and is not a large revenue maker for Google.

Google announced that it is expanding that program by limiting the number of free pages to up to five per user on a given day, depending on how publishers set up their sites. That limit prevents users from evading registration or subscription by simply googling the headline of every story they want to read in order to avoid registration or a subscription, while still offering people a good chunk of free content every day.

Google is also now allowing subscription-only sites to show just portions of stories — the headlines and first few paragraphs — and have those snippets show up in searches. However, links to such sites will have the label “subscription” next to them in search results.

These changes come as Murdoch continues to accuse search engines, semi-automated aggregation sites and bloggers who rewrite stories of stealing content from his newspaper empire. News organizations continue to cut back on staff and bureaus as ad dollars continue to flee newsprint. Meanwhile, the Federal Trade Commission is also looking into newspapers’ ongoing trouble adapting their business models to the internet, where information wants to be free (as in beer).

To clear up what this means to readers and publishers, we put together the following FAQ about Google’s changes to the “First Click Free” program:

What has changed?

Publishers and website owners can now limit the number of times you can access their websites for free through Google to five times per day. Google put the new limit in place because “some [publishers] are worried about people abusing the spirit of First Click Free to access almost all of their content.” The previous version didn’t limit the number of pages on a subscription-required (or registration-required) site that users could access from Google’s News or web search. Unless you use Google or Google News to access the same website over and over rather than sampling the full breadth of what’s available, this rule change won’t affect you.

What gives Google the right to index anything for search?

The Digital Millennium Copyright Act allows search engines to list indexes and search results of whatever websites they want, so long as they remove them if asked to do so by copyright holders. Sites that don’t want search engines to index their pages can use a little file called robots.txt to tell search engine spiders to stay away — a voluntary but generally respected system.

What do publishers have to do to take advantage of the program?

If they’re using it for Google News, they need to tell Google. If you want to apply the system to your website in the general Google web search, you don’t need to alert Google; you just need to set up your website so that regular users — and Googlebot — can read at least one entire page (or as many as five) for free during a visit to your website. You would need to set up a request for registration or subscription only after a user clicks through to a subsequent page on your site from Google results.

Are there technical ways around the limits?

Certainly. The limit will likely be cookie-based, so simply clearing your cookies or changing browsers will help. A more clever workaround involves changing your browser’s user agent so it looks to the website like you are Google’s indexing spider. It’s a surprisingly easy change in Firefox, but might be overkill for most users.

Does this make any sense for a site that doesn’t have a paywall?

Yes, sites that require a free registration for repeat users will find this limit useful, and it will let them better track and target users for ads.

What would the immediate effect be if implemented? On Google? On the site? On the reading public?

For most users it won’t change anything. For website owners, it offers a way to monetize content through subscriptions, without disappearing from Google.

Are there any other industries which might want the same deal?

The music industry already has a similar deal with MySpace and Lala on the music side, where you get one free listen of any particular song. Depending on how this new five-page limit goes, record labels and music publishers could experiment with surfacing subscription services through Google.

The same way MySpace and Lala use free Google plays as a teaser for a song purchase, music-subscription services could copy the text-publisher model by offering five songs from various collections, playlists or subscription services for free before asking listeners to subscribe. Video subscriptions could potentially be offered in the same way.

What about using Google to search news and websites by including “site:[domain].com” in a search query (for example, “google news” site:wired.com)?

If a publisher or website joins the First One Free program, you will only be able to use Google’s web search to get to the site’s web pages up to five times a day.

Are there any implications in this modified policy for Google Books or even Google Editions?

Probably not. Google Books is a program under which Google scans works and hosts the digital versions of books in and out of print, so it needs no cooperation from other sites. This policy could be used as an option for wary authors and publishers who decide to have their in-copyright books remain in the search, but want to limit how many books of theirs are seen a day by the same user. Page limits per book are already built in to the system, which we explain in greater depth in our Google Books FAQ. Google Editions is a browser-based e-reader initiative, and the content will be entirely opt-in, so there should be no friction with anyone whatsoever.

Would this affect search only? What about all those stories I see on Google News?

If Google can see an article for indexing purposes, it can display the headline and first sentence on Google News, so nothing changes on that front. This will only affect you if you click through to view news stories or websites.

How does this affect linking? What if I link to a story in my post that that someone can’t read because of this?

Other sites may not have the ability to link to web pages without encountering a registration page or subscription pay wall. Using the link that Google or Google News uses should make it possible for your readers to get to the story without getting the login page.

Who’s most likely to bear the brunt of customer anger when things seem broken? Google or the site?

Things will only seem broken if a user accesses the same news publisher or subscription site through Google for the second to sixth time in a day, depending on where the site has set its limit.

Why is five the “magic” number? Or is it “up to five?”

It’s up to five. According to Google, this achieves a balance between monetizing content and allowing Googlebot and Google users some degree of free access.

Why doesn’t Google just pay every publisher 1 percent of whatever it makes on every click as a good-faith gesture?

Google might answer this question with another question: Why should Google pay 1 percent of its search-engine ad revenue to the websites of the world when the DMCA allows it to display search results for free? Or why shouldn’t websites pay Google for all the traffic that flow to their sites from its searches?


That will be $19.95 please...