This morning, Carla Thompson, senior analyst at The Guidewire Group, had some glowing things to say about her experiences with Surf Canyon:

“I’ve been using the plug-in for several weeks now and have quickly grown to love it. It’s one of my favorite types of technology – I downloaded it and forgot about it until it made my life easier.”

“Clicking on a link that interested me returned a related result from page 16 of the Google results – one I never would have seen without Surf Canyon.”

“[It is] an easy-to-use service on the front and complex algorithms on the back that make my existing search habits much more productive.”

Erick Schonfeld at TechCrunch also had nice things to report:

“[Surf Canyon] offer[s] improvements to the pared-down search interface that we are all used to and point[s] to areas where search can be made better.”

“… what I like best about Surf Canyon is the interface. It doesn’t take you to another Web page. The recommended results just appear underneath the appropriate link. It feels more like an application than a cumbersome Website where you have to click through multiple pages to find what you want. Google could take a lesson in interface design from Surf Canyon here with all of its Ajax goodness.”

Elinor Mills at CNET kicked off the day at 5am PST with this:

“[Surf Canyon] was a much quicker way to find [a colleague’s] articles than scanning through pages of Web results.”

Charles Knight at AltSearchEngines, provided a very thorough review of the product, the problem it addresses and the solution it delivers. VentureBeat and InformationWeek also covered the launch as did StreetReporters.net in Paris.

Update (2/20/08) : Christophe Deschamps at Outils Froids has updated his review:

“What is certain is that I’ve been using [Surf Canyon] for two months and it has become totally indispensable.”

Mike Gunderloy at Web Worker Daily offered this:

“… if you need to quickly narrow down on a set of results from an ambiguous heap, [Surf Canyon] shines.”

Finally, Cyndy Aleo-Carreira at Profy observed that:

“[Surf Canyon] speeds up that whole paging-through-results dance or the far nerdier insert-boolean spasm we all do when trying to find something.”

Update (2/21/08) : A couple more reviews came in from France. Alain Joannes at Journalistiques has returned with this:

“Surf Canyon implemented in my Firefox browser is simply fulgurant.” (Editor’s note: We had to look it up. Fulgurant, adj., amazingly impressive; suggestive of the flashing of lightning.)

Rue89 also posted an article entitled “Welcome to the Third Age of Online Search” which offered Google Personalization and Surf Canyon as two examples of how personalization is the next major improvement in the evolution of search technology.

Update (9/10/08): Today Alt Search Engines posted an English translation on the Rue89 article.

Surf Canyon’s browser extension significantly accelerates the search process, enables searchers to find relevant information buried among results

OAKLAND, CALIF. February 19, 2008 – Surf Canyon today announced the official release of its Discovery Engine for Search™ at www.SurfCanyon.com. The browser extension for Internet Explorer and Firefox, designed to disambiguate user queries in real time, significantly accelerates the search process.

As the volume of content on the internet expands exponentially, so has the challenge of sifting through mountains of search results in order to pinpoint the most relevant. By using Surf Canyon’s semantic real-time implicit personalization technology while searching on Google, Yahoo! or MSN, searchers are able to more quickly and easily find relevant information buried within the usually overwhelming quantity of results. Rather than leaving users to manually filter through static sets of links, this innovative application uses real-time behavior signals to calculate “instantaneous relevancies” within a result set in order to dynamically pull forward the most pertinent results while pushing back those less germane. The technology is similarly suited for targeting sponsored links, benefiting both users and advertisers.

“Like many innovations, the concept was born out of frustration,” said Mark Cramer, Surf Canyon CEO. “While existing search technology is certainly more than adequate for many needs, when looking for something relatively obscure I often felt somewhat alone digging through all the results and, as such, decided that I could benefit from a search technology that would work with me during the process. With our product, search is no longer a static lecture being pushed to the user, but a participatory conversation that yields better and faster results.”

“Surf Canyon significantly improves my search experience and the deeper results it delivers are spot on virtually every time,” said Carla Thompson, senior analyst with The Guidewire Group. “I also really like that it functions within my existing search; once installed, I’m able to enjoy the advantages without having to adjust my habits.”

About Surf Canyon
Surf Canyon develops real-time implicit personalization technology for internet search. Its patents-pending Discovery Engine for Search™ transforms static result pages into dynamic knowledge resources, enabling users to more quickly and easily find pertinent information buried among all the irrelevant results, significantly accelerating the search process.
Surf Canyon has headquarters in Oakland, California and is recognized by DEMO.com as an “Innovator to Watch.”

It has been observed for some time that search engine users rarely venture past the first page of results. Various studies have shown that only 10% of the time does a user click on a result with a rank greater than 10. A typical user would thus rather reformulate the query or abandon the search when they can’t find what they need on page one.

This is particularly problematic since, as discussed previously, it’s increasingly difficult, if not impossible, to map two- and three-word queries to the exploding content on the internet. It is thus increasing unlikely that the top 10 will contain the information sought without some form of additional input from the user, either explicitly (reformulation suggestions or clusters) or implicitly (personalization).

But what if there was a search technology that was able to dig into the results and fetch the relevant information for you?

Surf Canyon’s Discovery for Search™ technology is designed to find the “hidden treasures” in the subsequent search results, and Beta testing has confirmed this. The graph below compares the initial rank of a result (x-axis) with the frequency with which it is clicked (y-axis). The AOL data is from 2006 and the Surf Canyon data is from the most recent 10,000 queries. The result is that Surf Canyon doubles (17% vs. 9%) the users’ ability to find relevant information buried in the search results.

“Personalization,” which is the tailoring of a product to the specific interests and needs of an individual, has been widely heralded as an important web innovation as search engines, and websites in general, work to deliver greater value to users. There are, in general, two types of personalization, explicit and implicit, which must both be employed to achieve the greatest benefit for the user. Through the examination of some of the limitations of the former, however, we can gain a greater appreciation for the importance of the latter.

• Explicit personalization, also referred to as “customization,” is the process through which users shape their experiences by actively selecting preferences or responding to questions. Within search, voting for, rating or deleting particular results, or any adjustment on a ‘Preferences’ or ‘Options’ page, are obvious examples of explicit personalization. Less obvious, however, are things such as presenting suggested search terms or similar queries to the user, such as with Yahoo! Search Assist or Google Suggest. Clustering (the process of suggesting possible query reformulations through the analysis of similarities between documents in the result set) is also a form of explicit personalization. In all of these cases the onus is upon to user to make a determination as to the path forward and then actively select an option.

• Implicit personalization is the process through which the users’ experiences are shaped by inferences of intent generated through the careful observation of behavior. Google’s Personalized Search, which delivers a modified set of results based on past search activity, is a clear example. Leveraging behavioral analysis or social relationships to target anything from advertising to search results to any other sort of content is also implicit personalization.

One is not better than the other. They are simply different and, to a great extent, compatible. Explicit personalization by itself, however, suffers from a number of limitations that can be alleviated through the addition of implicit personalization:

1. Explicit personalization, by definition, requires effort on the part of the user and many people merely will not bother. It seems that users want to enter their queries and then simply go straight to the results, scanning from top to bottom.
2. Even if users are willing to make the effort, they don’t necessarily know with enough precision what it is that they want. Furthermore, in some cases they might not yet possess enough knowledge to comprehend the clusters or suggestions. While clusters may sometimes help with reformulation, since it is often easier to recognize terms rather than recall them, search is frequently a process of discovery, with each result leading to better understanding of the subject matter and a greater ability to express intent. Knowing all of this a priori is not always possible.
3. Explicit personalization through a classification or clustering of search results may also create additional work if the user’s intent spans multiple contexts or sub-contexts. Selecting a cluster, for example, will enable the user to ‘zoom in’ on a particular sub-context. However, if the sub-context is not a precise match, or if the nature of the search requires a body of information that covers multiple sub-contexts, then the user will have to expend additional effort going up and down the classification tree.
4. Even if users think they know what they want, they still might be mistaken. Or worse, either intentionally or not, they are not always honest with themselves. Based on the ability to observe user behavior on all of Yahoo!’s properties, Dr. Usama Fayyad stated, “I know more about your intent than any 1000 keywords you could type,” which of course alludes to the difficulty of both accurately knowing and then expressing one’s own intent.

It is for these reasons that implicit personalization is an important compliment to explicit personalization. One could compare it to the focus adjustments of a telescope, with the course adjustment being explicit personalization and the fine adjustment being implicit personalization; you could certainly make out some celestial objects with only the course adjustment, but you need the fine adjustment to really discern the details.

According to DEMO.com:

“Surf Canyon is a service that tweaks primary search results for greater accuracy than big-name search engines provide making it an Innovator to Watch.”

Perhaps lost in all the coverage surrounding the extraordinary financial successes of search-related companies is the story regarding growing dissatisfaction with internet search. These stats from an InformationWeek article entitled “The Ultimate Search Engine” are nonetheless becoming more familiar:

“People search for 11 minutes on average before finding what they’re looking for, and half abandon searches without getting that far, according to Microsoft. By Gartner’s estimate, half of potential Web sales are lost because visitors simply can’t find what they want.”

Furthermore, a May 2007 study by Autobytel and Kelton Research asserts that “72% of consumers report experiencing ‘search engine fatigue’ (i.e., a high level of frustration) when researching a topic on the Internet” and that “65% of consumers say they’ve spent two or more hours in a single sitting searching for specific information on search engines.”

Search is not “broken,” but if half of users abandon searches without ultimately being satisfied, half of potential sales are lost and 3 out of 4 people are feeling frustration, then clearly there is substantial room for improvement.

It is our contention that there are two major factors driving this dissatisfaction:

1. The quantity of content on the internet is exploding. Google announced 6 billion items in their index way back in 2004, and one may only speculate at where it is today. Yahoo! has since proclaimed 20 billion items and, more recently, Microsoft declared a quadrupling of their index to 20 billion items. As discussed previously, it is virtually impossible to navigate that much information with two- and three-word queries.
2. As users become more familiar with search and the internet, their expectations increase. As John Battelle noted in his book “The Search”, “Surfers [are moving] from a stance of exploration (‘What’s out there?’) to expectation (‘I want to find something that I know is out there’).” When people know that the information is out there and expect to find it, but yet they can’t because it’s buried under an avalanche of irrelevant results, frustration naturally ensues.

There are, not surprisingly given the enormity of the stakes at hand, many strategies for combating this problem. Yet it is particularly instructive to note what users are actually doing during those 11 minutes to over two hours. Since most everyone has already been there, we know that it consists primarily of clicking on various results and, most importantly, query reformulation.

Query reformulation is a major challenge facing internet searchers today. 28% of all queries are reformulations of a previous query and, in such cases, the average query is reformulated 2.6 times. While experienced users know how to manipulate terms, use quotes, employ Boolean operators and even filter out documents, the outcome is that there is a direct relationship between the quality of the results and the skill of the searcher.

The major search engines, recognizing this problem, have implemented functionality to facilitate query reformulation. Yahoo!’s Search Assist will suggest keywords and related concepts as users enter queries. Google will often provide links to “related searches” at the top or bottom of their results page and they are also working on Google Suggest, a feature similar to Search Assist. Live Search has “related searches” on their results page and Ask.com has “search suggestions” during the entering of the query as well as “narrow your search” suggestions on the results page.

Nevertheless, these features and others like them presuppose a certain level of knowledge on the part of the user and require user time and effort to explicitly refine the query. Even experienced users with the aid of query suggestions might not be able to find the exact, magical combination of words and operators to accurately express intent, either because the vocabulary is unknown or the objective is imprecise.

As such, current search technologies do not yet do enough to assist users during the search process. The search results page, with its results, maps, photos, query suggestions and other offerings, is treated as a static creation presented to the end user as a final product. Once delivered, the onus is upon the user to dig through all of the pages of results and either find the information sought or reformulate the query.

It is precisely for this reason that Surf Canyon has developed Discovery for Search™ technology to work with searchers to help them find what they need. There is great benefit to the user in allowing the search results page to be dynamic, altering and refining the presentation of information as a search is being conducted. What might have been the most efficient ranking of millions of matched documents at the moment the user pressed the ‘search’ button is most likely not the case after the user’s very next action. Personalizing the search results in real time, on the fly, not only enables people to find information faster and with less effort, diminishing ‘search engine fatigue,’ but actually encourages people along the way by actively assisting with their efforts.

Surf Canyon recently received some positive coverage in the Hexagon, with nice reviews at Outils Froids and Journalistiques. While both are in French, we have taken the liberty of translating a sentence from the conclusion of M. Deschamps’ article:

“For two days I have tested [Surf Canyon] with all my searches as opposed to using Google and I must recognize that I am particularly carried away by the quality of the results supplied as well as the speed with which one discovers that which one seeks.”

Update (12/11/07): Another online journal in France, Street Reporters, has posted a nice article.

Here’s a quick 1, 2, 3 to help you get the most out of our Discovery Engine for Search™.

Step 1: Enter you query on our website or, if you like, download our add-on and then go to your favorite search engine (Google, Yahoo! or Bing), just like you always do, and enter your keywords.

Our example will use a search for the term “dolphins” since it is often used to illustrate one of the classic problems with information retrieval. No matter how much information is gathered about the subject, the individual, or all the individuals who have ever searched for “dolphins”, it is impossible to know a priori whether or not the searcher’s intent is related to the animal or to the football team. While search engines may suggest alternate queries or reformulations through the use of keyword suggestions or clusters, the search results produced will always be a mix of the animal and the football team.

Step 2: Review the results and select the one that corresponds to your interests. If it’s the Miami Dolphins that you like, click #2: MiamiDolphins.com. If your search is satisfied, great! If not, return to the search results and we will provide you with some “Recommendations”. You may, if you like, also get the “instant” Recommendations by simply clicking the bull’s eye next to the particular link of your choice.

Recommendations are dug out of the search results based on the Discovery Engine’s inference of our real-time intent. Put another way, we figure out what you want and then goes past the first page of the results and fetches it for you.

Step 3: The process is cumulative, so continue clicking on links and bull’s eyes that are of interest to you. The longer you work on the result set, the better we understand your intent and the more accurate are the Recommendations. The Recommendations can nest to three levels, enabling you to “drill down” on not just your high-level intent, but your sub-intents as well, such as “Miami Dolphins news.”

Click on “More Results” and you’ll get a second page of results automatically customized to your intent. In this case, they are naturally all related to the Miami Dolphins football team and are geared towards news.

Keep going and we will keep working to help you find what you need.

That’s Discovery!

Lastly, don’t worry about ‘mistakes’: clicking the links that are perhaps not of interest. We are smart enough to handle it.

After 1½ years of development, Surf Canyon has emerged from “stealth” mode. Last week a number of online journals, including CNET, Mashable and VentureBeat, published articles about the technology and company. Coverage was picked up in places like Egypt, Russia, Japan and Germany.

For people familiar with Google’s “similar pages” links, it is important to note that they are not at all related to Surf Canyon’s Recommendations.

The “similar pages” links on Google give users a sampling of websites that are linked to by the same websites linking to that particular result. Confusing? For example, if website A is one of the results and website B links to website A, then clicking on “similar pages” for website A will give you other websites that are linked to by website B. The theory is that if website B is linked to website A then other websites linked to by website B will be “similar” to website A. This is, of course, not necessarily the case and is, perhaps, rarely the case. The Search Engine Watch Forum has a good description of “similar pages”.

Surf Canyon’s Recommendations, on the other hand, are based on implict intent inferred from user behavior and, as a result, are considerably more valuable. Query Discovery™ technology provides the power that enables users to unlock mountains of search results in order to find relevant information buried deep within.