The History of Search Engines

Jeeves walking the Lycos dog

Today, Google and other search engines are smarter than ever—they use machine learning to help process and rank information, and can understand natural human speech. But the internet wasn’t always so easy to navigate! There was a time when you had to know the exact wording of a website’s title to find it, the search results were often riddled with spam, and getting new content indexed by the search engines could take weeks to complete.

Search engines certainly have changed! In this timeline you can learn about the history and evolution of search engines from 1990 to the present day.

How Search Engines Work

Information Retrieval

A user query prompts engines to return results, which are ranked hierarchically using trust and relevance signals.

Web Crawling

Browses the web in a methodical, automated manner.

Indexing

Pages are analyzed by titles, headings and specific fields. This is the fastest form of search.

magnifying glass on a website
    1990
  • Archie Logo
    Archie
    • The first search engine - it searched FTP sites to create index of downloadable files
    • Due to limited space, only the listings were available and not for the contents for each site
  • 1991
  • World Wide Web Virtual Library (VLib)
    • Tim Berners-Lee set up a Virtual Library
    • CERN webserver hosted a list of webservers in the early age of the Internet
  • 1992
  • Veronica
    • Searched file names and titles stored in Gopher index systems
  • 1993
  • Jughead
    • Also searched file names and titles in Gopher index systems, but only searched a single server at a time
  • World Wide Web Wanderer
    • Created by Matthew Gray; a bot counted active web servers and “measured the growth” of the Internet. Bot was soon upgraded to capture actual URLs
    • Database was called Wandex
    • Bot accessed same page hundreds of times a day and caused lag
  • Primitive Web Search
    • JumpStation: Info about a page’s title and header using simple linear search
    • World Wide Web Worm: Indexed titles and URLs (these two listed results in the order they were found without ranking)
    • RBSE Spider (had a rating system)
    • Unless exact title was a match, it was extremely hard to find anything
  • 1994
  • Infoseek Logo
    Infoseek
    • Webmasters could submit a page in realtime
  • EINet Galaxy
    • Efficient in its utilization of different web search features
    • Unnecessary library due to small size of Internet
  • ALIWEB
    • Created by Martijn Koster, it crawled meta info and allowed the user to submit the pages they wanted indexed along with a description
    • No bot & no using excessive bandwidth
    • People didn’t know how to submit their sites
  • Yahoo Logo
    Yahoo! Search
    • Created by David Filo and Jerry Yang, beginning as a collection of favorable web pages that incuded a man-made description with each URL
    • Increasing size influenced them to become a searchable directory
    • Informational sites added for free, but they expanded to included commercial sites. This is still available for $300/year
    • Long wait time to be included
  • Webcrawler Logo
    WebCrawler
    • First crawler that indexed entire pages, but was too popular to be used during daytime
  • Lycos Logo
    Lycos
    • Went public with catalog of 54,000 documents
    • Ranked Relevance retrieval and used prefix matching and word proximity
    • By August 1994 they had identified 394,000 documents; 1.5 million by January 1995
  • 1995
  • LookSmart Logo
    LookSmart
    • Competed with Yahoo! by increasing inclusion rates back and forth
  • Excite Logo
    Excite
    • Created by six Stanford undergrads
  • AltaVista Logo
    AltaVista
    • Unlimited bandwidth (for the first time)
    • First to allow natural language queries
    • Advanced searching techniques
    • Add or delete your own URL within 24 hours
    • Search tips and new features
  • AOL buys WebCrawler
  • Netscape begins using Infoseek as their default search
  • 1996
  • Backrub Logo
    Google’s beginnings: BackRub
    • Larry and Sergey began working on BackRub, a search engine which utlized backlinks for search
    • It ranked pages using citation notation, meaning any mention of a website on another site would count it as a vote toward the mentioned site
    • A website’s “authority” or reliability came from how many people linked to that site, and how trustworthy the linking sites were
  • Inktomi Logo
    Inktomi: HotBot
    • Search engine Hotbot listed on Hotwire
    • Inktomi pioneered paid inclusion model, but was not as efficient as pay per click by Overture
  • Lycos identifies 60 million documents (more than any other search engine)
  • Ask Jeeves Logo
    AskJeeves/Ask.com
    • Launch of a natural language search engine where human editors tried to match search queries
    • Powered by DirectHit, which aimed to rank links by popularity. Easy to spam
    • Uses clustering to organize sites by subject specific popularity (local web communities)
  • 1997
  • Excite buys out WebCrawler
  • 1998
  • MSN Logo
    MSN Search
    • Relied on Overture, Looksmart, and Inktomi until Google proved their backlinks model
  • Google Logo
    Google Launches
  • Overture Logo
    Overture
    • Formerly Goto.com, it was the first company to successfully provide a pay-per-click placement search service
  • 1999
  • Excite is bought by @Home for $6.5 billion
  • AllTheWeb Logo
    AllTheWeb
    • Sleek interface with advanced features that was was eventually rolled into Yahoo! search
  • Google gets funding from Sequoia Capital as well as from a few other investors
  • AOL selects Google as a search partner
  • 2000
  • LookSmart buys non-commercial directory Zeal for $20 million
  • Teoma Logo
    The Teoma engine is released
  • 2001
  • Excite’s bankruptcy leads to Infospace purchasing it for $10 million
  • AskJeeves buys Teoma to replace DirectHit
  • Inktomi is exposed for accidentally allowing public to access database of spam sites (over one million)
  • 2002
  • Yahoo! starts working on their search engine again, beginning to acquire other search directories. Until this time, they had outsourced their search services
  • LookSmart transitions into a pay per click provider, destroying reliability.
  • LookSmart buys WiseNut
  • 2003
  • Overture plans to purchase AltaVista for $80 million in stock, $60 million in cash
  • AllTheWeb bought by Overture for $70 million
  • Inktomi is bought out by Yahoo! for $235 million
  • LookSmart feels the sting of rejection when they were dumped by Microsoft and lose more than 65% of their annual revenue
  • Yahoo! buys Overture for $1.63 billion
  • 2004
  • MSN launches preview of new search engine
  • Lycos is sold to Daum Communications, the second largest Internet portal in Korea
  • 2005
  • MSN drops Yahoo! organic results for own in-house technology in January 2005
  • IAC (owner of ticketmaster.com, match.com) buys Ask Jeeves for $1.85 billion, changes name to Ask.com and drops Teoma
  • Snap Logo
    Snap
    • Overture owner Bill Gross launches Snap search engine which shows search volumes, revenues, and advertisers
    • Proved to be too complicated, not simplistic enough for the average web surfer
  • 2006
  • MSN drops Yahoo!’s search ad program
  • LookSmart shuts Zeal down
  • Microsoft announces launch of Live Search Product
  • 2007
  • 2008
  • Cuil Logo
    Cuil
    • Managed and developed by former Google employees, it had indexed 127 billion web page as of February 2009
  • 2009
  • Bing Logo
    Bing
    • Rebranding of MSN/Live Search
    • Inline search suggestions for related searches directly in result set
  • 2010

The History of Search Engines

Modern search engines are pretty incredible – complex algorithms enable search engines to take your search query and return results that are usually quite accurate, presenting you with valuable information nuggets amidst a vast information data mine.

Search engines have come a long way since their early prototypes, as our Internet Search Engines History infographic illustrates. From improvements in web crawlers and categorizing and indexing the web, to introducing new protocols such as robots.txt so that webmasters have control what web pages get crawled, to the introduction of voice search, the development of search engines has been the culmination of multiple search technologies that developed from different search engines. Alta Vista was the first search engine to process natural language queries; Lycos started strong with a system categorizing relevance signals, matching keywords with prefixes and word proximity; and Ask Jeeves introduced the use of human editors to match actual user search queries.


How Do Search Engines Work?

First of all, let's ask what is a search engine? A search engine is a program that searches the web for sites based on your keyword search terms. The search engine takes your keyword and returns search engine results pages (SERP), with a list of sites it deems relevant or connected to your searched keyword.

The goal for many sites is to appear in the first SERP for the most popular keywords related to their business. A site's keyword ranking is very important because the higher a site ranks in the SERP, the more people will see it.

SEO, or search engine optimization, is the method used to increase the likelihood of obtaining a first page ranking through techniques such as link building, SEO title tags, content optimization, meta description, and keyword research.

Google search engines and other major search engines like Bing and Yahoo use large, numerous computers in order to search through the large quantities of data across the web.

Web search engines catalog the world wide web by using a spider, or web crawler. These web-crawling robots were created for indexing content; they scan and assess the content on site pages and information archives across the web.


Algorithms and Determining the Best Search Engines

Different internet search engines use different algorithms for determining which web pages are the most relevant for a particular search engine keyword, and which web pages should appear at the top of the search engine results page.

Relevancy is the key for online search engines – users naturally prefer a search engine that will give them the best and most relevant results.

Search engines are often quite guarded with their search algorithms, since their unique algorithm is trying to generate the most relevant results. The best search engines, and often the most popular search engines as a result, are the ones that are the most relevant.


Search Engine History

Search engine history all started in 1990 with Archie, an FTP site hosting an index of downloadable directory listings. Search engines continued to be primitive directory listings, until search engines developed to crawling and indexing websites, eventually creating algorithms to optimize relevancy. 

Yahoo started off as just a list of favorite websites, eventually growing large enough to become a searchable index directory. They actually had their search services outsourced until 2002, when they started to really work on their search engine. 


History of Google Search Engine

Google's unique and improving algorithm has made it one of the most popular search engines of all time. Other search engines continue to have a difficult time matching the relevancy algorithm Google has created by examining a number of factors such as social media, inbound links, fresh content, etc.

As evidenced by the above infographic, Google appeared on the search engine scene in 1996. Google was unique because it ranked pages according to citation notation, in which a mention of one site on a different website became a vote in that site's favor. This was something that search engines

Google also began judging sites by authority. A website's authority, or trustworthiness, was determined by how many other websites were linking to it, and how reliable those outside linking sites were.

Google search history can be witnessed by taking a look at Google's homepage progressions over the years. It's remarkable to see how basic and primitive the now most popular search engine once was.


Google Search Engine History: Looking In To the Past

Google Search Engine History

A picture of the original 1997 Google search engine homepage, back when Google was part of stanford.edu.

Google Search Engine Picture

Google search engine homepage in 2005

Search Engine Screenshot

The modern, minimalist Google of 2011.


List of Alternative Search Engines

While Google is widely regarded as the most popular search engine, there are also a number of alternative search engines one can use.

Different search engines exist for unique needs. For example, you may want a search engine to help you search for blogs specifically, or perhaps you want search engines for kids that only return sites appropriate for children.

Here is a list of search engines that cater to particular interests:

These free search engines enable you to easily find information that caters to your unique needs.


Deep Web Search Engines

In addition to standard search engines, there are deep web search engines as well.

The deep web refers to areas of the internet that are not susceptible to normal indexing, and so can't easily be found and indexed by crawlers. These alternative search engines specialize in this not-easy-to-find data.

  • Yippy is a popular deep web search engine, especially useful for niches and obscure interests
  • Scirus is a science specific deep web search engine
  • Deep Dyve is a deep web search engine for scholarly articles and journals

Top 10 Search Engines

Below is our top 10 internet search engines list. While deciding which is the "best" search engine depends on your unique wants and needs, below is a list of popular search engines, some more unique than others.

  • Google - Offering everything from image searches, map searches, news searches, etc. With impressive keyword relevancy and a continuously improving search algorithm, it's easy to see why Google is still the reigning champ.
  • Mahalo - Mahalo is a unique 'human-powered' search engine that employs a group of editors to manually sift and organize thousands of pieces of content. Poor guys. 
  • Yahoo - While Yahoo has been suffering as of late, it's still a classic and a popular search engine.
  • Bing - The Microsoft powered search engine prides itself on being a "decision engine" by offering search suggestions on the side column and providing extra search options.
  • Ask - Clean layout and handy results grouping.
  • AOL Search - AOL continues to be used, primarily by people who still use AOL. They're out there somewhere.
  • Blekko - Blekko's clean, minimalist layout is easy to navigate, and /tags allow for grouping searches.
  • DogPile - the once alternative to Google is getting a comeback and is a great alternative to bigger search engines.
  • Duck Duck Go - Doesn't track your search history and is avoids spammy sites.
  • The Internet Archive - This search engine lets users travel back in time to see how web pages looked in years gone by. A very fun search engine to play around with.

WordStream is a provider of PPC management software and SEO tools for continuously optimizing and expanding Internet marketing efforts, involving large numbers of keywords, such as long tail keywords and search engine keywords. WordStream provides a scalable, private, online keyword workbench—which includes a keyword analyzer and a keyword suggestion tool—to better find keywords, group keywords, conduct keyword analysis and for and turning your research into action. WordStream also provides a FREE keyword tool, which is faster, more accurate and gives you thousands more keyword suggestions than the Google AdWords Keyword Tool for both SEO and  PPC marketing.