A Brief History of Search Engines

The internet is different from anything we have known. Traditionally, if you wanted to buy a book, you would go down to the local book store and walk around the aisles and perhaps spend an hour trying to figure out the DOS computer at the Book stores search terminal. But in cyberspace, it's a whole different story. Search engine marketing consultants will tell you that the most valuable commercial property in 2020 is going to be the first page of the search engines. The search engine provides a way to search and buy products without ever leaving the comfort of your home. It's potentially one of the most widely used strategies for navigating in cyberspace. Considering the quantity of information that's available from a good search engine, it's like having the Yellow Pages, a guide book and a road map all rolled into one.

Search engines can locate reviews, help to compare prices, and even find if there have been any reported problems with the product or the manufacturer. Typing in "books" into the Google search engine returns about 9,140,000 search results. The possiblities are endless.

For many of us, using search engines has become routine. Not bad for a technology that isn't even twenty years old. But how did search engines come into being? What are the origins of the search engine that scowers the outer reaches of cyberspace?

Note : This is in no means an exhaustive history of search engines. There is a resource list at the end of this article for more detailed research. The Early Beginnings of the web and the World Wide Web ( or Your Tax Greenbacks at Work).

In 1957, after the U.S.S.R. launched Sputnik (the first artificial earth satellite), the United States created the Advanced Research Projects Agency (ARPA) as a part of the Department of Defense. Its purpose was to establish U.S. leadership in science and technology applicable to the military.

Part of ARPA's work was to prepare a plan for the United States to maintain control over its missiles and bombers after a nuclear attack. Through this work the ARPANET - a.k.a. the Internet - was born. The first ARPANET connections were made in 1969 and in October 1972 ARPANET went 'public.'

Almost 20 years after the creation of the Internet, the World Wide Web was born to allow the public exchange of information on a global basis. It was built on the backbone of the Internet.

"The Internet [Net] is a network of networks. Basically it is made from computers and cables.... The [World Wide] Web is an abstract imaginary space of information. On the Net, you find computers - on the Web, you find documents, sounds, videos, ... information. On the Net, the connections are cables between computers; on the Web, connections are hypertext links. The Web exists because of programs which communicate between computers on the Net. The Web could not be without the Net. The Web made the Net useful because people are really interested in information and don't really want to have [to] know about computers and cables.

With information being shared worldwide, there was eventually a need to find that information in an orderly manner. Archie, Veronica, and Jughead
(or The History of Search Engines Beginning at Riverdale High)

The very first tool used for searching on the Internet was called "Archie". (The name stands for "archives" without the "v", not the kid from the comics). It was created in 1990 by Alan Emtage, a student at McGill University in Montreal. The program downloaded the directory listings of all the files located on public anonymous FTP (File Transfer Protocol) sites, creating a searchable database of filenames.

While Archie indexed computer files, "Gopher" indexed plain text documents. Gopher was created in 1991 by Mark McCahill at the University of Minnesota. (The program was named after the school's mascot). Because these were text files, most of the Gopher sites became Web sites after the creation of the World Wide Web.

Two other programs, "Veronica" and "Jughead," searched the files stored in Gopher index systems. Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives) provided a keyword search of most Gopher menu titles in the entire Gopher listings. Jughead (Jonzy's Universal Gopher Hierarchy Excavation And Display) was a tool for obtaining menu information from various Gopher servers.
I, Robot

In 1993, MIT student Matthew Gray created what is considered the first robot, called World Wide Web Wanderer. It was initially used for counting Web servers to measure the size of the Web. The Wanderer ran monthly from 1993 to 1995. Later, it was used to obtain URLs, forming the first database of Web sites called Wandex.

According to The Web Robots FAQ, "A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. Web robots are sometimes referred to as web wanderers, web crawlers, or spiders. These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them."

Initially, the robots created a bit of controversy as they used large amounts of bandwidth, sometimes causing the servers to crash. The newer robots have been tweaked and are now used for building most search engine indexes.

In 1993, Martijn Koster created ALIWEB (Archie-Like Indexing of the Web). ALIWEB allowed users to submit their own pages to be indexed. According to Koster, "ALIWEB was a search engine based on automated meta-data collection, for the Web."
Enter the Accountants

Eventually, as it seemed that the Web might be profitable, investors started to get involved and search engines became big business.

Excite was introduced in 1993 by six Stanford University students. It used statistical analysis of word relationships to aid in the search process. Within a year, Excite was incorporated and went online in December 1995. Today it's a part of the AskJeeves company.

EINet Galaxy was was established in 1994 as a part of the MCC Research Consortium at the School of Texas, in Austin. It was finally bought from the Varsity and, after being transferred through several firms, is another company Today. It was made as a directory, containing Gopher and telnet search features as well as its Web search feature.

Jerry Yang and David Filo made Yahoo in 1994. It started out as a listing of their favorite Web sites. What made it different was that each entry, as well as the URL, also had a description of the page. Inside a year the two received funding and Yahoo, the corporation, was created.

Later in 1994, WebCrawler was introduced. It was the first full-text search engine on the web; the entire text of each page was indexed for the first time.

Lycos introduced significance retrieval, prefix matching, and word proximity in 1994. It was a giant search engine, indexing over 60 million documents in 1996; the biggest of any search engine at the time. Like other search engines, Lycos was made in a college atmosphere at Carnegie Mellon Varsity by Dr. Michael Mauldin.

Infoseek went online in 1995. It did not truly bring anything new to the search engine scene. It is now owned by the Walt Disney Net Group and the domain forwards to Go.com.

Alta Vista also started in 1995. It was the first search engine to allow natural language inquires and advanced searching techniques. It also offers a multimedia search for footage, music, and videos.

Inktomi started in 1996 at UC Berkeley. In June of 1999 Inktomi introduced a directory search engine powered by "concept induction" technology. "Concept induction," according to the company, "takes the experience of human analysis and applies the same habits to a computerized analysis of links, usage, and other patterns to figure out which sites are most popular and the most productive." Inktomi was purchased by Yahoo in 2003.

AskJeeves and Northern Light were both launched in 1997.

Google was launched in 1997 by Sergey Brin and Larry Page as an element of a research project at Stanford School . It uses inbound links to rank sites. In 1998 MSN Search and the Open Directory were also started. The Open Directory, according to its site, "is the largest, most comprehensive human-edited directory of the Web. It is built and maintained by a vast, global community of volunteer editors." it tries to become the "definitive catalogue of the Web." The entire directory is maintained by human input.

Adobe Flash Player not installed or older than 9.0.115!

Here is another great place to learn about the history of the Search Engines: Wiki-pedia http://en.wikipedia.org/wiki/Search_engines

About the Author: Taylor Reaume is a fortune 500 SEO Consultant from Santa Barbara, CA. Specialities include: ethical search engine optimization, internet marketing courses, Google search marketing, SEO copywriting services

Set as favorite

Bookmark

Email This

Trackback(0)

TrackBack URI for this entry

Comments (7)

...

Search Engine is every bodies requirement. And you give excellent detail of search engines. great

Viny , July 01, 2009

...

this is a great article, I really enjoyed what you said

robin , September 22, 2009

...

Taylor's a great SEO guy...he runs a team of Search Engine Optimization Specialists that help Boost Organic SEO, based in Santa Barbara, CA. Search engine optimization companies like the search engine pros are very effective at helping boost organic SEO. Organic SEO will help maximize your traffic levels and profits.. I definitely recommend Him. And surely you will be happy for the great results.
Keep it up Taylor!!!

Debbie , September 27, 2009

...

Very comprehensive article on Search Engine's history. Not a lot know it's importance in the cyber world. This article is surely helpful and very informative.

SHERYL , September 30, 2009