Contact Us

  • You're Smart. So Why Isn't Your Web Strategy?
  • This field is for validation purposes and should be left unchanged.

Search Engine Optimization: A Brief History

Search Engine Optimization: A Brief History

What is Search Engine Optimization?

The internet is different from anything we have known.   Traditionally, if you wanted to buy a book, you would go down to the local book store and walk around the aisles and perhaps spend an hour trying to figure out the DOS computer at the Book stores search terminal. But in cyberspace, it’s a whole different story. Search engine marketing consultants will tell you that the most valuable commercial property in 2020 is going to be the first page of the search engines. The search engine provides a way to search and buy products without ever leaving the comfort of your home. It’s potentially one of the most widely used strategies for navigating in cyberspace.  Considering the quantity of information that’s available from a good search engine, it’s like having the Yellow Pages, a guide book and a road map all rolled into one.

Search engines can locate reviews, help to compare prices, and even find if there have been any reported problems with the product or the manufacturer. Typing in “books” into the Google search engine returns about 9,140,000 search results. The possibilities are endless. And because of this, webmasters have become dependent on them for generating traffic. The field of search engine optimization has evolved to meet that need.

For many of us, using search engines has become routine. Not bad for a technology that isn’t even twenty years old. But how did search engines come into being? What are the origins of the search engine that scours the outer reaches of cyberspace?

Note: This is by no means an exhaustive history of search engines. There is a resource list at the end of this article for more detailed research. The Early Beginnings of the web and the World Wide Web  ( or Your Tax Greenbacks at Work).

In 1957, after the U.S.S.R. launched Sputnik (the first artificial earth satellite), the United States created the Advanced Research Projects Agency (ARPA) as a part of the Department of Defense. Its purpose was to establish U.S. leadership in science and technology applicable to the military.

Part of ARPA’s work was to prepare a plan for the United States to maintain control over its missiles and bombers after a nuclear attack. Through this work the ARPANET – a.k.a. the Internet – was born. The first ARPANET connections were made in 1969 and in October 1972 ARPANET went ‘public.’

Almost 20 years after the creation of the Internet, the World Wide Web was born to allow the public exchange of information on a global basis. It was built on the backbone of the Internet.

“The Internet [Net] is a network of networks. Basically it is made from computers and cables…. The [World Wide] Web is an abstract imaginary space of information. On the Net, you find computers – on the Web, you find documents, sounds, videos, … information. On the Net, the connections are cables between computers; on the Web, connections are hypertext links. The Web exists because of programs which communicate between computers on the Net. The Web could not be without the Net. The Web made the Net useful because people are really interested in information and don’t really want to have [to] know about computers and cables.

With information being shared worldwide, there was eventually a need to find that information in an orderly manner.  Archie, Veronica, and Jughead
(or The History of Search Engines Beginning at Riverdale High)

The very first tool used for searching on the Internet was called “Archie”. (The name stands for “archives” without the “v”, not the kid from the comics). It was created in 1990 by Alan Emtage, a student at McGill University in Montreal. The program downloaded the directory listings of all the files located on public anonymous FTP (File Transfer Protocol) sites, creating a searchable database of filenames.

While Archie indexed computer files, “Gopher” indexed plain text documents. Gopher was created in 1991 by Mark McCahill at the University of Minnesota. (The program was named after the school’s mascot). Because these were text files, most of the Gopher sites became Web sites after the creation of the World Wide Web.

Two other programs, “Veronica” and “Jughead,” searched the files stored in Gopher index systems. Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives) provided a keyword search of most Gopher menu titles in the entire Gopher listings. Jughead (Jonzy’s Universal Gopher Hierarchy Excavation And Display) was a tool for obtaining menu information from various Gopher servers.

In 1993, MIT student Matthew Gray created what is considered the first robot, called World Wide Web Wanderer. It was initially used for counting Web servers to measure the size of the Web. The Wanderer ran monthly from 1993 to 1995. Later, it was used to obtain URLs, forming the first database of Web sites called Wandex.

According to The Web Robots FAQ, “A robot is a program that automatically traverses the Web’s hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. Web robots are sometimes referred to as web wanderers, web crawlers, or spiders. These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them.”

Initially, the robots created a bit of controversy as they used large amounts of bandwidth, sometimes causing the servers to crash. The newer robots have been tweaked and are now used for building most search engine indices.

In 1993, Martijn Koster created ALIWEB (Archie-Like Indexing of the Web). ALIWEB allowed users to submit their own pages to be indexed. According to Koster, “ALIWEB was a search engine based on automated meta-data collection, for the Web.”
Enter the Accountants

Eventually, as it seemed that the Web might be profitable, investors started to get involved in a bid to determine how to make money from the internet.

Excite was introduced in 1993 by six Stanford University students. It used statistical analysis of word relationships to aid in the search process. Within a year, Excite was incorporated and went online in December 1995. Today it’s a part of the AskJeeves company.

EINet Galaxy was was established in 1994 as a part of the MCC Research Consortium at the University of Texas at Austin. It was finally bought from the Varsity and, after being transferred through several firms, is another company Today. It was made as a directory, containing Gopher and telnet search features as well as its Web search feature.

Jerry Yang and David Filo made Yahoo in 1994. It started out as a listing of their favorite Web sites. What made it different was that each entry, as well as the URL, also had a description of the page. Inside a year the two received funding and Yahoo, the corporation, was created.

Later in 1994, WebCrawler was introduced. It was the first full-text search engine on the web; the entire text of each page was indexed for the first time.

Lycos introduced significance retrieval, prefix matching, and word proximity in 1994. It was a giant search engine, indexing over 60 million documents in 1996; the biggest of any search engine at the time. Like other search engines, Lycos was made in a college atmosphere at Carnegie Mellon University by Dr. Michael Mauldin.

Alta Vista also started in 1995. It was the first search engine to allow natural language inquires and advanced searching techniques. It also offers a multimedia search for footage, music, and videos.

Inktomi started in 1996 at UC Berkeley. In June of 1999 Inktomi introduced a directory search engine powered by “concept induction” technology. “Concept induction,” according to the company, “takes the experience of human analysis and applies the same habits to a computerized analysis of links, usage, and other patterns to figure out which sites are most popular and the most productive.” Inktomi was purchased by Yahoo in 2003.

Google was launched in 1997 by Sergey Brin and Larry Page as an element of a research project at Stanford University. It used inbound links to rank sites. In 1998 MSN Search and the Open Directory were also started. The Open Directory, according to its site, “is the largest, most comprehensive human-edited directory of the Web. It is built and maintained by a vast, global community of volunteer editors.” It tries to become the “definitive catalog of the Web.” The entire directory is maintained by human input.

About the Search Engine Pros:
The Search Engine Pros are a search engine optimization consultancy located in Santa Barbara, California. The Search Engine Pros specialize in all aspects of search engine optimization and search engine marketing, and they have helped their clients achieve internet success through the power of organic search. Other areas in which the Search Engine Pros specialize are Pay Per Click advertising, email advertising, and SEM.

Enjoyed this post?
Get an email alert when we publish another awesome blog just like this one!

Facebook Comments

No Comments

Post a Comment


Do you love our service?
Share a Facebook review in the box below!

Not on Facebook? That’s okay, review us here instead.