Before ranking: crawlers, directories and the trespass lawsuit
The battle over search starts with a legal fight. eBay filed suit against Bidder's Edge in December 1999 over the site's crawling of eBay pages, citing an "ancient trespass theory known to legal scholars as trespass to chattels, basically a trespass or interference with real property — objects, animals, or, in this case, servers." To qualify as trespass to chattels, eBay had to prove harm, which it argued was the strain on its servers. Blocking the crawlers directly had failed, with proxied IPs replacing blocked ones each time.

Judge Ronald M. Whyte granted an injunction halting Bidder's Edge's crawling until the matter was resolved, a case whose record included the phrase "rude robots" — crawlers that ignored requests to stop spidering and tried to get around countermeasures. After appeals and countersuits, Bidder's Edge paid an undisclosed sum and shut down. As the history would show, that victory did not settle anything: the robots were already there.
Symbolic systems and the AI lineage of search
Stanford is the birthplace of Yahoo!, Google and Excite and ran the servers for the first search engines; both Yahoo!'s and Google's founders are alumni. Many prominent search figures, however, came out of Stanford's symbolic systems program. Founded in 1985 as a study of the "relationship between natural and artificial systems that represent, process, and act on information," it drew on linguistics, mathematics, semiotics, psychology, philosophy and computer science — the same fields at the core of mid-20th-century artificial intelligence research. Program alumni include several Excite founders and Srinija Srinivasan, Yahoo!'s fourth employee.
Srinivasan went from her studies to a position at the AI research lab Cyc. Marissa Mayer, a Google early employee and later Yahoo! CEO, also drew on AI research there; her thesis used natural language processing to help users find flights through a conversation with a computer. "You look at how people learn, how people reason, and ask a computer to do the same things. It's like studying the brain without the gore," she later said of the program.

The two early approaches diverged. Web crawlers like Excite would map sites programmatically to answer queries; Yahoo! guided computers with human perspective. At Google, AI was an explicit stated goal — Steven Levy, author of In the Plex, calls Google a "vehicle to realize the dream of artificial intelligence in augmenting humanity." Founders Larry Page and Sergey Brin mentioned AI at their first press conference. The conflict that dominated search for half a decade was the directory versus the crawler, human precision versus machine completeness.
Indexing: from 130 sites to the whole web
The earliest spiders were crude. Matthew Gray's World Wide Wanderer at MIT was built to count sites, not read them: 130 in the summer of 1993, 3,000 a year later, and just shy of 30,000 by 1995. The work belongs to information retrieval, which crawls digital documents, parses them, and stores them in a centralized index so a search need not re-crawl every document each time. Keeping that index current requires constant re-crawling — a hard problem for a theoretically infinite web.
Jonathan Fletcher, a University of Stirling graduate and IT employee, built Jumpstation in 1993 to avoid relying on manual lists. His crawler followed as many links as it could into a searchable, centralized database, then started over; to keep the index small, he crawled only titles and some metadata. That limitation shows in this sample of tools mapped at the time. WebCrawler, launched from the University of Washington in April 1994, was the first search engine to crawl entire webpages and make them searchable, serving 1 million queries by November. At Carnegie Mellon, Michael Maudlin released Lycos, named for the Latin translation of wolf spider; by 1995 it had indexed over a million webpages.

Commercial pressure reshaped search quickly, since search engines were a common starting point for web sessions. InfoSeek, initially a paid search option, was bought by Disney and became Netscape's default search engine; AOL bought WebCrawler; and Lycos became a fully commercial enterprise under a venture capitalist. Netscape Navigator, the browser of choice for new web users, linked directly to search engines on its homepage. Excite, another Stanford-founded crawler, was offered $3 million at launch; its six co-founders talked it over on two couches, then opted to keep the product, bring in a new CEO, and seek a far larger payoff.

Digital Equipment Corporation's AltaVista arrived at the end of 1995, built initially to demonstrate DEC hardware. Its multithreaded crawler — one researcher called it a "brood of spiders" — indexed thousands of sites at once, indexing nearly 10 million webpages at launch and over 100 million by 1996. That reach allowed full-content search, and it re-crawled every few weeks where competitors took months, setting the standard for depth and scope.

AltaVista experimented with natural language processing, translation tools and multi-lingual search, and its site search sample shows video search offered years before it became expected. Most other early search engines died of a mundane cause: universities hosting them did not like their bandwidth consumed by unrelated traffic and shut them down — all except Stanford.
Yahoo!: directory, voice, and scale
Stanford prepared for this. By the second half of the 20th century, the university was losing contracts to Harvard and MIT. In 1951, Dean of Engineering Frederick Terman brokered a deal with Palo Alto to annex 700 acres for an industrial park, which drew Hewlett-Packard and helped form the Stanford research park; Xerox and, later, Facebook were among its other occupants. The land transformed the university into an entrepreneurship and technology center.
David Filo was Jerry Yang's teaching assistant in Stanford's computer science program; the pair became, in Yang's description, "both extremely tolerant of each other, but extremely critical of everything else. We're both extremely stubborn, but very unstubborn when it comes to just understanding where we need to go." In 1994, Filo showed Yang the web, and the two shelved their thesis to surf and trade links. They combined their lists into a site hosted on Stanford's internet connection: Jerry and David's Guide to the World Wide Web, launched first to Stanford students in 1993 and to the world in January 1994. The directory sorted sites into categories made up on the fly; split when too big, it gave the site the personality Yang called the "voice of Yahoo!" — a smaller subset of the web, but the "cool web," easiest to navigate.
At the end of 1994 they renamed the site Yahoo! (an awkward forced acronym for Yet Another Hierarchical Officious Oracle). It was getting almost a hundred thousand hits a day, occasionally taking down Stanford's internet. Unlike other universities, Stanford kept the server running and encouraged the founders to go commercial. Netscape had included Yahoo! in its browser via a toolbar button labeled "Net Directory," and Marc Andreessen agreed to host the site on Netscape's servers. By spring 1996 Yahoo! had a new CEO and a record-setting IPO.
The web soon outgrew two friends swapping links. "I picture Jerry Yang as Charlie Chaplin in Modern Times," one journalist wrote, "confronted with an endless stream of new work that is only increasing in speed." Srinivasan, a symbolic systems graduate Yang and Filo had met while studying abroad in Japan, joined as Ontological Yahoo; her founders took the Chief Yahoos title. Ontology, the study of being associated with Plato and Aristotle, had a second life in artificial intelligence as a formal way for machines to categorize and share knowledge — the definition she knew from Stanford, applied to Yahoo!'s hierarchy.
Her task was to teach Yahoo! to infer what users meant and to scale the founders' point of view: "This is not a perfunctory file-keeping exercise. This is defining the nature of being," she said. "Categories and classifications are the basis for each of our worldviews." Within a few months she had reorganized the makeshift categories into a new, more concrete hierarchy, subdividing subjects and cross-linking results into multiple categories. The hierarchy, however, was a guideline; about 50 content managers, the surfers, did the curation. Coached in Yahoo!'s methodology but given wide editorial freedom, they added hundreds of sites daily and deliberated over classification, keeping personal judgment — and thus the site's voice — in the hierarchy.

When Netscape swapped its directory button for Excite late in 1995, Yahoo!'s staff feared the loss of traffic. It never came; Yahoo! remained the most popular website. Newsweek later described Yahoo! as a guide, an early warning, and a checklist item for web entrants — a directory with the largest readership among early search tools, covering hundreds of thousands of the millions of pages.

The site became a portal: Yahoo! Mail, Yahoo! Geocities, Yahoo! Calendar and more, all branded, built out through product expansion and acquisitions. In April 1996 Yahoo!, Lycos and Excite all went public to soaring prices, and InfoSeek followed a few months later. The press dubbed the race the "portal wars," with the future of the web and a billion-dollar advertising industry at stake. Excite positioned itself as a more accurate, larger alternative; Lycos abandoned the search engine that made it and chased the portal plan. When Excite announced a merger with @Home, an ISP, one investor noted "AOL and Yahoo! were already in the lead, and there was no room for a number three portal." AOL had enough muscle for the #2 slot. Everyone else was competing against Goliath, and none pulled it off. Most search engines lost track of search along the way.
Google eventually built a massive empire on a sophisticated, world-class infrastructure of its own design, fueled by the most elaborate and complex (and arguably invasive) advertising mechanism ever built. Few companies loom larger. This one, too, started at Stanford.
Even among the most renowned artificial intelligence experts, Terry Winograd, a computer scientist and Stanford professor, stands out in the crowd. He was also Larry Page's advisor and mentor when he was a graduate student in the computer science department. He recalled unique proposals he would receive from Page for his thesis — some involving "space tethers or solar kites" — "It was science fiction more than computer science," he would later remark — but Page kept returning to the web's hyperlinks. With tens of thousands of sites added weekly in 1996, one-way links decentralized the web but made it nearly impossible to assemble a list of pages linking to a given page. Page wanted a graph of who linked to whom.
His insight: a hyperlink is a citation. A site's worth depends on how often it is linked (its backlinks), and not merely on count — 100 links from pages each linked once carry far less weight than five links from pages each linked 100 times. Aggregating both dimensions yields an ordered list. As John Battelle recounts in The Search:
Page realized that a raw count of links to a page would be a useful guide to that page's rank. He also saw that each link needed its own ranking, based on the link count of its originating page. But such an approach creates a difficult and recursive mathematical challenge — you not only have to count a particular page's links, you also have to count the links attached to the links. The math gets complicated rather quickly.
Sergey Brin, a math prodigy who had crossed paths with Page a few times, took the problem. "Basically we convert the entire Web into a big equation, with several hundred million variables," he later explained, the page ranks, "and billions of terms, which are the links. And we're able to solve that equation." Scott Hassan, the seldom mentioned third co-founder who wrote the first web crawler, put it plainly: it would "surf the web backward!" The result was PageRank — named for Larry Page — and only afterward did the pair match ranks to queries and put PageRank in a search engine called Google, launched on Stanford's internet connection in August 1996.

For a shortcut, the search engine's front end let a user with a known site just type the address into the box and search, so a common destination need not rank well.
Google solved the relevancy problem: crawlers returned matching pages but could not sort them, so results lay buried. Google's first page usually had what you needed, so much so that the site added an "I'm Feeling Lucky" button that skipped straight to the top result. Growth spread by word of mouth; in 1997 it strained Stanford's network, and again Stanford encouraged its founders toward commerce. They sought to license the algorithm to Yahoo!, Infoseek and Excite, each of which passed on buying Google for a million dollars or less. A Stanford faculty member connected them to investors including Jeff Bezos and David Cheriton; they incorporated in September 1998, moved into a friend's garage with early employees including symbolics systems alum Marissa Mayer, and two checks for hundreds of thousands of dollars sat in a desk drawer or a car trunk for weeks.

Google stayed frugal, simple and fast despite investors' push toward portals, focusing on search and on machines. It aimed to turn a phrase into meaning. Google's offices were even built to simulate a college dorm. Steven Levy, in Newsweek, described it as a "high-tech version of the Oracle of Delphi, positioning everyone a mouse click away from the answers to the most arcane questions — and delivering simple answers so efficiently that the process becomes addictive." "Googling" entered the vernacular.
In 1999, Google served ten thousand searches a day; a year later, seven million. As the web grew, skeptics predicted Google could never keep up. Google kept up. After securing a further round in 1999, it moved headquarters, hired heavily — among them former AltaVista engineers and AI expert Peter Norvig — and invested in better servers, faster spiders and bigger indexes, inventing web infrastructure that had previously been theoretical.

In 2001, Page and Brin brought in Eric Schmidt as CEO for a decade. Employee #4 Heather Cairns recalled his first address: "I want you to know who your real competition is," he said. "It's Microsoft." Bill Gates later said Google "blew away the early innovators," but in the late 1990s Microsoft was watching another upstart: its war with Netscape would subsume the web for over half a decade.



