The Search Engine: From Archie to AI-Augmented Search
Photo: Pixabay / Pexels
Before Google, before Yahoo, before anyone had heard the phrase "just search for it," there was a graduate student at McGill University manually trawling FTP servers for useful files. His solution to that drudgery became the world's first search engine — and set in motion an invention that now processes an estimated 8.5 billion queries every day. This is the story of how humans taught machines to find information, and what that capability has become.
What a Search Engine Actually Does
A search engine is a software system that indexes content across a network, accepts a query from a user, and returns a ranked list of results it judges relevant. That three-part cycle — crawl, index, rank — is the skeleton every major search engine still runs on, even when AI layers sit on top of it.
The crawl phase uses automated programs (called crawlers, spiders, or bots) that follow links from page to page, downloading content as they go. The index phase stores that content in a structured database so queries can be answered in milliseconds rather than hours. The rank phase applies algorithms to sort results by estimated relevance to the query.
Think of it like a library, but one where the librarians have read every book, catalogued every sentence, and can cross-reference any word against every other book in 0.3 seconds. The difference is that a library's catalogue is static; a search index is updated continuously.
Definition: PageRank is an algorithm that estimates a webpage's importance by counting how many other pages link to it and weighting those links by the linking page's own importance. It treats links as votes, with more authoritative pages casting heavier votes.
The Story: From FTP Lists to the Web's Memory
Archie and the Pre-Web Internet (1990)
In 1990, Alan Emtage was a graduate student and systems administrator at McGill University in Montreal. Part of his job was to manually find useful software on public FTP (File Transfer Protocol) servers, which were then the internet's primary file-distribution infrastructure. The process was tedious: connect to one server, browse its files, connect to the next, browse again.
Emtage automated the task. Working with colleagues Bill Heelan and J. Peter Deutsch, he wrote a program that would connect to public FTP servers on a schedule, collect the names of available files, and store them in a searchable database. He called it Archie — "archive" with the 'v' dropped — and released it from McGill's servers in 1990.
Source: McGill University Bicentennial — Creation of the first internet search engine
Archie did not index the content of files, only their names and locations. It also had nothing to do with the World Wide Web, which Tim Berners-Lee would not release publicly until 1991. But as the first tool that automated the discovery of internet resources, Archie established the conceptual template: let a machine survey the network so humans do not have to.
By the early 1990s, non-proprietary Archie attracted roughly half of all internet traffic in Canada. In 1992, Emtage and Deutsch co-founded Bunyip Information Systems to commercialize a licensed version of the tool.
Source: Capitol Technology University — Alan Emtage, Creator of ARCHIE
| Year | Milestone | Significance |
|---|---|---|
| 1990 | Archie (Alan Emtage, McGill) | First internet search tool; indexed FTP file names |
| 1991 | World Wide Web goes public | Created the content landscape search engines would need to index |
| 1994 | WebCrawler (Brian Pinkerton, U. of Washington) | First full-text web search engine |
| 1995 | AltaVista (Digital Equipment Corporation) | First to index tens of millions of pages at speed |
| 1995 | Yahoo Directory | Human-curated category hierarchy; dominant through late 1990s |
| 1998 | Google (Brin and Page, Stanford) | PageRank algorithm; link-based quality scoring |
| 2008 | DuckDuckGo (Gabriel Weinberg) | Privacy-first search; no user tracking |
| 2009 | Bing (Microsoft) | Microsoft's rebrand of Live Search |
| 2023 | AI-augmented search | ChatGPT Search, Perplexity, Google AI Overviews |
WebCrawler, AltaVista, and the First Web Search Era (1994–1997)
When the web exploded in the early 1990s, Archie's FTP-file-name approach was insufficient. Web pages contained full documents — articles, product descriptions, research papers — and users needed to search within text, not just file names.
Brian Pinkerton at the University of Washington launched WebCrawler in April 1994. It was the first search engine to implement full-text indexing of web pages, crawling over 4,000 sites at launch and recording every word on every page. It served its one-millionth query by November 14, 1994.
Source: History of Information — Brian Pinkerton Develops the WebCrawler
In December 1995, Digital Equipment Corporation launched AltaVista. It indexed tens of millions of pages and could handle millions of queries daily — revolutionary at a time when most engines crashed under heavy load. AltaVista was the first engine to feel fast at scale, and it dominated professional research use through the late 1990s.
Yahoo, which launched its directory in 1994, took a different approach entirely. Rather than automated crawling, Yahoo hired humans to categorize websites into a browsable hierarchy. This worked when the web contained thousands of sites. It became unsustainable when it contained millions.
BackRub, PageRank, and the Birth of Google (1996–1998)
In 1996, Larry Page and Sergey Brin were PhD students at Stanford University. Page began a research project — initially called BackRub — with a specific goal: rank web pages by analyzing their backlink structure rather than just their text content.
The insight was that a link from one page to another is an implicit endorsement, like a citation in an academic paper. A page that many other pages link to is probably more authoritative than one that few pages link to. Better still, a link from an authoritative page should count more than a link from an obscure one. Page and Brin formalized this logic into PageRank.
Source: NSF — On the Origins of Google
The first version of Google ran on Stanford's network in August 1996, consuming nearly half the university's total bandwidth. Page and Brin published their foundational paper, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," in 1998. They incorporated Google Inc. on September 7, 1998, in a friend's garage in Menlo Park, California.
Source: Engineering and Technology History Wiki — PageRank and the Birth of Google, 1996–1998
The quality difference was immediately apparent to users. Where AltaVista and other engines returned pages that merely contained a query's keywords — regardless of whether those pages were useful — Google returned pages that both contained the keywords and had been judged authoritative by the broader web's link structure. It was the difference between a keyword search and something resembling a quality filter.
The single most counterintuitive fact about search engine history: PageRank was not about the page being searched — it was about the pages linking to it. Google's genius was that it outsourced relevance judgments to millions of website authors who had never heard of Stanford.
The Yahoo Directory Model and Why It Failed
Yahoo's human-curated directory dominated through the late 1990s not because it was more accurate than crawling engines, but because it was more trusted. Editors had vetted every listing, which meant fewer spam results. In an era when "link farms" and keyword stuffing were already polluting crawler-based results, a human editor's judgement carried weight.
The model failed on scale. By 2001, the web contained an estimated one billion indexable pages. No editorial team could keep pace. Yahoo responded by licensing Google's search technology for its results — an arrangement that lasted until 2004, by which point Google had surpassed Yahoo in market share and Yahoo began developing its own crawler-based index again.
Source: A Brief History of Search Before Google — Sebo Marketing
How PageRank Works
The PageRank algorithm treats every web page as a node in a graph. Every hyperlink from one page to another is a directed edge. The algorithm computes each page's rank as a weighted sum of the ranks of the pages linking to it, divided by the number of outbound links on each linking page.
Imagine a web with three pages. Page A links to pages B and C. Page B links only to C. Page C links to A. The algorithm runs iteratively: it starts with each page at an equal rank, then redistributes ranks according to the link structure, repeating until the values stop changing significantly. Pages that receive links from many high-rank pages accumulate high rank themselves.
The metaphor that Page and Brin used was a "random surfer" — imagine someone clicking links at random, infinitely. The probability that this surfer ends up on any given page, in the long run, is that page's PageRank. A page that many surfers would naturally land on (because it is linked from many others) is probably important.
Source: PageRank — Wikipedia
In practice, PageRank was one signal among hundreds that Google's algorithm weighed. By the 2010s, Google reported using over 200 ranking factors. But PageRank was the architectural innovation that made everything else possible: it gave the engine a principled way to distinguish quality from noise at internet scale.
How a Modern Search Query Is Processed
When a user types a query today, the engine does not perform a fresh crawl of the internet. It queries a pre-built index — a massive, constantly updated database of known pages and their content — and applies ranking algorithms to that index in real time. The sequence, simplified:
- Query parsing: The query is tokenized and expanded (synonyms, spelling corrections, intent classification).
- Candidate retrieval: The index returns a large set of pages that contain the query terms.
- Ranking: Hundreds of signals — PageRank, freshness, user location, query context, click-through rates — are applied to sort the candidates.
- Results assembly: Snippets, titles, and structured data (knowledge panels, featured snippets) are assembled for display.
- Delivery: The ranked page is returned, typically in under 300 milliseconds.
Source: Google Search — How Search Works
The People Who Built Search
Alan Emtage (born 1964, Barbados) came to McGill University for graduate study and built Archie while working as a systems administrator. He is rarely mentioned in mainstream technology history, a gap that researchers in the history of computing have noted as a significant omission. Archie's architectural logic — automated network scanning, central index, query interface — maps directly onto every search engine that followed.
Larry Page (born 1973, Michigan) joined Stanford's computer science PhD program in 1995. His supervisor was Terry Winograd, who encouraged Page's interest in the link structure of the web as a data source. Page's contribution was primarily algorithmic: the mathematical formulation of PageRank.
Sergey Brin (born 1973, Moscow) was already at Stanford when Page arrived. Brin's background was in data mining and pattern extraction. Together, they brought engineering and mathematical rigor that most other search projects lacked.
Brian Pinkerton built WebCrawler as a research project at the University of Washington. He demonstrated, a year before Google existed, that full-text indexing of the web was technically feasible.
Gabriel Weinberg (born 1979) founded DuckDuckGo on February 29, 2008, after selling his previous startup NamesDatabase. His original goal was to reduce SEO spam in results; privacy became the differentiating focus from 2010 onward. DuckDuckGo now serves several billion queries monthly without tracking individual users.
Patent Disputes and the Legal Architecture of Search
Search engines have largely avoided the destructive patent litigation that consumed other technology sectors, in part because the core algorithmic innovations — crawling, inverted indexing, link analysis — were developed in academic settings where patenting was not a priority.
Google did patent PageRank. US Patent 6,285,999, titled "Method for node ranking in a linked database," was filed in January 1998 and assigned to Stanford University, which licensed it to Google. The patent expired in 2018.
Source: PageRank — Wikipedia
The more significant legal battles have been about distribution and market access rather than algorithm patents. The European Commission fined Google €2.42 billion in 2017 for favoring its own shopping comparison results over those of competitors — a decision upheld on appeal in 2024. The US Department of Justice brought an antitrust suit against Google in 2020, arguing that its exclusive default-search agreements with Apple and other device manufacturers illegally foreclosed competition. A federal judge ruled in August 2024 that Google had maintained an illegal monopoly in the general search market.
Unintended Consequences: What Search Changed That Nobody Predicted
The most significant unintended consequence of search was the commoditization of attention. Before efficient search, information was scarce and distribution was the bottleneck. After efficient search, distribution became free and attention became the scarce resource. Every business that relied on consumers not being able to easily compare alternatives — travel agents, classified ad newspapers, brick-and-mortar encyclopedias — found itself structurally undermined.
The SEO industry emerged as an unintended side effect of PageRank itself. Once webmasters understood that inbound links determined rankings, they began manufacturing artificial links — through link farms, paid placements, and comment spam. Google has spent the better part of 25 years building algorithmic countermeasures (Panda in 2011, Penguin in 2012, ongoing quality updates) to suppress this behavior.
Source: Google Search Central — History of Google algorithm updates
Filter bubbles were identified as a social consequence by technologist Eli Pariser in his 2011 book of the same name. Personalized search — showing each user results calibrated to their past behavior — means different users querying the same term may receive substantially different results. The concern is that personalization reinforces existing beliefs rather than exposing users to contrary evidence.
The knowledge gap and digital divide widened in an unexpected direction. Effective search requires sophisticated query formulation. Studies have shown that users who know how to phrase queries precisely, evaluate source credibility, and iterate on failed searches extract dramatically more value from the same tool. Search engines democratized access to information but could not democratize the skill of using them.
Where Search Stands Today
Google processes approximately 8.5 billion queries per day as of 2024 and holds roughly 91% of the global search market. Bing holds approximately 3–4%. DuckDuckGo accounts for less than 1% by volume but several billion monthly searches by absolute count.
Source: Statcounter Global Stats — Search Engine Market Share
The most structurally significant development since PageRank is the emergence of AI-augmented search. Perplexity AI, launched in 2022, presents synthesized answers rather than a ranked list of links, with inline citations. Google deployed AI Overviews (previously called Search Generative Experience) broadly in 2024, placing AI-generated summaries above traditional link results for many queries. Microsoft integrated OpenAI's technology into Bing in 2023.
This shift carries a tension the industry has not resolved: AI-generated summaries provide faster answers but risk displacing the traffic that funds the publishers whose content the summaries are synthesized from. Publishers including major newspapers have filed copyright suits against AI search products. The legal and economic equilibrium of AI-augmented search was, as of 2025, unsettled.
Source: The New York Times v. Microsoft and OpenAI — Filed December 2023
Q: Who invented the first search engine? Alan Emtage, a graduate student at McGill University in Montreal, created Archie in 1990. It indexed file names on public FTP servers rather than web pages, since the World Wide Web did not yet exist publicly, but it established the core model of automated network indexing that all subsequent search engines followed.
Q: How does Google's PageRank algorithm work? PageRank treats hyperlinks as votes. Every page that links to another page is casting a vote for that page's importance. Links from pages that are themselves highly linked count more heavily. The algorithm runs iteratively until rankings stabilize, producing a score for every page that reflects its authority as judged by the collective linking behavior of the web.
Q: When was Google founded? Larry Page and Sergey Brin incorporated Google Inc. on September 7, 1998, in Menlo Park, California. The project began as a Stanford PhD research project called BackRub in 1996, published as a research paper in 1998.
Q: What is the difference between a search engine and a web directory? A search engine uses automated crawlers to index content across the web algorithmically. A web directory is a human-curated categorized list of websites. Yahoo's original directory (1994) was the dominant example of the latter. Directories were more accurate but could not scale; crawlers could scale but required algorithmic quality filters to remain useful.
Q: How does DuckDuckGo protect user privacy? DuckDuckGo does not store IP addresses or search history, does not build user profiles, and does not use personalized tracking. It returns the same results to all users for the same query, meaning results are not filtered based on past behavior. The service launched in 2008 and adopted privacy as its core differentiator from 2010 onward.
Q: What is AI-augmented search? AI-augmented search uses large language models to synthesize answers from indexed web content and present them directly in the search interface, rather than returning a list of links. Products including Perplexity AI, Google AI Overviews, and Microsoft Copilot in Bing represent this approach. The model reads relevant retrieved pages and generates a summary with citations, reducing the need for users to click through to source pages.
The Takeaway
The search engine's lineage runs from Alan Emtage's graduate-school automation project in 1990 to systems that now field billions of daily queries using machine learning models trained on the collective output of human writing. The core architecture — crawl, index, rank — has not changed; what has changed is the sophistication of each step. PageRank replaced keyword counting with link-based quality scoring. Neural ranking models later replaced keyword proximity with semantic understanding. AI generation is now overlaying synthesis on top of retrieval. Each generation solved the dominant failure mode of the previous one: garbage results, then inconsistent quality, then the friction of link navigation. The next unsolved problem is accuracy — AI systems that synthesize fluently but sometimes hallucinate — and the economics of a system where fewer users click through to the sources that made the synthesis possible.
Read next:
Weekly digest
Stay ahead of the curve
Get the best science, AI, and tech stories delivered to your inbox every week. No spam — unsubscribe any time.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. Articles are drafted against peer-reviewed literature and primary sources, fact-checked, and edited by a human before publication. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles
The Sewing Machine: Howe, Singer, and the First Patent War
The sewing machine's full history: Elias Howe's 1846 lockstitch patent, Isaac Singer's 1851 improvements, the first US patent pool in 1856, and today's global garment industry.
The Saxophone: How a Belgian Misfit Instrument Changed Music
The saxophone was patented in 1846 by Adolphe Sax, a man who nearly died 12 times before age 20. Here's how his misfit instrument conquered jazz.
The Invention of the World Wide Web — SciHub101
How Tim Berners-Lee's 1989 proposal at CERN became the Web, released to the public domain in 1993, and grew into the infrastructure of modern life.
The Modem: From SAGE Air Defense to the 56K Era
The modem started as military hardware in 1958 for SAGE air defense. Here's how it became the Hayes Smartmodem, the 56K standard, and the backbone of the internet age.
AI Search vs Traditional Search: How Discovery Is Changing
Traditional search ranks pages; AI search composes answers. This article explains the difference and what it means for publishers, users, and brands.
How to Use NotebookLM — Complete Guide (2026)
How to use Google NotebookLM in 2026: upload sources, ask grounded questions, generate study guides, quizzes, mind maps, and Audio Overviews step by step.