The Messy Birth of the Address You Type Every Day
In early 1982, the ARPANET faced a crisis of identity. With 455 hosts connected and the network about to transition from its original NCP protocol to TCP/IP, the era of interconnected networks — an “Inter-net” — was dawning. On January 11th of that year, 22 computer scientists gathered in a meeting to tackle what might seem today like a mundane problem: how to address computer mail.
The attendees read like a who’s who of computing history — the future founder of Sun Microsystems, the creator of Zork, the architect of NTP, and the man who persuaded the government to fund Unix. Their challenge was that other networks were sprouting up, with names like “COMSAT,” “CHAOSNET,” “UCLNET,” and “INTELPOSTNET,” each maintained by universities and corporations willing to lease 56k lines and invest in PDP-11s to handle routing.
Under the original ARPANET design, the addressing problem was simple: centralize everything. A Network Information Center (NIC) kept a single master file, known as HOSTS.TXT, which enumerated every host on the network — much like the modern /etc/hosts file on Linux or OS X. Every update required the NIC to use FTP (a protocol dating back to 1971) to push changes to each and every host, placing an unsustainable burden on their infrastructure.
Despite what you might guess, the hot priority wasn’t web pages or video — the internet didn’t exist as we know it. It was email that drove the agenda. The group’s conclusion: establish a hierarchy with an external query system for each domain, allowing networks to resolve their own addresses. Their proposal extended the user@host identifier to [email protected], where “domain” could itself nest deeper hierarchies. Instead of a worldwide HOSTS file update, you could look up just the slice you needed.
Pragmatism, Not Vision
Wired today might expect those early architects to proceed with a prescient vision of the web, e-commerce, and the metaverse. Nothing could be further from the truth. The group deliberately selected the least disruptive solution, explicitly choosing “one causing least difficulty for existing systems.”
The earliest alternative proposal opted for a format like <user>.<host>@<domain>. That convention might have evolved into something like zack.cloudflare@com — had the precedent of ‘.’ puncturing usernames not already been established.
Before There Was DNS: UUCP and Bang Paths
Before the Internet was publicly available, computer communication relied on direct point-to-point dial-up connections. You would have your modem call another modem, and the two machines would transfer files. To turn this into a network of sorts, the UUCP tool was created in 1976 — it is very likely still installed on your OS X or Linux machine as uucp.
In this system, each computer kept a file listing the hosts it was aware of, their phone numbers, and a username and password on that host. You would craft a “path” from your current machine to your destination through hosts that each knew how to connect to the next:
sw-hosts!digital-lobby!zackThis address wasn't just for file transfers — it was also your email address. In this era before “mail servers,” if your computer was off, you weren't receiving email. While ARPANET (introduced in 1969) was restricted to top-tier universities, UUCP created a bootleg Internet for everyone else. It formed the basis for both Usenet and the BBS system.
A failed proposal for ARPANET itself involved separating domain components with the exclamation mark (!), so that connecting to the ISIA host would mean connecting to !ARPA!ISIA. This wasn't a crazy divergence — it was an attempt to maintain consistency with the UUCP-style exclamation-separated addresses that were already in use.
DNS and the Question of What “IN” Means
The DNS system we still use today was proposed in 1983. If you run a DNS query today with the dig tool, you'll see a response like this:
;; ANSWER SECTION:
google.com. 299 IN A 172.217.4.206This tells us that google.com is reachable at 172.217.4.206. The A indicates an address record mapping a domain to an IPv4 address. The 299 is the time to live, in seconds. But what does the IN mean?
IN stands for “Internet.” Like so much of this, the field dates back to an era with several competing computer networks. Other potential values were CH for CHAOSNET or HS for Hesiod, the name service of the Athena system. CHAOSNET is long dead, but a much-evolved version of Athena is still used by MIT students. The full list of DNS classes is on the IANA website — unsurprisingly, only one is in common use today.
Top-Level Domains and the Root
It is extremely unlikely that any other TLDs will be created.
— Jon Postel,
1994
Once domains were arranged hierarchically, a root had to be chosen. That root is traditionally signified with a single '.' — ending all your domain names with a '.' is semantically correct, and google.com. works in your browser.
The first top-level domain was .arpa, allowing users to address old ARPANET hostnames during the transition — hfnet would become hfnet.arpa. After the transition, server administrators had to choose among just five TLDs: .com, .gov, .org, .edu, or .mil.
The hierarchy works through a set of root DNS servers responsible for turning .com into the .com nameservers, who in turn answer for google.com. There are only 13 server clusters because that's all that fits in a single UDP packet — DNS historically operated over UDP, where responses could never exceed 512 bytes.
; This file holds the information on root name servers needed to
; initialize cache of Internet domain name servers
; (e.g. reference this file in the "cache . "
; configuration file of BIND domain name servers).
;
; This file is made available by InterNIC
; under anonymous FTP as
; file /domain/named.cache
; on server FTP.INTERNIC.NET
; -OR- RS.INTERNIC.NET
;
; last update: March 23, 2016
; related version of root zone: 2016032301
;
; formerly NS.INTERNIC.NET
;
. 3600000 NS A.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET. 3600000 A 198.41.0.4
A.ROOT-SERVERS.NET. 3600000 AAAA 2001:503:ba3e::2:30
;
; FORMERLY NS1.ISI.EDU
;
. 3600000 NS B.ROOT-SERVERS.NET.
B.ROOT-SERVERS.NET. 3600000 A 192.228.79.201
B.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:84::b
;
; FORMERLY C.PSI.NET
;
. 3600000 NS C.ROOT-SERVERS.NET.
C.ROOT-SERVERS.NET. 3600000 A 192.33.4.12
C.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:2::c
;
; FORMERLY TERP.UMD.EDU
;
. 3600000 NS D.ROOT-SERVERS.NET.
D.ROOT-SERVERS.NET. 3600000 A 199.7.91.13
D.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:2d::d
;
; FORMERLY NS.NASA.GOV
;
. 3600000 NS E.ROOT-SERVERS.NET.
E.ROOT-SERVERS.NET. 3600000 A 192.203.230.10
;
; FORMERLY NS.ISC.ORG
;
. 3600000 NS F.ROOT-SERVERS.NET.
F.ROOT-SERVERS.NET. 3600000 A 192.5.5.241
F.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:2f::f
;
; FORMERLY NS.NIC.DDN.MIL
;
. 3600000 NS G.ROOT-SERVERS.NET.
G.ROOT-SERVERS.NET. 3600000 A 192.112.36.4
;
; FORMERLY AOS.ARL.ARMY.MIL
;
. 3600000 NS H.ROOT-SERVERS.NET.
H.ROOT-SERVERS.NET. 3600000 A 198.97.190.53
H.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:1::53
;
; FORMERLY NIC.NORDU.NET
;
. 3600000 NS I.ROOT-SERVERS.NET.
I.ROOT-SERVERS.NET. 3600000 A 192.36.148.17
I.ROOT-SERVERS.NET. 3600000 AAAA 2001:7fe::53
;
; OPERATED BY VERISIGN, INC.
;
. 3600000 NS J.ROOT-SERVERS.NET.
J.ROOT-SERVERS.NET. 3600000 A 192.58.128.30
J.ROOT-SERVERS.NET. 3600000 AAAA 2001:503:c27::2:30
;
; OPERATED BY RIPE NCC
;
. 3600000 NS K.ROOT-SERVERS.NET.
K.ROOT-SERVERS.NET. 3600000 A 193.0.14.129
K.ROOT-SERVERS.NET. 3600000 AAAA 2001:7fd::1
;
; OPERATED BY ICANN
;
. 3600000 NS L.ROOT-SERVERS.NET.
L.ROOT-SERVERS.NET. 3600000 A 199.7.83.42
L.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:9f::42
;
; OPERATED BY WIDE
;
. 3600000 NS M.ROOT-SERVERS.NET.
M.ROOT-SERVERS.NET. 3600000 A 202.12.27.33
M.ROOT-SERVERS.NET. 3600000 AAAA 2001:dc3::35
; End of fileRoot DNS servers are guarded in safes inside locked cages, with a clock on the safe to prove the camera feed hasn't been looped. Given how slowly DNSSEC implementation has proceeded, an attack on those servers could redirect Internet traffic for a portion of users.
The TLD nameservers don't change often. In fact, 98% of requests the root servers receive are errors, mostly from broken and toy clients that don't cache. Several root operators had to spin up special servers just to return “go away” to people asking for reverse DNS lookups on local IP addresses.
Punycode: Encoding Unicode Into Domains
Punycode is the system used to encode Unicode into domain names. The problem is simple: how do you write 比薩.com when the entire system is built on ASCII?
Domains were locked to ASCII because the original documents governing them specify that encoding. Every piece of internet hardware from the last forty years assumes it. The web itself was never ASCII-only — it was originally conceived to speak ISO 8859-1, adding characters like ¼ and ä but no non-Latin scripts. That restriction on HTML was removed in 2007, the year Unicode became the most popular character set on the web. But domains stayed ASCII-only.
Punycode wasn't the first attempt at a solution. In 2000, several IETF members proposed UTF-5, encoding Unicode into five-bit chunks mapped onto A-V and 0-9. The site 日本語.com would become the cryptic M5E5M72COA9E.com. The method had real disadvantages: A-V and 0-9 being used in the output meant those characters also had to be encoded, producing very long domains — a serious problem when each segment is limited to 63 characters. A Myanmar-language domain would be limited to no more than 15 characters.
There was also the question of how clients would know the domain was encoded. One suggestion used an unused bit in the DNS response — the “last unused bit in the header,” which DNS folks were “very hesitant to give up.” Another suggested starting every such domain with ra--, as no domains matched that prefix at the time. The ultimate conclusion, reached in 2003, was Punycode, which uses delta compression to shorten encoded domains dramatically.
Delta compression works well because characters in a domain tend to cluster in the same general Unicode area. The three characters [1620, 1584, 1597] get sorted to [1584, 1597, 1620], then stored as the lowest value plus the deltas (13 and 23) — significantly less to transmit. Punycode efficiently encodes those integers into domain-compatible characters and prepends xn-- so consumers know it's encoded. The website 熱狗sales.com becomes xn--sales-r65lm0e.com.
The transformation could be transparent, but that introduces a security problem: many Unicode characters print identically to ASCII ones. Cyrillic “а” and Latin “a” look the same. Register Cyrillic аmazon.com (xn--mazon-3ve.com) and tricking users onto the wrong site gets easy. That's why your browser shows xn--vi8hiv.ws rather than a friendly Unicode rendering of ??.ws.
Protocol and Port
The first portion of a URL is the protocol. The most common is http, the document transfer protocol Tim Berners-Lee invented for the web. Some believed Gopher should win instead. Gopher was designed to send structured data resembling a file tree — requesting /Cars might return:
1Chevy Camaro /Archives/cars/cc gopher.cars.com 70
iThe Camaro is a classic fake (NULL) 0
iAmerican Muscle car fake (NULL) 0
1Ferrari 451 /Factbook/ferrari/451 gopher.ferrari.net 70identifying two cars, their metadata, and where to connect for more. The earliest popular protocol was FTP (1971), for listing and downloading files. Gopher extended that with metadata, enabling news feeds and simple databases, but lacking the freedom and simplicity of HTTP and HTML.
HTTP is simple, especially compared to FTP or the emerging HTTP/3. It's entirely text-based rather than binary, which Tim Berners-Lee correctly intuited would be easier for generations of programmers to develop and debug. HTTP makes few assumptions about content — despite accompanying HTML, it lets you specify any type via the MIME Content-Type.
A request:
GET /index.html HTTP/1.1
Host: www.example.comMight respond:
HTTP/1.1 200 OK
Date: Mon, 23 May 2005 22:38:34 GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: 138
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
ETag: "3f80f-1b6-3e1cb03b"
Accept-Ranges: bytes
Connection: close
<html>
<head>
<title>An Example Page</title>
</head>
<body>
Hello World, this is a very simple HTML document.
</body>
</html>IP gets small packets (around 1500 bytes) between computers. TCP handles larger blocks, sending them reliably via many IP packets. On top of that, HTTP or FTP specifies the format of the data sent via TCP — TCP/IP sends bytes, and the protocol says what they mean.
The default port numbers tell part of the story: Gopher is 70, HTTP is 80. The HTTP port was assigned at the request of Berners-Lee between 1990 and 1992. Ports predate the Internet — the original NCP protocol on ARPANET used 40-bit addresses, the last eight being an AEN (“Another Eight-bit Number”), used like ports today to separate messages for different processes. Users were asked to register socket numbers to avoid collisions; when TCP/IP expanded ports to 16 bits, registration continued. The logic of allowing manual port specification — local development, hosting multiple services — was also the basis for the www. prefix. Giving users a specific machine name (dx3.cern.ch) causes trouble when that machine must be replaced; a common subdomain (www.cern.ch) can point wherever needed.
The Bit In-Between
URL syntax places a double slash (//) between protocol and the rest:
http://cloudflare.comThat double slash was inherited from the Apollo computer system, one of the first networked workstations, which faced the same problem of separating a path from the machine hosting it. The Apollo team created a special path format:
//computername/file/path/as/usualBerners-Lee copied it, and now regrets it, wishing the domain (example.com) were the first portion:
http:com/example/foo/bar/bazIn 1992, Berners-Lee created three things that gave birth to what we consider the Internet: HTTP, HTML, and the URL. His goal was Hypertext — documents linking to one another. Documents were hosted in many formats and accessed via multiple protocols, so he needed a consistent way to reference a file encoding its protocol, its host, and its location on that host. At the original presentation in March 1992, he called it a “Universal Document Identifier” (UDI). Many formats were considered:
protocol: aftp host: xxx.yyy.edu path: /pub/doc/README
PR=aftp; H=xx.yy.edu; PA=/pub/doc/README;
PR:aftp/xx.yy.edu/pub/doc/README
/aftp/xx.yy.edu/pub/doc/READMEThe same document explains why spaces must be encoded in URLs (%20):
The use of white space characters has been avoided in UDIs: spaces are not legal characters. This was done because of the frequent introduction of extraneous white space when lines are wrapped by systems such as mail, or sheer necessity of narrow column width, and because of the inter-conversion of various forms of white space which occurs during character code conversion and the transfer of text between applications.
The URL was fundamentally an abbreviated way of referring to the scheme, domain, port, credentials, and path that previously had to be understood contextually per system. It was first officially defined in an RFC published in 1994.
scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]This system made it possible to refer to different systems from within Hypertext. But now that virtually all content is over HTTP, it may be less necessary. As early as 1996, browsers inserted http:// and www. automatically.
Breaking, and Not Fixing, URLs
I do not think the question is whether people can learn the meaning of the URL, I just find it it morally abhorrent to force grandma or grandpa to understand what, in the end, are UNIX file system conventions.
— Israel del Rio
1996
The slash-separated path should be familiar to anyone who has used a computer in fifty years. The hierarchical filesystem was introduced by MULTICS — its creator attributes it to a two-hour conversation with Albert Einstein in 1952. MULTICS used the greater-than symbol (>) to separate components:
>usr>bin>local>awkThat was logical, but Unix chose > for redirection and gave the forward slash (/) to paths.
The consequence of fragile URLs is measurable: One half of the URLs referenced by US Supreme Court opinions no longer work. An academic paper written in 2001 has better than even odds of having broken links by 2011.
There was a fervent belief in 1993 that the URL would die in favor of the “URN” — a permanent reference that, unlike a URL, never breaks. Berners-Lee described the “urgent need” for them as early as 1991. The simplest approach — a cryptographic hash of content, e.g. urn:791f0de3cfffc6ec7a0aacda2b147839 — didn't meet the web community's needs: it wasn't clear who to ask to resolve such a hash into content, and it couldn't account for format changes (compressed vs. uncompressed) that still represent the same document.
The URN format was finalized in 1997 and essentially never used. Each URN has two parts: an authority who resolves a given type, and the specific document ID. For example, urn:isbn:0131103628 identifies a book as a permanent link that your local isbn resolver can hopefully turn into URLs. Given search engines' power, the best URN format today might simply let files point to their former URLs for search engines to index:
<!-- On http://zack.is/history -->
<link rel="past-url" href="http://zackbloom.com/history.html">
<link rel="past-url" href="http://zack.is/history.html">Query Parameters and Their Accidental History
The "application/x-www-form-urlencoded" format is in many ways an aberrant monstrosity, the result of many years of implementation accidents and compromises leading to a set of requirements necessary for interoperability, but in no way representing good design practices.
Query parameters follow the path and encode options like ?name=zack&state=mi. It may seem odd that queries use the ampersand (&), the same character used in HTML entities. HTML authors regularly have to encode ampersands inside URLs — http://host/?x=1&y=2 becoming http://host/?x=1&y=2. That confusion has always existed. Cookies follow a similar format with a semicolon (x=1;y=2), which doesn't conflict with HTML encoding at all — the W3C encouraged implementers to support both separators as early as 1995.
Originally, queries were strictly for searching “indexes.” The Web was funded to aid collaboration among high-energy physicists. Berners-Lee created the <ISINDEX> tag to support searching. When <ISINDEX> appeared on a page, the browser showed a search field and sent the query to the server. That query was formatted as keywords separated by plus signs (+):
http://cernvm/FIND/?sgml+cmsThe tag was quickly abused for everything from search to calculating square roots. Proposals for a general-purpose <input> tag emerged, using plus signs to separate what looks like a modern GET query:
http://somehost.somewhere/some/path?x=xxxx+y=yyyy+z=zzzzSome wanted a way to declare link targets searchable:
<a HREF="wais://quake.think.com/INFO" INDEX=1>search</a>Berners-Lee favored strongly-typed queries:
<ISINDEX TYPE="iana:/www/classes/query/personalinfo">The real work on <INPUT> began in January 1993. It was decided <SELECT> inputs needed a separate, richer structure:
<select name=FIELDNAME type=CHOICETYPE [value=VALUE] [help=HELPUDI]>
<choice>item 1
<choice>item 2
<choice>item 3
</select>Reusing <li> rather than a new <option> element was absolutely considered. One alternative resembled what Angular would eventually do with variable substitution:
<ENTRYBLANK TYPE=int LENGTH=length DEFAULT=default VAR=lval>Prompt</ENTRYBLANK>
<QUESTION TYPE=float DEFAULT=default VAR=lval>Prompt</QUESTION>
<CHOICE DEFAULT=default VAR=lval>
<ALTERNATIVE VAL=value1>Prompt1 ...
<ALTERNATIVE VAL=valuen>Promptn
</CHOICE>Inputs were checked against the type attribute, with VAR values available for string substitution in URLs:
http://cloudflare.com/apps/$appIdAnother proposal used @ rather than = as the separator:
name@value+name@(value&value)It was Marc Andreessen who suggested the current format, based on what he had already built into Mosaic:
name=value&name=value&name=valueJust two months later, Mosaic added method=POST support, and modern HTML forms were born. Andreessen's company Netscape also created the cookie format, a decision whose consequences still surface today.
The Fragment
Another part of the syntax is the data rather than the location: the URL that preceded that first RFC. Some parts are not sent to a server — like the component that follows. That ambiguity — does the identifier always refer to the thing at the identifier's location, regardless of its semantics — is referred to as the httpRange-14 problem.
