This is part 2 of a 4-part series called “HTTP considered harmful”:

1. Motivation and scope
2. Overview of networks
3. Streaming (work in progress)
4. A streaming framework for distributed applications (work in progress)

Overview of networks

The kinds of network faults we encounter with HTTP are baked into the fabric of the Internet. However, many similar problems have plagued various other layers of the networking stack. Let’s take a dive into the evolution of the Internet and see how they’ve been addressed.

The Human Connection

The word communication will be used here in a very broad sense to include all of the procedures by which one mind may affect another.

~Warren Weaver, Recent Contributions to The Mathematical Theory of Communication

A simple example of a connection-oriented interaction is meeting someone in person. You get in close proximity to the other person, exchange some gestures and words, and then leave. This interaction is synchronous and we have worked out protocols for retransmission of lost information (i.e. “could you repeat that please?”).

In contrast, sending letters is connectionless. Each letter carries some information, but it arrives with some delay, sometimes out of order, and sometimes letters get lost altogether. Piecing together the timeline and accounting for failures in delivery sometimes requires significant mental effort on either side and has been yielding both brilliant novels and failed relationships.

Both kinds of interactions are the underlying foundation of a more abstract “human connection”, where each party maintains some “state” in their minds consisting of information and feelings about the other person. You could say that a physical layer of air, electromagnetic waves, and paper supports a transport layer of a conversation, which in turn is the building block for yet another layer of a social network. When we talk about “connecting” with someone, it’s usually that last layer we’re referring to.

The Internet

Multiple computers can be connected together directly, typically with copper cables, optical fibers, and electromagnetic fields. The physical layer of computer networks is connection-oriented: machines get in the wireless range, or an Ethernet cable is plugged in. Factors such as noise or a pair of scissors yielded by a toddler make this layer inherently unreliable in terms of correctness and availability.

To account for the variable rate of information density, error detection, as well as the desire to multiplex between receivers (interesting aside: the original Ethernet allowed connecting more than 2 computers with a single cable!), information is sent in packets, much like letters are sent by humans, forming a network access layer. Those packets arrive in order and without errors but not necessarily reliably. Thus, this layer is connectionless and offers in-order but unreliable delivery at local scale.

Multiple networks can be connected together, forming an inter-network, i.e. the internet layer. (In OSI, this is confusingly referred to as the network layer, as if the other layers didn’t participate in forming a computer network.) The modern Internet owes its success to its military roots where one of the key objectives was reliability, so instead of continuing to use multidrop, token ring, or star topologies, the network layer is a more general graph. Machines following routing protocols probe the topology and communicate the findings across the network. The system is able to recover from multiple failures of individual links and nodes. It’s still connectionless, but trades in-order delivery for increased reliability. The internet layer is connectionless and offers out-of-order, unreliable delivery but is otherwise global and highly fault-tolerant.

The transport layer attempts to solve the above issues for limited spans of time. The Transport Control Protocol describes how we can create a connection-oriented channel facilitating in-order, reliable delivery. It solves those issues with elaborate techniques involving acknowledgements, retransmissions, sequencing, and flow control. The abstraction lasts only as long as the TCP connection: when the network or service become unavailable, the connection will eventually be lost. This also affects the reliability of the last message—if the connection is broken, the sender and the receiver may not necessarily agree whether the message has been successfully transmitted.

The Web

A great majory of the modern IT industry is built on HTTP. I belive that the key reason for this is not that it’s the most suitable protocol for its contemporary applications, but because it was immediately convenient when distributed applications were becoming commonplace.

HTTP is a mature, well-built protocol that serves a single purpose: to perform point-in-time reads and mutations of individual resources.

It was originally built around the following concepts:

• stateless (connectionless) operation
• resources, identifiable with URIs
• operations on those resources, including a selection of methods and status codes
• arbitrary request and response content
• headers with metadata
  • content types, adapted from Multipurpose Internet Mail Extensions (MIME)
  • caching (Expires, If-Modified-Since, and later Cache-Control headers)
  • arbitrary application-specific metadata that lives outside of the content

We later added support for cookies, TLS, and curiously, TCP connection reuse—but we didn’t expose any new connection-oriented primitives; this was done purely as a performance optimization.

Reinventing the wheel

Most of those concepts bring tons of utility to builders of applications. However, the stateless nature of the protocol poses one critical issue: if a request fails, it’s left to the application developer to figure out what to do. Did the failed request affect the state or not? “Requests should be idempotent”, says common wisdom. This covers GET, PUT, and DELETE. What about POST? “Generate a UUID and deduplicate server-side”. Easier said than done—isn’t the logic tier supposed to be stateless? Are we basically storing parts of the networking stack’s state together with application data?

Before you say it’s not that significant, ask yourself: would you rely on REST alone for replicating writes to a database? I think not. You’d rather deal with write-ahead logs (WAL), Change Data Capture (CDC), state machine replication, and message queues than simply proxying the REST request to two endpoints. Just thinking about the possible side effects gives me chills. Now, if those techniques are so successful for data-management systems, why shouldn’t we use them more broadly?

As you’ve been reading this post, you might have noticed that the terms “state” and “connection-oriented” appear together, and so do “stateless” and “connectionless”. It makes intuitive sense. It should follow, then, that perhaps the most suitable protocol for stateful applications should be connection-oriented.