On almost every project I’ve worked on, everyone has been telling themselves the web services are RESTful services. Most of them don’t really follow RESTful principles, at least not strictly. But the basic ideas of REST are the driving force of how the APIs are written. After working with them for years, and reading into what a truly RESTful interface is, and what the justification is, I am ready to say:
REST has no reason to exist.
It’s perfectly situated in a middle ground no one asked for. It’s too high-level to give raw access to a database to make arbitrary queries using an actual query language like SQL, and it’s too low-level to directly drive application activities without substantial amounts of request forming and response stitching and processing.
It features the worst of both worlds, and I don’t know what the benefit is supposed to be.
Well, let’s look at what the justification for REST is. Before RESTful services, the API world was dominated by “remote procedure call” (RPC) protocols, like XML-RPC, and later Simple Object Access Protocol (SOAP). The older devs I’ve worked with told horror stories about how painful it was to write requests to those APIs, and they welcomed the “simplicity” of REST.
The decision to use REST is basically the decision to not use an RPC protocol. However, if we look at the original paper for REST, it doesn’t mention RPC protocols a single time. It focuses on things like statelessness, uniformity, and cacheability. But since choosing this architectural style for an API is just as much about not choosing the alternative, the discussion began to focus on a SOAP vs. REST comparison.
On the wiki page for SOAP, it briefly mentions one justification in the comparison:
SOAP is less “simple” than the name would suggest. The verbosity of the protocol, slow parsing speed of XML, and lack of a standardized interaction model led to the dominance of services using the HTTP protocol more directly. See, for example, REST.
This point tends to come up a lot. For example, this page repeats the idea that by using the capabilities of HTTP “directly”, REST can get away with defining less itself, which makes it “simpler”. So it seems the idea is that protocols like SOAP add unnecessary complexity to APIs, by reinventing capabilities already contained in the underlying communication protocols. The RPC protocols were written to be application-layer agnostic. As such, they couldn’t take advantage of the concepts already present in HTTP. Once it became clear that all these RPC calls are being delivered over HTTP anyway, they’ve become redundant. We can instead simply use what HTTP gives us out of the box to design APIs.
See, for example, the answers on this StackOverflow question:
SOAP is a protocol on top of HTTP, so it bypasses a lot of HTTP conventions to build new conventions in SOAP, and is in a number of ways redundant with HTTP. HTTP, however, is more than sufficient for retreiving, searching, writing, and deleting information via HTTP, and that’s a lot of what REST is. Because REST is built with HTTP instead of on top of it, it also means that software that wants to integrate with it (such as a web browser) does not need to understand SOAP to do so, just HTTP, which has to be the most widely understood and integrated-with protocol in use at this point.
Bottom line, REST removes many of the most time-consuming and contentious design and implementation decisions from your team’s workflow. It shifts your attention from implementing your service to designing it. And it does so without piling gobbledygook onto the HTTP protocol.
It’s curious that this “SOAP reinvents what HTTP already gives us” argument did not appear in the original REST paper. It’s a bad argument, which leads directly to the no-man’s land between raw database access and high-level application interfaces.
HTTP does already gives us what we need to make resource-based requests to a server; in short, CRUD. They claim that a combination of the HTTP request method (GET, PUSH, DELETE, etc.), path elements, and query parameters, already do for us what the RPC protocols stuff into overcomplicated request bodies.
The problem with this argument is that what HTTP provides and what RPC provides are not the same, either in implementation or in purpose. Those features of HTTP (method, path, and query parameters) expose a much different surface than what RPC calls expose. RPC is designed to invoke code (procedure) on another machine, while HTTP is designed to retrieve or manipulate resources on another machine.
Somewhere along the line, the idea arose that REST tells you how to map database queries and transactions to HTTP calls. For example, a SELECT of a single row by ID from a table maps to a GET request with the table name being a path element, and the ID being the next (and last) path element. An INSERT with column-cell value pairs maps to a POST request, again with the table in the path, and this time no further path elements, and the value pairs as body form data.
This certainly didn’t come from the original paper, which doesn’t mention “database” a single time. It mentions resources. The notion most likely arose because REST was shoehorned into a replacement for RPC, used to build APIs that almost always sits on top of a database. If you’re supposed to “build your API server RESTfully”, and that server is primarily acting as a shell with a database at its kernel, then a reading of “REST principles”, which drives your API toward dumb data access (rather than execution of arbitrary code), will inevitably become a “HTTP -> SQL” dictionary.
The mapping covers basic CRUD operations on a database, but it’s not a full query language. Once you start adding WHERE clauses, simple ones may map to query parameters, but there’s no canonical way to do the mapping, and there’s no way to do more sophisticated stuff like subqueries. There’s not even a canonical way to select specific columns. Then there’s joining. Since neither selecting specific columns nor joining map to any HTTP concept, you’re stuck with an ORM style interface into the database, where you basically fetch entire rows and all their related rows all at once, no matter how much of that data you actually need.
The original paper specifically called out this limitation:
By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application’s needs. The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction.
So, basically, this manifestation of “REST as querying through HTTP” gives us a lot less than what query languages already gave us, and nothing more. I’ll bet if you ask people why they picked REST over just opening SQL connections from the clients, they’ll say something about security, i.e. by supplying an indirect and limited interface to the database, REST allows you to build a sort of firewall in the server code on the way to actually turning the requests into queries. Well, you’re supposed to be able to solve that with user permissions. Either way, it’s nothing about the interface actually being nicer to work with or more powerful.
It’s only barely more abstract. You could probably make a fair argument it’s really not more abstract. REST is just a stunted query language. We can say this is a straw man, since the paper never said to do this. But unless the very suggestion to make application servers RESTful in general is a straw man (making the vast majority of “RESTful” services a misapplication of REST), I don’t see how it could have ended up any differently.
Claiming that this serves as a replacement for RPC fundamentally misunderstands what RPC protocols do. It’s in the name: remote procedure call. It’s a protocol to invoke a function on another machine. The protocol is there to provide a calling convention that works over network wires, just like a C compiler defines a calling convention for functions that works within a single process on a single machine. It defines how to select the function (by name), how to send the parameters, and how to receive the return value.
How much does HTTP help with this? Well, I guess you can put the class name, or function name (or both) into the URL path. But there’s no one way to do this that jumps out to me as obviously correct. The HTTP methods aren’t of much use (functions don’t, generally, correspond to the short list of HTTP methods), and query parameters are quite inappropriate for function parameters, which can be arbitrarily complex objects. Any attempt to take what SOAP does and move “redundant” pieces into HTTP mechanisms isn’t going to accomplish much.
We can, of course, send RPC calls over HTTP. The bulk, if not entirety, of the calling convention goes into the request body. By limiting HTTP to RESTful calls, we’re foregoing the advantage of the very difference between an API and direct querying: that it triggers arbitrary code on the server, not just the very narrowly defined type of work that a database can do. We can raise the abstraction layer and simply invoke methods on models that closely represent application concerns, and execute substantial amounts of business logic on top of the necessary database queries. To demand that APIs be “RESTful” is to demand that they remain mere mappings to dumb data accesses, the only real difference being that we’re robbed of a rich query language.
What you get is clients forced to do all the business logic themselves, and make inefficient coarse fetches of entire rows or hierarchies of data, or even worse a series of fetches, each incurring an expensive round trip to the server, to stitch together a JOIN on their side. When people start noticing this, they’ll start breaking the REST rules. They’ll design an API a client really wants, which is to handle all that business logic on the server, and you end up with a /api/fetchStuffForThisUseCase endpoint that has no correspondence whatsoever to a database query (the path isn’t the name of any table, and the response might be a unique conglomerate and/or mapping of database entities into a different structure).
That’s way better… and it’s exactly what this “REST as HTTP -> SQL” notion tries to forbid you from doing.
The middle of the road is, as usual, the worst place to be. If the client needs to do its own queries, just give it database access, and don’t treat REST as a firewall. It’s very much like the fallacy of treating network address translation as being a firewall. Even if it can serve that role, it’s not designed for it, and there are much better methods designed specifically for security. Handle security with user permissions. If you’re really concerned with people on mobile devices MITM attacking your traffic and seeing your SQL queries, have your clients send those queries over an end-to-end encrypted request.
If you’re a server/database admin and you’re nervous about your client developers writing insane queries, set up a code review process to approve any queries that go into client code.
If the client doesn’t need to do the business logic itself, and especially if you have multiple clients that all need the same business logic, then implement it all on the server and use an RPC protocol to let your clients invoke virtual remote objects. You’re usually not supposed to hand-write RPC calls anyways. That’s almost as a bad as handwriting assembly with a C compiler spec’s calling convention chapter open in your lap. That can all be automated, because the point of RPC protocols is that they map directly to function or method calls in a programming language. You shouldn’t have to write the low-level stuff yourself.
What, then, is the purpose of the various features of HTTP? Well, again it’s in the name: hypertext transfer protocol. HTTP was designed to enable the very first extremely rudimentary websites on the very first extremely rudimentary internet to be built and delivered. It’s designed to let you stick HTML files somewhere on a server, in such a way they can be read back from the same location where you stuck them, to update or delete them later, and to embed links among them in the HTML.
The only reason we’re still using HTTP for APIs is the same reason we’re still using HTML for websites: because that’s the legacy of the internet, and wholesale swapping to new technology is hard. Both of them completely outdated and mostly just get in our way now. Most internet traffic isn’t HTML pages anymore, it’s APIs, and it’s sitting on top of a communication protocol built for the narrow purpose of sticking HTML (to be later downloaded directly, like a file) onto servers. Even TCP is mostly just getting in our way, which is why it’s being replaced by things like QUIC. There’s really no reason to not run APIs directly on the transport protocol.
Rather than RPC, it’s really HTTP that’s redundant in this world.
Even for the traffic that is still HTML, it’s mostly there to bootstrap Javascript, and act as the final delivery format for views. Having HTML blend the concepts of composing view components and decorating text with styling attributes in, I believe, outdated and redundant in the same way.
To argue that APIs need to use such a protocol, to the point of being restricted to that protocol’s capabilities, makes no sense.
The original REST paper never talked about trying to map HTTP calls to database queries or transactions. Instead it focused on resources, which more closely correspond to (but don’t necessarily have to be) the filesystem on the server (hence why they are identified with paths in the URL). It doesn’t even really talk about using query parameters. The word “query” appears a single time, and not in the context of using them to do searches.
The driving idea of REST is more about not needing to do searches at all. Searches are ways to discover resources. But the “RESTful” way to discover resources is to first retrieve a resource you already know about, and that tells you, with a list of hyperlinks, what other resources exist and where to find them. If we were strict, a client using a RESTful API would have to crawl it, following hyperlinks to build up the data needed to drive a certain activity.
The central justifications of REST (statelessness, caching and uniformity) aren’t really ever brought up much in API design discussions… well, caching is, and I’ll get to that. RPC protocols can be as stateless (or stateful) as you want, so REST certainly isn’t required to achieve statelessness. Nor does sticking to a REST-style interface guarantee statelessness. Uniformity isn’t usually a design requirement, and since it comes at the cost of inefficient whole-row (or more) responses, it usually just causes problems.
That leaves caching, the only really valid reason I can see to make an API RESTful. However, the way REST achieves caching is basically an example of its “uniformity”: it sticks to standard HTTP mechanisms, which you get “for free” on almost any platform that implements HTTP, but it comes at the cost of being very restricted. For it to work, you have to express your query as an HTTP GET, with the specific query details encoded as query parameters. As I’ve mentioned, there’s not really a good way to handle complex queries like this.
Beside, what does HTTP caching do? It tells the client to reuse the previous response for a certain time range, and then send a request with an ETag to let the server respond with a 304 and save bandwidth by not resending an identical body. The first part, the cache age, can be easily done an any system. All you need is a client-side cache with a configurable age. The second part… well what does the server do when it gets a request with an ETag? Unless the query is a trivial file or row lookup, it has to generate the response and then compute the ETag and compare it. For example, any kind of search or JOIN is going to require the server to really hit the database to prove whether the response will change.
So, what are you really saving by doing this? Some bandwidth in the response. In most cases, that’s not the bottleneck. If we’re talking about a huge search that returns thousands (or more) of entities, and your customers are mostly using your app on slow cell networks in remote locations, then… sure, saving that response bandwidth is a big deal. But the more typical use case is that response bodies are small, bandwidth is plentiful, but the cloud resources needed to compute the response and prove it’s unchanged are what’s scarce.
You’ll probably do a much better job optimizing the system in this way by making sure you’re only requesting exactly what you need… the very capability that you lose with REST. This even helps with the response body size, which is going to be way bigger if you’re returning entire rows when all you need is a couple column values. Either that, or the opposite, where you basically dump entire blobs of the database onto the client so that it can do its querying locally, and just periodically asks for diffs (this also enables offline mode).
Again, the caching that REST gives us is a kind of no-man’s land middle ground that is suboptimal in all respects. It is, again, appropriate only for the narrow use case of hypertext driven links to resource files in a folder structure on a server that are downloaded “straight” (they aren’t generated or modified as they’re being returned).
The next time I have the authority to design an API, there’s either not going to be one (I’ll grant direct database access to the clients), or it will be a direct implementation of high-level abstract methods that can be mapped directly into an application’s views, and I’ll pick a web framework that automates the RPC aspect to let me build a class on one machine and call its methods from another machine. Either way, I’ll essentially avoid “designing” an API. I’ll either bypass the need and just use a query language, or I’ll write classes in an OOP language, decide where to slice them to run on separate machines, and let a framework write the requisite glue.
If I’m really bold I might try to sell running it all directly on top of TCP or QUIC.