Introduction
So, it’s just a normal day at work, you’re writing some Swift code, and the need arises to model an abstract type: there will be multiple implementations, and you want users to be able to use those implementations without knowing which implementation they are using.
So, of course, you reach for the tool that Swift provides for modeling abstract types: a protocol. If you were writing Kotlin, C#, Java or TypeScript, the comparable tool would be an interface.
As an example, let’s say you’re writing the interface to your backend, which exposes a few endpoints for fetching data. You are going to want to swap out the real or “live” backend for a fake backend during testing, so you’re at least going to need two variations. We’ll therefore write it as a protocol:
protocol NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing]
func fetchApplicants(for listing: JobListing) async throws -> [Applicant]
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing]
}We’ll write each flavor as a conforming concrete type. The “live” one:
struct NetworkInterfaceLive: NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [JobListing].self, from: data ?? .init())
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings/\(listing.id)/applicants")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [Applicant].self, from: data ?? .init())
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("applicants/\(applicant.id)/listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type:[JobListing].self, from: data ?? .init())
}
init(
urlSession: URLSession = .shared
host: URL
) {
self.urlSession = urlSession
self.url = url
}
let urlSession: URLSession
let host: URL
}And then a fake one that just returns canned successful responses:
struct NetworkInterfaceHappyPathFake: NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing] {
StubData.jobListings
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
StubData.applicants[listing.id]!
}
func fetchJobListings(for applicant: Applicant) async throws {
StubData.applicantListings[applicant.id]!
}
}Simple enough, right?
But wait! You start hearing that this isn’t the only way to implement this design. You can, in fact, build abstract types in Swift without using protocols at all! How could this be possible!?
We use what are called “protocol witnesses”. We replace the protocol with a struct, replace function declarations with closure members, and replace the concrete implementations with factories that assign the closures to the implementations provided by that concrete type. First the abstract type:
struct NetworkInterface {
var fetchJobListings: () async throws -> [JobListing]
var fetchApplicants: (_ listing: JobListing) async throws -> [Applicant]
var fetchJobListingsForApplicant: (_ applicant: Applicant) async throws -> [JobListing]
}Then the live implementation:
extension NetworkInterface {
static func live(
urlSession: URLSession = .shared
host: URL
) -> Self {
func fetchJobListings() async throws -> [JobListing] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [JobListing].self, from: data ?? .init())
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings/\(listing.id)/applicants")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [Applicant].self, from: data ?? .init())
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("applicants/\(applicant.id)/listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type:[JobListing].self, from: data ?? .init())
}
return .init(
fetchJobListings: fetchJobListings,
fetchApplicants: fetchApplicants,
fetchJobListingsForApplicant: fetchCandidates(for:)
)
}(If you haven’t seen this before, yes you can write “local functions” inside other functions, and they’re automatically closures with default implicit capture, which is how they have access to urlSession and host. If you need to explicitly capture anything you have to switch to writing them as closures, as in let fetchJobListings: () async throws -> [JobListing] = { [...] in ... }).
And the happy path fake:
extension NetworkInterface {
var happyPathFake: Self {
.init(
fetchJobListings: { StubData.jobListings },
fetchApplicants: { listing in StubData.applicants[listing.id]! },
fetchJobListingsForApplicant: { applicant in StubData.applicantListings[applicant.id]! }
)
}
}Take a moment to study those to understand that they are, in fact, equivalent. The only difference is that when you want to instantiate a live instance, instead of writing let network: NetworkInterfaceProtocol = NetworkInterfaceLive(url: prodUrl), we write let network: NetworkInterface = .live(url: prodUrl). And similarly when instantiating the happy path fake. Other than that (and the small caveat that you can’t have named function arguments), it’s equivalent.
And, in fact it is more powerful: I can, for example, mix and match the individual functions from different implementations (i.e. make an instance whose fetchJobListings is the happyPathFake one, but the other two functions are live). I can even change the implementation of one function after creating an instance, substituting new implementations inline in other code, like test code, and those functions will close over the context, so I can capture stuff like state defined on my test case file, to make a test double that, say, returns job listings that depends on how I’ve configured the test.
This technique is discussed by PointFree here. I don’t subscribe to their content, so I only see the free portion of this article, and similarly for other articles they link to from there. From this limited vantage point, it appears to me their position is that “protocol oriented programming”, a popular paradigm in the Swift community, is not a panacea, that there are situations where it is not the right tool, and the protocol witness approach can be a better and more powerful alternative. They specifically say that some things are harder to do with protocols than with protocol witnesses, and some things are not possible to do with protocols than can be done with protocol witnesses.
Now, the fundamental starting point of this discussion seems to be that these two options (the protocol with implementing structs, and the struct with settable closure members) are different implementations of the same design. That is, we’re attempting to model the same system, with the same requirements and behavior. And we’re not even debating two different conceptual models of the system. We agree there is an abstract type of network interface (and this, not somewhere else, is where the abstraction lies… this is important) and it has multiple concrete implementations. Rather, we are only comparing two mechanisms of bringing this conceptual model into reality.
As an example of what I’m talking about, here is two different models. This:
struct JobListing {
let id: String
let role: String
let applicants: [Applicant]
let candidates: [Candidate]
...
}
struct Applicant {
let id: String
let person: Person
...
}
struct Candidate {
let id: String
let person: Person
...
}
struct Person {
let id: String
let firstName: String
let lastName: String
...
}vs. this:
struct JobListing {
let id: String
let role: String
let applicants: [Applicant]
let candidates: [Candidate]
...
}
struct Applicant {
let id: String
let isCandidate: Bool
let firstName: String
let lastName: String
...
}These are both attempts to conceptually model the same requirements, but they are different models (i.e. the former allows applicants and candidates to have different attributes, the latter does not). Contrast with the following, which is two implementations, with competing language features, of the same model. This:
struct JobListing {
let id: String
let role: String
var applicants: [Applicant] { ... }
var candidates: [Candidate] { ... }
...
}vs. this:
struct JobListing {
let id: String
let role: String
func getApplicants() -> [Applicant] { ... }
func getCandidates() -> [Candidate] { ... }
...
}These both model the domain in the exact same way. The difference is in whether we use the language feature of computed read-only variables vs. functions. That’s purely mechanical.
I believe PointFree are framing this choice of modeling our problem as a protocol with implementing structs, vs. a struct with closure members, as the latter: the same conceptual model but with different tools to construct the model. This is why the discussion focuses on what the language can and can’t do, not the exact nature of the requirements or system, since it is fundamentally about choosing language constructs, not correctly modeling things.
I think this framing is mistaken.
I rather consider the choice over these two implementations to be a discussion over two different conceptual models. This will become clear as I analyze the differences, and I will point out exactly where the differences have implications for the conceptual model of the system we’re building, which goes beyond mere selection of language mechanisms. In the process, we need to ask and satisfactorily answer:
- What is the nature of this “extra power”, and what are the implications of being able to wield it?
- What exactly are the things you “can’t” do (or can’t do as easily) with protocols, and why is it you can’t do them? Is it a language defect, something a future language feature might fix?
- Does the protocol witness design actually eliminate protocols?
- Why is this struct called a “protocol witness”? (that’s very significant and reveals a lot about what’s going on)
About That Extra Power
One, if not the, fundamental arguments to justify protocol witnesses is that it is “more powerful” than protocols. Now, based on the part of the article I can see, I have a suspicion this is at least partially based on mistaken beliefs about what you can do with protocols. But there’s at least some truth to this, simply considering the examples they show (which I mentioned above in my own version of this): you can reconfigure individual pieces of functionality in the object, and even compose functionality together in different combinations, where in the protocol -> implementer approach you can’t (well, sort of… we’ll get there). With protocols, the behaviors always come as a set and can’t be mixed and matched, and they can’t be changed later.
This is, indeed, much more “powerful” in the sense you are less restricted in what you can do with these objects.
And that’s bad. Very bad.
“More powerful”, which we can also phrase as “more flexible”, always balances with “less safe”. Being able to do more in code can very well mean, and usually does mean, you can do more wrong things.
After all, why are these restrictions in place to begin with? Is it a language defect? Have we just not figured out how to improve protocols to make them this flexible? Can you imagine a Swift update in which protocols enable you to do this swapping out of individual requirements after initializing a concrete implementing type?
No, it’s the opposite. Unfettered flexibility is the “stone age” of software programming. What’s the most flexible code, “powerful” in the sense of “you can implement more behaviors”, you could possibly write?
Assembly.
There, you can do lots of things higher level languages (where even BASIC or C is considered “high level”) won’t let you do. Some examples are: perform addition on two values in memory that are execution instructions and store the result in another place that is later treated as an instruction, not decrement the stack before jumping to the return address, decrement the stack by an amount that’s not equal to the amount it was incremented originally, and have a for loop increment the index in the middle of an iteration instead of at the end.
These are all things you can’t do in higher level languages. And thank God for that! They’re all bugs. That’s the point of higher level languages being more restricted: they stop you from writing incorrect code, which is the vast majority of code.
See, the main challenge of software engineering is not the inability to write code easily. It’s the ability to easily write incorrect code. The precise goal of raising the abstraction level with higher level languages is to restrict you from writing code that’s wrong. The restrictions are not accidents or unsolved problems. They are intentional guardrails installed to stop you from flying off the side of the mountain.
This is the point of strongly typed languages. In JavaScript you can add an integer to a string. You can’t do this in C. Is C “missing” a capability? Is JavaScript “more powerful” than C? I guess in this sense yes. But if adding an integer to a string is a programmer error, what is that extra power except a liability?
This is what the “footgun” analogy is about. Some footguns are very very powerful. They’re like fully automatic heat-seeking (or maybe foot-seeking?) assault footguns. And that’s even worse than a single action revolver footgun because, after all, my goal is to not shoot myself in the foot.
While a looser type system or lower level language is “more powerful” at runtime, these languages are indeed less powerful at design time. Assembly is the most powerful in terms of what behavior you can create in the running program, but it is the least powerful in terms of what you can express, at design time, is definitely wrong and should be reported at design time as an error.
There’s no way to express in JavaScript that adding a network client to a list of usernames is nonsense. There’s no way to express that calling .length on a DOM node is nonsense. You can express this in Swift, in a way the compiler understands so that it prevents you from doing nonsensical things. This is more powerful.
So more power at runtime is less power at design time, and vice versa. Computer code is just a sequence of instructions, and the vast vast majority of such sequences are useless or worse. Our goal is not to generate lots of those sequences quickly, it’s to sift through that incomprehensibly giant space of instruction sequences and filter out the garbage ones. And because of the halting problem, running the program to exercise every pathway is not a viable way of doing so.
Is this relevant to this discussion about protocol witnesses? Absolutely. All this is doing is turning Swift into JavaScript. In JavaScript, there are no static classes with fixed functions. “Methods” are just members of an object that can be called. “Objects” are just string -> value dictionaries that can hold any values at any “key” (member name).
The only difference between this and the structs PointFree is showing us here is that JavaScript objects are even more powerful still: you can add or remove any functions, at any name, you want during runtime. How can we make Swift do this?
@dynamicCallable
struct AnyClosure {
// This will be much better with parameter packs
init<R>(_ body: () -> R) {
_call = { _ in body() }
}
init<T, R>(_ body: (T) -> R) {
_call = { args in body(args[0] as! T) }
}
init<T1, T2, R>(_ body: (T1, T2) -> R) {
_call = { args in body(args[0] as! T1, args[1] as! T2) }
}
init<T1, T2, T3, R>(_ body: (T1, T2, T3) -> R) {
_call = { args in body(args[0] as! T1, args[1] as! T2, args[2] as! T3) }
}
...
@discardableResult
func dynamicallyCall(withArguments args: [Any]) -> Any {
_call(args)
}
private let _call: ([Any]) -> Any
}
@dynamicMemberLookup
struct NetworkInterface {
private var methods: [String: AnyClosure]
subscript(dynamicMember member: String) -> AnyClosure {
get { methods[member]! }
set { methods[member] = newValue }
}
}In a language that doesn’t have stuff like @dynamicCallable and @dynamicMemberLookup, you can still do this, but you have to settle for ugly syntax: network.fetchApplicants(listing.id) will have to be written as network.method("fetchApplicants").invoke([listing.id]).
With this, we can not only change the implementation of fetchJobListings or fetchApplicants, we can add more methods! Or delete one of those methods. Or change the parameters those methods take, or the type of their return value. Talk about being more powerful!
So that’s even better! Well, if you consider this added “power” to be a good thing. I don’t. What’s the point of adding a new method, or even worse changing the parameters of an existing one? It’s not like production code is going to call that new method, or send those new types of parameters in. Well, you might misspell fetchListings as fechListings or forget or reorder parameters, and now that’s a runtime error instead of a compile time error.
I like statically typed languages, and I like them because they restrict me from writing the bugs I can easily write in JavaScript, like calling a method that doesn’t exist, or changing the definition of sqrt to something that doesn’t return the square root of the input.
And this is very controversial. I’ve met devs who strongly dislike statically typed languages and prefer JavaScript or Ruby because the static type system keeps stopping them from doing what they want to do. I don’t want to be completely unfair to these devs: it is more work to design types that allow everything you need but also design bugs out of the system. Rather, it’s more work at first, and then much much less work later. It’s tempting to pick the “easier now and harder later” option because humans have time preference.
(Again to be fair to them, no static type system in existence today is expressive enough to do everything at compile time that dynamic languages can do at runtime. Macros, and more generally metaprogramming, will hopefully bridge the gap).
What bugs are no longer impossible to write when we use protocol witnesses? Exercising any of these new capabilities that are just for tests in production code. You can now swap an individual function out. You should never do that in production code. If you do it’s a bug. If your response to this is “why would anyone ever do that?”, my counter-response is “LOL“.
Indeed this design, by being “more powerful” at runtime, is less powerful at compile time. I simply can’t express that a flavor of the object exists where the definitions of the two methods can’t vary independently. With the protocol I can still express that a flavor exists where they can vary independently: make the same witness struct but make it also conform to the protocol. So I actually have more expressiveness at compile time with the protocol. I can even say, for example, that one particular function or type only works with one concrete type (like a test that needs the happy path type, I can express that: func runTest(network: NetworkInterfaceHappyPathFake)), or that two parts of code must work with the same type (make it a generic parameter).
I can’t do any of this with protocol witnesses because the compiler has no idea about my types. It only knows about the abstract type (represented by the witness struct), instead of knowing not only about all the types (the abstract type, as the protocol, and the live concrete type, and the happy path concrete type, etc.) but also their relationship (the latter are all subtypes of the abstract type, expressed by the conformance). So, as it turns out, and specifically because the protocol witness can do things the protocol design can’t do at runtime, the protocol design can do things at compile time the protocol witness design cannot.
“More” or “less” powerful depends on whether you’re looking at compile time or runtime, and one will be the opposite of the other.
These Are Different Models
As I said at the beginning, the very framing here is that these are alternative language mechanisms for the same model. The abstract type and concrete implementing types still exist in the protocol witness code. You just haven’t told the compiler about them, so it can’t enforce any rules related to them.
But whether or not you should be able to mix and match, and by extension swap out, individual functions in the NetworkInterface, is a matter of what system we’re modeling: what exactly the NetworkInterface concept represents, and what relationship, if any, its functions have to each other. Why, after all, are we even putting these three functions in the same type? Why did we decide the correct conceptual model of this system is for fetchJobListings and fetchApplicants(for:) to be instance methods on one type? Why didn’t we originally make three separate protocols, each of which defines only one of these functions?
Well, because they should vary together! Presumably the jobListing you pass into fetchApplicants(for:) is going to be, in fact needs to be, one you previously retrieved from fetchJobListings on the same NetworkInterface instance… or at least an instance of the same type of NetworkInterface. If you grabbed a jobListing from one implementation of NetworkInterface then asked another implementation of NetworkInterface what it’s applicants are, do you think it’s going to be able to tell you?
This means we really should try to express that a particular JobListing type is tied back to a particular NetworkInterface type, and that it is a programmer error that we ideally want to catch at compile time to send a JobListing retrieved from one type of NetworkInterface into another type of NetworkInterface. How would we do that? Definitely not with protocol witnesses. We not only need protocols, we need to exercise more of their capabilities:
protocol NetworkInterfaceProtocol {
associatedtype JobListing
associatedtype Applicant
func fetchJobListings() async throws -> [JobListing]
func fetchApplicants(for listing: JobListing) async throws -> [Applicant]
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing]
}This way, each NetworkInterface type has to specify its own JobListing and Applicant types, and (as long as we pick different types, which we should if we want strong typing) it will become a compile time error to try to pass one type of NetworkInterface‘s job listing into another type of NetworkInterface, which we know is not going to produce a sensible answer anyways (it might crash or throw an exception due to not finding a matching job listing, or even worse it might successfully return a wrong result because both backend systems happen to contain a job listing with the same id).
Obviously, then, it’s nonsense to mix and match the individual functions. The whole point of grouping these functions together is that they form a cohesive unit, and the implementation of one needs to “match” the implementation of another. Modeling this in a way where we are allowed to compose different individual functions is just plain wrong.
But if these three functions happened to be totally separate, like let’s say we have separate backends for different resources, they have no relationship to each other (no foreign keys from one to the other, for example), and it should be possible, and might eventually be a production use case, to mix and match different backends, then it would be just plain wrong to model all those functions as being a cohesive unit, incapable of being individually composed.
See, these are different models of a system. Which one is correct depends on what system we’re trying to build. This is not about language mechanics.
Even with the system as it is, where these functions clearly are a cohesive unit… why should NetworkInterface be a protocol? Why should we be able to define two interfaces that have entirely different implementations? Look at what’s defined in the Live implementation: we implement each one by doing an HTTP GET from a particular path added to a shared base endpoint (each one can’t have a separate base URL, they all have to go to the same backend system). Why would we want to replace this entire implementation? Are we going to have a backend that doesn’t use HTTP? If so, yes it makes sense this needs to be abstract. If not… well then what do we expect to vary?
The base URL? Because you have a Development, Staging and Production backend hosted at different URLs?
Well then just make that a parameter of a concrete type. If you want to be really fancy and make it a compile time error to send a Development JobListing into a Staging NetworkInterface, you can make NetworkInterface generic and use that to strongly type the response entities by their environment:
protocol Environment {
static var baseURL: URL { get }
}
enum Environments {
enum Development: Environment { static let host = ... } }
enum Staging: Environment { static let host = ... } }
enum Production: Environment { static let host = ... } }
}
struct JobListing<E: Environment> {
...
}
struct Applicant<E: Environment> {
...
}
struct NetworkInterface<E: Environment> {
func fetchJobListings() async throws -> [JobListing<E>] {
let (data, response) = try await urlSession.data(for: E.host.appendingPathComponent("listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [JobListing<E>].self, from: data ?? .init())
}
func fetchApplicants(for listing: JobListing<E>) async throws -> [Applicant<E>] {
let (data, response) = try await urlSession.data(for: E.host.appendingPathComponent("listings/\(listing.id)/applicants")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [Applicant<E>].self, from: data ?? .init())
}
func fetchJobListings(for applicant: Applicant<E>) async throws -> [JobListing<E>] {
let (data, response) = try await urlSession.data(for: E.host.appendingPathComponent("applicants/\(applicant.id)/listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type:[JobListing<E>].self, from: data ?? .init())
}
init(
urlSession: URLSession = .shared
) {
self.urlSession = urlSession
}
let urlSession: URLSession
}So here, we use protocols, but in a different place: we’re expressing that the variation is not in the network interface itself. There’s only one network interface. The variation is the environment. Making NetworkInterface abstract is not correct because we don’t want that to vary. We’re not going to build a system that connects to anything other than one of these environments, who all have an HTTP backend with identical RESTful APIs. The web protocol, the resource paths, the shape of the responses, none of that varies. Only the environment varies. Not only can we not swap out individual function implementations, we can’t even swap out a network interface. There’s only one concrete type (parameterized by its environment).
In fact… why did we even think to make NetworkInterface abstract to begin with? We don’t even have multiple environments yet!
Ohh, right… testing.
The Interaction of Testing and Design
That’s a weird use case for this. We’re making things abstract specifically so we can break them… if we consider faking to be a breakage, which we should, as our customers certainly would if we accidentally shipped that to them. So then all these extra capabilities, like individually hot-swapping functions… sure, that creates network interfaces that are obviously broken, as any mix-and-matched one would have to be. But that’s the goal: we want to write a test where we can inject fake (a type of broken) behavior into specific parts of the code.
Testing is uniquely challenging in Swift, compared to other industry languages. In other languages the runtime system is dynamic enough that you can mess with objects at runtime by default. This is how you’re able to build mocking frameworks that can take any method on any class and rewrite it while the test is running. Back in Objective-C days we had OCMock. A major pain point in converting ObjC code with tests to Swift is that you can’t do this anymore.
Why is Swift uniquely difficult in this way (it is comparable to C++)? Because it is so much more strongly typed and compile time safe, which also affects how it is compiled. In other languages basically everything is dynamic dispatch, but in Swift a lot of stuff is static dispatch, so there is literally no place for tests to hook into and redirect that dispatch (at runtime at least). To allow a test to redirect something at runtime, you have to write Swift code specifically to abandon some of that compile time safety… or write Swift code that is compile time polymorphic instead of run time polymorphic, a.k.a. generics.
It’s very unfortunate, then, that we’ve introduced the ability to construct broken network interfaces into the production code just so we can do these things in tests. Wouldn’t it be much better if we could isolate this God mode cheat where we can break the invariants to just when the tests run? How might we do that?
Well, first let’s back up a moment here. What are we writing tests for? To prove the system works? To test-drive the features? It’s a popular idea in TDD circles that “testability” should dictate design. Now this evolved out of a belief that “testable design” is really just synonymous for “good design”… that a design that is untestable is untestable because it has design flaws. Testing is therefore doubly valuable because it additionally forces design improvements. Basically, testability reveals a bunch of places where you hardcoded stuff you shouldn’t have (like dependencies, which should be injected because that makes all your code far more reusable anyways).
But that concept can get carried way, to the point that your design becomes a slave to “testability”. You introduce abstractions, the capability for variation, customization, swapping out dependencies, all over the place just so you can finely probe the running system with test doubles. Even though you not only don’t need that variation, it would be fundamentally incorrect for those things to vary, we introduce a capability for variation just so we can construct intentionally broken (not shippable to production) configurations of our code just to run tests against them.
Again, this wasn’t really a concern in industry languages where TDD evolved in, because it’s not even possible to not have the capability for variation literally everywhere (people eventually found out how to mock even ostensibly static calls in Java). C++ devs might have known about it, and Swift brought this issue to the surface for a different audience.
There is a lot of debate over different testing strategies, including whether it’s a good idea in the first place for tests to “see” things customers would never be able to see, or for tests to run against anything other than the exact code that customers are going to use. The argument is simple: the more you screw with the configuration of code just to test it, the less accurate that test will be because no one in the wild is using that configuration. On the other hand, black box testing the exact production build with no special test capabilities or probes inserted is really hard and tends to produce brittle, slow, shallow tests that can’t even reliably put the system into a lot of the initial conditions we want to test.
So it’s really is worth asking ourselves: what exactly do we need to test that requires replacing implementations of NetworkInterface functions with test doubles? After all, this is the interface into a different system that we communicate with over the wire. Can’t we do the interception and substitution at the wire? Can’t we take full control over what responses NetworkInterface is going to produce by controlling the HTTP server at the base URL? Is that maybe a better way to test here, because then we don’t have to make NetworkInterface abstract, because arguably it shouldn’t be, there should not be a NetworkInterface that does anything other than make those specific HTTP calls?
You can do all the mixing, matching and hot swapping you want this way. As long as you can control your HTTP server in your tests, you’re good to go. If you run the server in the test process, you can supply closures that capture state in your test functions.
Okay so this is a pretty specific issue with this specific example. But then again, this is the specific example used to motivate the desire to mix and match and hot swap methods in our network interface. I think if we ignored the testing use case, it would become clear that this is a capability we very much do not want, and then protocol witnesses are plainly the wrong design specifically because they allow this.
But let’s say you really don’t want to spin up a local HTTP server to power your tests. I get it: it’s a pain in the a** to do that, and a heck of a lot easier to just hot-swap the functions. On the other hand, you aren’t exercising your production NetworkInterface, even though it would be nice for your tests to uncover problems in there too. And then it’s possible for your faked implementations to do things the real one simply wouldn’t be able to do. For example, the live implementation can throw a BadRequest error, whatever URLSession.data(for:) can throw, and whatever JSONDecoder.decode can throw. If you swap out a fake function that throws something else, and an explosion occurs… so what? Why does the rest of the code need to prepare for that eventuality even though it would never (and perhaps should never) happen?
Well, if NetworkInterface is a protocol, you can’t be sure that would never happen. After all, by having production code couple to a protocol, it’s coupling to implementations that can do anything. If NetworkInterface were concrete we’d know that isn’t possible, and we’d be wasting time ensuring production code deals with it. Correspondingly, we wouldn’t be able to write a test that throws something else. We could only control the response by controlling what the server returns, which can only trigger one of the errors that live implementation can throw… and we’ll be sure whatever scenario we created is one that can really happen and that we really need to deal with.
And so on… but anyways, you’re just not going to spin up an HTTP server, so can we somehow isolate these special hot-swapping capabilities to just the tests? Yes. To do so we still need to make NetworkInterface a protocol, which makes the production code potentially way more open-ended than we want it to be… but we can try to express that:
- The entire production code should all use one implementation of
NetworkInterface. That is, we shouldn’t be able to switch from one type to another in the middle of the app - The only implementation that should be available in production code is the live implementation
The second part is easy: we’re going to use the protocol model, the Live implementation will be in production code, and a Test implementation will be defined in the test suite. The Live implementation will look just like it did originally. The Test one will look like this:
struct NetworkInterfaceTest: NetworkInterfaceProtocol {
var fetchJobListingsImp: () async throws -> [JobListing]
func fetchJobListings() async throws -> [JobListing] {
try await fetchJobListingsImp()
}
var fetchApplicantsImp: (_ listing: JobListing) async throws -> [Applicant]
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
try await fetchApplicantsImp(listing)
}
var fetchJobListingsForApplicantImp: (_ applicant: Applicant) async throws -> [JobListing]
func fetchJobListing(for applicant: Applicant) async throws -> [JobListing] {
try await fetchJobListingsForApplicantImp(listing)
}
}
extension NetworkInterfaceTest {
var happyPathFake: Self {
.init(
fetchJobListingsImp: { StubData.jobListings },
fetchApplicantsImp: { listing in StubData.applicants[listing.id]! },
fetchJobListingsForApplicantImp: { applicant in StubData.applicantListings[applicant.id]! }
)
}
}See, we’re still using the protocol witness approach, and we’re still building factories for creating specific common configurations, which we can always reconfigure or customize later. But we’re concentrating it in a specific implementation of the protocol. This expresses that in general a NetworkInterface has cohesion among its functions, and they therefore cannot be individually selected. But this individual configuration capability exists specifically in one implementation of NetworkInterface. By defining NetworkInterfaceTest in the test target instead of the prod target, we’re keeping this, and all its added capabilities, out of the production code.
For the first point, since NetworkInterface is a protocol, there’s nothing stopping you from creating new implementations and using them in specific places in production code. Maybe it’s enough to define only the Live one in prod code, but substituting a different (and definitely wrong) one at one specific place (like in a Model for one screen, or component on a screen, buried deep in the structure of the app) would be easy to miss because you’d have to go drill down into that specific place to see the mistake.
Is it not a requirement of the production code that everything uses the same type of network interface? It would always be incorrect to switch from one type to another for one part of the app. Even in tests, aren’t we going to swap in the fake server for everything? Can we somehow express this in the design, to make it a compiler error to mix multiple types? If we could, then picking the wrong type would affect the app everywhere, and that would probably be immediately obvious upon the most trivial testing.
The reason it would be possible to switch network types mid-way is because every UI model stores an instance of the protocol existential, like let network: NetworkInterfaceProtocol, which is really shorthand for let network: any NetworkInterfaceProtocol. That any box is able to store, well, any implementing type, and two different boxes can store two different types. How do we make them store the same type?
With generics.
Instead of this:
final class SomeComponentModel {
...
let otherComponentModel: OtherComponentModel
private let network: any NetworkInterfaceProtocol
}
final class OtherComponentModel {
...
private let network: any NetworkInterfaceProtocol
}We do this:
final class SomeComponentModel<Network: NetworkInterfaceProtocol> {
...
let otherComponentModel: OtherComponentModel<Network>
private let network: Network
}
final class OtherComponentModel<Network: NetworkInterfaceProtocol> {
...
private let network: Network
}See, the generic parameter links the two network members of the two component models together: they can still vary, we can make component models for any network interface type, but when we select one for SomeComponentModel, we thereby select the same one for OtherComponentModel. Do this throughout all the models in your app, and you express that the entire app structure works with a single type of network interface. You select the type for the entire app, not for each individual component model.
This is an example of “fail big or not at all”. If you can’t eliminate an error from the design, try to make it a nuclear explosion in your face so you immediately catch and correct it. The worst kinds of failures are subtle ones that fly under the radar. If your response to this is “just cover everything with tests”, well… I’ve yet to see any GUI app come anywhere close to doing that. Plus if you aren’t categorizing in your mind compilation errors as failed tests, and the type system you create as a test suite, you’re not getting the point of static typing.
Whether this is worth it to you, well that depends. How determined are you to prevent mistakes like creating new network interface types and using them in tucked away parts of the app (is this something you think is appreciably risky?), and how much does making all the types in your app generic bother you (maybe you don’t like the bracket syntax, or you have other aesthetic or design philosophy reasons to object to a proliferation of generics in a codebase)?
I have been steadily evolving for the past 3 years toward embracing generics, and I mean really embracing them: I have no problem with every single type in my app being generic, and having multiple type parameters (if you want to see code that works this way, just look at SwiftUI). I used to find the brackets, extra syntactic boilerplate, and cognitive load of working out constraints (where clauses) alarming or tedious. Then I got used to it, it became second nature to me, and now I see no reason not to. It leads to much more type safe code, and greatly expands the scope of rules I can tell the compiler about, that the compiler then enforces for me. I don’t necessarily think inventing a new network interface for an individual component is a likely bug to come up, but the cost of preventing it is essentially zero, and I additionally document through the design the requirement that the backend selection is app-wide, not per-component.
(The same evolution is happening in my C++ codebases: everything is becoming templates, especially once C++20 concepts showed up).
And this all involves using the two most powerful, expressive, and, in my opinion, important, parts of Swift: protocols and generics.
It’s Not a Choice Over Protocols
Is this “protocol oriented programming”? And if it is, does this mean I believe protocol oriented programming is a panacea? I don’t know, maybe. But the protocol witness design hasn’t removed protocols, it’s just moved them around. The issue isn’t whether to use protocols or not, it’s where they properly belong, which is determined by the correct conceptual model of your system, specifically in what should really be abstract.
Where are the protocols in the protocol witness version? After all, the keyword protocol is nowhere to be found in that code. How can I claim it’s still using protocols?
Because closures are protocols.
What else could the be? They’re abstract, aren’t they? I can’t write let wtf = (() async throws -> [JobListing]).init(). They express a variation: a closure variable holds a closure body, but it doesn’t specify which one. Even the underlying mechanisms are the same. When you call a closure, the compiler has to insert a dynamic dispatch mechanism. That means a witness table (there’s that word “witness”, we’ll get to that later), which lists out the ways a particular instance fulfills the requirements of a protocol.
This is more obvious if we think about how we would implement closures if the language didn’t directly support them. We’d use protocols:
protocol AsyncThrowingCallable0<R> {
associatedtype R
func callAsFunction() async throws -> R
}
protocol AsyncThrowingCallable1<T, R> {
associatedtype T
associatedtype R
func callAsFunction(_ arg0: T) async throws -> R
}
struct NetworkInterface {
var fetchJobListings: any AsyncThrowingCallable0<[JobListing]>
var fetchApplicants: any AsyncThrowingCallable1<JobListing, [Applicant]>
var fetchJobListingsForApplicant: any AsyncThrowingCallable1<Applicant, [JobListing]>
}
extension NetworkInterface {
static func live(
urlSession: URLSession = .shared
host: URL
) -> Self {
struct FetchJobListings: AsyncThrowingCallable0 {
let urlSession: URLSession
let host: URL
func callAsFunction() async throws -> [JobListing] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [JobListing].self, from: data ?? .init())
}
}
struct FetchApplicants: AsyncThrowingCallable1 {
let urlSession: URLSession
let host: URL
func callAsFunction(_ listing: JobListing) async throws -> [Applicant] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings/\(listing.id)/applicants")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [Applicant].self, from: data ?? .init())
}
}
struct FetchJobListingsForApplicant: AsyncThrowingCallable1 {
let urlSession: URLSession
let host: URL
func callAsFunction(_ applicant: Applicant) async throws -> [JobListing] {
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("applicants/\(applicant.id)/listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type:[JobListing].self, from: data ?? .init())
}
}
return .init(
fetchJobListings: FetchJobListings(urlSession: urlSession, host: host),
fetchApplicants: FetchApplicants(urlSession: urlSession, host: host),
fetchJobListingsForApplicant: FetchJobListingsForApplicant(urlSession: urlSession, host: host)
)
}This is something I think every Swift developer should see at least once, because it shows the underlying mechanism of closures and helps demystify them. We have to implement callability, which we do as a protocol requirement (even if we didn’t have callAsFunction to improve the syntax, we could name the function something like invoke and it still works, we just have to write out invoke to call a closure). We also have to implement capturing, which is fundamentally why these are protocols (whose implementations can be any struct with any amount of instance storage) and not just function pointers.
Now, am I just being pedantic? Can’t I just as well conceive of closures as being function pointers, instead of protocols? No, the specific reason a closure is not just a function pointer is because of capture. I could not keep urlSession and host around and accessible in the live implementation’s methods if they were just function pointers. And what exactly gets captured varies across the implementations.
The key capability of the protocol design is that each implementing type can have its own set of (possibly private) fields, giving each one a different size and layout in memory. The protocol witness itself can’t do this. It’s a struct and its only fields are the methods. Every instance is going to have the exact layout in memory. How, then, can we possibly give the live version more instance fields? By stuffing them inside the captures of the methods, which requires the methods to support variation in their memory layout. The protocoliness of closures is, in fact, crucial to the protocol witness approach working at all.
So, with this clear, we can plainly see that we aren’t choosing protocols vs. no protocols. Rather, we’re choosing one protocol with three requirements at the top vs. three protocols with one requirement (being callable) on the inside. It’s no different than any other modeling problem where you try to work out what the correct way to conceptualize the system you’re building. What parts of it are abstract? Do we have a single abstraction here, or three separate abstractions placed into a concrete aggregate?
If this is your way to avoid being “protocol oriented”, well… I have some bad news for you!
Since we’re on the subject, could we implement either one of these conceptual models (the single abstraction with multiple requirements or the multiple abstractions, each with a single requirement) without any protocols? That might seem plainly impossible. How can we model abstraction without using the mechanism of abstraction that Swift gives us? Well, you can do it, because of course abstraction itself has an implementation, and we can always do that by hand instead of using the language’s version of it.
I really, really hope no one actually suggests this would ever be a good idea. And I’m more nervous about that than you might think because I’ve seen a lot of developers who seem to have forgotten that dynamic dispatch exists and insist on writing their own dispatch code with procedural flow, usually by switching over enums. I hesitate to even show this because I might just be giving them ideas. But if you promise me you’ll take this as educational only and not go start replacing protocols in your code with what you’re about to see… let’s proceed.
This is, of course, how you would model this if you were using a language that simply didn’t have an abstract type system… like C. How would you implement this system in C? Well, that would be a syntactic translation of what you’re about to see in Swift:
enum NetworkInterfaceType {
case live(urlSession: URLSession, host: URL)
case happyPathFake
}
struct NetworkInterface {
let type: NetworkInterfaceType
func fetchJobListings() async throws -> [JobListing] {
switch type {
case let .live(urlSession, host):
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [JobListing].self, from: data ?? .init())
case .happyPathFake:
return StubData.jobListings
}
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
switch type {
case let .live(urlSession, host):
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("listings/\(listing.id)/applicants")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [Applicant].self, from: data ?? .init())
case .happyPathFake:
return StubData.applicants[listing.id]!
}
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
switch type {
case let .live(urlSession, host):
let (data, response) = try await urlSession.data(for: host.appendingPathComponent("applicants/\(applicant.id)/listings")
guard let response, response.statusCode == 200 else {
throw BadResponse(response)
}
return try JSONDecoder().decode(type: [JobListing].self, from: data ?? .init())
case .happyPathFake:
return StubData.applicantListings[applicant.id]!
}
}
}For those curious: how would you do this in C, particularly the enum with associated values, something that does not translate directly to C? You would use a union of all the associated value tuples (structs) for each case, and have a plain enum for tracking which value is currently stored in the union (this is basically how enums with associated types are implemented under the hood in Swift).
How would you even add a new test double to this? You’d have to define a new enum case, and add the appropriate switch cases to each method. Yes, you’d have to do this for each and every “concrete implementation” you come up with. The code for every possible implementation of the network interface would all coexist in this giant struct, and in giant methods that cram all those different varying implementations next to each other.
How would you capture state from where you define these other implementations? Ohh… manually, by adding the captured values as associated values of that particular enum case, that you’d then have to pass in by hand. Every variation of state capture would need its own enum case!
Please, please please don’t start doing this, please!
Again, my anxiety here is well founded. I’ve seen plenty of Swift code where an enum is defined whose name ends with Type, it’s added as a member to another type X, and several methods on X are implemented by switching over the enum. If you’ve ever done this… I mean if you’ve ever written an enum whose name is WhateverType and made one a member of the type Whatever, then you’ve invented your own type system. After all, wouldn’t it make more sense to embed the enum inside the other type and remove the redundant prefix?
struct Whatever {
enum Type {
...
}
let type: Type
...
}Try it. See what the compiler tells you. Whatever already has an inner typealias named Type, which is the metatype: literally Whatever.Type, that Swift defines for every type you create.
You’re inventing your own type system!
Why you’re doing this instead of using the type system the language ships with… I just don’t know.
There are other ways you could do this that might be slightly less ridiculous than cramming all the bodies of all the implementations of the abstract type next to each other in a single source file, and even in single functions. You could, for example, use function pointers, and pass Any parameters in to handle captures. By that point you’re literally just implementing dynamic dispatch and abstract types.
To be clear, this implements the first design of a single protocol with three requirements. If we wanted to implement the protocol witness this way, we’d need three separate structs with their own Type enums, so they can vary independently.
So… yeah. Good thing we decided to use protocols. I don’t know about you, but I’m very happy we don’t write iOS apps in C.
I do see this as being peripherally related to the protocol witness stuff, particularly the conception of it as a choice over language mechanics, because that is also inventing your own type system. As I pointed out earlier, the abstract type and multiple concrete types still exist. The compiler just doesn’t know about it. You’re still typing things. You’re just doing it without making your conceptual “types” literally types as defined in Swift.
I don’t know why you’d want to invent your own type system, and I doubt there is ever a good reason to. Doing so has very surprising implications, it’s an expert level technique that rapidly gets complicated beyond only the most trivial use cases, and it only shuts off a large part of compile time verification and forces you to do type validation at runtime (and probably fatalError or something on validation failures). If you’re doing this to circumvent the rules of a static type system, like for example that you can’t replace the definition of a type’s function at runtime or construct instances that are stitched together from multiple different types (in other words, static types are static)… well you absolutely should not be doing that, the type system works the way it does for a reason and you should stop trying to break it.
What’s a “Witness” Anyways?
Speaking of inventing your own version of what Swift already gives you… why is this technique called a “protocol witness”? If you Google “Swift” and “witness” you’ll likely encounter discussions about how Swift is implemented, something involving a “witness table”. You may have even seen a mention of such a thing in certain crashes, either the debug message that gets printed or somewhere in the stack trace.
To understand that, let’s think about what happens when we declare a variable of type let network: any NetworkInterfaceProtocol. Now, today, you may or may not have to write any here. Whether you do is based on some rather esoteric history (albeit recent history) about Swift. Basically, if you were allowed to write this at all before Swift 5.7, you don’t have to write any, but you can. If something about the protocol prevented you from using it as a variable type before Swift 5.7, specifically before this proposal was implemented, then you have to write any.
I’m harping on this because it’s of critical importance here. This is the correct way to think about it: the any is always there, and always has been, wherever you use a protocol as a variable type. This just was, and still is in many places, implied by the compiler, so writing it out is optional. After all, the compiler can already tell that a protocol is being used as a variable type, so there’s no syntactic ambiguity by omitting it (similar to how you can omit the type of a variable if it gets initialized inline).
How else can you use a protocol beside as the type of a variable? To declare conformance (i.e. to say NetworkInterfaceLive conforms to NetworkInterfaceProtocol) and in a generic constraint. Those two places involve the protocol itself. That is completely different than the protocol name with any before it. If P is a protocol, then any P is an existential box that holds an instance of any concrete type that implements P.
What is an “existential box”? I don’t want to get too deep into the implementation details of Swift’s type system (I’m writing another article series for that), but just think of it this way: P is abstract, which literally means no variable can have a type P. But there has to be some concrete type, and it has to be static: the type of a variable var theBox: any P can’t change as you put different concrete implementations of P inside it. So the type can’t be whatever concrete type you put in it.
That’s why we have to have a special concrete type, any P, for holding literally any P. Imagine for a moment the compiler just didn’t let you do this. You just aren’t allowed to declare a variable whose type is a protocol. But you need to be able to store any instance that implements a protocol in a variable. What would you do? Let’s say we have the protocol:
protocol MyProtocol {
var readOnlyMember: Int { get }
var readWriteMember: String { get set }
func method1() -> Int
func method2(param: Int) -> String
}Now, the point of trying to declare a variable var theBox: MyProtocol is so we can call the requirements, like theBox.readOnlyMember, or theBox.method1(). We need to create a type that lets us do this, but those calls actually dispatch to any implementing instance we want. Let’s try this:
struct AnyMyProtocol {
var readOnlyMember: Int { readWriteMember_get() }
var readWriteMember: String {
get { readWriteMember_get() }
set { readWriteMember_set(newValue) }
}
func method1() -> Int { method1_imp() }
func method2(param: Int) -> String { method2_imp(param) }
init<P: AnyMyProtocol>(_ value: P) {
readOnlyMember_get = { value.readOnlyMember }
readWriteMember_get = { value.readWriteMember }
readWriteMember_set = { newValue in value.readWriteMember = newValue }
method1_imp = { value.method1() }
method2_imp = { param in value.method2(param: param) }
}
private let readOnlyMember_get: () -> Int
private let readWriteMember_get: () -> String
private let readWriteMember_set: (String) -> Void
private let method1_imp: () -> Int
private let method2_imp: (Int) -> String
}What’s happening here is we store individual closures for each of the requirements defined by the protocol (all getters and setters of vars and all funcs). We define a generic initializer that takes an instance of some concrete implementation of P, and we assign all these closures to forward the calls to this implementation.
This is called a type eraser, because that’s exactly what it’s doing. It’s dropping knowledge of the concrete type (whatever the generic parameter P was bound to when the init was called) but preserving the capabilities of the instance. That way, whoever is using the AnyMyProtocol doesn’t know the type of MyProtocol those calls are being forwarded to.
But hold on… this isn’t correct. If you tried compiling this code, you’ll see it failed. Specifically, we’re trying to assign readWriteVar on the value coming into the initializer. But value is a function parameter, and therefore read-only. We need to store our own mutable copy first, and make sure to send all our calls to that copy. We can do that by simply shadowing the variable coming in:
init<P: AnyMyProtocol>(_ value: P) {
var value = value
readOnlyMember_get = { value.readOnlyMember }
readWriteMember_get = { value.readWriteMember }
readWriteMember_set = { newValue in value.readWriteMember = newValue }
method1_imp = { value.method1() }
method2_imp = { param in value.method2(param: param) }
}What exactly is happening here? Well, when the init is called, we create the local variable, which makes a copy of value. All those closures capture value by reference (because we didn’t explicitly add [value] in the capture list to capture a copy), so we are able to write back to it. Since those closures are escaping, Swift sees that this local variable is captured by reference in escaping closures, which means that variable has to escape too. So it actually allocates that variable on the heap, allowing it to live past the init call. It gets put in a reference counted box, retained by each closure, and the closures are then retained by the AnyMyProtocol instance being initialized. When this instance is discarded, all the closures are discarded, the reference count of this value instance goes to 0, and it gets deallocated.
In a rather convoluted way, this effectively sneaks value into the storage for the AnyMyProtocol instance. It’s not literally inside the instance, it just gets attached to it and has the same lifetime.
Now think about what happens here:
let box1 = AnyMyProtocol(SomeConcreteStruct())
var box2 = box1
box2.readWriteMember = "Uh oh..."
print(box1.readWriteMember)What gets printed? This: "Uh oh...".
We assigned readWriteMember on box2, and it affected the same member on box1. Evidently AnyMyProtocol has reference semantics, even though the value it’s erasing, a SomeConcreteStruct, has value semantics. This is wrong. When we assign box2 to box1 it’s supposed to make a copy of the SomeConcreteStruct instance inside of box1. But above I explained that the actual boxed value is the var value inside the init, which is placed in a reference counted heap-allocated box to keep it alive as long as the closures that capture it are alive. This has to happen, this value must have reference semantics, because the closures have to share the value. When the readWriteMember_set closure is called and writes to value, that has to be “seen” by a subsequent call to the readWriteMember_get closure.
But while we need the value to be shared among the various closures in a single instance of AnyMyProtocol, we don’t want the value to be shared across instances. How do we fix this?
We have to put the erased value in the AnyMyProtocol instance as a member, and we need that member to be the one closures operate one, which means it needs to be passed into the closures:
struct AnyMyProtocol {
var readOnlyMember: Int { readWriteMember_get(erased) }
var readWriteMember: String {
get { readWriteMember_get(erased) }
set { readWriteMember_set(&erased, newValue) }
}
func method1() -> Int { method1_imp(erased) }
func method2(param: Int) -> String { method2_imp(erased, param) }
init<P: AnyMyProtocol>(_ value: P) {
erased = value
readOnlyMember_get = { erased in (erased as! P).readOnlyMember }
readWriteMember_get = { erased in (erased as! P).readWriteMember }
readWriteMember_set = { erased, newValue in
var value = erased as! P
value.readWriteMember = newValue
erased = value
}
method1_imp = { erased in (erased as! P).method1() }
method2_imp = { erased, param in (erased as! P).method2(param: param) }
}
private var erased: Any
private let readOnlyMember_get: (Any) -> Int
private let readWriteMember_get: (Any) -> String
private let readWriteMember_set: (inout Any, String) -> Void
private let method1_imp: (Any) -> Int
private let method2_imp: (Any, Int) -> String
}Here we see the appearance of Any. What’s that? It’s Swift’s type erasing box for literally anything. It is what implements the functionality of keep tracking of what’s inside the box and properly copying it when assigning one Any to another. The closures now have what is effectively the self parameter. The closures are going to be shared among copies of the AnyMyProtocol, we can’t change that. So if they’re shared, and we don’t want them to operate on a shared P instance, we have to pass in the particular P instance we want them to operate on.
In fact, these aren’t closures anymore because they aren’t capturing anything. And that’s fortunate, because we’re trying to solve the “Swift doesn’t let you use abstract types as variable types” hypothetical, which would ban closures too. By eliminating capture, these are now just plain old function pointers, which aren’t abstract types.
Now, we can collect these function pointers into a struct:
struct AnyMyProtocol {
var readOnlyMember: Int { witnessTable.readWriteMember_get(erased) }
var readWriteMember: String {
get { witnessTable.readWriteMember_get(erased) }
set { witnessTable.readWriteMember_set(&erased, newValue) }
}
func method1() -> Int { witnessTable.method1_imp(erased) }
func method2(param: Int) -> String { witnessTable.method2_imp(erased, param) }
init<P: AnyMyProtocol>(_ value: P) {
erased = value
witnessTable = .init(
readOnlyMember_get: { erased in (erased as! P).readOnlyMember },
readWriteMember_get = { erased in (erased as! P).readWriteMember },
readWriteMember_set = { erased, newValue in
var value = erased as! P
value.readWriteMember = newValue
erased = value,
method1_imp = { erased in (erased as! P).method1() },
method2_imp = { erased, param in (erased as! P).method2(param: param) }
)
}
struct WitnessTable {
let readOnlyMember_get: (Any) -> Int
let readWriteMember_get: (Any) -> String
let readWriteMember_set: (inout Any, String) -> Void
let method1_imp: (Any) -> Int
let method2_imp: (Any, Int) -> String
}
private let erased: Any
private let witnessTable: WitnessTable
}The specific WitnessTable instance being created doesn’t depend on anything except the type P, so we can move into an extension on MyProtocol:
extension MyProtocol {
static var witnessTable: AnyMyProtocol.WitnessTable {
.init(
readOnlyMember_get: { erased in (erased as! Self).readOnlyMember },
readWriteMember_get = { erased in (erased as! Self).readWriteMember },
readWriteMember_set = { erased, newValue in
var value = erased as! Self
value.readWriteMember = newValue
erased = value,
method1_imp = { erased in (erased as! Self).method1() },
method2_imp = { erased, param in (erased as! Self).method2(param: param) }
)
}
}Then the init is just this:
init<P: AnyMyProtocol>(_ value: P) {
erased = value
witnessTable = P.witnessTable
}Does this idea of a table of function pointer members, one for each requirement of a protocol, sound familiar to you?
Hey! This is a protocol witness! “Witness” refers to the fact that the table “sees” the concrete P and its concrete implementations of all those requirements, recording the “proof”, so-to-speak, of those implementations in the form of the function pointers. When someone else comes along and asks, “hey, does you value implement this requirement?”, the witness answers, “yes! I saw that earlier. Here, look at this closure, this is what its implementation is”.
Well, this is exactly what the Swift compiler writes for you, for every protocol that you write. The only difference is that theirs is called any MyProtocol instead of AnyMyProtocol. The compiler creates a witness table for every protocol and uses it in its existential boxes to figure out where to jump to.
At any point an existential box will hold a pointer to a particular witness table, and when you call a method on it, the compiled code goes to the witness table, grabs the function pointer at the right index (depending on what requirement you’re invoking), and then jumps to that function pointer.
The box is, in fact, implementing dynamic dispatch, which always requires some type of runtime lookup mechanism. The members of the witness table are function pointers, which are just the addresses of the first line of the compiled code for the bodies. Calling one just jumps the execution to that address. Every function has a known compile time constant address, so if you want dynamic dispatch, you have to store a function pointer in a variable. That’s what the witness table is.
There are things the compiler can make its box do that we can’t make our box do, and vice versa. The compiler makes its box transparent with respect to casting: we can directly downcast an instance of any P to P1 (where P1 is a concrete implementer of P) and the compiler checks what it has in the box and, if it matches, pulls that out and gives it to us. We can’t make our own box do that, at least not transparently. On the other hand, the compiler never conforms its box to any protocols (not even the protocol being erased: any P does not conform to P, you may have seen compiler errors about this that, they used to be much worse messages like P as a type cannot conform to itself, which is pretty freaking confusing!). You can conform your box to whatever protocols you want.
If we wrote our own existential box for NetworkInterfaceProtocol, it would look like this:
struct AnyNetworkInterface: NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing] {
try await fetchJobListings_imp()
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
try await fetchApplicants_imp(listing)
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
try await fetchJobListingsForApplicant_imp(applicant)
}
init<Erasing: NetworkInterfaceProtocol>(erasing: Erasing) {
fetchJobListings_imp = { try await erasing.fetchJobListings() }
fetchApplicants_imp = { listing in try await erasing.fetchApplicants(for: listing) }
fetchJobListingsForApplicant_imp = { applicant in try await erasing.fetchJobListings(for: applicant) }
}
private let fetchJobListings_imp: () async throws -> [JobListings]
private let fetchApplicants_imp: (JobListing) async throws -> [Applicant]
private let fetchJobListingsForApplicant_imp: (Applicant) async throws -> [JobListings]
}
// Convenience function, so we can box an instance with `value.erase()` instead of `AnyNetworkInterface(erasing: value)`
extension NetworkInterfaceProtocol {
func erase() -> AnyNetworkInterface {
.init(erasing: self)
}
}Well this is almost exactly like the Test implementation I showed earlier, isn’t it!?
“Protocol witness” is a reference to the fact this is, for all intents and purposes, a type erasing box. The only difference is the one PointFree shows us is made to be configurable later. We can do that too:
struct ConfigurableAnyNetworkInterface: NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing] {
try await fetchJobListings_imp()
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
try await fetchApplicants_imp(listing)
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
try await fetchJobListingsForApplicant_imp(applicant)
}
init<Erasing: NetworkInterfaceProtocol>(erasing: Erasing) {
fetchJobListings_imp = { try await erasing.fetchJobListings() }
fetchApplicants_imp= { listing in try await erasing.fetchApplicants(for: listing) }
fetchJobListingsForApplicant_imp = { applicant in try await erasing.fetchJobListings(for: applicant) }
}
var fetchJobListings_imp: () async throws -> [JobListings]
var fetchApplicants_imp: (JobListing) async throws -> [Applicant]
var fetchJobListingsForApplicant_imp: (Applicant) async throws -> [JobListings]
}
extension NetworkInterfaceProtocol {
func makeConfigurable() -> ConfigurableAnyNetworkInterface{
.init(erasing: self)
}
}With this, we can do something like start off with the Live implementation, put it in a configurable box, then start customizing it:
var network = NetworkInterfaceLive(host: prodHost)
.makeConfigurable()
network.fetchJobListings_imp = {
...
}What we’re really doing here is writing our own existential box. That’s what I mean when I say this technique is getting into the territory of creating our own versions of stuff the compiler already creates for us. It’s just a less absurd form of building our own type system with Type enums.
Now, there are reasons why you need to build your own type erasing box. Even Apple does this in several of their frameworks (AnySequence, AnyPublisher, AnyView, etc.). It usually comes down to making the box conform to the protocol it’s abstracting over (AnySequence conforms to Sequence, AnyPublisher conforms to Publisher, AnyView conforms to View, etc.). This is a language limitation, something we expect, or at least hope, will be alleviated in future versions (i.e. allowing extensions of the compiler-provided box: extension any Sequence: Sequence), and sometimes we just need to work around language limitations by doing stuff ourselves the compiler normally does.
(…well, not necessarily. Whether the type erasing box should ever conform to the protocol it abstracts over is not so obvious. Remember how we used a generic to ensure two component models use the same concrete network interface type? If any NetworkInterfaceProtocol automatically counted as a concrete network interface, you could pick that as the type parameter and then you’re able to mix and match, or switch mid-execution. Then what’s the point of making it generic? Maybe what we really need is a way to specify that a generic parameter can be either a concrete type or the existential).
However, writing a type erasing box that you can then start screwing with is not an example of this. This is not a missing language feature. Allowing methods of an any P to be hot-swapped or mix-and-matched would break the existential box. You’d never be sure, when you have an any P, if you have a well-formed instance conforming to P, and everyone has the keys to reach in and break it.
This is why I would only want to expose an implementation of NetworkInterfaceProtocol that allows such violations to tests, and call it Test to make it clear it’s only for use as a test double. Now that’s a fine approach. I’ve actually gone one step further and turned such a Test implementation into a Mock implementation by having it record invocations:
typealias Invocation<Params> = (time: Date, params: Params)
final class NetworkInterfaceMock: NetworkInterfaceProtocol {
private(set) var fetchJobListings_invocations: [Invocation<()>] = []
var fetchJobListings_setup: () async throws -> [JobListing]
func fetchJobListings() async throws -> [JobListing] {
fetchJobListings_invocations.append((.now, ())
try await fetchJobListings_setup()
}
private(set) var fetchApplicants_invocations: [Invocation<JobListing>] = []
var fetchApplicants_setup: (_ listing: JobListing) async throws -> [Applicant]
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
fetchApplicants_invocations.append((.now, listing)
try await fetchApplicants_setup(listing)
}
private(set) var fetchJobListingsForApplicant_invocations: [Invocation<Applicant>] = []
var fetchJobListingsForApplicant_setup: (_ applicant: Applicant) async throws -> [JobListing]
func fetchJobListing(for applicant: Applicant) async throws -> [JobListing] {
fetchJobListingsForApplicant_invocations.append((.now, applicant)
try await fetchJobListingsForApplicantImp(listing)
}
func reset() {
fetchJobListings_invocations.removeAll()
fetchApplicants_invocations.removeAll()
fetchJobListingsForApplicant_invocations.removeAll()
}
}This kind of boilerplate is a perfect candidate for macros. Make an @Mock macro and you can produce a mock implementation, that records invocations and is fully configurable from the outside, with as a little as @Mock final class MockMyProtocol: MyProtocol {}. You can probably do the same with the type erasing boxes.
Remember I said earlier that Swift’s compile time safety shut off a lot of easy configurability in tests that all executed at runtime (i.e. rewriting methods)? Well, macros is the solution, where this same level of expressiveness is recovered at compile time.
What Can’t You Do with Protocols?
But let’s rewind a little bit. Do you actually need to create this kind of open-ended type whose behavior you can then change, which is a strange thing to be able to do to any type (changing the meaning of its methods) in order to cover the examples shown?
No. For example, in the case where we want a test to be able to swap in its own implementation of fetchJobListings, we don’t have to make any concrete NetworkInterfaceProtocol whose behavior can change ex-post facto. Instead, we can build transforms that take one type of network interface, and create a new type of network interface with new behavior. The key is that we stick to the usual paradigm of a static type having static behavior. We don’t make a particular instance, of course whose type doesn’t change, dynamic to this degree. We create a new instance of a new type:
struct NetworkInterfaceReplaceJobListings<Base: NetworkInterfaceProtocol>: NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing] {
try await fetchJobListings_imp()
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
try await base.fetchApplicants(for: listing)
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
try await base.fetchJobListings(for: applicant)
}
init(
base: Base,
fetchJobListingsImp: @escaping () async throws -> [JobListings]
) {
self.base = base
self.fetchJobListingsImp = fetchJobListingsImp
}
private let base: Base
private let fetchJobListingsImp: () async throws -> [JobListings]
}
extension NetworkInterfaceProtocol {
func reimplementJobListings(`as` fetchJobListingsImp: @escaping () async throws -> [JobListings]) -> NetworkInterfaceReplaceJobListings<Self> {
.init(base: self, fetchJobListingsImp: fetchJobListingsImp)
}
}
...
let network = NetworkInterfaceHappyPathFake()
.reimplementJobListings {
throw TestError()
}The key difference here is that the network interface with the swapped out implementation is a different type than the original one. This can interact in interesting ways with the generics system. For example, if you implemented your component models to be generics in order to constrain all component models in your app to always use the same network interface type, then this is ensuring that either the entire app uses the network interface with the swapped out implementation, or no one does.
This is more strongly typed. It expresses something I think is very reasonable: a network interface with a swapped out implementation of fetchJobListings is a different type of network interface. However, there’s still an element of dynamism. Each type we create a new NetworkInterfaceReplaceJobListings instance, we can supply a different reimplementation. Everything else is static, but the jobListings implementation is still instance-level varying. This is clear from the fact the type has a closure member. Closures are abstract, so that’s dynamic dispatch. Can we get rid of that exception and our classes fully static? Can we instead make it so that each specific reimplementation of jobListings produces a distinct type of NetworkInterface?
Yes, but it requires a lot of boilerplate, and we unfortunately lose automatic capture so we have to implement it ourselves. Let’s say in a test we’re calling reimplementJobListings inside a method either for a test, or for test setup:
final class SomeTests: XCTestCase {
var jobListingsCallCount = 0
func testJobListingsGetsCalledOnlyOnce() {
let network = NetworkInterfaceHappyPathFake()
.reimplementJobListings { [weak self] in
self?.jobListingsCallCount += 1
return []
}
...
XCTAssertEqual(jobListingsCallCount, 1)
}
}We can replace this with defining a local type of network interface:
final class SomeTests: XCTestCase {
var jobListingsCallCount = 0
func testJobListingsGetsCalledOnlyOnce() {
struct NetworkInterfaceForThisTest: NetworkInterfaceProtocol {
func fetchJobListings() async throws -> [JobListing] {
parent?.jobListingsCallCount += 1
return []
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
try await base.fetchApplicants(for: listing)
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
try await base.fetchJobListings(for: applicant)
}
init(
parent: SomeTests?
) {
self.parent = parent
}
private let base = NetworkInterfaceHappyPathFake()
private weak var parent: SomeTests?
}
let network = NetworkInterfaceForThisTest(parent: self)
...
XCTAssertEqual(jobListingsCallCount, 1)
}
}Here, we’re going all the way with “static types are static”, meaning a type’s behavior is statically bound to that type. If we want different behavior in a different test, we make a different type. But having to type all of this out, and deal with capturing (in this case weak self manually) is a lot of tedium. Especially in tests, I probably wouldn’t bother, and would bend the rules of a static type having fixed behavior to avoid all this boilerplate.
What I would love to be able to do, though, is be able to do this fully static approach, but avoid having to invent a throwaway name for this local type, and have it close over its context, using the same syntax as always to specify capture:
final class SomeTests: XCTestCase {
var jobListingsCallCount = 0
func testJobListingsGetsCalledOnlyOnce() {
let network = NetworkInterfaceProtocol { [weak self] in
func fetchJobListings() async throws -> [JobListing] {
parent?.jobListingsCallCount += 1
return []
}
func fetchApplicants(for listing: JobListing) async throws -> [Applicant] {
try await base.fetchApplicants(for: listing)
}
func fetchJobListings(for applicant: Applicant) async throws -> [JobListing] {
try await base.fetchJobListings(for: applicant)
}
private let base = NetworkInterfaceHappyPathFake()
}
...
XCTAssertEqual(jobListingsCallCount, 1)
}
}This isn’t a very odd language capability, either. Java and Typescript both support this very thing.
Another minor issue here is performance, which I’ll be very frank, I would be shocked if this ever actually matters for something like an iOS app. But it’s interesting to point it out.
The protocol witness pays various runtime costs, in terms of both memory and CPU usage, because it is fundamentally, and entirely, a runtime solution to the problem of swapping out implementations. First, what is the representation of the protocol witness NetworkInterface in memory? Well, it’s a struct, and it looks to be a pretty small one (just three members, which are closures), so it’s likely to be placed on the stack where it’s a local variable. However, the struct itself isn’t doing anything interesting. The real work happens inside the closure members. Those are all existential boxes for a closure, which as we saw above is a concrete type that holds all the captured state and implements a callAsFunction requirement. Depending on how much gets captured, the instances of those concrete types may or may not fit directly into their type erasing boxes. If they don’t, they’ll be allocated on the heap and the box will store a pointer to its instance. This will result in runtime costs of pointer indirection, heap allocation and cache invalidation.
When methods are called, the closure members act as a thunk to the body of the closure, so that’s going to be another runtime cost of pointer indirection.
Contrast that with the first approach shown above (which is really an example of the Proxy Pattern), where the static typing is much stronger. The type that we end up with is a NetworkInterfaceReplaceJobListings<NetworkInterfaceHappyPathFake>. Notice the way the proxy type retains information about the original type that it is proxying. In particular, it does not erase this source type. Furthermore, NetworkInterfaceHappyPathFake is a static type, not a factory that constructs particular instances.
The implication of this is that the network variable can never (even if it was made var) store another type of network interface. We can’t change the definitions of any of the functions on this variable at any point in its life. That means it is known at compile time what the precise behavior is, except for the closure provided as the reimplementation of fetchJobListings. The fact that, for example, the other two calls just call the happy path fake version, is known at compile time. It is known at compile time that the base member is a NetworkInterfaceHappyPathFake. The size of this member, and the closure member, is known at compile time. If it’s small enough it can be put on the stack. There is no runtime dispatch of any of the calls, except for the closure.
If we go to the fully static types where we define new ones locally, we eliminate the one remaining runtime dispatch of the closure, and literally everything is hardwired at compile time.
The cost is all paid at compile time. The compiler looks at that more complex nested/generic type and figures out, from that, how to wire everything up hardcoded. This the difference, in terms of runtime performance, between resolving the calls through runtime mechanisms as in the protocol witness, and resolving the calls through compile mechanisms of generics.
(Caveat: when you call generic code in Swift, supplying specific type parameters, the body of the generic code can be compiled into a dedicated copy for that type parameter, with everything hardwired at compile time, and there will be no runtime cost, but only if the compiler has access to that source code during compilation of your code that calls it, which is generally true only if you aren’t crossing module boundaries. If you do cross a module boundary, when the compiler compiled the module with the generic code, it compiled an “erased” copy of the generic code that replaces the generic parameter with an existential and dynamic-dispatches everything. That’s the only compiled code your module will be able to invoke, and you’ll end up paying runtime costs).
Again, I’m sure this will never matter in an iOS app. We used to write apps in Objective-C, where everything is dynamic dispatch (specifically message passing, the slowest type), on phones half the speed (or less) of what we write them on now. I rather think it’s interesting to point out the trivial improvement in runtime performance (and corresponding, probably also trivial, degradation of compile time performance) of the strongly typed implementation because it further illustrates that the behavior of the program is being worked out at compile time… and it therefore does more validation, and rejection of invalid code, at compile time.
These kinds of techniques where you start building up more sophisticated types, typically with some kind of type composition (through generic parameters), of protocols, with the goal of eliminating runtime value-level (existential) variation and try to express the variation on the type level, is where the power of protocols can really yield fruit, especially in terms of expanding your thinking about what they can do. It’s easy to just give up and say “this can’t be done with protocols or at the type level”, but you should always try. You might be surprised what you can do, and even if you revert to existentials later, you’ll probably learn something useful.
Conclusion
So, do I think you should ever employ this struct witness pattern?
Well, that depends on what it means exactly.
As a refactor of a design with protocols on the basis that it works around language limitations? No, that’s fundamentally confused: those are not language limitations, that’s called a static type system, and if you’re going to throw that away for dynamic typing at least frame it honestly and accurately. Then do your best to convince me dynamic typing is better than static typing (good luck).
What about if we realize those individual functions are where the abstractions should be, because it is correct modeling of the system we’re building to let them vary independently? Well, in that case, the question is: should those members be just closures, or should you define protocols with just one function as their requirements? For example, if you define fetchJobListings to simply be any () async throws -> [JobListing] closure, to me this communicates that any function that has that signature is acceptable. I could have two such functions, both defined as mere closures, but they mean different things and it’s a programmer error to assign one to another. If I introduce two protocols, each of which having a single function requirement, well first I can name that function, thereby indicating what the meaning of this function is. Second, it’s strongly typed: the compiler won’t let me use one where another is expected.
So even in this case, I would want to think carefully about whether closures are sufficient, or if I want to have stronger typing than this and instead define a protocol that implements callAsFunction. Being as biased toward strong typing as I am, it’s likely I’ll choose to write out protocols. That extra boilerplate? The names of those protocols, and possibly the names of their one function requirement? I want that. Terseness has never been my goal. If it were I’d name my variables things like x or y1.
As a hand-written type eraser, intending to be a replacement for the language-provided existential where I need more flexible behavior? Never, I would consider that to be breaking the existential by enabling it to circumvent and violate the static type system.
To implement test doubles? Yes I’d do something that’s effectively the same as a protocol witness, but I wouldn’t replace the protocol with this, I would add to the protocol based design by having this witness implement the protocol.
So, ultimately, as it is presented: no, I never would, and I would consider it to be broadly in the same category as the style of coding where you define your own TheThingType enums that appear as an instance member in a type TheThing, which is a bad style of Swift that tries to replace the language’s type system with a homegrown type system that, however good or even better (which I doubt it will be) it is, will never be verified by the compiler, and that’s a hard dealbreaker for me.
Where might you see me writing code that involves simple structs with a closure member where I then create a handful of different de-facto “types” by initializing the closure in a specific way for each one? In prototyping. Like I’m scaffolding out some code, it’s early and I’m not sure about the design, and because it’s less boilerplate, whipping up that struct is just a little bit faster than writing out the protocol and conforming type hierarchy. Once I’m more sure this abstract type is here to stay, I’m almost certainly going to refactor it to a protocol with the closure member becoming a proper function.
If I can tie together all my decisions regarding this into a single theme, it is: I completely, 100% favor compile time safety over runtime ease or being able to quickly write code without restriction. Because perhaps my central goal of design is to elevate as many programmer errors as possible into compilation failures, and because generally the way to do that is to use statically typed languages and develop a strong (meaning lots of fine-grained) type system, I immediately dislike the protocol witness approach because it degrades the type system that the compiler sees. My goal is not and has never been to avoid or minimize the boilerplate that defining more types entails.
If you’re interested in pursing this style of programming, where you are trying to reject as much invalid code as possible at compile time, my #1 advice, if you’re working in Swift, is to embrace generics and spend time honing your skills with protocols, especially the more advanced features like associated types. If there’s a single concrete goal to drive this, try your best to eliminate as many protocol existentials in your code base as possible (this will almost always involve replacing them with generics). As part of this, you have to treat closures as existentials (which, fundamentally, they are).
You will get frustrated at times, and probably overwhelmed. Run into the challenge, not away. You’ll be happy you did later.
In a follow-up article I will explore a more nontrivial conceptual model of an abstract type with concrete implementations, and what happens (specifically what breaks down, especially related to compile time safety) if you try to build it will protocol witnesses.