It’s Global Until It Isn’t

I’ve finally found a illustration of a problem I’ve wanted to describe. I like the tools I’m using to illustrate this problem, so this is not me denigrating the individual pieces. Instead, I’m drawing attention to how we compose pieces together.

Consider a content negotiation middleware like the one in muuntaja. It will consume the InputStream that is the body and then add :body-params to the req. You add it to your root handler, and start on your merry way:

(def app
  (-> some-handler
      (muuntaja.middleware/wrap-format)))

Later, you’re integrating with an external service that needs to do webhooks. These webhooks are authenticated using HMAC in order to verify the request. This HMAC is calculated based on the body, you attempt to slurp the :body… but it’s empty! Muuntaja has already consumed the body, so you no longer can.

How do you disable Muuntaja, just for this handler?

The obvious "hack" is to do something like this, but this code is certainly not easy to understand.

(defn wrap-format
  "Like muuntaja.middleware/wrap-format, but doesn't apply to webhooks"
  [handler & args]
  (let [fmt-handler (apply muuntaja.middleware/wrap-format handler args)]
    (fn [req]
      (if (= (:uri req) "/some/webhook")
        handler
        fmt-handler))))

This code displays a number of downsides:

Complected to which URI the handler is mounted at
Complected to which handlers should or shouldn’t be part of the process.
Your webhook handler will behave differently than others, so copying code from it won’t necessarily work
There’s a big disconnect between the webhook handler, and this piece of middleware. In most applications these won’t even be in the same file, the webhook handler will appear to magically have the body intact. This is known as logical coupling.
If we want to parse the body (we do, that’s the pith in a webhook) we’ll need to implement the pieces that the muuntaja middleware was providing. We need to make sure we take the same options that the muuntaja middleware would, copying and risking being out of sync, or perhaps convoluting the handler to take the params somehow.

We’ve created all of these problems, because we had lost the option to compose pieces.

We could use something similar to route data to introduce a flag of some kind at the handler. The flag might be :webhook? or :skip-body-parsing?. This leads to one of my favourite quotes, from the wrong abstraction.

Programmer A sees duplication.

Programmer A extracts duplication and gives it a name.

This creates a new abstraction. It could be a new method, or perhaps even a new class.

Programmer A replaces the duplication with the new abstraction.

Ah, the code is perfect. Programmer A trots happily away.

Time passes.

A new requirement appears for which the current abstraction is almost perfect.

Programmer B gets tasked to implement this requirement.

Programmer B feels honor-bound to retain the existing abstraction, but since isn’t exactly the same for every case, they alter the code to take a parameter, and then add logic to conditionally do the right thing based on the value of that parameter.

What was once a universal abstraction now behaves differently for different cases.

Another new requirement arrives.

Programmer X.

Another additional parameter.

Another new conditional.

Loop until code becomes incomprehensible.

You appear in the story about here, and your life takes a dramatic turn for the worse.

It’s not quite a perfect description of the pattern here, but it’s similar. Conditional parameter logic is icky given time, especially in an application where abstractions do not go through hammock time. I would draw a distinction, for example, between Yada^[1] and an ad-hoc system designed inside an application.

A technique I use to overcome this is composition, rather than inheritance. This is one of the subtle shifts that takes longer when moving from imperative languages compared to objects and type modelling. I found a beautiful description of this on ClojureVerse by didibus.

If A, B, C are the operations you need performed. Lets assume all of them need no input data and simply print out their names.
(defn A [] (println "A"))
(defn B [] (println "B"))
(defn C [] (println "C"))
Now say you want to print ABC? It’s often tempting to do:
(defn A [] (println "A") (B))
(defn B [] (println "B") (C))
(defn C [] (println "C"))

(A)
Creating a deep/nested call stack, where you’ve coupled your operations and their flow together.

Instead favor a shallow/flat call stack:
(defn A [] (println "A"))
(defn B [] (println "B"))
(defn C [] (println "C"))

(do (A) (B) (C))
With top level orchestration.

— didibus

The earliest possible design that could have avoided this problem, would be to compose explicitly at each handler. This way, each handler has total control over how it is negotiated:

(def handler-a
  (-> (fn [req] …)
      (muuntaja.middleware/wrap-format)))

(def handler-b
  (-> (fn [req] …)
      (muuntaja.middleware/wrap-format)))

(def handler-webhook
  (fn [req] …))

Although at this point, the middleware seems silly, you might as well just do:

(defn handler-a
  [req]
  (let [body (muuntaja.core/decode (:body req) (content-type req))]
    …))

That would make it pretty clear how you can use it from your webhook handler:

(defn handler-webhook
  [req]
  (let [his (HashingInputStream. (:body req))]
        body (muuntaja.core/decode his (content-type req))]
     …))

This solution reduces logical coupling, but also gives flexibility in how it’s applied. The downside is increased tedium at each handler, although it’s hard to argue that (:body-params req) is significantly better than a helper, (body-params req) which calls out to muuntaja for you. Going from only having body-params to also having decode-input-stream is much simpler than the previous transition to disable the middleware conditionally:

before

(def ^:private m (m/create …))

(defn body-params
  "Return decoded body using API defaults for req."
  [req]
  (m/decode m (content-type req) (:body req)))

after

(def ^:private m (m/create …))

(defn decode-input-stream
  "Return decoded InputStream using API defaults."
  [is content-type]
  (m/decode m content-type is))

(defn body-params
  "Same as (decode-is (:body req) (content-type req))"
  [req]
  (decode-is (:body req) (content-type req)))

Avoiding inheritance even makes the ideas in Spec-ulation easier to follow! We’re less likely to need a breaking change if we can simply add a new function to rework the interface. Consumers can use whichever suits their situation.

The general principle is to focus on composable pieces, rather than one-size-fits-all solutions. This is really what "simple" is about in "simple made easy."

As we make things simpler, we get more independence of decisions because they’re not interleaved, so I can make a location decision. It’s orthogonal from a performance decision.

— Rich Hickey

We need to design our programs to use simple pieces which can be composed into others. In Design, Composition, Performance Rich Hickey goes deeper into how you can do this, I won’t cover that here in as much depth.

The problem with the ring design above is that we took away the choice to compose them differently in the future. It’s not as simple a solution as calling a function. Be mindful of whether the choices you’re making to your program will inhibit future change by constraining alternative choice.

1. Although Yada has the same problem of consuming the body, but no simple solution (only one involving interceptor stuff).