Part I: Everything sucks

Technology organisations have adopted APIs at the heart of their IT strategy. However, with teams moving faster than ever, and microservices on the rise, the number of integration touchpoints is exploding.  Our current patterns of API consumption simply don't cut it.

Distribute information, not APIs

Organisations don't really want APIs - they want easy access to information and capability. APIs are really just a means to an end.

On one hand, the tooling for publishing APIs to make data and services accessible has matured massively - it's never been easier to simply expose a RESTful endpoint, documented with Swagger, Gitops'd out to your fancy elastic Kubernetes cluster.  Boom...check... APIs delivered, right?  Wrong.

While we've managed to commoditize the process of publishing APIs, sadly the overhead of consuming APIs is crippling. The reality is that most API strategies today put too much cost on the consumer:

  • The cost of discovery - finding the right API which serves the data you're looking for in a large organisation can be tough
  • The cost of integration - The initial build out of integration takes time.  OpenAPI tooling helps, but it's wins are very short-lived.
  • The cost of maintenance - The silent killer.  As producers change their APIs, this burden of change flows downstream to all consumers.

The worst part of consumer facing costs are they are repeated - integrating APIs is a tax that dev teams pays repeatedly in order to get and retain access to information.

Organisations with internal API strategies are paying this tax over and over, for every new consumer that spins up.

...Integrating APIs is a tax that dev teams pays repeatedly in order to get and retain access to information.

This high cost of change leads to a paradox of success - the more consumers an API has, the higher the number of points to change, meaning the more expensive it is to evolve.  As a result, some services simply become too critical to evolve, and their unmaintained APIs continue to spread - like a forgotten ham sandwich in the bottom of  kids schoolbag.

Swagger is not a (consumer) strategy

Swagger/OpenAPI - and their newest incarnation, Service Catalogues - are great tools.  However, the traditional adoption patterns in using OpenAPI results in building horribly brittle architectures.  Swagger's "SDKs for APIs" is really just tooling that helps tightly couple interfaces across the enterprise.

Whether you're using OpenAPI codegen tools, service catalogues to help find APIs, or UI based drag-and-drop integrators, these are all tools that facilitate tight coupling between consumers and producers.  As a result, whenever upstream APIs change, consumers are obligated to update along with them.

Swagger/OpenAPI - and their newest incarnation, Service Catalogues - are great tools for building horribly brittle architectures.

That's not to say that OpenAPI documentation isn't great - it is.  So is Protobuf,  gRPC, GraphQL - even WSDLs for that matter ... we need more, richer, better API documentation.  (Heck, that's why we built Taxi)

And, API publishers should be held accountable for producing accurate detailed API docs, in a machine readable format. (README.md and whatever Wiki software exists in your enterprise doesn't count).

But, our thinking around consumer patterns needs to evolve beyond expecting consumers to hitch their wagon to whatever producers publish.

Put simply, today's API consumer patterns facilitate rapidly producing tightly coupled services.  That's clearly nuts.

Consumers want information.  APIs are a producers concern - and one we should abstract away from.

Consumers just need access to information - they shouldn't have to care about the mechanics of APIs.  Really, a consumer just needs to define two things:

  • Defining the data they want
  • Defining the shape they want it

Everything outside of this is an implementation concern - and one that we should be abstracting consumers away from.

If a producer has done a good enough job of documenting their APIs - the data they expose and their "rules of engagement" - then software should be handling the API interaction.  

Consumer patterns need to evolve towards a dynamic knowledge broker that facilitates delivering data, as the consumer needs it, without coupling consumers and producers together. (Spoiler alert: this is what we build, but I'll come back to that).

Producers should be free to evolve their APIs without worrying about the cascading change of all their consumers.  After all - that's what microservices and Agile practices promised, right - small, autonomous teams iterating and delivering at pace.  Instead, what we got was a mess of integration code and change committees to ensure releases don't break consumers.

...[we need] a dynamic knowledge broker that facilitates delivering data, as the consumer needs it, without coupling consumers and producers together.

Federated GraphQL isn't the answer either

Thoughtworks recently phrased this beautifully - GraphQL is certainly "having it's moment".  

GraphQL was designed as a tool for reducing HTTP traffic for Web traffic to a backend, and cherry-picking data attributes from a predefined domain model.  And, for this purpose, it really is a remarkable tool.  

However, GraphQL was never intended as a back-end federation strategy. And, while it seems some tools are pushing in this direction, it's a much less compelling offering.

Importantly, different services across an enterprise have different data domain models - and that's a Good Thing - teams should be free to express ideas in ways that are meaningful to them, regardless of duplication.  GraphQL doesn't try to solve this problem.

And while GraphQL does allow consumers to opt-out of the data they don't care about, accessing new data, or data in a different shape still incurs an integration tax that adds no value.

Globally shared domain models - whether that's a GraphQL schema, or an "Enterprise Domain Model" - is not an effective knowledge distribution strategy:  

  • It restricts how information can be distributed - limiting only to what the schema can express.
  • It requires that all consumers consume data in the same shape.  Attempting to feed data downstream to another?  You're out of luck, it's a one-size-fits-all policy around here, bub.
  • It ultimately results in a (deferred) cascading change, when the schema ultimately needs to evolve.

Summary

APIs are great, if you're a producer.  But consumer patterns haven't evolved, and today lead to tightly coupled services.

Enterprises need new middleware knowledge brokers, that facilitate discovery of data, without tightly coupling producers and consumers, and without incurring the massive integration tax.

In our next post, we'll talk about how to achieve this, using Vyne.