At Redox, we’re taking a pragmatic approach to building microservices, especially in how we handle cross-cutting concerns. Most refer to this part of a microservice ecosystem as the chassis—things that every microservice needs to have in order for us to be confident that it’s a quality product. The chassis includes:
- Logging
- Metric reporting and instrumentation
- Fault tolerance (a-la Hystrix)
- Error reporting
- Tracing (especially through multiple services)
- Defaults for everything (how to build a project, run, test, format code, etc.)
And that’s just the first six I could think of. To complicate matters, you need a chassis for each language/platform you have in your stack.
At this point, you’re either feeling overwhelmed or sympathetic (because you’ve been through this). There are numerous frameworks out there to make things easier, such as Gizmo, Finagle, or Microdot, each of which has various levels of convention over configuration type decisions they’ve made for you.
Note: Our Microservices Journey, Part One: The Beginning and From the Engineering Blog: Redox Microservices Journey Part Two, Principles have great stories about parts of this microservice project.
Make it easy to do the right thing
Frameworks ultimately exist to make it easy for developers to do the right thing. Microservice frameworks like those above need to reflect the existing values and processes of the organization; if they don’t fit, or they don’t match pieces of the infrastructure that you already have, you need to either fix them or go your own way.
That’s the main reason we’re not using one of these frameworks. We already have a very robust infrastructure that handles some of these concerns, and we’ve built tools into the monolith which are generally extensible outside the monolith.
In general, these cross-cutting concerns fall into two buckets:
- Functionality that you get for running things on the Redox infrastructure for free
- Functionality that requires developers to implement a pattern in software
An example of the first is logging—anything logged in any of our services automatically gets piped to a log aggregator.
An example of the second is using Hystrix—we wrap common calls in libraries that make this easy to do, but developers still need to know how to design a circuit breaker, and so they use Hystrix for new endpoints.
Make it desirable to do the right thing
Another feature of frameworks for microservices is the “implement an interface” pattern. In other words, you must implement things in an opinionated way, and then you get a bunch of other stuff for free. Instead of making it easy to do the right thing, this approach makes it impossible to do the wrong thing—your code won’t compile.
There’s nothing wrong with a more opinionated approach, but there are tradeoffs. When I was a developer at Epic, I was impressed by their desktop and web frameworks. As a newly minted Epic developer, you slap together a UI, implement a few interfaces, and your new view can be integrated with the monolithic application. Allowing thousands of developers to work on the same platform is a testament to good architecture. Despite this, the only option was VB6 or the Epic version of web development (lots of server-side C# rendering).
Redox faces a lot of challenging healthcare problems and we need to be able to pick the right tool for the right job. We love serverless (see AWS Lambda has been HIPAA Eligible for a Month, and it’s Awesome), and we also love Docker, which makes it relatively easy for us to build new things in new languages.
For us, it makes more sense to not enforce a top-down set of rules, but rather make it desirable to do the right thing. These can come in the form of carrots or sticks, but the underlying theme is to make a real, tangible incentives for people to do the right thing.
Examples of making it desirable to the right thing include:
- (Carrot) Standing up the phenomenal Hystrix dashboard so that developers can see the impact of doing the right thing
- (Stick) Reject requests that don’t have tracing included so that it will never work
- (Carrot) Good starting points, especially when it comes to defaults
- (Stick) Detect errors from the logs and page developers when they happen
First Steps
Our approach to building these incentives is already underway (and dovetails nicely with future posts in this series). Our first steps looked like this:
- Have a few developers sit down and hash out differences between our existing microservices. If you don’t have an inventory of services, start with that.
- Find the differences and standardize (hopefully you pick developers open to compromise). For example, we had a long discussion about compiling typescript vs ts-node. (Drop me a line if you want to hear what we decided.)
- Put all of the recommendations into one service—it could be a real one, or just a simple service that is intended to be copied from.
- Identify things that need more time and make sure people track that work as development
You can start to identify these patterns and defaults you want to use anytime, and integrate them as your infrastructure, security, CI/CD and other essential components come online.
Oh, and if you’d like to help us in our microservices journey, we’re hiring!