10 Practical Lessons Learnt from Implementing Microservices

Photo by HENCE THE BOOM on Unsplash

I recently created my first microservice architecture for a home project with a couple of friends and thought I might document the lessons we learnt along the way. I wanted to do this from the perspective of someone first trying the approach, rather than the perfect example presented by huge organisations like Netflix. Hopefully, some of the info will be useful for teams looking to start creating their own microservices.

We ended up creating a modest architecture, with six microservices deployed via cloudformation in AWS using containers. As well as the services, we created multiple batch and stream processors, with a react web app for the front end. All of the different components interacted with the different microservices, which bounded all of the domain contexts. No database or cache was ever shared by two different services and no data was requested that didn’t go through a service.

1: Kill All Humans!

I have been in the business long enough to know that automation is key. Fortunately, this means we made some good decisions early on! I would strongly advise that if you don’t have the following in place or are not willing to put in the time in with your first service to rethink the microservice approach. Well, at least put your focus on automation before going down that route. I have put in brackets what we used.

  1. Decent branching strategy (gitflow)
  2. Continuous Integration (bitbucket pipelines)
  3. Continuous Deployment (cloudformation, docker, aws ecs)
  4. Package Management (npm)

You really can’t half do the things above or you are going to spend countless hours repeating yourself. The numbering is on purpose too, as generally speaking it’s good to attack them in the above order.

2: Service Creation Automation

After you create your first service, I suggest that you take a minute to write down a list of everything you had to do in order to make the first one. This will no doubt include things such as creating repos, setting up ci/cd pipelines, static analysis, documentation, security settings, etc, etc. The list will be way longer than you think!

You may decide when you create your second service, to follow all these steps and that it isn’t so bad…. you will probably be right. But you will end up “having a bad time” once you take this approach with your third, fourth and nth number service. The longer you take this approach, the worse it gets and the more of a hurdle there is to spin up a new service.

After creating my fourth, I realised I really needed to invest a bit more in the setup of this process and create a generator that would automatically create all the steps above. That means, whatever supporting tools you use, you need to make sure they have a nice API behind them to make this possible. If you choose the right tools and create a good generator, you are going to save yourself 90% of the work when creating new services.

A service started off taking a couple of days to create and eventually went down to a few minutes.

3: Service/Host Coupling

One concept that is really nice with microservices, is the idea that they can be hosted anywhere. Today they might be hosted using express and using docker, but tomorrow you might want to utilize serverless with something like AWS Lambda.

To make this transition nice and easy, services should be coded in such a way that they are agnostic to the host. This means that authorization, validation and any other logic are encapsulated within the service, with properties like tokens and headers being passed into them.

We learnt a big lesson from this though and that is not to split out the host and the service into different packages. The cost of making small changes to the service and then updated the package every time in the host is not worth the benefit of being a purist and keeping them separate. You will no doubt end up creating some kind of host for integration testing purposes in the service package anyhow, which will look almost identical to your host package. A total waste of time!

4: Service Boilerplate

You are going to end up sharing a lot of code between different services. It goes without saying that you need to apply the ‘Don’t Repeat Yourself’ (DRY) principles here. You often need to balance getting stuff out the door and doing things the right way. In this case though, if you don’t do things the right way, you are going to cause yourself a lot of pain. Whatever framework you create or use, you want your services to only contain the code that makes that service unique. The killer of not doing it this way is actually the unit tests. You don’t want to be testing multiple times within your different services common code that could be shared.

A mistake we made was to use base class methods to do various parts of the service request pipeline (authz, validation, etc), rather than passing through a function to a generic service to be called at the right time in this pipeline. The approach we took was a very similar method used in Web API by Microsoft and at the time I thought it was a good trade-off between flexibility and shareability to do it this way. For microservices, however, this is not the case. You really want to have next to no code that is duplicated. I ended up having pointless copied unit tests and code to test a pipeline that was never likely to change.

In the end, we obviously created our own microservice framework, which gave us full control over our service request pipeline.

5: Host Boilerplate

After a while and as my project began to mature, I realised that there was not only a lot of service boilerplate but host boilerplate too. Stuff like retrieving tokens, adding correlation ids, security headers, logging and configuration. If you are working with any kind of dependency injection, you also end up creating a lot of very similar code to create similar services.

6: Diagnosing Problems

One thing I did from the start, was to implement correlation ids that are passed between my services with a request. This made my life a million times easier when it came to debugging; I can only imagine what a terrible place you would be in if you didn’t do this!

7: API Gateways

One of the big mistakes I made early on was not understanding the proper bounded contexts of my domain. I instead, in true Plato style, tried to apply them to the ‘perfect form’ of a conceptual object like ‘user’ or ‘account’ and not how my product domain thought of them. This meant I ended up stitching multiple different stores and therefore in essence services together.

I realised later on, that I was just creating mini API gateways. To do things differently now, I would create the individual microservices for the domain context and then have an overarching API gateway to create composite objects. This also has a big benefit for mobile applications, where network bandwidth is more of an issue, as you can return only the data that is needed and the traffic becomes less ‘chatty’.

8: Security

If this is your first time working with microservices and especially if you don’t know OAuth 2, OpenID connect or JWT, then you need to not only learn about these things but implement them straight away.

Although I had some knowledge around the area, I did not implement any of it right until the end of the project and caused myself a lot of pain. I had to not only change how a lot of the service boilerplate worked but also the underlying infrastructure. Other components, such as stream processors also had to be updated and changed.

9: Caching Locally

It took us a while to figure out exactly how we were going to handle caching in a microservice architecture. There are a few main places you can cache:

  • The component calling the microservice (batch processor, web app, api gateway, etc)
  • Shared cache of the microservice (e.g. redis)
  • Local cache on a singular instance of the microservice

Within a monolithic architecture, it is quite normal to have a shared cache between multiple instances of the same component. This is especially the case if retrieving and transforming data is expensive. You can speed this up even further, by having a local cache on the individual instances. Within a microservice architecture, we learnt that local caching is not only completely unnecessary but also limits the capabilities of different calling components.

Rather than implementing local caching within the service, the calling component should decide its own caching strategy. So for example, a stream processor dealing with high throughput will want to cache data for at least a short period of time. But components that require the data to be as up to date as possible, but have low throughput (say user roles) can choose not to cache at all. In either case, there is still shared caching going on within the service and so the data stores are not being queried.

10: Polyglot Sharing

We used a number of different languages and tools in our project. They all did some very similar things, like logging, metrics, etc and so we created multiple libraries for the different languages to handle this. This was once again a waste of time.

There are many different approaches you could take to dealing with this, but one we found worked was to create a microservice around one of these common tasks (logging). Then, each component, no matter its language, could just call this service and it would handle it. One major benefit we found later on by doing it this way, is when we added another strategy to where our logs went, no component had to care. We just updated the service and everything started working automatically.

There is one problem with this approach though and that is the additional latency that it can cause, which is going to be a problem for components like stream processors. There are workarounds to this though (such as using queues), which I will leave for another article. But seriously consider what you are going to do for these shared functionalities early on, because we live in an increasingly polyglot world.

Summary

In summary, automate everything, don’t repeat yourself and create a full test service (including security) before creating lots of real ones.

Please share with all your friends on social media and hit that clap button below to spread the word. Leave a response with lessons your learnt. 👏

If you liked this post, then please follow me and check out some of my other articles.

About

Matthew Bill is a passionate technical leader and agile enthusiast from the UK. He enjoys disrupting the status quo to bring about transformational change and technical excellence. With a strong technical background (Node.js/.NET), he solves complex problems by using innovative solutions and excels in implementing strong DevOps cultures.

He is an active member of the tech community, writing articles, presenting talks, contributing to open source and co-founding the Norwich Node User Group. If you would like him to speak at one of your conferences or write a piece for your publication, then please get in touch.

Find out more about Matthew and his projects at matthewbill.gihthub.io

Thanks for reading!

Technical Leader | Agile Coach | Polyglot Software Engineer | Solution Architect | DevOps Enthusiast | Speaker & Writer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Use Alfred to launch Safari URL in Chrome

CS373 Fall 2020: Warren Wang

Jupyter is the new Excel

Creating a Serverless Data Pipeline in GCP

A Data Migration Story, Part II: Implementing a Databricks Migration

Dealing with copyright infringement and plagiarism

Curing Mac’s High CPU + Fan Noise + Heat

Ark Desktop Wallet v1.4.2 Released

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Matthew Bill

Matthew Bill

Technical Leader | Agile Coach | Polyglot Software Engineer | Solution Architect | DevOps Enthusiast | Speaker & Writer

More from Medium

Best Database Management Software for 2022

Just another day at Engineering

Scale Applications by optimizing DNS Configuration

PM2 — Making a good experience with this process manager