Engineering @ Bitpanda: Improving software releases by using feature flags

In this article, I want to give you an overview of feature flags, what use cases we see and some considerations on how to manage them properly. In a follow up article, we will run through an example on how we integrated a feature flag system into our services. Let’s get started.

Bitpanda’s Engineering department is organised into scrum teams and each team is contributing heavily to the Bitpanda platform. All teams aim to deliver working software in a reliable and fast manner, so users receive features frequently and can use them early.

Deploying often and reliably requires a high confidence in development quality and, of course, good practices. One of the ways to minimise issues and release features in a confident manner is to use feature flags. Feature flags are also known as feature toggles, feature switches and some other names.

What are feature flags?

Feature flags allow functionality inside a software application to be enabled or disabled without deploying code. Some systems use feature flags via a REST API and, most of the time, an additional web interface for changing the state of the flag. With feature flags, the software deployment process is completely decoupled from feature releases. That means new features can be integrated gradually into the application and be deployed without being visible to the users.

Main reasons to use feature flags

There are many reasons why you should use feature flags and how you can benefit from them. In this article, I want to highlight the most important ones.

1 - Breaking down big features into small manageable chunks

Big feature releases are scary. Often, a lot of people from different departments are involved. Many development teams are working together and have to integrate their services with each other. Additionally, everything needs to work properly so that users are happy. As the user base of Bitpanda grows steadily, this tends to get harder to achieve. But how can feature flags help here?

First, the feature can be developed in smaller chunks and be integrated in the code base without breaking existing logic. This also means that the merge requests/pull requests can be smaller and live for a shorter period of time. Smaller changes are easier to review by team members and more team members are willing to review it as it can be done in some minutes instead of many hours. Do not expect your colleagues to sit down for a whole day in order to review a huge refactoring with a bunch of new features.

Second, work can be distributed better inside one team or over multiple teams when using smaller chunks. Let’s take the Bitpanda Crypto Index feature as an example. If only one team were to implement the feature, it would probably not be finished today. To achieve big results, all Bitpanda teams have to contribute their parts and deliver a piece of the puzzle guided by a feature flag, so users see it when it is done.

2 - Decouple deployments from feature releases

Software deployments and feature releases are decoupled from each other, which allows software engineers and the business to move at their own pace. Software engineers are able to make the application stable and scalable before the marketing department starts their efforts to promote the new feature.

In combination with the above point of releasing smaller updates, every software update is easier to identify. If an error slipped through the review, the change can be rolled back immediately and fixed, before it has an impact on users. Additionally, the software engineers have the context of the change in mind and its resolution might be much faster.

The whole industry tends to move to the practice of deploying often. In the “State of DevOps” report, there is a nice graph illustrating how high-performing teams release multiple times a week and produce fewer issues over time.

3 - Test in production

Testing in production? Yes, you read it right. Sometimes the test environment does not represent the full spectrum of functionality that is available in production. This could be the amount of traffic generated, where it is hard and costly to run a test environment at the same size as the production environment. Or, in our case, it is more about third-party integrations.

At Bitpanda, we support a lot of payment providers and blockchain nodes which do not always support a test environment that is a replication of the production environment. It is even sometimes the case that the vendor test system behaves differently to their production system.

For this purpose, it makes sense to enable the new provider or blockchain nodes only for employees to test it properly, before everyone is able to use it.

Full integration can be tested up front and the rollout can happen gradually to more and more users. It is important to keep an eye on monitoring and support tickets during this time. With this approach, issues can be found early and if something goes wrong only a small percentage of users are affected.

Using feature flags

To use feature flags yourself, you can either create your own implementation, or use one of the available feature flag services. Every application has to integrate the system separately. Mobile apps, the web frontend and the backend services must be able to access the feature flag system.

When Bitpanda was small, we had our own implementations for each of our products.

The amount of products and the engineering department grew, therefore we are currently in the process of using a common system and want to move all products to use LaunchDarkly. LaunchDarkly provides a highly-available API and a UI for managing feature flags, plus some advanced features.

We will run through an example of how we utilise LaunchDarkly in a follow up article. If you want to host a system yourself or you want to use an open-source tool, that is also possible. For example, Unleash is a popular open-source feature flag system which is also integrated with GitLab.

Considerations when using feature flags

Now that we know the benefits of feature flags we should also look at some downsides and how to avoid them.

One consideration on using feature flags properly is to have good test coverage and a continuous integration pipeline that executes automated tests on every code change. Using this, the software engineering team can minimise the risk of breaking functionality on new code changes. Running automated tests is good practice anyway and regarding feature flags, every permutation should be tested to ensure compatibility. The drawback here is the increased complexity of maintaining additional tests and permutations.

To keep complexity in check, it makes sense to not have too many feature flags and to ensure that they are not dependent on each other. To keep the amount of feature flags low, they should be as short-lived as possible and then be removed when the feature is released fully. I would recommend keeping a ticket in the product backlog for removing the flag. Additionally, some feature flag systems can alert if the feature flag is ready for removal as no application is attempting to fetch the state.

Another complexity issue is dependent feature flags. They should be avoided! It makes it really hard to know the state of the system. Try to keep the structure clean and tidy. If a feature needs a new REST API endpoint, then secure the endpoint with a feature flag. If the calculation logic changes, try to create one code path where the decision between the new and old logic happens.

Who is responsible for enabling/disabling features?

As software deployments and feature releases are better decoupled by using feature flags, the responsibility for enabling/disabling features can be shifted. The product owner could get the permissions to rollout a feature in coordination with other departments. Each team has to decide on its own if they want to have this and make sure to have some audit in place to trace what feature has been enabled when. We aim for bug-free releases, but the possibility is always there that some bugs can sneak in. Even though automated tests are in place and the feature was tested properly. With good coordination and great monitoring, it is very likely that your next feature release will be successful.

Summary

Feature flags can help to increase the number of successful feature releases quite significantly by decoupling the software deployments from feature releases. The process fits better into the continuous delivery approach with higher success rates. Releasing small parts step by step makes it easier to keep track of what is going on. If the number of feature flags stay limited and the application has a great test coverage, there is no reason to not use feature flags.