AWS Lambda Feature Toggles Made Simple
“Feature Toggles are a powerful technique, allowing teams to modify system behavior without changing code…. Toggles introduce complexity. We can keep that complexity in check by using smart toggle implementation practices and appropriate tools to manage our toggle configuration, but we should also aim to constrain the number of toggles in our system” — Martin Fowler
Feature toggles (or feature flags) are a powerful tool but as Martin Fowler mentioned in this article, they can also be quite hard to implement.
In this blog I’ll demonstrate how feature toggles in AWS can become a simple task and a great tool that you must include in your AWS Lambdas.
I’ll implement both static and dynamic feature toggles. I’ll demonstrate that by using Python, AWS AppConfig service, AWS Lambda Powertools and Pydantic, you can start using feature toggles in your lambdas with ease right away.
Static hardcoded feature toggles
A simple implementation for static feature toggle would be to use Lambda environment variables. Their values are set during Lambda deployment time and can only be changed by a new Lambda deployment CI/CD pipeline.
You can find below a cool one liner trick to parse & validate Lambda environment variables with Pydantic.
In this example, we have 2 environment variables: env_var_1 and env_var_2. They are modelled into a Pydantic schema class in line 4 as two strings parameters.
In line 18, the lambda_handler function calls the check_env_vars which parses the os.enviorn variable (line 15) and extracts from it only the defined environment variables, env_var_1 and env_var_2. In case of a missing parameter, a detailed ValidationError exception is raised.
CONF is now initialised with the environment variables and can be used across the lambda code in the same manner it’s used in line 19, as a dataclass.
You can read more about Pydantic and it’s parsing prowess in my first blog.
Ok great, this works BUT…
The major problem with this approach is that you can’t change the feature flags without redeploying the entire lambda (hence the term static) which means going through your pipeline which takes time. A lot of time.
There’s a better way, use dynamic feature toggles. Use AWS App Config.
Dynamic Feature Toggle Configuration
“Using a general purpose DB which is already part of the system architecture to store toggle configuration is very common; it’s an obvious place to go once Feature Flags are introduced and start to gain traction. However nowadays there are a breed of special-purpose hierarchical key-value stores which are a better fit for managing application configuration…..
These services form a distributed cluster which provides a shared source of environmental configuration…. Configuration can be modified dynamically whenever required..” — Martin Fowler
What is AWS AppConfig?
AWS AppConfig is a self managed service that can be used to store plain text/YAML/JSON configuration in AWS to be consumed by multiple clients. The consumers can be containers or lambdas. We will use it in the context of feature toggles but it can store any other configurations that you might require.
How does it work?
AppConfig is consisted of configuration hierarchies. First, you create an application.
An application has a list of environments (dev, stage, production etc.)
Then you create an environment.
Each environment can have multiple configurations profiles.
Each profile defines the current version of a configuration, its’ values (in JSON/YAML/plain text format) and the deployment strategy to use when deploying it. You can then finally deploy the configuration profile which is listed in the deployments tab. Our configuration is consisted of 3 parameters: 2 booleans and one string.
You can read more about deployment strategies here.
I’d like to stop here and not dive too deep into AppConfig specifics and move on. AppConfig is awesome and you should read more about it here.
Let’s consume the configuration
Now that we got the basics out of the way, let’s assume for a second that we have deployed our application’s configuration as a JSON document which has all the feature toggles, and now want to consume it in our lambda.
The lambda will fetch the configuration from AWS AppConfig. If a new configuration is deployed into AppConfig, the lambda will fetch the new configuration and change its’ behavior (according to the fetched values) without the need to redeploy the Lambda itself, hence being dynamic.
This sounds simple, but as Martin Fowler stated, this is no easy feat. So how do we do that?
AWS Lambda Powertools to the rescue!
Recently, I had the pleasure of contributing a new AppConfig parameter utility code to an amazing project on Github: AWS Lambda Powertools.
This repo, which started mostly Python oriented (but now supports other languages such as Java and more to follow) provides an easy to use solutions for lambda logging, tracing (with CloudWatch metrics), SSM utilities, and even validation and advanced parsing for incoming AWS Lambda events. Check out my previous blog on the latter.
The new AppConfig utility provides an easy way to consume configuration from AWS AppConfig and save it in a dedicated local cache. The cache also has a configurable TTL.
Once the configuration is received, we will parse it with Pydantic and create a singleton global instance of the feature toggles schema in a similar way we did with the environment variables CONF instance.
Building upon our first environment variable example with Pydantic, we will now add 3 new environment variables: application name, environment name, and configuration name which define the configuration we wish to extract from AWS AppConfig.
These variables will determined during lambda deployment. If you deploy to a dev environment — environment name should be ‘dev’, for test ‘test’ and so forth. This will allow you to deploy to multiple environment while using a different configuration per environment.
In our case the configuration name is ‘test-application-profile’, the application name is ‘test-service’ and the environment name is ‘dev’ which are represented by the APP_CONF_APPLICATION, APP_CONF_ENVIORMENT and APP_CONF_CONFNAME environment variables at lines 8–10 below.
Now let’s define our feature toggles schema. We define 2 boolean toggles: feature1, feature2 and 1 string log_level parameter. I’ll explain later on why these are defined as Optional with a default value.
Now, let’s import AppConfigProvider class from Lambda Powertools and use it to load a configuration from AWS AppConfig according to the environment variables: APP_CONF_APPLICATION, APP_CONF_ENVIORMENT and APP_CONF_CONFNAME.
In line 8, AppConfigProvider is configured with the AWS AppConfig parameters: application and environment.
Line 11 is the actual call to AWS AppConfig. The utility AppConfigProvider get function is used to retrieve a specific configuration, to store it in its’ internal cache for 600 seconds and to also return the value as a JSON Python Dict object which will be saved to conf_json variable.
Each configuration ‘get’ call to AWS AppConfig costs money, which makes the cache option very useful. As long as the 600 seconds TTL haven’t passed from the first ‘get’, you wont connect again to AppConfig and won’t be charged. Very nice indeed.
Line 14 causes the configuration Dict, conf_json to be parsed, validated and loaded into the FEATURE_TOGGLES global class instance of type AppConfToggles. In case of an error, a Pydantic ValidationError exception is raised with a detailed error message.
Putting it all together
Now that we have consumed the feature toggles configuration, parsed and validated them, we can load them at line 53 by calling load_feature_toggles and then use them in lines 56 and 59 accordingly.
How do I change my AppConfig configurations?
I’m a firm believer in the configuration as code methodology. It allows you to track your changes, automating the feature toggles updates process and eliminating manual changes in production.
Following this logic, the AWS AppConfig creation and configuration must be managed though a different code repository and CI/CD pipeline. This separation will allow to deploy changes to AppConfig without triggering the entire Lambda pipeline and make this solution truly dynamic.
Feature Toggles Use Cases
Keep in mind that Any change must be reflected in both repositories. The Lambda repo will define the Pydantic schema (AppConfToggles) that will load the supported feature toggles. The toggles’ repo will define and push the toggles configuration values into AWS AppConfig. This creates a coupling between the two repositories.
However, since the toggles are defined in the AppConfToggles class as Optional with a default value, toggles can be removed or added without fear of breaking the code during runtime. This definition minimizes the coupling.
Adding a new feature toggle
- Add a new toggle i.e feature3: bool to the toggles repo with a value. Run CI/CD pipeline, update AWS AppConfig.
- AppConfToggles doesn’t include feature3 yet so it won’t crash when parsing the new configuration from AWS AppConfig (it ignores feature3 since it’s not defined in the AppConfToggles schema). Add the new toggle to the AppConfToggles class with type Optional and set a default value. Run CI/CD pipeline, deploy the lambda.
Modifying a value of an existing toggle, i.e feature1
- Change feature1’s value in the toggle’s repo. Run CI/CD pipeline.
- Lambda’s repo doesn't require any change. It will load the new value once the AppConf provider’s cache expires (600 seconds in our case).
Removing a toggle, i.e feature2
- Remove feature1 from the toggle’s repo. Run CI/CD pipeline.
- Since all toggles are defined as Optional with a default value in the Pydantic schema, removing feature1 from AppConf won’t break the Lambda in runtime. Remove feature1 from the Lambda repo, run CI/CD pipeline.
As you can see, all possible changes are safe and won’t crash your Lambda during runtime.
Summary
In this blog post we implemented two types of feature toggles, static and dynamic.
For the dynamic version, we used AWS AppConfig in combination of the excellent PowerTools utility to easily consume and validate the feature toggles.
Pros:
- We Didn’t write new code or services or even a DB. We used existing proven tools (Pydantic, Lambda Powertools) and self managed services (AWS AppConfig).
- Feature toggle changes are dynamic and take short time to deploy via a dedicated & tracked CI/CD pipeline.
- Once the new toggle’s configuration is deployed and AppConfigProvider cache expires, the lambda will use the new configuration.
4. No change can break your lambda during runtime in any scenario.
Cons:
- The solution requires to maintain another code repository with a dedicated CI/CD pipeline.
- Each feature toggle change requires a PR and a code review (can also be listed under pros).
I hope you found this blog useful, feel free to leave any comment below.
If you liked this blog, you should definitely check out part of this blog where we continue to improve this solution. Click here.
Acknowledgements
Martin Fowler’s article
Alon Sadovski
Heitor Lessa
Want to learn more about Serverless?
Check out my website https://www.ranthebuilder.cloud
About Me
Hi, I’m Ran Isenberg, an AWS Community Builder (Serverless focus), a Cloud System Architect and a public speaker based in Israel.
I see myself as a passionate Serverless advocate with a love for innovation, AWS and smart home applications.
Connect with me at https://www.linkedin.com/in/ranisenberg/
