Managing configuration information in a complex, cloud-native application can be daunting. There is seemingly configuration everywhere.
- There’s configuration describing the network interconnections in your system, including routing rules and port blocking.
- There’s configuration for your load balancers, determining where to send traffic destined for your service.
- There’s configuration for security permissions needed for databases, caches, servers, third-party applications, and other systems.
- There’s configuration for your application itself, describing database connections, service connections, and various secrets and other configuration values.
In a cloud-native application using a microservice-based architecture, the configuration problem is multiplied. Configurations are everywhere. Some configurations are well-known and managed, sometimes in a revision control system. However, some of these configurations are stored within the system or component that requires the information. And some of it may be simply unknown and completely unmanaged.
Face it, there’s configuration everywhere. And it’s making your application and its infrastructure more complex than necessary.
How do you take control and reduce the complexity associated with this large quantity of configuration information in a cloud-native environment? Here are five best practices to help you get a hand on your configuration mess.
1. Keep a Single Source of Truth for All Configuration
Sometimes, configurations are copied all over the place. A configuration used in one server is copied to another server, then another server. Eventually, there might be hundreds of copies of the same or nearly the same configuration all over the place. The risk that a change to one copy won’t make it to other copies is high. As time goes on, the configurations will drift from each other, leading to configuration errors, application failures, and even security vulnerabilities.
Instead, for each type of configuration information (such as a specific type of configuration file), keep a single master copy of the configuration. This single master copy is where all changes to the configuration should be made. Then, if the configuration is needed in many (or hundreds) of different locations, automate a process to push the configuration out to those other locations.
This best practice keeps all the same (or similar) configurations properly aligned and makes sure that required changes that are made to one are made to all copies.
2. Use Automation To Make Variations
In the previous best practice, for some configuration files, their configuration is almost the same across all devices. Sometimes, small changes are required in each copy of the configuration.
Rather than keeping a separate, unique copy of the configuration for each component, use a single copy and automate the changes for each component using macros and variables.
For instance, consider a specific cloud security policy that requires a copy of a server IP address within the rule. The configuration is virtually the same across multiple use cases. But the rule contains an IP address that changes from use-case to use-case. Instead of creating many configurations, create one configuration with a variable containing the IP address. During the process of deploying the security policy, replace the variable with the proper IP address in each use case.
This automation will ensure that similar configuration files are maintained identically and that change drift doesn’t occur within each use of the configuration.
3. Use Revision Control
Almost all configurations ultimately can be managed as simple text files. Put those text files in your revision control system, such as git. Then, when you need to deploy the configuration change, get the appropriate version out of the revision control system.
This practice will keep a running history of all changes to the configuration. When a change is made, it will keep track of who made that change and the reason for the change (such as from a support ticket or project plan).
This can be helpful when a problem occurs. If a system suddenly starts acting strangely, a simple check of the revision control system can see what changes have occurred in the configuration recently. Those changes, if in error, can be rolled back. Additionally, the changes can be analyzed to see why they caused a problem to avoid future issues.
4. Centralize Your Configurations
As a corollary to the previous best practice, centralize all your configurations in a single, well-known location. Using a revision control system to manage your configurations makes this easier to do.
By centralizing your configurations, you’ll never have to guess where the “primary source” configuration for a given component exists. They will be easy to review during problem situations, and if the location is well known, it will be more natural to use and expand. If a need for a new configuration file crops up, it’ll be easier to follow the best practices outlined above if the other configurations are all in the same location. This reduces the likelihood of untracked configurations being used out of convenience, and it ensures that everyone follows the best practices and maintains accountability.
5. Use automated systems to distribute configuration information
Your configurations should be deployed using automated systems, just like your software is deployed. Using a CI/CD pipeline is standard policy for modern cloud-native applications, and that deployment pipeline should deploy all configurations along with all code. This enables the consistent application of variable and macros changes described above, and it ensures that all changes are deployed across all components that require the configuration. Automation ensures consistency and accountability in managing large quantities of configurations.
Configurations Are Assets
Configurations are as valuable as code, and incorrect changes can be just as dangerous to your application. You should treat your configurations with the same level of discipline that you apply to your source code.
By using shared, centralized, templatized configurations that are tracked by version control and automatically deployed, you will go far in reducing the configuration complexity inherent in modern cloud-native applications.
This article is a reprint of my article from Sept 18, 2023 (revised).
Frequently Asked Questions (Ask SAI)
Do you have questions about software architecture, cloud computing, or application modernization? Make sure to ask SAILee!
SAILee is an AI-powered automated assistant that will provide high-quality, professional replies to questions based on the books, articles, courses, and other content created by Lee Atchison. Here are some sample questions and answers about this article, asked and answered by SAILee:
Here are three questions readers might have after reading the article that are not explicitly answered:
- What tools or platforms are best suited for implementing these configuration management best practices in a cloud-native environment?
For implementing configuration management best practices in a cloud-native environment, several tools and platforms can be highly effective. For example, tools like HashiCorp Consul and AWS Systems Manager Parameter Store allow for centralized and automated management of configuration data. CI/CD pipelines can be integrated with tools such as Jenkins, GitLab CI/CD, or GitHub Actions to automate configuration distribution alongside code deployments. Configuration as code (CaC) tools like Terraform and Pulumi are particularly helpful for managing infrastructure and application settings in a systematic, version-controlled way. - How should sensitive configurations, such as secrets or API keys, be managed securely in alignment with these best practices?
Sensitive configurations, such as secrets or API keys, require additional measures to ensure security while adhering to the best practices outlined. These should not be stored in plaintext files or traditional revision control systems. Instead, dedicated tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault are ideal for securely managing sensitive configuration data. These tools provide encryption, access control, and audit logs, ensuring that sensitive information is both accessible and protected. When integrating secrets into automated pipelines, it’s crucial to use environment variables or secret management integrations offered by the pipeline tools. - What are some common challenges or pitfalls when transitioning to a centralized configuration management system, and how can they be mitigated?
Transitioning to a centralized configuration management system can present challenges. Teams might encounter resistance due to the initial learning curve or a reluctance to move away from long-established practices. Legacy systems may have decentralized configurations that are difficult to consolidate. To mitigate these issues, it is important to involve all stakeholders in the planning phase, provide training, and start with a pilot project to demonstrate the benefits. Gradually integrating these practices, rather than implementing them all at once, can reduce disruption and allow teams to adapt incrementally. Additionally, having clear documentation and support during the transition phase can alleviate confusion and build confidence in the new system.
