Every entity that is part of an IT infrastructure has a unique identity. Users and devices/machines have their own identities. Secrets Management can help authenticate the identities of users and devices/machines, verify them before they are authorized, and keep the identities secret. Secrets Management helps organizations secure their IT infrastructure.

Secrets and Secrets Management

Before we discuss Secrets Management, we should explain what Secrets are. Most of us know that passwords are supposed to be kept secret. However, you should be aware of the distinction between Secrets and sensitive data. Sensitive data includes personal information such as someone’s name, date of birth, social security number, and salary. Secrets, on the other hand, are digital authentication credentials that grant access to certain resources, systems, and/or data. The following are examples of Secrets:

• Database connection strings
• System-to-system passwords
• API keys
• SSH keys
• Encryption keys
• X.509 certificates

The above Secrets can initiate communications between users and devices/machines. Communications from users to machines, and from machines to machines use these Secrets to ensure they are trusted entities.

Secrets Management involves a set of centralized tools, methods, and workflows organizations use to securely store, retrieve, and manage their digital authentication credentials or the Secrets mentioned above.

Why Secrets Management is Important

The main benefits of using Secrets Management include:

• Security. Not only is your data encrypted, but your data storage is also encrypted. This data encryption is one-way encryption. If a Secret is exposed, breached, or discarded, the only way to retrieve it is to create a new one.
• Audits. Organizations may have their systems externally or internally exposed or breached due to leaked Secrets. Audit logs are essential for identifying and understanding these leaks, and for preventing similar leaks in the future.
• Centralization. As mentioned earlier, secrets sprawl occurs when Secrets are scattered throughout the organization. Keeping Secrets in a centralized location enables organizations to take swift action when Secrets are exposed or breached. In addition, Secrets centralization means that organizations don’t have to depend on teams to locate them.
• Lease, revoke, rotate. This is one of the most important features of Secrets management. Some security compliance requirements specify that Secrets must be rotated within a specific period of time. Secrets Management solutions enable organizations to revoke all Secrets that are breached or exposed (with the help of audit logs and centralization features). In addition, Secrets management assists organizations in revoking passwords when an employee leaves.
• Policies. You don’t want all employees to have access to all resources. For example, junior developers might need read-only access to production databases rather than write access. System admins might not need access to database systems, but SREs do. And so on. You can use policies to grant appropriate access. The ability to define policies for each employee or team member based on their particular requirements helps organizations keep their systems secure.
• Access levels. Secrets Management allows organizations to grant full or limited access to resources, enabling organizations to follow the principle of least privilege.

Types of Secrets

By definition, Secrets enable a user to gain access to other resources, systems, and/or data. Let’s take a look at the different types of Secrets that are out there.

Authentication Secrets

According to the NIST (National Institute of Standards and Technology), “authentication secrets” is a generic term for any Secret value that an attacker could use to impersonate a subscriber in an authentication protocol.

This type of Secret is to authenticate or verify the identity of the entity who initiates the communication. Once all checks are passed or authenticated, the initiator is authorized or granted access to specific resources, systems, and data based on the request.

Encryption Secrets

Similar to Authentication Secrets, the main idea of having Encryption Secrets is to authenticate users, devices/machines, or systems. Encryption Secrets allow organizations to encrypt their Secrets with encryption keys. There are several different types of encryption that organizations can use:

• Symmetric encryption, or Secret key encryption, uses one key for encrypting and decrypting Secrets.
• Asymmetric encryption, or public key encryption, is encryption using two keys. This method uses public and private keys. Digital certifications that are signed by Certificate Authorities (CAs) can distribute the public key. Only individual owners should know private keys.
• Hashing is one-way data/Secret encryption that is irreversible. If you do not remember your passwords or Secrets (and there is no “forgot password” feature or mechanism to receive a hint), the only way to “update” your Secret is to create a new one.

Application Secrets

Application Secrets are the most familiar type for most IT personnel and developers. As the name implies, these Secrets authenticate users or systems, allowing them to gain access to other systems. Many developers think that allowing DevSecOps to manage these Secrets could slow down their development process, so they tend to store them as part of their development code, either by hard-coding them in the application logic or by storing them in a config file (like config.php, config.xml, or config.json). Storing Secrets this way is a bad practice that could lead to compromised or breached applications and/or systems.

Secrets Management Solutions

At this point, you know the importance of Secrets Management and the different types of Secrets that can lead to compromised or breached systems if they fall into the wrong hands. It becomes evident that Secrets Management should be part of your organization’s IT playbook.

How Secrets Management Solutions Work

From the previous section, you know that storing or hard-coding Secrets in your application is a bad practice. Be wary of storing them in version control tools like Git, for example. Hackers can scan GitHub repos to find Secrets you’ve committed (intentionally or unintentionally) and other similar information. Then, they can use those Secrets to do something malicious.

Secrets Management solves these issues. A Secrets Management solution safely stores all your Secrets in a centralized location and allows you to apply policies and access levels. It also gives organizations the ability to lease, revoke, and rotate Secrets when systems are breached/compromised or as part of meeting their compliance requirements (such as having to rotate passwords every 60 days).

Having a Secrets Manager is also very helpful for developers since they’ll always know where to retrieve or save Secrets. Developers can utilize Secrets Managers for working locally, for CI/CD, and every step of the software development lifecycle.

Key Management Systems

It is a best practice to secure your data and Secrets by using encryption (either in transit or at rest). In key management systems, you create cryptographic keys for encrypting and decrypting your data/Secrets. You can attach a policy to these cryptographic keys that will dictate who can access your data as well as how and when.

Some Key Management systems provide the ability to identify Key Management actions, lifecycle events, and usage. These features help organizations increase their security posture and audit their systems as part of their compliance requirements.

Secrets Management Platforms

Cloud and on-prem infrastructure, Kubernetes, and containers all need to interact with Secrets. There are several Secrets Management platforms and tools on the market. We’ll discuss a few of them below.

HashiCorp Vault is a centralized, identity-based Secrets and encryption management system. It encrypts data at rest (inside the Vault) and in transit, and it also offers ACL capabilities that enable organizations to write access policies based on their needs. The Vault’s audit trail gives organizations more visibility into how their Secrets are being used, accessed, and rotated, among other things.

Common use cases for Vault include:

• Secrets Management
• Dynamic Secrets
• Kubernetes Secrets
• Database credential rotation
• Automated PKI infrastructure
• Identity-based access
• Data encryption & tokenization
• Key Management

With web applications, for example, it’s possible Secrets can be exposed by the way developers write an exception (by throwing the username and password), by a logging system that captures the API key, or by an application monitoring performance system that shows the connection strings or encryption keys. While you can’t entirely prevent this from happening, you can use HashiCorp Vault’s dynamic Secrets to create short-lived and temporary credentials to mitigate the problem.

There is a lot of documentation and tutorials to help you understand and master Vault. One of the most important things to know about Vault is that it uses tokens, which are associated with policies. You can define policies based on your organization’s needs. Such policies can be for read, write, create, update, and/or delete, or a combination of them. Vault also gives you the ability to revoke, renew, and rotate the tokens based on your needs.

You can use HashiCorp Vault in cloud-based or non-cloud infrastructure, and it is cloud provider agnostic and supports many third-party integrations.

Cloud-Based Secrets Management Services

While HashiCorp Vault is not cloud-vendor specific, several cloud-based Secrets Management services are available, including AWS Secrets Manager and Amazon Simple Systems Manager (SSM).

AWS Secrets Manager is a centralized AWS Secrets Management system that allows your application or system to create, delete, rotate, encrypt, decrypt, and use Secrets through API calls. Instead of hard-coding passwords in your code, you make API calls to AWS Secrets Manager to retrieve your credentials, API tokens, or other Secrets.

AWS Secrets Manager offers features including:

• Secrets/password retrieval at runtime. You can dynamically retrieve stored credentials from AWS Secrets Manager when you need them.
• Stores different types of Secrets. AWS Secrets Manager allows you to store encrypted Secrets and database connection strings.
• Encrypts Secret data. AWS Secrets Manager uses AWS Key Management Service (KMS) to encrypt Secret data.
• Automated Secrets rotation. Some compliance requirements dictate that you rotate passwords or Secrets. AWS Secrets Manager allows you to set up and schedule Secrets rotation without user intervention. You define the Secrets rotation with AWS Lambda functions.
• Database Secrets rotation. AWS Secrets Manager supports database Secrets rotation for Amazon Relational Database Service (RDS) databases such as Amazon Aurora, MySQL, PostgreSQL, Oracle, MariaDB, and Microsoft SQL Server (these are configured in Amazon RDS ).
• Supports rotating Secrets in other AWS services. AWS Secrets Manager supports rotating Secrets in Amazon DocumentDB and Amazon Redshift, for example.
• Access Control. You can attach IAM policies for fine-grained control over who (users, groups, and roles) can access what (Secrets). For example, you could attach a fully-managed policy to a specific group while applying read-only policies to others.
• Compliance. AWS Secrets Manager adheres to compliance standards, including HIPAA, ISO, Payment Card Industry Data Security Standard (PCI DSS), System and Organization Control (SOC), Federal Risk and Authorization Management Program (FedRAMP), Department of Defense (DOD) Cloud Computing Security Requirements Guide (SRG), Information Security Registered Assessors Program (IRAP), and Outsourced Service Provider’s Audit Report (OSPAR).

AWS Systems Manager (formerly Amazon Simple Systems Manager — SSM — and Amazon EC2 Systems Manager) is an agent-based application, meaning you must install an SSM agent in your systems. When you are about to issue a command, the SSM agent, acting as a client, will initiate the connection to the SSM server and process the command that the SSM server receives. Once the SSM server completes the command, it returns the result to the SSM agent, and the SSM agent passes the result to the client application or system.

SSM allows you to update, manage, and configure your resources: e.g., Amazon EC2 instances, edge devices, on-prem servers, and virtual machines. The communication between these entities is through SSM agents. Some examples that you can do through SSM by using IAM policy are restricting root-level access command, specifying tag in IAM policy to restrict user’s ability to run commands, and restricting access based on tags.

There are a few critical differences between AWS Secrets Manager and AWS Systems Manager (SSM). SSM does not support key rotation or policy attachment; it also has a 4KB size limit (while AWS Secrets Manager has a 10KB limit). So, before choosing which one to use, make sure that you know what your application will need (for example, whether you will need to rotate Secrets, how big the Secrets will be, and whether or not you must meet other compliance requirements).

For Secrets Management in Kubernetes, a Secret is a type of object used to store tokens, SSH keys, certificates, credentials, and other things that can be considered Secrets. To prevent Secrets sprawl, Kubernetes Secrets are separate from the application code that defines the Pod lifecycle. So, as Pods come and go, the Secrets are still intact. A newly created Pod simply mounts the Secrets (data) as a volume. Characteristics of Kubernetes Secrets include:

• Kubernetes Secrets are base64 encoded.
• They can be mounted as data volumes.
• They can also be defined as environment variables.
• They are namespaced.
• Their size limit is 1MiB.

To properly secure Kubernetes Secrets, you must utilize role-based access control (RBAC). With RBAC, you can control access to Kubernetes Secrets based on user roles. You can also use ClusterRoles to define what users can perform (verbs) within a cluster and a namespace.

The following example from the Kubernetes website shows how a ClusterRole can be used to grant read access to Secrets in any namespace or across all namespaces (depending on how it’s bound):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: secret-reader
rules:
- apiGroups: [""]
  #
  # at the HTTP level, the name of the resource for accessing Secret
  # objects is "secrets"
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Best Practices for Secrets Management

Effective Secrets Management boils down to following best practices. When planning to implement Secrets Management, you must consider your organization’s needs, security requirements, and ease of use of the solution you choose, among other things. Let’s narrow the list down to the four most important things:

• Secure storage and distribution. When choosing between Secrets Management solutions, you should focus on all the great features they offer, and how they store and transmit data and Secrets. Encrypt data and Secrets at rest and also in transit.
• Access control and governance. Access control and policies have become integral to IT infrastructure, and one could argue that they’re the most critical part. They give your organization the ability to limit who can access what. You can create any policy (such as granting read-only, read/write, or admin access) that you need to secure your Secrets, data, and infrastructure.
• Regular rotation and expiration of Secrets. There might be a time when one of your team members leaves the organization, but the API key or other vital credentials belong to that individual, and no one else on the team knows them. Secrets rotation is essential not only for this scenario but also for preventing the possibility of a system compromise.
• Auditing and monitoring for compliance. Auditing and monitoring enable organizations to know which Secrets were accessed, how, and by whom. With this information, they can improve their security posture by adding or modifying access control and policies. Besides, many organizations must implement auditing and monitoring to meet compliance requirements.

Conclusion

Secrets Management helps organizations secure their systems and infrastructure. Users-to-Machines and Machines-to-Machines communications require identity checks. The identity of Users and Machines could be passwords, API keys, encryption keys, or certificates, and they are called Secrets. Implementing Secrets Management could be a daunting task to start. If Secrets Management is your organization’s next priority, you should start small for a more significant impact. Start by securing Secrets using the tool described above. Then improve your Secrets Management by having at-rest and in-transit data encryption and implementing access through policies; regular Secrets rotations should be next on your list; then auditing and monitoring Secrets behavior to help your organization analyze your infrastructure security posture.