The Top 6 Kubernetes Security Issues and How to Fix Them

Ahhh, Kubernetes. Can't live with it, can't live without it.

On the one hand, it's the single most popular platform to manage containers, and for good reason. It's incredibly powerful, endlessly flexible, and easily portable across various infrastructures and environments.

But on the other hand, it can be painfully complex. "K8s configurations are not living, majestic trees," says Regis Wilson, founding engineer at Release. "They are a bunch of dead chopped wood."

Suffice it to say that most developers (and security professionals) share a love-hate relationship with everyone's favourite container orchestration system.

To make your lives easier, we've compiled a list of 7 of the most common security issues found on Kubernetes, plus some best-practices on how to get rid of them.

Thank us later.

1. Access Control & Privilege Escalation

One of the most common attack vectors you can expect in Kubernetes is privilege escalation. This happens when a user gets access to resources or data they're not authorised for.

Most of the time, attackers are looking for ways to escalate privileges in your container so they can easily access, modify, or steal data from your system.

How does privilege escalation work in Kubernetes?

Kubernetes uses a system called Role-Based Access Control (RBAC) to determine which users get access to what.

For example, user "Harold" has read-only access to Kubernetes pods, while user "Kumar" has access patch, update, or delete pods.

Different users get assigned different 'roles', so in this example, Harold is assigned the role of 'pod-reader', while Kumar is in the role of 'pod-writer' (you can name the roles whatever you want).

The real problem starts when Harold manages to get the privileges of Kumar, a role which he should not have the authorisation to access.

I'm sure you can see the problem here.

Privilege escalation is one of those classic security vulnerabilities that's existed for years, and has now found a new home in Kubernetes.

There's tons of ways to exploit this flaw, from kernel privilege escalation to impersonating groups, so there's no single solution to fix them all.

Best practices to prevent privilege escalation

The first (and most obvious) precaution to take is to keep your Kubernetes up to date. Older versions might have bugs or flaws that are easily exploitable, so this will altogether eliminate at least some vulnerabilities.

You'll also need to restrict access to the Kubernetes API by using mechanisms like RBAC and admission controllers limit requests to create, modify, and delete objects in your containers.

In addition, major cloud providers like AWS and Azure (the two biggest) offer services like Identity and Access Management (IAM) and Active Directory to control how much access users have to resources in your containers.

Need to quickly get some hands-on skills in Kubernetes Access Control? Check out our in-depth course!

2. Insufficient Logging and Monitoring

One of the biggest mistakes you can make with your Kubernetes deployment is to overlook Kubernetes audit logs.

It's almost never the case that a security attack happens 'overnight'. Usually, the attacker spends days, sometimes months, doing reconnaissance on the target software, gathering information and waiting for the perfect moment to strike.

Monitoring your Kubernetes environment at runtime and logging that information can detect and help you quickly shut down an attack before it does too much damage.

Kubernetes audit systems work much like physical surveillance: they monitor all activity in the cluster, collecting and logging the data like IP addresses, time of operation, component calls, and so on.

If that doesn't sound exciting enough for you, think of it as detective work! You have all the puzzle pieces laid out in front of you, and your job is to put them together and find the masked culprit.

Fun, right?

Best practices with Kubernetes audit and logging

Kubernetes lets you audit and log activity in your environment, but doesn't actually analyse the logs.

To do that, you can use an open source threat detection engine like Falco. This tool detects unexpected behaviour, intrusions, and data theft in real time.

It even lets you define rules so that it triggers alerts when it spots the specific sort of activity you're looking for.

3. Security Misconfigurations & Default Configurations

Kubernetes is a container orchestration platform, which means its primary purpose is to manage containers for you while you're off doing more important things like, you know, running a business.

It's designed to take care of the nitty-gritty for you, and offers a wide array of controls and configurations that help make your life a lot easier.

But as with most one-size-fits-all solutions, the default configurations don't really hold up under serious scrutiny.

For example, let's consider network policies, which determine which pods can communicate with each other as well as endpoints.

Kubernetes doesn't apply a network policy on pods by default, so every pod in your environment can talk to every other one. This can wind up being a serious security risk.

Too many developers make the dangerous assumption that default Kubernetes configurations are sufficiently secure. If you're building apps on Kubernetes, this is the first thing to check before deployment.

Best practices to prevent security misconfigurations

Every application is different, so there's no one-size-fits-all guide to securely configuring your containers. But vulnerabilities can crop up even from seemingly innocuous or minor errors.

If you want to learn more about specific security misconfigurations and how to avoid them, check out this article by Datree.io, or even this one by Bridgecrew.

4. Improper Secrets Management

Pretty much any application you build is bound to store sensitive data of some kind, whether it's the organisation's or the users'.

These secrets are often the only thing standing between an attacker and complete access to your internal systems.

Like with security misconfigurations, using default solutions is a no-go. Built-in secrets management systems in Kubernetes are easily exploitable, and necessitate the use of third-party tools like Hashicorp Vault.

How should you NOT store secrets?

The absolute worst way to manage sensitive data is to store them as plain-text files on your container. Kubernetes also includes a resource called Secrets to store sensitive information like passwords, tokens, or keys.

However, Secrets are an object just like any other in Kubernetes, and since they're stored as base-64 encoded strings, they're not inherently secure, either.

The only truly secure way to store secrets is to encrypt them. The Kubernetes API server accepts an argument called --encryption-provider-config that encrypts secret data at rest. This isn't configured by default, it needs to be specifically enabled.

Note that we don't recommend you use this in-built encryption provider. Many applications make use of dynamic secrets, version control requirements, and various access control scenarios. The --encryption-provider-config isn't robust enough to deal with such extensive requirements.

Best practices for Kubernetes secrets management

The most reliable methods for securely encrypting and storing secrets is to use third party services. They create a centralised mechanism for secrets management and access control.

One of the most popular open source secrets management tools is Vault by Hashicorp, which offers features like identity-based access, dynamic secrets, and key management. Other popular open-source services are Conjur and Confidant.

Major cloud providers offer robust secrets managements services as well. Amazon has AWS KMS, Azure has Key Vault, and Google Cloud has Cloud Key Management.

Pssst! We've got an exclusive Intro to Kubernetes Secrets course with hands-on labs. Check it out!

5. Compromised Container Images & Registries

Container images are static files with executable code that are used to build containers. They contain everything you need for an application to run — the container engine, system libraries, configurations, etc.

If the container image itself is compromised, the containers created from it will in turn have security issues that can leave it vulnerable to container breakout.

Here are 2 ways a container images can pose a security risk.

Insecure public registry images

Plenty of organisations use open source base images when building their own container images.

These can't be blindly trusted, as they might contain vulnerabilities or security misconfigurations, and in some cases, even hidden malware that could cripple your build.

Even official Docker images aren't totally foolproof, since vulnerabilities can be introduced later on in development.

Outdated libraries or packages

Container images are immutable, which means their contents cannot be changed once built. This makes it ideal for deploying consistently in any environment.

The downside to this is that once packages or libraries in the image become obsolete, they can become a security risk for the image as a whole.

However, there are situations where an outdated library is needed for compatibility with older systems.

Best practices to secure compromised container images

In both cases, the most effective security solution is to run comprehensive security scans on all your container images before you start building in them.

Here's some of the most popular vulnerability scanning tools for container images (in no particular order):

• Clair
• Trivy
• Anchore

Remember, it's far more efficient to fix bugs in an image before deployment than have to remove it out of production.

When you incorporate security scans into your CI/CD pipeline, you'll both streamline the remediation process and speed up software delivery. It's a win-win.

6. Runtime Threats

If you thought your security problems with Kubernetes ended at the deployment phase, boy, do I have news for you. This is where the fun really starts

Now that your cluster is out in the wild, it's fair game for attackers to try and find every tiny little chink in your armour.

It could be a minor misconfiguration in your RBAC policies that allows your attacker to gain unauthorised privileges. Or even a malware in a container image you used that gives them backdoor access at runtime.

There's any number of things that could go wrong with your Kubernetes deployment.

Best practices to ensure security at runtime

Here's where auditing and monitoring your containers really comes in handy. Note that Kubernetes only audits and logs activity on your build, but doesn't analyse the logs.

As mentioned above, you can use Falco, an open source threat detection engine that alerts you when it notices an attack happening.

Beyond Kubernetes audits, you should even look into runtime security enforcement tools like Seccomp, SELinux, and AppArmor.

• Seccomp (short for 'secure computing mode') is a Linux kernel feature that allows you to restrict the calls a user can make within a container. Learn more about how it works.
• SELinux is a Linux kernel security module that allows users and administrators more control over access control. Check out the SELinux Project Wiki for more information.
• AppArmor is another Linux kernel security module that protects the OS and applications from both external and internal threats, and zero-day attacks by restricting a program's capabilities. Check out their website to learn more.