K07: Network Segmentation
Overview
When operating Kubernetes with multiple microservices and tenants, a key area of concern is around control of network traffic. Isolating traffic within the context of a Kubernetes cluster can happen on a few levels including between pods, namespaces, labels, and more.
Description
Kubernetes networking is flat by default. Meaning that, when no additional controls are in place any workload can communicate to another without constraint. Attackers who exploit a running workload can leverage this default behavior to probe the internal network, traverse to other running containers, or invoke private APIs.
How to Prevent
Workloads in a cluster eventually need to communicate to one another as well as a variety of internal and external endpoints. The goal of network segmentation within Kubernetes is to minimize the blast radius if a container is compromised and stop lateral movement while allowing valid traffic to route as expected.
Native Controls (Multi-Cluster): One way to truly enforce network isolation within Kubernetes is to utilize separate clusters when appropriate. This adds complexity when working with tightly coupled microservices but is a viable option when separating different tenants based on risk.
Native Controls (NetworkPolicies): Network policies are built into Kubernetes itself and behave like firewall rules. They control how pods communicate. Without network policies, any pod can talk to any other pod. Network Policies should be defined as to limit pod communication to only defined assets while denying everything that isn’t explicitly configured. Below is an example of a network policy prevents backend egress between pods running the “default” namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-backend-egress
namespace: default
spec:
podSelector:
matchLabels:
tier: backend
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
tier: backend
Service Mesh: There are a number of different service mesh projects
available for different use cases including Istio,
Linkerd, and Hashicorp
Consul. Each of these service mesh technologies
offer different ways to segment network traffic within a Kubernetes cluster and
all come with pros and cons. Below is an example of an Istio
AuthorizationPolicy
:
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: "shoes-writer"
namespace: default
spec:
selector:
matchLabels:
app: shoes
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/inventory-sa"]
to:
- operation:
methods: ["POST"]
- The
selector
onshoes
means we’re enforcing all Deployments labeled withapp:shoes
. - The
source
workload we’re allowing has theinventory-sa
identity. In a Kubernetes environment, this means that only pods with theinventory-sa
Service Account can access shoes. - The only allowed HTTP operation is
POST
, meaning that other HTTP operations, likeGET and PUT
will be denied.
CNI Plugins:
Container Network Interface (CNI) is an open source specification that is used to configure access to networking resources. CNI is a software-defined mechanism for allowing or disallowing network access within Kubernetes and has a wide variety of supported plugins. Solutions such as Project Calico and Cilium all offer different mechanisms for isolating network traffic within the context of Kubernetes. A CNI is typically needed if an operator would like to implement Kubernetes Network Policies (above).
When choosing a CNI, it is most important to understand the feature-set that you are seeking from a security perspective and the resource overhead and maintenance related to using the plugin.
Example Attack Scenarios
A WordPress pod is compromised on a cluster that has no network segmentation and
the attacker is able to utilize built-in networking utilities such as dig
and
curl
to explore the network (it is an Ubuntu base image after all). They
discover an internally accessible API running on port 6379
which is typically
Redis. They are able to probe the Redis microservice which was intended to be
internal and only used by backend APIs using curl
. Data is stolen and
modified.
A locked-down NetworkPolicy
or service mesh implementation would have made the
network connectivity to Redis from something like WordPress impossible.
A not-so-critical web application gets compromised on a cluster which has no network segmentation and the attacker is able to make a request to metadata URL to grab kube-env file containing certificate keys which has all the details for the bootstrap process. The attack can be performed to register itself as a node and steal secrets for further escalation.
A simple NetworkPolicy
mentioned below can block users from making calls to
metadata URL
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: block-1
spec:
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32
podSelector: {}
policyTypes:
- Egress
References
Istio Authorization: https://istiobyexample.dev/authorization/
Kubernetes CNI Explained: https://www.tigera.io/learn/guides/kubernetes-networking/kubernetes-cni/
Kubernetes Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Hacking kubelet on GKE: https://www.4armed.com/blog/hacking-kubelet-on-gke/