There are many ways by which Kubernetes allows you to expose applications (pods) to users. At Grofers, we like to allow our developers to do this without needing intervention from the DevOps team. We find ourselves using LoadBalancer type services often. At the same time, we also want to ensure our developers don’t end up introducing any new security risks while provisioning LoadBalancer services. In this post, we’ll talk about how we augment the default provisioning process of Kubernetes LoadBalancers to suit those needs.
At Grofers, we follow the “You build it, you run it” philosophy. This means that developers deploy applications and provision services independently. This philosophy has enabled us to move fast but it comes with its own set of challenges. Case in point — Kubernetes LoadBalancers. When Kubernetes LoadBalancer are provisioned, they are public by default. As a developer, you can choose to override this configuration and declare your LoadBalancers as private by adding the ”service.beta.kubernetes.io/aws-load-balancer-internal” annotation. However, this creates an inherent risk. The security of our setup hinges on whether a developer decides to add this annotation every time they create a new service. Should a developer forget to add this annotation, they would end up exposing internal applications to the public network. We don’t want that.
We needed better safety nets, better security controls in our service provisioning process in Kubernetes.
Policy Enforcement and Governance using Kyverno
Kubernetes has an interesting concept of admission controllers that can be used to intercept requests to Kuberenetes’ API server and decide whether they should be allowed. This kind of control pattern can be leveraged to enforce policies and govern your Kubernetes cluster. We started looking for frameworks that had been built for this purpose and arrived at Kyverno.
What is Kyverno and how is it useful?
Kyverno is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations using admission controls and background scans. Kyverno policies are Kubernetes resources and do not require learning a new language. You can write policies declaratively. The policies themselves are objects in Kubernetes. Another, more popular, alternative to Kyverno that deserves a mention is Open Policy Agent (which we use as well) but it requires you to learn Rego — an OPA specific language — to implement policy logic.
How do we use Kyverno for secure provisioning of Services?
Let us refer back to our problem statement. Kyverno allows us to write a policy that makes sure that developers are not able to provision any public facing LoadBalancers. The policy instructs Kyverno to inspect all the incoming requests for the creation of a new Service object. These requests are then rejected if they don’t contain the annotation required to declare them as private. Ofcourse, there are cases when we genuinely want to expose certain services to the public internet. For this purpose, we maintain a whitelist of service in a configmap that our Kyverno policy can refer to while considering if the request should be allowed.
Note: To whitelist your load balancer, add an entry like namespace/service_name.
Let’s look at our policy specification line-by-line:
- Line 2: You can see that it is a cluster policy which means it is going to be applied cluster-wide irrespective of the namespace.
- Line 6–7: These lines represent that this policy is going to be of an “enforced” nature which means that it will block any incoming resources which will violate this policy.
- Line 10–14: These lines make the policy fetch the values from a configmap named allowed-elbs from the security namespace and become aware of whitelisted services.
- Line 15–18: These lines make sure that the policy only acts upon incoming requests of type Service.
- Line 19–31: These lines consist of the core logic of blocking any incoming service if it matches all of the following conditions:
– The service is not annotated with the annotation service.beta.kubernetes.io/aws-load-balancer-internal.
– The service is of the type load balancer.
– The namespace/service_name is not present in the configmap of whitelisted services.
As Kubernetes is deployed in organizations to enable more and more developers to deploy applications, governance, and security of Kubernetes clusters are becoming an area of concern. Writing policies for security and governance with Kyverno is easy and the barrier to entry is extremely low. We are looking at adopting Kyverno for more governance-related use-cases.
Ankit Saini used to work as a Security Engineer at Grofers. Follow him on Twitter.
We are hiring across various roles! If you are interested in exploring working at Grofers, we’d love to hear from you.
Secure provisioning of LoadBalancer Services on Kubernetes using Kyverno was originally published in Lambda on Medium, where people are continuing the conversation by highlighting and responding to this story.