Skip to content

Free Compute: Scenario 1 Defense

Backstory

Name: Blue

  • Overworked
  • Can only do the bare minimum
  • Uses defaults when configuring systems
  • Usually gets blamed for stability or security issues
  • Has no experience operating Kubernetes clusters

Motivations

  • Blue gets paged at 1am with an “urgent” problem: the developers say “the website is slow”
  • Blue reluctantly agrees to take a “quick look”
  • Blue wants desperately to get back to sleep. Zzz

Defense

Blue looks at the page with an unsurprising lack of details, and spends a few minutes getting the answer to exactly which website they are referring to that is underperforming. It's "the one running in Kubernetes", they said. Blue leverages their Cloud Shell terminal to begin the process of troubleshooting the issue.

Identifying the Issue

The first step is to determine the name for the web application deployment in question. From the terminal, Blue runs the following to see a listing of all pods in all namespaces:

kubectl get pods --all-namespaces

The cluster is relatively small in size, but it has a couple deployments that could be the site in question. The development team mentions performance is an issue, so Blue checks the current CPU and Memory usage with:

kubectl top node

and

kubectl top pod --all-namespaces

It appears that a suspcious deployment named bitcoinero is running, and its causing resource contention issues. Blue runs the following to see the pod's full configuration:

kubectl get deployment -n svc bitcoinero -o yaml

It was created very recently, but there are no ports listening, so this looks unlikely to be part of the website. Next, Blue grabs a consolidated listing of all images running in the cluster:

kubectl get pods --all-namespaces -o jsonpath="{..image}" | tr -s '[[:space:]]' '\n' | sort -u

Confirming the Foreign Workload

Blue sends a message back to the developers asking for confirmation of the suspicious bitcoinero image, and they all agree they don't know who created the deployment. Blue looks at the audit logs for the AKS cluster in the Azure Portal. 

AKSAuditAdmin
| where RequestUri startswith "/apis/apps/v1/namespaces/dev/" 
    and Verb == "create" 
    and ObjectRef contains "bitcoinero"
| project User, SourceIps, UserAgent, ObjectRef, TimeGenerated
Audit logs showing the bitcoinero deployment was created from command line by someone with admin credentials

Blue sees that the bitcoinero deployment was created by the cluster admin using the kubectl commandline interface. The IP addresses also show that whoever this was connected from outside the company network.

Cleaning Up

Unsure of exactly who created the bitcoinero deployment, Blue decides that it's now 3am, and the commands are blurring together. The website is still slow, so Blue decides to delete the deployment:

kubectl get deployments -n dev
kubectl delete deployment bitcoinero -n dev

Stopping further intrusions

Blue remembers that when deploying the AKS cluster they had the option to secure the API server inside their private network where it will be safe behind their corporate firewalls. Perhaps now is the time to make that change: 

SUBNET_ID=$(az network vnet subnet create --resource-group <resource-group> --vnet-name <vnet-name> --name <subnet-name> --address-prefix <subnet-prefix> --query id -o tsv)
az aks update -n <cluster name> -g <cluster resource group> \
    --enable-apiserver-vnet-integration \
    --apiserver-subnet-id $SUBNET_ID

# add ip whitelisting

Giving the "All Clear"

Seeing what looks like a "happy" cluster, Blue emails their boss that there was a workload using too many resources that wasn't actually needed, so it was deleted. Also, they added some additional "security" just in case.