K8s

Question 1

Service Account Tokens

Answer 1

• ServiceAccount: A resource that provides an identity for processes running in pods
• Every Pod has a service account. Created automatically
• Allows apps to authenticate with the K8s API and access cluster resources
• Service accounts are associated with access tokens that the pods can use to authenticate themselves to the k8s API server, enabling them to perform actions like reading or modifying resources within the cluster
• When a pods starts, K8s mounts a Service Account token at a predefined location, eg /var/run/secrets/k8s.io/serviceaccount/token
• This token is signed by the Kubernetes Certificate Authority. This token is signed
• When decoded the ServiceAccount token contains things like namespace, pod name and uuid, service account name
• Once the K8s API receives ServiceAccount token from the pod, it validates existence of the service account and checks if the system has any RBAC rules attached that allow/deny the current action

Question 1

OIDC

Answer 1

Used the OIDC authentication layer to validate and authenticate our external users managed on our corporate identity provider, JumpCloud

We used OIDC integration to access the K8s cluster

K8s uses OIDC validation mechanism to confirm it’s the correct user

Use the OIDC group and/or claims which get returned for looking up the RoleBinding

Question 2

Roles, Bindings, Subjects

Answer 2

Role: Grant permissions within a specific namespace. Contains rules that represent a set of permissions. There are no DENY rules, always additive

ClusterRole: Same idea but for the whole cluster and granting permissions for non-namespaced resources (like nodes)

RoleBinding: Grants the permissions defined in a Role to a user or set of users. Applies only within a specific namespace. Has a list of Subjects

ClusterRoleBinding: Same idea. Use this to grant permissions across the whole cluster

Subjects: Users, Groups, or ServiceAccounts

• These Bindings get bound to subjects and they can be authenticated users returned from the OIDC integration or ServiceAccounts with a corresponding namespace
• Within the Role you would define the rules which can be accessed, eg get, list, create, delete (these are the verb rules) against a set of resources (eg pod, services)
• To change roles you need to remove the Binding, change the role, then rebind. Will get an error if you attempt to change a role when it’s already bound

Question 3

Other authentication strategies

Answer 3

• Bootstrap tokens. Allow servers to authenticate for the first time
• Static tokens: Only use for simple stuff
• Authentication Proxy: K8s api server delegates the full authentication process to a proxy server
• Webhook tokens: Similar to Auth Proxy
• X.509 certs: Uses mTLS and extracts the username from the Common Name and the RBAC groups from the Organization fields. Each “user” has this cert and the k8s API verifies these certs using a certificate authority
  Impersonation:

Question 4

RBAC

Answer 4

• Adds security to a K8s cluster
• Role Based Access Controls
• RBAC defines roles, which are collections of permissions, and subjects which are service accounts or users. A subject is granted a role
• What attributes on an object get used during RBAC authorization?
  User
  Group
  Resource
  Name

^ These attributes are validated against policies

Verbs: The operations we want to do with the resources

Question 5

Node Authentication and Authorization in EKS

Answer 5

• Nodes in an EKS cluster authenticate themselves using AWS IAM.
• Each node is associated with an IAM role that has specific permission
• Node authorization is managed to K8s, specifically through the kubelet component running on each node. Can also use RBAC

Question 6

K8s Components

Answer 6

Two primary components:

Control Plane (master nodes)
- API Server
- Control Manager
- Etcd
- Scheduler

Data Plane (worker nodes)
- Kube Proxy
- Kubelet
- Container runtime

Question 7

K8s Component - API Server

Answer 7

• All interactions with the cluster are made through the API server
• The API server is responsible for validating all API requests and ensuring they are authorized
• Adheres to a client-server model
• Clients can be software developers, devops engineers, nodes, k8s scheduler, etc

Authentication: Verify the identity of the entity making the request, whether it’s a user, pod, or machine

Authorization: Checks if the requesting entity possesses adequate permissions for the operation

Only service which interacts directly with etc

Question 8

K8s Component - Control Manager

Answer 8

Runs controllers to regulate the state of the cluster

Ensure desired state matches current state

Creates service accounts, new endpoints, manages load balancers, etc

Manage storage volumes

It encompasses several controllers, each focusing on a specific aspect of the cluster’s behavior. These controllers continuously monitor the cluster’s state and work towards reconciling any deviations from the desired state.

Examples: like the ReplicaSet controller ensuring the desired number of pod replicas are running, the Deployment controller managing the lifecycle of deployments, and the Node controller handling node-related tasks like detecting and responding to node failures.

Question 9

K8s Component - Etcd

Answer 9

Distributed KV store which only has cluster state

Always insert, never delete. Though runs compaction

No application data

Build on raft consensus algo

Question 10

K8s Component - Scheduler

Answer 10

Watches for newly created Pods and assigns objects or pods to nodes

Knows about the quality and configuration of the nodes

It uses various policies to select the most suitable node for a pod based on factors such as resource requirements, node capacity, and pod affinity/anti-affinity

Question 11

K8s Component - Kubelet

Answer 11

• An agent running on each worker node
• Communicates with control plane
• Receives pod definitions from master and interacts with container runtime. Connects to container runtime
• Monitors health of the pods

Question 12

K8s Component - Kube Proxy

Answer 12

• The network agent
• Runs on each node
• Handles dynamic updates and maintenance of all networking rules on the node.
• Configures iptables
• Forwards networking requests to the pods
• Allows services to communicate with each other
• Managed by a DaemonSet

Question 13

Worker node

Answer 13

Has container runtime. Examples: docker, containerd, rkt

Pods are scheduled on worker nodes

Has networking

Pod is smallest scheduling unit in K8s

Worker node components
- Kubelet
- Kubeproxy

Question 14

Master node

Answer 14

Contains control plane components

Makes global decisions about the cluster

Manages cluster operations

Must be up all the time

If active master fails, then one of the replicas will take its place

Question 15

K8s alternatives

Answer 15

Nomad
Docker swarm: very easy to install and use but designed for small scale

Question 16

K8s evolution history

Answer 16

K8s allows you to set boundaries on the applications running on physical servers. Without it apps would eat all resources, poor resource utilization

VM helped and allowed you to set resources utilization on each VM. Each VM is heavy, each have their own OS

Containers are light weight and can be ported across different clouds or environment

Portable isolated application environment

Can be densely packed, very good resources utilization

Question 17

Microservices

Answer 17

Light weight application which has its own environment requirements and dependencies

Question 18

K8s Namespace

Answer 18

• A k8s cluster is comprised of multiple projects/applications with multiple teams
• Allows isolation of resources within a cluster
• A namespace is a logical isolation of resources, network policies, RBAC, and everything else
• Each namespace can have different authentication policies, restrictions, etc

Question 19

How do you perform maintenance on the K8s node?

Answer 19

• Often need to run administrative tasks due to upgrades, security patches, etc
• To drain the k8s node can use the following

Marks the node as “un-schedulable”. Prevents pods from being scheduled on it
kubectl cordon

Removes existing pods from the node. Flag will skip over any pods that are part of a DaemonSet, which ensures that essential system services are not disrupted
kubectl drain –ignore-daemon set

Question 20

K8s Pod

Answer 20

• A pod encapsulates one or more containers which share the same network namespace and can communicate with each other over the localhost interface
• Each pod has a unique IP
• All containers within a pod share the same IP
• Can run a pod on a specific node by using node affinity. A node is assigned an arbitrary label and they are configured to be assigned to that node as per the label created

Question 21

Explain Daemonset

Answer 21

• A type of controller that ensures a specific pod runs on each node in the cluster
• A DaemonSet runs one pod per node
• Used to run background services
• Useful for system-level tasks like logging, monitoring, or node specific services.

Example: Use a DaemonSet to ensure that a logging agent runs on every node in the cluster to collect logs locally

Question 22

How to stop pods

Answer 22

• K8s will first send a SIGTERM signal to the container’s main process and wait for a certain period of time, determined by the terminationGracePeriodSeconds parameter before issuing a SIGKILL

Question 23

What is a service?

Answer 23

An abstraction that defines a logical set of Pods and a policy by which to access them.

Enable communication between different parts of an application and provide load balancing and service discovery

Achieved through labels and selectors

How to find the Service?
1) env vars which automatically get added when resource is created
2) enable k8s add-on, core dns.
- Creates DNS record for each service
- Can find service in same namespace by just the name
- Other namespaces can access by adding suffix namespace

Cluster IP:
- Default service type
- Exposes the service on a cluster internal IP.
- Means that only the services inside the cluster can access it

Node Port:
- Exposes the service on a static port on each node in the cluster.
- Port is mapped to a particular service
- Node redirects to cluster IP of the service then request is forwarded to app
- This makes the service accessible from outside the cluster.
- Anyone who has the node IP/port can access it

Load Balancer:
- Provisions an external load balancer in the cloud infra and directs traffic to the K8s service.
- Allows you to expose your service to the internet

External IP:
- Use an external load balancer outside of K8s
- Not managed by K8s

Ingress: A K8s object which defines to expose services to external traffic. Can provide features like load balancing, SSL termination, etc

A service provides a stable and consistent IP address and DNS name for the pods

Question 24

Horizontal Pod Autoscaler (HPA)

Answer 24

- A controller that automatically scales the number of pods in a ReplicaSet, Deployment, StatefulSet, or any other custom resources based on the observed CPU utilization of other metrics - Peregrine used keda (Kuberenetes Event Driven Auto scaling)

Question 25

Give an example of how K8s can be used to deploy a highly available application

Answer 25

- Create a Deployment that defines the desired state of your application, such as the number of replicas, the image, the ports, etc. - Create a Service that exposes your application to other pods or external clients. The service acts as a load balancer that distributes traffic among your pods. - Create an Ingress that defines rules to expose your service to external traffic. The ingress controller routes the traffic to your service based on the host name or path. - Configure health checks for your pods using liveness and readiness probes. These probes allow Kubernetes to monitor the health and availability of your pods and restart them if they fail or become unresponsive. - Configure rolling updates for your deployment using updateStrategy parameters. These parameters allow you to perform zero-downtime updates to your pods without affecting the availability of your application. - Configure horizontal pod autoscaling for your deployment using HPA parameters. These parameters allow you to automatically scale your pods based on CPU utilization or other metrics.

Question 26

Admission Controller

Answer 26

Part of the auth flow 1. Client Request 2. Send to API server 3. Authentication 4. Authorization 5. Optional admission controller 6. Persist to etcd Can accept or reject the request based on additional policies. For example resource quota, request count, custom business logic. Very granular

Question 27

K8s Job

Answer 27

- One or more pods to create a task - Ensure task is completed successfully

Question 28

Affinity / Anti-Affinity

Answer 28

Pod affinity and pod anti-affinity allow you to specify rules about how pods should be placed relative to other pods The rules are defined using custom labels on nodes and label selectors specified in pods Anti-affinity rules allow you to prevent pods of a particular service from scheduling on the same nodes as pods of another service that are known to interfere with the performance of the pods of the first service. Or, you could spread the pods of a service across nodes or availability zones to reduce correlated failures

Question 29

Liveness / Readiness Probes

Answer 29

Liveness - A probe which checks if an application is running based on any arbitrary business logic. If it fails the container gets restarted - Is the container in a healthy state Readiness Readiness probes enable developers to instruct Kubernetes that a running container should not receive traffic until additional tasks are completed, such as loading files, warming caches, and establishing network connections

Question 30

Tainting a node

Answer 30

Taints are used to repel pods from being scheduled onto certain nodes unless the pods have corresponding tolerations. They are typically used to mark nodes with certain characteristics, such as hardware capabilities or organizational policies. Taints are set by administrators on nodes to influence where pods can be scheduled.

Question 31

ConfigMap

Answer 31

- Has configuration data as key value pairs - Often get set as env vars on the pod - Can be used to mount a file. Key is the mount path, value is the contents of the file - We used sops for secrets.yaml

Question 32

Volume

Answer 32

A volume is a directory backed by a storage medium Volume is attached to pods and outlives the life of the pod Volume type decides the type of directory to create EmptyVolume - Tied to the life of the pod - It's created when the pod is assigned to a node - Useful for sharing files between containers within the same pod or for temporary data - eg "/tmp", "/var/run/shared_sockets" HostPath - Mounts a file or directory from the host node's filesystem into your pod - Often not used in prod given security risks (pod has access to underlying node), node coupling (makes it difficult to migrate pots and limits flexibility), etc PersistentVolumeClaim - Allows Pods to request storage resources without needing to know the details of the underlying storage infra - A way to provision storage dynamically - Types include: ReadWriteOnce, ReadOnlyMany, ReadWriteMany, - With a PVC a user requests storage CSI - Allows 3rd parties to develop their own plugins to manage storage systems

Question 33

Ingress

Answer 33

Allows one to route and expose multiple services through a single IP address and port combination Used to manage external access to services within a cluster. Provides HTTP/S routing to services based on hostnames and URL paths Rules: Define rules that specify how incoming requests should be routed. Example, route requests with a specific hostname (example.com) to a particular service Handles TLS termination NGINX is a common ingress controller, ISTIO too Allows you to decouple rules from the application

Question 34

Secrets

Answer 34

Secrets object allow us to encode sensitive information before sharing it Same idea of ConfigMaps Keep in mind that by default etcd data is not encrypted and therefore the secret can be viewed However, on EKS you can encrypted the underlying EBS volume

Question 35

Helm / Helmfile

Answer 35

Helm is a package manager for Kubernetes that helps you manage, version, and deploy applications using pre-configured packages called charts Helmfile allows you to manage Helm releases (deployments) as code. Makes it easy to manage and version your Helm deployments

Question 36

ArgoCD

Answer 36

We integrated Helmfile into ArgoCD to automate the deployment of Helm releases using GitOps principles

Question 37

What are resource quotas in K8s?

Answer 37

A tool for managing the resource consumption in a namespace. It sets hard limits on the amount of compute resources, such as CPU and memory, that can be used by the objects within a namespace. This ensures that no single team or application consumes more than its fair share of resources, promoting efficient utilization and preventing resource contention. Can specify limits on: - CPU and Memory - Storage: limits on persistent volume claims - Object count: maximum number of objects like pods, services, and secrets

Question 38

What is a namespace in K8s?

Answer 38

A namespace is a logical partition which allows cluster resources to be divided between teams and/or applications running in the cluster Key points about namespaces: - Isolation: Namespaces isolate resources (eg pods, deployments) from each other - Resource quotas - Access Control: Using RBAC to restrict user access to certain namespaces -