Devops on Adur

LLMOps: integrating LLMs into DevOps workflows

Sun, 15 Jun 2025 00:00:00 +0000

LLMs have moved beyond chatbots. They’re now embedded in engineering workflows where they automate tedious tasks, speed incident response, and boost developer productivity. But deploying an LLM into a production DevOps pipeline is fundamentally different from using ChatGPT in a browser.

This guide covers what LLMOps means in practice, where LLMs fit into DevOps, architecture patterns that work, and pitfalls to avoid.

What is LLMOps?

LLMOps is the practices, tools, and infrastructure needed to operationalize LLMs. It extends MLOps but addresses challenges unique to language models:

Model selection vs. model training: Most teams consume pre-trained models (via APIs or self-hosted inference) rather than training from scratch. The operational focus shifts to prompt engineering, fine-tuning, and retrieval-augmented generation (RAG).
Cost management: LLM inference is expensive. Token-based pricing means costs scale with usage in ways that are harder to predict than traditional compute.
Non-determinism: LLMs produce variable outputs for the same input, which complicates testing, validation, and reproducibility.
Latency: Response times of seconds (not milliseconds) require different architectural patterns than traditional microservices.

LLMOps is not a separate discipline. It is an extension of your existing DevOps and MLOps practices, adapted for the specific operational characteristics of language models.

Practical use cases in DevOps

Here is where LLMs are delivering real value in DevOps workflows today:

Automated code review

LLMs can provide a first-pass review of pull requests, catching common issues like missing error handling, security anti-patterns, inconsistent naming, or missing tests. They do not replace human reviewers but reduce the burden of repetitive feedback.

Incident summarization

When an incident fires at 3 AM, the on-call engineer needs context fast. An LLM can ingest alert data, recent deployment logs, related runbooks, and previous incident reports to produce a concise summary of what is likely going wrong and what was done last time.

Log analysis

LLMs are surprisingly effective at pattern recognition in unstructured log data. Feed them a block of error logs and they can identify the root cause faster than manual grep sessions, especially for unfamiliar systems.

Documentation generation

Generating draft documentation from code, API schemas, or Terraform modules. The output needs human review, but it eliminates the blank-page problem and keeps docs closer to current state.

Infrastructure as Code generation

Given a natural language description of desired infrastructure, LLMs can generate Terraform, Ansible, or Kubernetes manifests as a starting point. Useful for scaffolding, not for production-ready code without review.

Architecture patterns for LLM integration

Pattern 1: API gateway to external LLM

The simplest approach. Your application calls an external LLM API (OpenAI, Anthropic, etc.) through a centralized gateway that handles authentication, rate limiting, logging, and cost tracking.

1
2
3
4
5


[CI/CD Pipeline] --> [API Gateway] --> [External LLM API]
 |
 [Logging & Metrics]
 |
 [Cost Tracking]

Pros: No infrastructure to manage, access to the most capable models, fast to implement. Cons: Data leaves your network, vendor lock-in, variable latency, ongoing API costs.

Pattern 2: Self-hosted inference

Run open-weight models (Llama, Mistral, etc.) on your own infrastructure using inference servers like vLLM or Ollama.

1
2
3


[CI/CD Pipeline] --> [Load Balancer] --> [vLLM / Ollama Instance(s)]
 |
 [GPU Node Pool]

Pros: Data stays internal, predictable costs at scale, no vendor dependency, full control over model versions. Cons: Requires GPU infrastructure, operational overhead, smaller models may be less capable.

Pattern 3: RAG-enhanced pipeline

Combine an LLM with a retrieval system that provides relevant context from your own knowledge base (runbooks, documentation, past incidents). This dramatically improves response quality for domain-specific tasks.

1
2
3
4


[Query] --> [Embedding Model] --> [Vector DB Search] --> [Context + Query] --> [LLM] --> [Response]
 |
 [Your Knowledge Base]
 (runbooks, docs, etc.)

This pattern is particularly powerful for incident response and documentation tasks where the LLM needs your organization’s specific context.

Key considerations

Cost

LLM API costs can be surprising. A code review pipeline that processes 50 PRs per day with large diffs can easily run hundreds of dollars per month. Strategies to control costs:

Set token limits per request
Cache common queries and responses
Use smaller models for simpler tasks (triage with a small model, escalate to a larger one)
Monitor token usage per pipeline and set alerts

Latency

LLM responses take seconds, not milliseconds. Design your integrations as asynchronous processes:

Post code review comments after the fact, do not block the PR
Process incident data in the background, push results to a Slack channel
Use streaming responses where possible to improve perceived performance

Hallucinations

LLMs will confidently generate plausible-sounding but incorrect information. This is a critical concern for DevOps tasks where bad advice can cause outages.

Mitigations:

Always present LLM output as suggestions, never as authoritative actions
Require human approval before any LLM-generated change is applied
Use RAG to ground responses in verified documentation
Implement output validation (e.g., lint generated IaC before presenting it)

Security

Data exposure: Anything you send to an external LLM API may be used for training or stored. Never send secrets, credentials, or sensitive customer data.
Prompt injection: Malicious content in code, logs, or user input can manipulate LLM behavior. Sanitize inputs and validate outputs.
Supply chain: LLM-generated code may introduce vulnerabilities. Run all generated code through your existing security scanning pipeline.

Tools and platforms

LangChain

A framework for building LLM-powered applications. Useful for orchestrating multi-step chains (e.g., retrieve context, format prompt, call LLM, parse output). Supports many LLM providers and has good tooling for RAG pipelines.

1
2
3
4
5
6
7
8


from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_template(
 "Review this code diff for security issues and suggest fixes:\n\n{diff}"
)
chain = prompt | ChatOpenAI(model="gpt-4o", temperature=0)
result = chain.invoke({"diff": code_diff})

vLLM

A high-throughput inference engine for self-hosted models. Supports PagedAttention for efficient memory management and continuous batching for high throughput.

1
2
3
4


# Start a vLLM server
python -m vllm.entrypoints.openai.api_server \
 --model mistralai/Mistral-7B-Instruct-v0.2 \
 --port 8000

Exposes an OpenAI-compatible API, so you can swap between self-hosted and external APIs with minimal code changes.

Ollama

The easiest way to run LLMs locally for development and testing. Great for prototyping pipelines before committing to infrastructure.

1
2
3
4
5
6
7


# Pull and run a model
ollama pull llama3
ollama run llama3 "Summarize this error log: [paste log]"

# Serve as an API
ollama serve
# Then call http://localhost:11434/api/generate

Example: Automated PR review pipeline

Here is a conceptual pipeline for automated PR review using an LLM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


# .github/workflows/llm-review.yml
name: LLM Code Review

on:
 pull_request:
 types: [opened, synchronize]

jobs:
 llm-review:
 runs-on: ubuntu-latest
 steps:
 - name: Checkout
 uses: actions/checkout@v4
 with:
 fetch-depth: 0

 - name: Get diff
 id: diff
 run: |
 git diff origin/${{ github.base_ref }}...HEAD > diff.txt

 - name: Run LLM review
 env:
 LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
 run: |
 python scripts/llm_review.py \
 --diff diff.txt \
 --model gpt-4o \
 --max-tokens 2000 \
 --output review.json

 - name: Post review comments
 uses: actions/github-script@v7
 with:
 script: |
 const review = require('./review.json');
 await github.rest.pulls.createReview({
 owner: context.repo.owner,
 repo: context.repo.repo,
 pull_number: context.issue.number,
 body: review.summary,
 event: 'COMMENT',
 comments: review.line_comments
 });

The review script would:

Read the diff
Split large diffs into chunks that fit within the model’s context window
For each chunk, construct a prompt asking for security issues, bugs, and style problems
Aggregate results and format as GitHub review comments
Include confidence scores and always mark output as AI-generated

Guardrails and responsible use

Label all LLM output clearly as AI-generated. Engineers should know when they are reading machine output.
Never auto-merge or auto-apply LLM suggestions. Keep a human in the loop for all changes.
Log all prompts and responses for debugging and audit purposes.
Set spending limits and alerts on LLM API usage.
Review prompt templates regularly to ensure they do not leak sensitive information.
Test for bias and errors with representative samples before deploying to production workflows.

Getting started recommendations

Pick one use case - Don’t try to LLM-enable everything at once. Start low-risk: documentation drafts, commit message suggestions.
Start with an external API - Don’t invest in GPU infrastructure until you’ve validated the use case. Use OpenAI or Anthropic to prototype.
Measure everything - Track cost per invocation, latency, user satisfaction, error rates from day one.
Build an evaluation framework - Create a test suite of known-good inputs and expected outputs. Run it against every prompt change or model update.
Plan your data strategy - Decide early what data you’ll and won’t send to external APIs. Document clearly.
Iterate on prompts - Prompt engineering is iterative. Version control prompts, treat as code.

LLMs are a powerful tool for DevOps automation, but they’re exactly that: a tool. They work best when thoughtfully integrated into existing workflows, with clear boundaries on what they can and cannot do autonomously.

Basic Kubernetes Commands

Thu, 29 Oct 2020 00:00:00 +0000

This post provides an overview of the basic kubectl CLI commands that can be applied to Kubernetes objects. Some examples of Kubernetes objects include Pods, ReplicaSets, Deployments, Namespaces, etc.

Namespaces

Namespaces are used in Kubernetes to organize cluster objects. Essentially, a namespace represents a folder containing a set of objects. By default, kubectl interacts with the default namespace. To use a different namespace, the --namespace flag is required, for example --namespace=example. To interact with all namespaces, use the --all-namespaces flag ¹.

Contexts

If you want to change the default namespace permanently, you can use a context. When used, it is recorded in the kubectl configuration file, stored at HOME/.kube/config. To create a context with a new default namespace name, run ¹:

1

kubectl config set-context my-context --namespace=nuevonamespacepordefecto

1

kubectl config use-context my-context

Kubernetes API objects

Every Kubernetes object is represented by a RESTful resource and exists at a unique HTTP path in the Kubernetes API. Resources are represented as JSON or YAML files. Through the kubectl command, you can access these objects. For example, using kubectl get you can access any resource in the default namespace ¹:

1

kubectl get <resource-name>

To get a more specific resource:

1

kubectl get <resource-name> <object-name>

To get more information about the object in JSON or YAML format, you can add the -o json or -o yaml flags respectively. This output is not very human-readable.

Another option to get human-readable details about an object is to use the kubectl describe command:

1

kubectl describe <resource-name> <object-name>

Creating, updating, or deleting Kubernetes objects

As mentioned earlier, Kubernetes objects or resources are represented by JSON or YAML files. To create, update, or delete these objects, such files are used. For example, to create or update an object stored in ejemplo.yaml, run:

1

kubectl apply -f ejemplo.yaml

If you prefer to make interactive edits instead of modifying the local file, you can use the kubectl edit command to download the latest version and launch an editor. After saving the file, it will be uploaded and automatically updated.

1

kubectl edit <resource-name> <object-name>

The kubectl apply command also saves the version history of configuration files. You can access these records using the edit-last-applied, set-last-applied, and view-last-applied options.

1

kubectl apply -f myobj.yaml view-last-applied

To delete an object, simply run:

1

kubectl delete -f ejemplo.yaml

Debugging

kubectl also has a set of commands for debugging your containers. To view the logs of a running container, run:

1

kubectl logs <pod-name>

If there are multiple containers in the pod, you can choose the container to inspect with the -c flag.

By default, kubectl logs lists the current logs and exits. If you want to continuously stream the logs to the terminal instead, you can add the -f (follow) flag to the command line.

You can also use the exec command to run a command in a running container:

1

kubectl exec -it <pod-name> -- bash

This will provide an interactive console inside the running container for more detailed debugging.

O’Reilly, Kubernetes: Up and Running ↩︎ ↩︎ ↩︎

Basic Kubernetes Objects

Thu, 29 Oct 2020 00:00:00 +0000

This post provides a description of the basic Kubernetes objects. Some examples of Kubernetes objects include Pods, ReplicaSets, Deployments, Namespaces, etc.

Basic objects

According to the Kubernetes documentation, Kubernetes objects are persistent entities. Kubernetes uses these entities to represent the state of the cluster. Each Kubernetes object is represented by a RESTful resource and exists at a unique HTTP path. Specifically, objects can describe ¹:

Which containerized applications are running (and on which nodes).
The resources available to those applications.
The behavioral policies for those applications, such as restart policies, upgrades, and fault tolerance.

Almost all Kubernetes objects include two fields that configure them: the spec or desired state of the object specification, and the status or actual/current state of the object. In the spec section, the intent or desired state of the object is declared. The control plane is responsible for attempting to match the actual state of the object with the desired state.

To create an object, the Kubernetes API must receive the object information in JSON format. Most of the time, the information is sent through kubectl in a .yaml file that will be converted to JSON format. An example of a .yaml file is as follows ¹:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
 name: nginx-deployment
spec:
 selector:
 matchLabels:
 app: nginx
 replicas: 2 # tells deployment to run 2 pods matching the template
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: nginx:1.14.2
 ports:
 - containerPort: 80

Required fields

The following mandatory fields must be set to create an object ¹:

apiVersion: which version of the Kubernetes API is being used to create the object.
kind: what type of object you want to create.
metadata: data that serves to uniquely identify the object, including a name, a UID, and an optional namespace.
spec: the desired state for the object.

Labels

Labels are key-value pairs attached to Kubernetes objects. They are used to organize and select subsets of objects based on predefined requirements. Many objects can share the same label, so labels do not provide uniqueness to objects ².

For example, to add the label color=green to a Pod called plant, you can run the following ³:

1

kubectl label pods planta color=verde

The above command will not overwrite an existing label, so you need to use the --overwrite flag. Or if you want to remove the color label, use the following command:

1

kubectl label pods bar color -

Label selectors

Controllers use label selectors to select a subset of objects. Kubernetes supports two types of selectors ⁴:

Equality-based selectors

Allow filtering objects based on label keys and values. Matching is achieved using the operators:

Equals = or == (there is no difference between the operators)
Not equals !=

Set-based selectors

Allow filtering objects based on a set of values. You can use in, notin operators for label values, and the exists operator for label keys.

Object types

Pods

A pod is the smallest scheduling unit in Kubernetes. It is a logical collection of one or more containers that ²:

Are scheduled on the same host.
Share the same network namespace, and therefore share a single IP address assigned to the Pod.
Have access to mount the same external storage (volumes).

Pods are ephemeral in nature and do not have self-healing capabilities. That is why controllers are used to manage Pod replication, fault tolerance, self-healing, etc. Some of these controllers include Deployments, ReplicaSets, etc.

ReplicaSets

Deployments

Getting Started with Minikube

Thu, 29 Oct 2020 00:00:00 +0000

Local Kubernetes installation

There are several tools that can be used to deploy Kubernetes on one or many clusters. Some of them include:

Minikube is the easiest and preferred method for setting up Kubernetes locally. It is used to manage a single-node cluster, although there is already an experimental feature that supports multi-node clusters.

Minikube

The minikube project is a local Kubernetes cluster implementation for Linux, macOS, and Windows. Its goal is to be the best tool for local Kubernetes application development ¹.

The first steps with minikube can be found in the official documentation and are as follows ²:

Requirements

2 CPUs or more
2GB of RAM
20GB of disk space
Internet connection
Container or virtual machine manager, such as: Docker, Hyperkit, Hyper-V, KVM, Parallels, Podman, VirtualBox, or VMware.

Installing minikube

For Linux there are three options:

Binary package:

1
2


 curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
 sudo install minikube-linux-amd64 /usr/local/bin/minikube

Debian package:

1
2


curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

RPM package:

1
2


curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-latest.x86_64.rpm
sudo rpm -ivh minikube-latest.x86_64.rpm

To start minikube, run:

1

minikube start

To stop minikube safely, run:

1

minikube stop

Installing Kubernetes

Kubernetes can be installed locally on virtual machines or directly on the operating system. Tools such as Ansible or kubeadm can be used to automate the installation.

The CLI tool kubectl can be used to manage, deploy, and configure the resources and applications of the Minikube cluster, and can be installed with the following commands:

1
2
3
4
5


sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2 curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

For details on kubectl commands, you can refer to the kubectl book, the official Kubernetes documentation, or its GitHub repository.

A common step after installation is to configure and enable kubectl command autocompletion:

1
2
3
4


sudo apt install -y bash-completion
source /usr/share/bash-completion/bash-completion
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >>~/.bashrc

Other relevant packages to install include:

kubeadm: used for managing or automating the installation
kubelet: an agent that runs on each node and communicates with the control plane components
kubernetes-cni: allows configuring network elements

1

sudo apt-get install kubelet kubeadm kubernetes-cni

Kubernetes Architecture Components

Wed, 28 Oct 2020 00:00:00 +0000

This post covers the components of the Kubernetes architecture. The main sources of information are the Introduction to Kubernetes course by The Linux Foundation on edX, authored by Chris Pokorni and Neependra Khare, and the official Kubernetes documentation.

Components of the Kubernetes architecture

A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications. Every cluster has at least one worker node. At a very high level of abstraction, Kubernetes has the following main components:

One or more master nodes, on the control plane side.
One or more worker nodes.

The following figure shows the architecture of the components of a Kubernetes cluster ¹:

The master node provides a runtime environment for the control plane responsible for managing the state of a Kubernetes cluster and is the brain behind all operations within the cluster. The control plane components are agents with very distinct roles in cluster management. To communicate with the Kubernetes cluster, users send requests to the control plane through a command-line interface (CLI) tool, a web user interface dashboard, or an application programming interface (API) ².

It is essential to keep the control plane running at all costs. Losing the control plane can cause downtime, resulting in service disruption to clients, with a potential loss of business. To ensure fault tolerance of the control plane, master node replicas can be added to the cluster, configured in high availability mode. While only one of the master nodes is dedicated to actively managing the cluster, the control plane components remain synchronized across the master node replicas. This type of configuration adds resilience to the cluster’s control plane, in case the active master node fails ².

To preserve the state of the Kubernetes cluster, all cluster configuration data is saved in etcd. etcd is a distributed key-value store that only holds data related to the cluster state, not client workload data. etcd can be configured on the master node (stacked topology) or on its dedicated host (external topology) to help reduce the chances of data store loss by decoupling it from the other control plane agents ².

With the stacked etcd topology, high availability master node replicas also ensure the resilience of the etcd data store. However, that is not the case with the external etcd topology, where etcd hosts must be replicated separately for high availability, a configuration that introduces the need for additional hardware.

A master node runs the following control plane components ¹:

kube-apiserver or API server
kube-scheduler or scheduler
kube-controller-manager or controller manager
etcd or data store

While a worker node has the following components:

Container Runtime
kubelet or node agent
kube-proxy or proxy
Addons for DNS, dashboard, cluster-level monitoring, and logging

Master node

kube-apiserver

All administrative tasks are coordinated by kube-apiserver, a central control plane component that runs on the master node. The API server receives RESTful requests from users, operators, and external agents, then validates and processes them. During processing, the API server reads the current state of the Kubernetes cluster from the etcd data store, and after the execution of a call, the resulting state of the Kubernetes cluster is saved in the distributed key-value data store for persistence. The API server is the only control plane component that communicates with the etcd data store, both for reading and saving Kubernetes cluster state information, acting as an intermediary interface for any other control plane agent querying the cluster state.

The API server is highly configurable and customizable. It can scale horizontally, and it also supports adding custom secondary API servers, a configuration that turns the primary API server into a proxy for all custom secondary API servers and routes all incoming RESTful calls to them based on custom-defined rules ².

kube-scheduler

The role of the kube-scheduler is to assign new workload objects, such as pods, to nodes. During the scheduling process, decisions are made based on the current state of the Kubernetes cluster and the requirements of the new object. The scheduler obtains from the etcd data store, through the API server, the resource usage data for each worker node in the cluster. The scheduler also receives from the API server the requirements of the new object that are part of its configuration data. Requirements may include constraints set by users and operators, such as scheduling work on a node labeled with disk == ssd as a key-value pair. The scheduler also takes into account Quality of Service (QoS) requirements, data locality, affinity, anti-affinity, dependent data location, taints, cluster topology, etc. Once all the cluster data is available, the scheduling algorithm filters the nodes with predicates to isolate potential candidate nodes, which are then scored with priorities to select the node that satisfies all the requirements for the new workload. The result of the decision process is communicated to the API server, which then delegates the workload deployment to other control plane agents.

The scheduler is highly configurable and customizable through scheduling policies, plugins, and profiles. Additional custom schedulers are also supported. A scheduler is extremely important and complex in a multi-node Kubernetes cluster ².

kube-controller-manager

A control plane component that runs the controllers to regulate the state of the Kubernetes cluster. Controllers are watch loops that run continuously and compare the desired state of the cluster (provided by the configuration data of objects) with its current state (obtained from the etcd data store through the API server). In case of a discrepancy, corrective actions are taken in the cluster until its current state matches the desired state. It runs controllers responsible for acting when nodes become unavailable, for ensuring the expected number of pods, for creating endpoints, service accounts, and API access tokens ². Logically, each controller is an independent process, but to reduce complexity, they are all compiled into a single binary and run in a single process. These controllers include ¹:

Node controller: responsible for detecting and responding when a node goes down
Replication controller: responsible for maintaining the correct number of pods for each replication controller in the system
Endpoints controller: builds the Endpoints object, i.e., joins Services and Pods
Service account and token controllers: create default accounts and API access tokens for new Namespaces.

etcd

A persistent, consistent, and distributed key-value data store used to store all Kubernetes cluster information ¹. New data is appended to the data store, never replaced. Obsolete data is periodically compacted to minimize the size of the data store.

Of all the control plane components, only the API server can communicate with the etcd data store.

The etcd CLI management tool, etcdctl, provides options for backups, snapshots, and restores. These are especially useful for a single-instance etcd Kubernetes cluster, common in development and learning environments. However, in Staging and Production environments, it is extremely important to replicate data stores in high availability mode.

Some Kubernetes cluster bootstrapping tools, such as kubeadm, provision stacked etcd master nodes, where the data store runs alongside the other control plane components on the same master node and shares resources with them ².

For data store isolation from the control plane components, the bootstrapping process can be configured for an external etcd topology. The data store is deployed on a separate dedicated host from the control plane, thus reducing the chances of an etcd failure ².

Both stacked and external etcd topologies support high availability configurations. etcd is based on the Raft consensus protocol, which allows a set of machines to survive the failure of some of them, including master node failures. At any given time, one of the nodes in the group will be the leader and the rest will be followers ².

etcd is written in the Go programming language. In Kubernetes, besides storing the cluster state, etcd is also used to store configuration details such as subnets, ConfigMaps, Secrets, etc.

Worker node

A worker node provides a runtime environment for client applications. Although they are containerized microservices, these applications are encapsulated in pods, controlled by the cluster’s control plane agents running on the master node. Pods are scheduled on worker nodes, where they find the necessary compute, memory, and storage resources to run, and networking to communicate with each other and the outside world. A pod is the smallest scheduling unit in Kubernetes. It is a logical collection of one or more containers scheduled together, and the collection can be started, stopped, or rescheduled as a single unit of work.

Additionally, in a multi-worker Kubernetes cluster, network traffic between client users and the containerized applications deployed in Pods is handled directly by the worker nodes and is not routed through the master node ².

Container Runtime

Although Kubernetes is described as a “container orchestration engine”, it does not have the ability to handle containers directly. To manage the lifecycle of a container, Kubernetes requires a container runtime on the node where a Pod and its containers will be scheduled. Kubernetes supports many container runtimes ²:

Docker: although it is a container platform that uses containerd as its container runtime, it is the most popular option used with Kubernetes
CRI-O: a lightweight container runtime for Kubernetes that also supports Docker image registries
containerd: a simple, portable container runtime that provides robustness
frakti: a hypervisor-based container runtime for Kubernetes

kubelet

The kubelet is an agent that runs on every node and communicates with the control plane components on the master node. It receives pod definitions, primarily from the API server, and interacts with the container runtime on the node to run containers associated with the pod. It also monitors the health and resources of the containers running in pods. The kubelet agent takes a set of Pod specifications, called PodSpecs, that have been created by Kubernetes and ensures that the containers described in them are running and healthy.

The kubelet connects to container runtimes through a plugin based on the Container Runtime Interface (CRI). The CRI consists of protocol buffers, gRPC APIs, libraries, and additional specifications and tools that are currently under development. To connect to interchangeable container runtimes, kubelet uses a shim application that provides a clear abstraction layer between kubelet and the container runtime.

From blog.kubernetes.io

As shown above, the kubelet acting as a gRPC client connects to the CRI shim, which in turn acts as a gRPC server to perform container and image operations. The CRI implements two services: ImageService and RuntimeService. ImageService is responsible for all image-related operations, while RuntimeService is responsible for all pod and container-related operations ².

kube-proxy

kube-proxy is the network agent that runs on every node, responsible for dynamic updates and maintenance of all network rules on the node. It extracts Pod network details and forwards connection requests to Pods.

The kube-proxy is responsible for TCP, UDP, and SCTP stream forwarding or round-robin forwarding across a set of pod backends, and it implements forwarding rules defined by users through Service API objects ².

Addons

Addons are cluster features and functionalities not yet available in Kubernetes, so they are implemented through third-party pods and services ².

DNS: the cluster DNS is a DNS server required to assign DNS records to Kubernetes objects and resources
Dashboard: a general-purpose web-based user interface for cluster management
Monitoring: collects cluster-level container metrics and stores them in a central data store
Logging: collects cluster-level container logs and stores them in a central log store for analysis.

Networking challenges

Decoupled microservices-based applications rely heavily on networking to mimic the tight coupling that was once available in the monolithic era. Networking, in general, is not the easiest to understand and implement. Kubernetes is no exception: as an orchestrator of containerized microservices, it must address several distinct networking challenges ²:

Container-to-container communication within pods
Pod-to-pod communication on the same node and across all cluster nodes
Pod-to-Service communication within the same namespace and across cluster namespaces
External-to-Service communication so that clients can access applications in a cluster.

Container to container within pods

By leveraging the virtualization features of the underlying host OS kernel, a container runtime creates an isolated network space for each container it starts. On Linux, this isolated network space is called a network namespace. A network namespace can be shared between containers or with the host operating system.

When a pod is started, the Container Runtime initializes a special pause container with the sole purpose of creating a network namespace for the pod. All additional containers, created through user requests, running within the Pod will share the Pause container’s network namespace so they can all communicate with each other via localhost.

Pod to pod across nodes

In a Kubernetes cluster, pods are scheduled on nodes in a nearly unpredictable manner. Regardless of their host node, pods are expected to be able to communicate with all other pods in the cluster, all without the implementation of Network Address Translation (NAT). This is a fundamental requirement of any Kubernetes networking implementation.

The Kubernetes networking model aims to reduce complexity and treats Pods as VMs on a network, where each VM is equipped with a network interface, so each Pod receives a unique IP address. This model is called “IP-per-Pod” and ensures pod-to-pod communication, just as virtual machines can communicate with each other on the same network.

However, let us not forget about containers. They share the Pod’s network namespace and must coordinate port assignments within the Pod just as applications would on a VM, while being able to communicate with each other on localhost within the Pod. However, containers are integrated with the overall Kubernetes networking model through the use of Container Network Interface (CNI)-compatible CNI plugins. CNI is a set of specifications and libraries that allow plugins to configure networking for containers. While there are some core plugins, most CNI plugins are third-party Software-Defined Networking (SDN) solutions that implement the Kubernetes networking model. In addition to addressing the fundamental networking model requirement, some networking solutions offer support for network policies. Flannel, Weave, and Calico are just a few of the SDN solutions available for Kubernetes clusters.

Pod to the outside world

A successfully deployed containerized application running in pods within a Kubernetes cluster may require accessibility from the outside world. Kubernetes enables external accessibility through Services, complex encapsulations of routing rule definitions stored in iptables on cluster nodes and implemented by kube-proxy agents. By exposing services to the external world with the help of kube-proxy, applications become accessible from outside the cluster through a virtual IP address.

Introduction to Kubernetes

Tue, 27 Oct 2020 00:00:00 +0000

This post covers the basic introductory concepts of Kubernetes. The main sources of information are the Introduction to Kubernetes course by The Linux Foundation on edX, authored by Chris Pokorni and Neependra Khare, and the official Kubernetes documentation.

What is Kubernetes and why is it used?

Kubernetes or “K8s” is a portable, extensible, open-source platform, licensed under Apache 2.0, for managing workloads and services. It is used for automating the deployment, scaling, and management of containerized applications and was originally designed by Google and donated to the Cloud Native Computing Foundation (part of the Linux Foundation). It supports different container runtime environments, including Docker ¹.

Currently, there is a trend toward running processes in what is known as the “cloud”. However, this was preceded by a model known as monolithic, which relied on outdated software architecture principles, with large components written in legacy programming languages and the entire system deployed on expensive hardware that was costly to manage.

Therefore, the current trend is to separate and simplify each software component to turn them into distributed components, described by their specific characteristics. This creates microservices that can be coupled together and are easy to replace or relocate. The microservices architecture is aligned with the principles of Event-Driven Architecture (EDA) and Service-Oriented Architecture (SOA).

Each microservice is developed in a modern programming language, selected as the most suitable for the type of service and its function. This offers great flexibility in combining microservices with specific hardware when necessary, enabling deployments on low-cost commodity hardware. Although the distributed nature of microservices adds complexity to the architecture, one of the greatest benefits of microservices is scalability. With the overall application becoming modular, each microservice can be scaled individually, either manually or automatically through demand-based autoscaling. Additionally, there is practically no downtime or service disruption for clients because updates are rolled out seamlessly, one service at a time, instead of having to recompile, rebuild, and restart an entire monolithic application ².

In summary, these microservices are deployed in containers and Kubernetes is a container orchestrator. Therefore, to understand what Kubernetes is, it is necessary to review the basic concepts of containers and container orchestrators.

What are containers?

Containers are OS-level virtual spaces that bundle application code with associated libraries and configuration files, along with the dependencies needed for the application to run. They provide scalability and high performance to applications on any infrastructure of your choice. They are best suited for delivering microservices by providing portable and isolated virtual environments so applications can run without interference from other running applications.

Benefits of using containers

The benefits of using containers include ³:

Agile application creation and deployment: Greater ease and efficiency in creating container images instead of virtual machines.
Continuous development, integration, and deployment: Allows container images to be built and deployed frequently and reliably, facilitating rollbacks since the image is immutable.
Dev and Ops separation of concerns: You can create container images at build time rather than at deployment time, decoupling the application from the infrastructure.
Observability: Surfaces not only OS-level information and metrics, but also application health and other signals.
Consistency across development, testing, and production environments: The application runs the same on a laptop as it does in the cloud.
Cloud and OS distribution portability: Runs on Ubuntu, RHEL, CoreOS, your physical datacenter, Google Kubernetes Engine, and everything else.
Application-centric management: Raises the level of abstraction from the OS and virtualized hardware to the application running on a system with logical resources.
Loosely coupled, distributed, elastic, liberated microservices: Applications are broken into smaller, independent pieces that can be deployed and managed dynamically, rather than as a monolithic application running on a single high-capacity machine.
Resource isolation: Makes application performance more predictable.
Resource utilization: Enables greater efficiency and density.

What are container orchestrators?

In development environments, running containers on a single host for application development and testing can be a viable option. However, when migrating to quality assurance (QA) and production (Prod) environments, it is no longer viable because applications and services must meet specific requirements ²:

Fault tolerance.
On-demand scalability.
Optimal resource usage.
Auto-discovery to automatically discover and communicate between components.
Accessibility from the outside world.
Seamless security updates or patches with zero downtime.

Container orchestrators are tools that group systems to form clusters where container deployment and management are automated at scale, meeting the requirements listed above.

There are several container orchestrator solutions, and some of the available ones are:

Amazon Elastic Container Service (ECS).
Azure Container Instances.
Azure Service Fabric.
Kubernetes.
Marathon.
Nomad.
Docker Swarm.

While it is feasible to manage a few containers manually, orchestrators greatly simplify administration for operators, especially when dealing with hundreds or thousands of containers. Most container orchestrators can perform the following actions ²:

Group hosts while creating a cluster.
Schedule containers to run on cluster hosts based on resource availability.
Allow containers in a cluster to communicate with each other regardless of the host they are deployed on in the cluster.
Bind containers and storage resources.
Group sets of similar containers and bind them to load-balancing constructs to simplify access to containerized applications, creating a level of abstraction between containers and the user.
Manage and optimize resource usage.
Allow the implementation of policies to secure access to applications running inside containers.

With all these configurable yet flexible features, container orchestrators are a great choice when it comes to managing containerized applications at scale.

Kubernetes features

Kubernetes offers a very broad set of features for container orchestration. Its main features are ²:

Automatic bin packing: Kubernetes automatically schedules containers based on resource needs and constraints, to maximize utilization without sacrificing availability.
Self-healing: Kubernetes automatically replaces and reschedules containers from failed nodes. It kills and restarts containers that do not respond to health checks, based on existing rules or policies. It also prevents traffic from being routed to unresponsive containers.
Horizontal scaling: With Kubernetes, applications scale manually or automatically based on CPU usage or custom metrics.
Service discovery and load balancing: Containers receive their own IP addresses from Kubernetes, while Kubernetes assigns a single Domain Name System (DNS) name to a set of containers to help load balance requests across all containers in the set.
Automated rollouts and rollbacks: Kubernetes seamlessly rolls out and rolls back application updates and configuration changes, constantly monitoring application health to prevent any downtime.
Secret and configuration management: Kubernetes manages sensitive data and application configuration details separately from the container image, to avoid rebuilding the respective image. Secrets consist of sensitive or confidential information passed to the application without revealing the sensitive content to the code configuration.
Storage orchestration: Kubernetes automatically mounts Software-Defined Storage (SDS) solutions to containers from local storage, external cloud providers, distributed storage, or network storage systems.
Batch execution: Kubernetes supports batch execution, long-running jobs, and replaces failed containers.

The architecture of Kubernetes is modular and pluggable. It not only orchestrates microservice-type applications as decoupled modules, but its own architecture also follows decoupled microservice patterns. Kubernetes functionality can be extended by writing custom resources, operators, custom APIs, scheduling rules, or plugins.