Monitoring on Adur

Observability stack: Prometheus, Grafana, and alerting

Sun, 25 Aug 2024 00:00:00 +0000

The Three Pillars of Observability

Observability is the ability to understand what is happening inside your systems by examining their external outputs. It rests on three pillars:

Metrics are numeric measurements collected over time: CPU usage, request latency, error rates, queue depths. They’re cheap to store, fast to query, and perfect for dashboards and alerts. Prometheus is the dominant tool here.

Logs are timestamped text records of discrete events: application errors, access logs, audit trails. They provide detailed context that metrics can’t. Loki, Elasticsearch, and Fluentd handle log aggregation.

Traces follow a single request as it moves through multiple services, showing latency at each hop. Jaeger and Tempo are the main open-source options. Traces are essential for debugging distributed systems, but they’re the most complex to instrument.

This guide focuses on metrics and logs using the Prometheus + Grafana + Loki stack, which covers the majority of observability needs for most teams.

Prometheus Architecture

Prometheus uses a pull-based model: instead of applications pushing metrics to a central collector, Prometheus scrapes HTTP endpoints at regular intervals. This design has some nice advantages:

Services don’t need to know about the monitoring system
Prometheus controls the scrape rate and detects when targets go down
You can easily run it locally against any service that exposes a /metrics endpoint

Core Components

Prometheus Server: scrapes targets, stores time-series data, evaluates alert rules
Exporters: translate metrics from third-party systems (node_exporter for Linux, mysqld_exporter for MySQL)
Pushgateway: accepts metrics pushed by short-lived batch jobs
Alertmanager: receives alerts from Prometheus and routes them to your notification channels

Scrape Configuration

A basic prometheus.yml configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


global:
 scrape_interval: 15s
 evaluation_interval: 15s

rule_files:
 - "alert_rules.yml"

alerting:
 alertmanagers:
 - static_configs:
 - targets:
 - "alertmanager:9093"

scrape_configs:
 - job_name: "prometheus"
 static_configs:
 - targets: ["localhost:9090"]

 - job_name: "node-exporter"
 static_configs:
 - targets: ["node-exporter:9100"]

 - job_name: "application"
 metrics_path: "/metrics"
 static_configs:
 - targets: ["app:8080"]

 # Kubernetes service discovery
 - job_name: "kubernetes-pods"
 kubernetes_sd_configs:
 - role: pod
 relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
 action: keep
 regex: true
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
 action: replace
 target_label: __metrics_path__
 regex: (.+)

In Kubernetes, service discovery automatically finds pods annotated with prometheus.io/scrape: "true". You don’t have to manually list every target anymore.

PromQL Basics

PromQL is Prometheus’s query language. Here are the most useful patterns:

Instant Vectors and Rate

1
2
3
4
5
6
7
8


# Current CPU usage per core
node_cpu_seconds_total{mode="idle"}

# Per-second rate of HTTP requests over the last 5 minutes
rate(http_requests_total[5m])

# Request rate by status code
sum(rate(http_requests_total[5m])) by (status_code)

Latency Percentiles with Histograms

1
2
3
4
5


# 95th percentile request latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# 99th percentile by endpoint
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler))

Error Rates

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Error rate as percentage
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

# Availability (inverse of error rate)
1 - (
 sum(rate(http_requests_total{status_code=~"5.."}[5m]))
 /
 sum(rate(http_requests_total[5m]))
)

Resource Utilization

1
2
3
4
5


# Memory usage percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk usage percentage
(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

Grafana Dashboards

Grafana connects to Prometheus as a data source and lets you build dashboards with panels for graphs, tables, gauges, and heatmaps.

Setup

Add Prometheus as a data source in Grafana either through the UI or via provisioning:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
 - name: Prometheus
 type: prometheus
 access: proxy
 url: http://prometheus:9090
 isDefault: true

 - name: Loki
 type: loki
 access: proxy
 url: http://loki:3100

Dashboard Provisioning

Instead of creating dashboards manually in the UI, store them as JSON files and provision them automatically:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# grafana/provisioning/dashboards/default.yml
apiVersion: 1
providers:
 - name: "Default"
 orgId: 1
 folder: ""
 type: file
 disableDeletion: false
 updateIntervalSeconds: 30
 options:
 path: /var/lib/grafana/dashboards
 foldersFromFilesStructure: true

Place your dashboard JSON files in the mounted directory. Export existing dashboards from the Grafana UI using the share/export feature and commit them to Git. This gives you version-controlled, reproducible dashboards.

Pro tip: when exporting dashboards for provisioning, replace hardcoded datasource UIDs with the variable ${DS_PROMETHEUS} so they work across environments.

Loki for Log Aggregation

Loki is Grafana’s log aggregation system. It’s designed to be cost-effective by indexing only metadata (labels) rather than full log content. It pairs naturally with Grafana, letting you correlate logs and metrics in the same dashboard.

Architecture

Loki uses the same label-based approach as Prometheus. Logs are tagged with labels like {job="myapp", namespace="production"} and queried using LogQL:

1
2
3
4
5
6
7
8


# All error logs from the payment service
{job="payment-service"} |= "error"

# JSON-structured logs, filter by level and extract fields
{job="api"} | json | level="error" | line_format "{{.msg}}"

# Count errors per service over time
sum(count_over_time({job=~".+"} |= "error" [5m])) by (job)

Log Collection with Promtail

Promtail is the agent that ships logs to Loki. A basic configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


server:
 http_listen_port: 9080

positions:
 filename: /tmp/positions.yaml

clients:
 - url: http://loki:3100/loki/api/v1/push

scrape_configs:
 - job_name: containers
 static_configs:
 - targets:
 - localhost
 labels:
 job: containers
 __path__: /var/log/containers/*.log

In Kubernetes, deploy Promtail as a DaemonSet to collect logs from all nodes automatically.

Alerting with Alertmanager

Alertmanager handles alert routing, grouping, deduplication, and silencing. Prometheus evaluates alert rules and fires alerts to Alertmanager, which then delivers notifications.

Alert Rules

Define alert rules in a file referenced by prometheus.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# alert_rules.yml
groups:
 - name: application
 rules:
 - alert: HighErrorRate
 expr: |
 sum(rate(http_requests_total{status_code=~"5.."}[5m]))
 /
 sum(rate(http_requests_total[5m]))
 > 0.05
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "High error rate detected"
 description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes"

 - alert: HighLatency
 expr: |
 histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
 > 1.0
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "High p95 latency"
 description: "95th percentile latency is {{ $value }}s"

 - alert: DiskSpaceLow
 expr: |
 (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
 > 85
 for: 15m
 labels:
 severity: warning
 annotations:
 summary: "Disk space above 85%"
 description: "Disk usage on {{ $labels.instance }} is {{ $value }}%"

Alertmanager Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


# alertmanager.yml
global:
 resolve_timeout: 5m

route:
 group_by: ["alertname", "severity"]
 group_wait: 30s
 group_interval: 5m
 repeat_interval: 4h
 receiver: "default"
 routes:
 - match:
 severity: critical
 receiver: "pagerduty-critical"
 repeat_interval: 1h
 - match:
 severity: warning
 receiver: "slack-warnings"

receivers:
 - name: "default"
 slack_configs:
 - api_url: "https://hooks.slack.com/services/XXX/YYY/ZZZ"
 channel: "#alerts"
 title: '{{ .GroupLabels.alertname }}'
 text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

 - name: "pagerduty-critical"
 pagerduty_configs:
 - service_key: "<pagerduty-service-key>"

 - name: "slack-warnings"
 slack_configs:
 - api_url: "https://hooks.slack.com/services/XXX/YYY/ZZZ"
 channel: "#warnings"

Docker Compose Deployment

Here’s a complete docker-compose.yml to spin up the full observability stack locally:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64


version: "3.8"

services:
 prometheus:
 image: prom/prometheus:latest
 volumes:
 - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
 - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml
 - prometheus_data:/prometheus
 ports:
 - "9090:9090"
 command:
 - "--config.file=/etc/prometheus/prometheus.yml"
 - "--storage.tsdb.retention.time=30d"

 alertmanager:
 image: prom/alertmanager:latest
 volumes:
 - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
 ports:
 - "9093:9093"

 grafana:
 image: grafana/grafana:latest
 volumes:
 - ./grafana/provisioning:/etc/grafana/provisioning
 - ./grafana/dashboards:/var/lib/grafana/dashboards
 - grafana_data:/var/lib/grafana
 ports:
 - "3000:3000"
 environment:
 - GF_SECURITY_ADMIN_PASSWORD=changeme

 loki:
 image: grafana/loki:latest
 volumes:
 - loki_data:/loki
 ports:
 - "3100:3100"

 promtail:
 image: grafana/promtail:latest
 volumes:
 - ./promtail/config.yml:/etc/promtail/config.yml
 - /var/log:/var/log:ro
 command: -config.file=/etc/promtail/config.yml

 node-exporter:
 image: prom/node-exporter:latest
 ports:
 - "9100:9100"
 volumes:
 - /proc:/host/proc:ro
 - /sys:/host/sys:ro
 - /:/rootfs:ro
 command:
 - "--path.procfs=/host/proc"
 - "--path.sysfs=/host/sys"
 - "--path.rootfs=/rootfs"

volumes:
 prometheus_data:
 grafana_data:
 loki_data:

Start the stack with:

1

docker-compose up -d

Access Grafana at http://localhost:3000, Prometheus at http://localhost:9090, and Alertmanager at http://localhost:9093.

Production Tips

Use recording rules for expensive PromQL queries that dashboards run frequently. Pre-compute and store the result as a new metric to reduce query load.
Set retention based on resolution: keep high-resolution data (15s intervals) for 15 days, then downsample to 5m for 90 days using Thanos or Cortex for long-term storage.
Label cardinality matters: avoid labels with unbounded values (user IDs, request IDs). High cardinality labels will blow up Prometheus memory usage.
Use Grafana folders and teams to organize dashboards by service or team. Skip the mega-dashboard that tries to show everything.
Alert on symptoms, not causes: alert on “error rate is high” rather than “Pod restarted.” Users care about the impact, not the internal mechanism.
Implement alert runbooks: every alert should link to a runbook describing what to check and how to mitigate. Add the link in the alert annotation.
Test your alerts: use promtool check rules alert_rules.yml to validate rule syntax. Use unit tests for complex PromQL expressions.
Secure your stack: put Grafana behind SSO/OAuth, restrict Prometheus access to internal networks, enable TLS between components in production.

The Prometheus + Grafana + Loki stack provides a solid observability foundation that scales well for most organizations. Start with metrics and alerting, add log aggregation when you need to correlate events, and introduce tracing when debugging cross-service latency becomes something you do regularly.

Introduction to AIOps: intelligent IT operations

Mon, 05 Dec 2022 00:00:00 +0000

What is AIOps?

AIOps (Artificial Intelligence for IT Operations) applies machine learning and data analytics to operational data (logs, metrics, events, traces) to automate and improve workflows. Gartner coined the term in 2017, but the idea is simple: use algorithms to handle the volume and complexity that humans can’t manage manually.

In practical terms, AIOps platforms ingest data from monitoring tools, APM systems, log aggregators, and event sources. They apply ML models to detect anomalies, correlate events, identify root causes, and in some cases trigger automated remediation. The goal is to reduce mean time to detection (MTTD) and mean time to resolution (MTTR) while freeing operations teams from alert fatigue.

Why traditional monitoring falls short

Monitoring used to work fine. You had a few servers, a handful of apps, and a limited set of metrics to watch. A static CPU threshold or log regex was enough.

Modern infrastructure broke that model:

Scale: A medium Kubernetes cluster generates millions of metrics and logs per minute. You can’t humanly watch dashboards at that scale.
Complexity: Microservices create tangled dependency graphs. One user request might touch dozens of services. Finding what caused a latency spike means correlating data across all of them.
Dynamic environments: Auto-scaling, ephemeral containers, and serverless functions mean baselines constantly shift. Static thresholds explode with false positives.
Alert fatigue: Teams get buried in alerts. When 90% is noise, that critical 10% disappears. Engineers start ignoring everything.

AIOps doesn’t replace monitoring. It layers on top of what you already have and makes it smarter.

Key capabilities

1. Anomaly detection

Instead of static thresholds, AIOps uses ML models (often time-series analysis, clustering, or autoencoders) to learn what “normal” looks like for each metric and service. When behavior deviates significantly from the learned baseline, an anomaly is flagged.

This handles the dynamic baseline problem. If your application normally sees a traffic spike every Monday at 9 AM, the model learns that pattern and does not alert on it. But an unexpected spike at 3 AM on a Wednesday gets flagged.

2. Event correlation

A single infrastructure issue can generate hundreds or thousands of related alerts across different monitoring tools. AIOps correlates these events — grouping them by time, topology, and causal relationships — to present a single incident instead of a wall of alerts.

For example, a network switch failure might trigger alerts on: the switch itself, all connected servers (connectivity lost), all applications on those servers (health check failures), and downstream services (timeout errors). An AIOps platform correlates all of these into one incident: “Network switch X failed.”

3. Root cause analysis

Beyond correlation, AIOps attempts to identify the root cause of an incident. By understanding the topology of your infrastructure and the causal chain of events, it can suggest that the network switch failure is the root cause, rather than presenting the application timeout as an independent issue.

This is where the value becomes tangible. Instead of an on-call engineer spending 30 minutes tracing through dashboards and logs, the platform surfaces the probable root cause immediately.

4. Auto-remediation

The most mature AIOps implementations close the loop by triggering automated remediation actions. If a known pattern is detected (disk filling up, a pod in CrashLoopBackOff, a runaway process consuming memory), the platform can execute predefined runbooks automatically.

Examples:

Restart a crashed pod or service.
Scale up a deployment when anomalous load is detected.
Clear a log directory when disk usage exceeds a dynamic threshold.
Trigger a failover when a primary database becomes unresponsive.

Auto-remediation requires careful design. Start with low-risk actions and expand as confidence grows.

Common platforms and tools

The AIOps landscape includes both commercial platforms and open-source building blocks:

Commercial platforms

Platform	Strengths
Dynatrace	Strong auto-discovery, AI engine (Davis), full-stack observability
Datadog	Unified monitoring + ML-powered alerting, Watchdog anomaly detection
Splunk ITSI	Powerful log analytics + ML toolkit, good for event correlation
Moogsoft	Pioneered AIOps space, strong event correlation and noise reduction
BigPanda	Event correlation and automation focused, integrates with existing tools
PagerDuty	Incident management with ML-driven noise reduction and smart grouping

Open-source building blocks

You can assemble an AIOps-like stack from open-source components:

Data collection: Prometheus, Grafana Agent, OpenTelemetry Collector, Fluentd/Fluent Bit.
Data storage: Prometheus (metrics), Elasticsearch/OpenSearch (logs), Jaeger/Tempo (traces).
Anomaly detection: Facebook Prophet, Isolation Forest (scikit-learn), luminol, Grafana ML.
Event correlation: Custom logic on top of event streams, or StackStorm for event-driven automation.
Alerting and automation: Alertmanager, Grafana OnCall, StackStorm, Rundeck.

Building a custom AIOps stack is significantly more work than using a commercial platform, but it gives you full control and avoids vendor lock-in. A reasonable middle ground is using a commercial platform for core AIOps capabilities while keeping your data pipeline open-source.

Practical use cases

Noise reduction in alert management

A team receiving 500+ alerts per day implements AIOps event correlation. Related alerts are grouped into incidents, duplicates are suppressed, and flapping alerts are silenced. Alert volume drops by 80%, and the on-call engineer can focus on actual incidents.

Proactive capacity planning

AIOps models analyze historical resource usage trends and predict when capacity limits will be reached. Instead of reacting to a disk-full alert at 2 AM, the platform predicts the issue two weeks in advance and creates a ticket for the team to address during business hours.

Faster incident response

During a production outage, the AIOps platform correlates alerts across the monitoring stack, identifies the root cause (a recent deployment that introduced a memory leak), and surfaces the relevant deployment commit. MTTR drops from 45 minutes to 10 minutes.

Automated scaling

The platform detects anomalous traffic patterns that deviate from the learned baseline. Instead of waiting for CPU to hit 80% (the static threshold), it triggers a scale-up action based on the rate of change, ensuring capacity is ready before users experience degradation.

How AIOps fits into DevOps workflows

AIOps is not a replacement for DevOps practices. It is an enhancement layer:

1
2
3
4
5


Code ──> CI/CD Pipeline ──> Deploy ──> Observe ──> AIOps Layer ──> Act
 │ │
 Monitoring Stack ML Models
 (metrics, logs, (anomaly detection,
 traces, events) correlation, RCA)

Developers benefit from faster root cause identification when their code causes issues in production.
Operations teams benefit from noise reduction, automated remediation, and proactive alerting.
SRE teams benefit from data-driven SLO tracking and error budget burn rate analysis.

AIOps works best when your observability foundation is solid. If you are not collecting good data (structured logs, meaningful metrics, distributed traces), ML models will not produce meaningful insights. Fix your observability first, then layer AIOps on top.

Getting started: A pragmatic path

If AIOps sounds useful, here’s a practical approach:

Audit your current observability stack. What data are you collecting? Do you have structured logs? Consistently labeled metrics? Traces across services? AIOps can only work with good data.
Start with noise reduction. This is the lowest-hanging fruit. Implement alert grouping and deduplication. Even basic rules-based correlation (before any ML) will reduce alert fatigue significantly.
Add anomaly detection to key metrics. Pick 3-5 critical business and infrastructure metrics. Apply a time-series anomaly detection model. Facebook Prophet or Prometheus recording rules with seasonal adjustments are good starting points.
Implement automated remediation for known issues. Identify the top 5 recurring incidents. Write runbooks for them. Automate the runbooks using StackStorm, Rundeck, or your platform’s automation engine.
Evaluate a commercial platform when complexity demands it. If you have hundreds of services, multiple monitoring tools, and a growing operations team, the investment in a commercial AIOps platform may be justified by the reduction in MTTR alone.
Measure the impact. Track MTTD, MTTR, alert-to-incident ratio, and false positive rate. Without metrics, you can’t prove AIOps is worth the investment.

AIOps isn’t magic. It’s a set of techniques that, applied to solid operational data, can reduce the burden on ops teams and improve reliability. Start small, measure everything, and scale what actually works.