Security on Adur

Shift-left security: integrating Trivy and Semgrep in GitLab CI

Mon, 16 Mar 2026 00:00:00 +0000

Why these two tools and not others

The security tooling landscape for pipelines is huge, and it is easy to end up with a pipeline that spends twenty minutes just on scans. After trying several combinations, I stick with Trivy and Semgrep for a simple reason: they cover two distinct attack surfaces with minimal friction.

Semgrep analyzes your source code looking for dangerous patterns — SQL injections, insecure deserialization, hardcoded secrets. It does this fast and without needing to compile anything. Trivy, on the other hand, takes care of everything that is not your code: dependencies with known CVEs, outdated base images, problematic IaC configurations. Between the two you cover your own code and third-party code.

Both are open-source, run without an external server, and produce JSON output that you can easily parse in CI. No licenses or third-party dashboards needed to get started.

Pipeline structure

The idea is that security should not be an isolated stage at the end, but something that runs in parallel with the rest of your checks. Here is the general layout:

1
2
3
4
5
6
7
8
9


stages:
 - test
 - security
 - build
 - deploy

variables:
 TRIVY_SEVERITY: "HIGH,CRITICAL"
 SEMGREP_RULES: "p/owasp-top-ten p/security-audit"

The security stage runs at the same level as test. If any scan fails, the pipeline stops before building the image or deploying anything.

Setting up Semgrep

Basic job

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


semgrep:
 stage: security
 image: semgrep/semgrep:latest
 script:
 - semgrep ci --config "$SEMGREP_RULES" --json --output semgrep-results.json
 artifacts:
 paths:
 - semgrep-results.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

This runs Semgrep on every merge request and on every push to the main branch. Results are always saved as an artifact, even if the pipeline fails — you will want to review them later.

Custom rules

Generic rulesets are fine to start with, but as soon as you have project-specific patterns you want to catch, you will need custom rules. Create a .semgrep/ directory at the project root:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# .semgrep/no-env-secrets.yml
rules:
 - id: no-os-environ-secrets
 patterns:
 - pattern: os.environ[$KEY]
 - metavariable-regex:
 metavariable: $KEY
 regex: ".*(SECRET|PASSWORD|TOKEN|KEY).*"
 message: "Direct access to secrets from environment variables. Use the secrets manager."
 languages: [python]
 severity: WARNING

Then reference it in the pipeline:

1
2


variables:
 SEMGREP_RULES: "p/owasp-top-ten p/security-audit .semgrep/"

Handling false positives

There will be false positives. That is unavoidable. What matters is how you handle them. The worst reaction is to disable the entire rule or slap allow_failure: true on the job. Instead, use inline annotations:

1
2


# nosemgrep: python.lang.security.audit.hardcoded-password
TEST_PASSWORD = "dummy" # test fixture, not used in production

Every suppression should have a comment explaining why it is safe to ignore. No exceptions. If you cannot justify it, do not suppress it.

For broader suppressions, use .semgrepignore:

1
2
3
4


# Exclude test fixtures
tests/fixtures/
# Exclude auto-generated code
*_generated.py

Setting up Trivy

Dependency scanning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


trivy-fs:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy fs --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-fs.json .
 artifacts:
 paths:
 - trivy-fs.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Trivy examines the project’s lockfiles (package-lock.json, requirements.txt, go.sum, etc.) and cross-references versions against vulnerability databases. The --exit-code 1 flag makes the job fail if it finds anything HIGH or CRITICAL.

Container image scanning

If you build Docker images, scan them before pushing to the registry:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


trivy-image:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 needs:
 - job: build-image
 artifacts: true
 script:
 - trivy image --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-image.json "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
 artifacts:
 paths:
 - trivy-image.json
 when: always
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

This job depends on the image build (via needs) and only runs on the main branch, not on every MR. Scanning images is slower than scanning the filesystem, so reserve that for what actually gets deployed.

IaC scanning

One often overlooked advantage of Trivy is that it also analyzes infrastructure configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


trivy-iac:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy config --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-iac.json .
 artifacts:
 paths:
 - trivy-iac.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 changes:
 - "**/*.tf"
 - "**/Dockerfile"
 - "**/*.yml"
 - "**/*.yaml"

It catches Dockerfiles running as root, Terraform files with overly open security groups, and Kubernetes configs without resource limits. The changes block ensures it only runs when relevant files are modified, avoiding unnecessary scans.

Blocking policies

Do not start blocking everything from day one. That breeds frustration, creative workarounds, and eventually someone puts allow_failure: true on every security job.

Better to do it in phases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Phase 1: Report only
semgrep:
 allow_failure: true

# Phase 2: Block critical only
semgrep:
 script:
 - semgrep ci --config "$SEMGREP_RULES" --json --output semgrep-results.json
 - |
 CRITICAL=$(jq '[.results[] | select(.extra.severity == "ERROR")] | length' semgrep-results.json)
 if [ "$CRITICAL" -gt 0 ]; then
 echo "Blocked: $CRITICAL critical findings"
 exit 1
 fi
 allow_failure: false

# Phase 3: Block high + critical
# ... adjust the jq filter

Same goes for Trivy. The --severity flag already lets you control which levels block the pipeline. Start with CRITICAL only, and once the team has adapted, add HIGH.

The complete pipeline

Putting it all together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


stages:
 - test
 - security
 - build
 - deploy

variables:
 TRIVY_SEVERITY: "HIGH,CRITICAL"
 SEMGREP_RULES: "p/owasp-top-ten p/security-audit"

semgrep:
 stage: security
 image: semgrep/semgrep:latest
 script:
 - semgrep ci --config "$SEMGREP_RULES" --json --output semgrep-results.json
 - |
 CRITICAL=$(jq '[.results[] | select(.extra.severity == "ERROR")] | length' semgrep-results.json)
 echo "Critical findings: $CRITICAL"
 if [ "$CRITICAL" -gt 0 ]; then
 echo "Pipeline blocked"
 exit 1
 fi
 artifacts:
 paths:
 - semgrep-results.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

trivy-fs:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy fs --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-fs.json .
 artifacts:
 paths:
 - trivy-fs.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

trivy-config:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy config --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-iac.json .
 artifacts:
 paths:
 - trivy-iac.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 changes:
 - "**/*.tf"
 - "**/Dockerfile"
 - "**/*.yml"

trivy-image:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 needs:
 - job: build
 artifacts: true
 script:
 - trivy image --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-image.json "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
 artifacts:
 paths:
 - trivy-image.json
 when: always
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Semgrep and the Trivy scans run in parallel within the security stage. If any of them fails, the pipeline stops before building or deploying.

Surfacing results in merge requests

JSON reports are fine for auditing, but developers need feedback visible directly in the MR. GitLab supports native security reports if you use official templates, but with external tools you can parse the JSON and comment on the MR:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


comment-results:
 stage: .post
 image: alpine:latest
 script:
 - apk add --no-cache curl jq
 - |
 SEMGREP_COUNT=$(jq '.results | length' semgrep-results.json 2>/dev/null || echo "0")
 TRIVY_COUNT=$(jq '.Results[]?.Vulnerabilities // [] | length' trivy-fs.json 2>/dev/null || echo "0")

 BODY="### Security summary\n- Semgrep: ${SEMGREP_COUNT} findings\n- Trivy: ${TRIVY_COUNT} vulnerabilities"

 curl --request POST \
 --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" \
 --header "Content-Type: application/json" \
 --data "{\"body\": \"$BODY\"}" \
 "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/notes"
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 allow_failure: true

Not the most elegant solution, but it works. If you are on GitLab Ultimate you get integrated security reports. If not, this gives enough visibility.

Cache and performance

Scans add time to the pipeline, and if that time is excessive the team will end up disabling them. A few things that help:

1
2
3
4
5
6
7
8


trivy-fs:
 variables:
 TRIVY_CACHE_DIR: ".trivycache/"
 cache:
 key: trivy-db
 paths:
 - .trivycache/
 # ...rest of the job

Caching Trivy’s vulnerability database avoids downloading it on every run. It is roughly 40MB downloaded from GitHub, and on shared runners that download can take a while.

For Semgrep, the binary itself is already fast, but if your repository is large, limit the paths:

1
2
3


semgrep:
 script:
 - semgrep ci --config "$SEMGREP_RULES" --include="src/" --include="app/" --json --output semgrep-results.json

No point scanning node_modules/, vendor/, or asset directories.

What I learned running this in practice

I have been using this setup in production for a while now, and some things only become clear with actual use.

Trivy flags a lot of CVEs in base images that have no fix available. If you do not filter with --ignore-unfixed, you will have constant noise. Better to add that flag and focus on what you can actually fix:

1

trivy image --ignore-unfixed --severity HIGH,CRITICAL my-image:latest

With Semgrep, community rulesets are a good starting point, but the rules that deliver the most value are the ones you write yourself, tailored to your project’s patterns. A single rule that catches unparameterized ORM queries is worth more than a hundred generic rules.

And the most important thing: do not try to plug every hole at once. Start with scans in report-only mode, review what comes up, tune the rules, and only then start blocking. If you block everything on day one, someone will have put allow_failure: true on every job by day two.

Lessons learned in DevSecOps

Mon, 20 Oct 2025 00:00:00 +0000

DevSecOps gets thrown around in job descriptions and conference talks a lot. But behind the buzzword are real lessons that only come from doing the work. From building pipelines that break when you add security gates, to watching teams ignore the tools you spent months deploying, to finally finding what actually works.

These are lessons we learned the hard way. They’re opinionated, practical, shaped by experience.

Security is everyone’s responsibility

Sounds like a break room poster, but it’s the most important lesson here. If security is only the security team’s job, you’ve lost.

Developers make security decisions every time they write code, whether they know it or not. How they validate input. How they handle secrets. How they configure network access. Every PR is a security event.

What works: make security part of the normal development workflow, not a gate at the end. Developers learn when they get fast feedback on security issues in their PR. They resent finding out three weeks later from an auditor.

We’ve seen this repeatedly: teams that treat security as shared responsibility find fewer critical vulnerabilities in production. Teams that silo it find them in the news.

Automate everything you can

Manual security processes do not scale. Period. If your security review is a human reading a checklist, it will be skipped under deadline pressure, inconsistently applied, and resented by everyone involved.

Automate the things that can be automated:

Dependency scanning in every CI build (Dependabot, Snyk, Trivy)
Static analysis on every pull request (Semgrep, SonarQube)
Secret detection as a pre-commit hook and CI check (gitleaks, detect-secrets)
Container image scanning before deployment (Trivy, Grype)
Infrastructure as Code scanning (tfsec, Checkov, KICS)
Compliance as Code for runtime policy enforcement (OPA, Kyverno)

The goal is not to catch everything automatically. The goal is to catch the easy stuff automatically so that human reviewers can focus on the hard stuff: business logic flaws, design-level security issues, threat modeling.

Start Small

One of the biggest mistakes we have made is trying to secure everything at once. You roll out SAST, DAST, SCA, container scanning, IaC scanning, and runtime protection in one quarter. The result? Alert fatigue, developer rebellion, and a wall of unresolved findings that nobody looks at.

Start with one tool, one pipeline, one team. Get it working well. Get developers comfortable with it. Resolve the false positives. Tune the rules. Then expand.

A practical progression:

Month 1: Secret detection in pre-commit hooks and CI. This is uncontroversial and catches real issues.
Month 2: Dependency scanning with automated PR creation for updates. Developers see the value immediately.
Month 3: Container image scanning blocking deployments of critical/high vulnerabilities.
Month 4+: Static analysis, gradually expanding rule sets.

Each step should be stable before moving to the next. Rushing creates noise, and noise teaches people to ignore alerts.

Blameless culture matters

When a security incident happens because someone pushed a secret to a public repo, or because a vulnerability was not patched in time, the response matters more than the incident itself.

If people get blamed, they hide things. They do not report near-misses. They cover up mistakes. And the next incident will be worse because nobody shared the lessons from the last one.

Blameless postmortems are not about letting people off the hook. They are about understanding systemic failures. Why was it possible to push a secret? Why was there no scanning? Why was the patching process slow? Fix the system, not the person.

We have found that teams with genuinely blameless cultures have significantly better security postures. People report suspicious things. They ask for help early. They flag risks before they become incidents.

Tooling is not enough without culture change

We once deployed a comprehensive security scanning pipeline with beautiful dashboards, Slack notifications, Jira ticket creation, the works. Six months later, there were 3,000 unresolved findings and the Slack channel was muted by every developer.

The tools were fine. The culture was not ready.

Before you deploy tooling, invest in:

Training: Developers need to understand why the tool exists and how to act on its findings.
Ownership: Someone needs to own the backlog of findings and triage them. If nobody owns it, nobody does it.
SLAs: Define clear timelines for remediating findings by severity. Critical gets 48 hours. High gets a week. Medium gets a sprint. Low gets a quarter.
Feedback loops: When a tool produces a false positive, there must be an easy way to report it and get the rule tuned. Otherwise, developers learn to ignore everything.

Invest in developer experience for security tools

If your security tool makes developers’ lives harder, they will find a way around it. This is not a character flaw. It is human nature and good engineering instinct: remove obstacles to shipping.

The security tools that get adopted are the ones that:

Run fast: A SAST scan that takes 20 minutes will be bypassed. One that takes 30 seconds will be tolerated.
Integrate natively: Show results in the PR, not in a separate portal. Nobody wants to log into another dashboard.
Have low false positive rates: Every false positive erodes trust. Invest time in tuning.
Provide actionable guidance: “SQL injection vulnerability on line 42” is useless without “here is how to fix it.”
Fail gracefully: If the scanner is down, the pipeline should warn, not block. Availability of the development pipeline is non-negotiable.

We think of it this way: if a developer has to change their workflow to accommodate a security tool, the tool has failed. The best security tooling is invisible.

Monitoring and observability are non-negotiable

You cannot secure what you cannot see. Security monitoring is not optional, and it is not something you bolt on after the fact.

What this means in practice:

Centralized logging: All application, infrastructure, and security tool logs in one place. If you have to SSH into a box to read logs, you are already behind.
Audit trails: Who did what, when, and from where. Every deployment, every config change, every access request.
Alerting on anomalies: Not just “is the service up?” but “is this access pattern normal?” Unusual API call volumes, access from new locations, privilege escalations.
Runtime security: Tools like Falco for container runtime monitoring. Know when something unexpected happens in production.

Monitoring is also how you prove to auditors and customers that your security controls are working. “Trust us” is not a compliance strategy.

Open source is your ally

Some of the best security tools available are open source. Trivy, Falco, OPA, Semgrep, gitleaks, cosign, KICS, Checkov. The ecosystem is rich and maturing fast.

Benefits of open source security tooling:

Transparency: You can read the rules and understand exactly what is being checked.
Community: Thousands of contributors finding edge cases and adding detection rules.
No vendor lock-in: You can switch tools without renegotiating a contract.
Cost: Start for free, scale as needed.

This does not mean commercial tools have no place. Some provide valuable aggregation, management, and support. But you can build a very solid security pipeline with open source tools alone, and we think every team should start there.

Continuous learning is essential

The threat landscape changes constantly. The tools change. The best practices evolve. What was considered secure two years ago might have a CVE today.

What we do to stay current:

Dedicate time for learning: At least a few hours per sprint for the team to read about new vulnerabilities, tools, and techniques. This is not a nice-to-have. It is a professional requirement.
Run internal CTFs and tabletop exercises: Nothing teaches security like trying to break things. Regular exercises keep skills sharp and reveal gaps in your defenses.
Participate in the community: Attend meetups, contribute to open source, read advisories. The security community is generous with knowledge. Take advantage of it.
Review and update: Quarterly reviews of your security tooling, policies, and incident response procedures. What worked last quarter may not work next quarter.

Final Thoughts

DevSecOps isn’t a destination. There’s no point where you say “we’re done, we’re secure.” It’s a continuous practice of reducing risk, improving visibility, building a culture where security is as natural as writing tests.

The most important lesson: perfect is the enemy of good. A basic security pipeline that developers actually use beats a comprehensive one they bypass. Start where you are, improve iteratively, never stop.

Supply chain security: SBOM and Sigstore

Mon, 10 Feb 2025 00:00:00 +0000

Supply chain attacks are no longer theoretical. The 2020 SolarWinds breach showed how one compromised build pipeline could hit thousands of organizations, including government agencies. Then Log4Shell in 2021 proved that a vulnerability deep in a transitive dependency could threaten every Java app overnight. The message was clear: we need visibility into what’s in our software and stronger integrity guarantees.

This guide covers the practical tools: SBOMs, Sigstore, and SLSA framework.

Why Supply Chain Security Matters

Traditional security focuses on your own code: static analysis, dependency scanning, penetration testing. Supply chain security extends that perimeter to everything your software depends on and every step in the process of building and delivering it.

The attack surface includes:

Source code repositories: compromised developer accounts, malicious commits
Dependencies: typosquatting, dependency confusion, compromised upstream packages
Build systems: tampered CI/CD pipelines, injected build steps
Artifact registries: replaced binaries, unsigned packages
Deployment pipelines: modified manifests, man-in-the-middle attacks

A single weak link in any of these stages can compromise the entire chain.

Software Bill of Materials (SBOM)

An SBOM is a formal, machine-readable inventory of all components in a piece of software. Think of it as an ingredients list for your application. It includes direct dependencies, transitive dependencies, their versions, licenses, and relationships.

Why You Need One

Vulnerability response: When a new CVE drops (like Log4Shell), you can instantly check if any of your applications are affected.
License compliance: Know exactly what licenses you are shipping.
Regulatory requirements: The US Executive Order 14028 and the EU Cyber Resilience Act both push toward mandatory SBOMs.
Transparency: Customers and partners can verify what they are running.

SBOM Formats

Two main formats dominate:

SPDX (Software Package Data Exchange): An ISO standard (ISO/IEC 5962:2021), originally focused on license compliance, now comprehensive. Supports JSON, RDF, YAML, and tag-value formats.
CycloneDX: An OWASP project, designed from the ground up for security use cases. Supports JSON and XML. Lighter weight and more opinionated.

Both are solid choices. CycloneDX tends to be easier to work with for security-focused workflows. SPDX has broader adoption in compliance-heavy industries.

Generating SBOMs with Syft

Syft from Anchore is one of the best tools for generating SBOMs. It supports container images, filesystems, and archives.

Install syft:

1

curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

Generate an SBOM from a container image:

1
2
3
4
5


# CycloneDX format (JSON)
syft packages registry.example.com/myapp:v1.2.3 -o cyclonedx-json > sbom.cdx.json

# SPDX format (JSON)
syft packages registry.example.com/myapp:v1.2.3 -o spdx-json > sbom.spdx.json

Generate an SBOM from a local directory:

1

syft packages dir:/path/to/project -o cyclonedx-json > sbom.cdx.json

You can then scan the SBOM for vulnerabilities using Grype:

1

grype sbom:sbom.cdx.json

The Sigstore Ecosystem

Sigstore is an open-source project that makes cryptographic signing and verification accessible. It eliminates the need to manage long-lived signing keys, which has historically been the main barrier to adoption of artifact signing.

The ecosystem has three core components:

cosign: Signs and verifies container images and other OCI artifacts.
Fulcio: A certificate authority that issues short-lived certificates based on OIDC identity (your existing identity provider).
Rekor: A transparency log that creates an immutable, tamper-resistant record of signing events.

How It Works

You authenticate with an OIDC provider (GitHub, Google, Microsoft, etc.).
Fulcio issues a short-lived certificate tied to your identity.
cosign uses that certificate to sign your artifact.
The signing event is recorded in Rekor’s transparency log.
Anyone can verify the signature using the transparency log, without needing your public key.

This is called “keyless” signing. No keys to rotate, no secrets to manage, no PKI infrastructure to maintain.

Signing Container Images with Cosign

Install cosign:

1
2
3
4
5
6


# Using Go
go install github.com/sigstore/cosign/v2/cmd/cosign@latest

# Or download a release
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 -o /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign

Sign an image (keyless mode):

1

cosign sign registry.example.com/myapp:v1.2.3

This will open a browser for OIDC authentication. In CI, you can use workload identity (e.g., GitHub Actions OIDC token) for non-interactive signing.

Verify an image:

1
2
3


cosign verify registry.example.com/myapp:v1.2.3 \
 --certificate-identity=user@example.com \
 --certificate-oidc-issuer=https://accounts.google.com

Attach an SBOM to an image and sign it:

1
2
3
4
5


# Attach the SBOM
cosign attach sbom --sbom sbom.cdx.json registry.example.com/myapp:v1.2.3

# Sign the SBOM attachment
cosign sign --attachment sbom registry.example.com/myapp:v1.2.3

The SLSA Framework

SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) is a framework that defines increasing levels of supply chain integrity guarantees.

SLSA Levels

Level 0: No guarantees. This is where most projects start.
Level 1: The build process is documented and produces provenance (metadata about how an artifact was built).
Level 2: The build is hosted on a hosted build service that generates authenticated provenance.
Level 3: The build platform provides hardened builds with tamper-resistant provenance. The build environment is isolated and ephemeral.

Each level builds on the previous one. The goal is not to jump to Level 3 immediately but to incrementally improve your posture.

SLSA Provenance

Provenance answers the critical questions: Who built this? What source was used? What build process was followed? Was the build environment tamper-proof?

SLSA provenance is a signed attestation in the in-toto format that captures this information.

CI/CD Integration: GitHub Actions Example

Here is a practical GitHub Actions workflow that builds an image, generates an SBOM, signs everything, and generates SLSA provenance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


name: Build, Sign, and Attest

on:
 push:
 tags:
 - 'v*'

permissions:
 contents: read
 packages: write
 id-token: write  # Required for keyless signing

env:
 REGISTRY: ghcr.io
 IMAGE_NAME: ${{ github.repository }}

jobs:
 build-sign-attest:
 runs-on: ubuntu-latest
 steps:
 - name: Checkout
 uses: actions/checkout@v4

 - name: Log in to GHCR
 uses: docker/login-action@v3
 with:
 registry: ${{ env.REGISTRY }}
 username: ${{ github.actor }}
 password: ${{ secrets.GITHUB_TOKEN }}

 - name: Build and push image
 id: build
 uses: docker/build-push-action@v5
 with:
 push: true
 tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}

 - name: Install cosign
 uses: sigstore/cosign-installer@v3

 - name: Install syft
 uses: anchore/sbom-action/download-syft@v0

 - name: Sign the image
 run: |
 cosign sign --yes \
 ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}@${{ steps.build.outputs.digest }}

 - name: Generate SBOM
 run: |
 syft packages \
 ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}@${{ steps.build.outputs.digest }} \
 -o cyclonedx-json > sbom.cdx.json

 - name: Attach and sign SBOM
 run: |
 cosign attach sbom --sbom sbom.cdx.json \
 ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}@${{ steps.build.outputs.digest }}
 cosign sign --yes --attachment sbom \
 ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}@${{ steps.build.outputs.digest }}

 - name: Verify signature
 run: |
 cosign verify \
 --certificate-identity-regexp="https://github.com/${{ github.repository }}/*" \
 --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
 ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}@${{ steps.build.outputs.digest }}

The id-token: write permission is what enables keyless signing in GitHub Actions. The GitHub OIDC token is automatically used by cosign without any manual key management.

Practical Adoption Roadmap

You do not need to do everything at once. Here is a sensible progression:

Week 1-2: Visibility

Start generating SBOMs for your most critical applications using syft.
Integrate Grype into your CI pipeline for vulnerability scanning against the SBOM.

Week 3-4: Signing

Set up cosign keyless signing in your CI/CD pipelines.
Sign your container images on every build.

Month 2: Verification

Enforce signature verification in your deployment pipelines (e.g., Kubernetes admission controllers like Kyverno or Sigstore policy-controller).
Attach SBOMs to images and sign them.

Month 3+: SLSA Provenance

Add SLSA provenance generation using slsa-github-generator.
Work toward SLSA Level 2, then Level 3.
Automate provenance verification in your deployment tooling.

Key Takeaways

SBOMs give visibility - You can’t secure what you can’t see. Generate SBOMs for every artifact.
Sigstore removes the excuse - Keyless signing kills key management overhead. No good reason not to sign.
SLSA is a maturity model - Use it to improve supply chain integrity incrementally, not as all-or-nothing.
Automate everything - These tools are built for CI/CD integration. Manual doesn’t scale.

The supply chain security ecosystem is maturing fast. Tools are production-ready, standards are solidifying, and regulatory pressure keeps rising. The best time to start was yesterday. The second-best is now.

Kubernetes security hardening guide

Sun, 03 Nov 2024 00:00:00 +0000

Kubernetes Attack Surface

Out-of-the-box Kubernetes isn’t secure. The cluster exposes multiple attack vectors that need systematic attention:

API Server - Central control point; unauthorized access = full cluster compromise
etcd - Stores all cluster state including secrets in base64 (not encrypted by default)
Kubelet - Node agent; exposed kubelet API leaks pod info and allows command execution
Container runtime - Breakout vulnerabilities give host access
Network - Pods can communicate freely with any other pod by default
Supply chain - Malicious or vulnerable images introduce backdoors

This guide covers the key hardening measures organized by attack vector.

API Server Hardening

The API server is the most critical component. Secure it first:

Disable anonymous authentication: set --anonymous-auth=false
Enable audit logging: track who did what (covered in detail below)
Restrict access: use firewall rules or security groups to limit who can reach the API server
Enable admission controllers: PodSecurity, NodeRestriction, ResourceQuota, and LimitRanger should all be active
Use OIDC for authentication: integrate with your identity provider instead of relying on client certificates

1
2


# Check current API server flags (on kubeadm clusters)
kubectl -n kube-system get pod kube-apiserver-<node> -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n'

RBAC Best Practices

RBAC is your primary authorization mechanism. Enforce least privilege rigorously.

Key Rules

Never use cluster-admin for workloads – it grants unlimited access
Use Roles (namespaced) over ClusterRoles whenever possible
Avoid wildcards in resource or verb specifications
Bind to ServiceAccounts, not users for automated workloads
Review bindings regularly – permissions accumulate over time

Example: Restricted Deployment Role

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 namespace: production
 name: deployment-manager
rules:
 # Can manage deployments
 - apiGroups: ["apps"]
 resources: ["deployments"]
 verbs: ["get", "list", "watch", "create", "update", "patch"]
 # Can view pods and logs
 - apiGroups: [""]
 resources: ["pods", "pods/log"]
 verbs: ["get", "list", "watch"]
 # Can view services
 - apiGroups: [""]
 resources: ["services"]
 verbs: ["get", "list", "watch"]
 # Explicitly NO access to secrets, configmaps/write, or exec
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 namespace: production
 name: deployment-manager-binding
subjects:
 - kind: ServiceAccount
 name: ci-deployer
 namespace: production
roleRef:
 kind: Role
 name: deployment-manager
 apiGroup: rbac.authorization.k8s.io

Audit existing RBAC with:

1
2
3
4
5
6


# List all cluster-admin bindings
kubectl get clusterrolebindings -o json | \
 jq '.items[] | select(.roleRef.name == "cluster-admin") | {name: .metadata.name, subjects: .subjects}'

# Check what a specific service account can do
kubectl auth can-i --list --as=system:serviceaccount:production:ci-deployer -n production

Pod Security Standards

Kubernetes Pod Security Standards (PSS) define three security restriction levels. Since 1.25, Pod Security Admission (PSA) is the built-in enforcement mechanism (replaces PodSecurityPolicy).

The Three Levels

Level	Description	Use Case
Privileged	No restrictions	System-level workloads (CNI, storage drivers)
Baseline	Blocks known privilege escalations	General-purpose workloads
Restricted	Maximum hardening	Sensitive workloads, multi-tenant clusters

Enforcing Pod Security Standards

Apply standards at the namespace level using labels:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: v1
kind: Namespace
metadata:
 name: production
 labels:
 # Enforce restricted: reject pods that violate
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/enforce-version: latest
 # Warn on baseline violations (shows warning but allows)
 pod-security.kubernetes.io/warn: restricted
 pod-security.kubernetes.io/warn-version: latest
 # Audit: log violations
 pod-security.kubernetes.io/audit: restricted
 pod-security.kubernetes.io/audit-version: latest

Compliant Pod Example

A pod that passes the restricted level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: v1
kind: Pod
metadata:
 name: secure-app
 namespace: production
spec:
 securityContext:
 runAsNonRoot: true
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: app
 image: myregistry.com/app:v1.2.3@sha256:abc123...
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 runAsUser: 1000
 runAsGroup: 1000
 capabilities:
 drop:
 - ALL
 resources:
 limits:
 memory: "256Mi"
 cpu: "500m"
 requests:
 memory: "128Mi"
 cpu: "250m"

Key security settings to always include:

runAsNonRoot: true – never run containers as root
readOnlyRootFilesystem: true – prevent filesystem modifications
allowPrivilegeEscalation: false – block setuid/setgid
capabilities.drop: ["ALL"] – remove all Linux capabilities
seccompProfile.type: RuntimeDefault – apply default seccomp profile
Always specify resource limits to prevent DoS

NetworkPolicies

By default, all pods can communicate with all other pods in a cluster. NetworkPolicies restrict this to only the traffic that is necessary.

Default Deny All

Start every namespace with a default deny policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: production
spec:
 podSelector: {}
 policyTypes:
 - Ingress
 - Egress

Allow Specific Traffic

Then explicitly allow only what is needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-api-to-db
 namespace: production
spec:
 podSelector:
 matchLabels:
 app: database
 policyTypes:
 - Ingress
 ingress:
 - from:
 - podSelector:
 matchLabels:
 app: api-server
 ports:
 - protocol: TCP
 port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: api-server-egress
 namespace: production
spec:
 podSelector:
 matchLabels:
 app: api-server
 policyTypes:
 - Egress
 egress:
 # Allow DNS resolution
 - to: []
 ports:
 - protocol: UDP
 port: 53
 - protocol: TCP
 port: 53
 # Allow connection to database
 - to:
 - podSelector:
 matchLabels:
 app: database
 ports:
 - protocol: TCP
 port: 5432
 # Allow external HTTPS
 - to:
 - ipBlock:
 cidr: 0.0.0.0/0
 except:
 - 10.0.0.0/8
 - 172.16.0.0/12
 - 192.168.0.0/16
 ports:
 - protocol: TCP
 port: 443

Note: NetworkPolicies require a CNI plugin that supports them (Calico, Cilium, Weave Net). The default kubenet does not enforce NetworkPolicies.

Secrets Management

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with read access to Secrets in a namespace can decode them trivially. Proper secrets management requires additional tooling.

Option 1: Sealed Secrets

Sealed Secrets (by Bitnami) encrypts secrets client-side so they can be safely stored in Git:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install kubeseal CLI
# Encrypt a secret
kubectl create secret generic db-creds \
 --from-literal=password=supersecret \
 --dry-run=client -o yaml | \
 kubeseal --format yaml > sealed-db-creds.yaml

# The sealed secret can be committed to Git
# Only the cluster's controller can decrypt it
kubectl apply -f sealed-db-creds.yaml

Option 2: External Secrets Operator

External Secrets Operator syncs secrets from external providers (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) into Kubernetes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: db-credentials
 namespace: production
spec:
 refreshInterval: 1h
 secretStoreRef:
 name: aws-secrets-manager
 kind: ClusterSecretStore
 target:
 name: db-credentials
 data:
 - secretKey: password
 remoteRef:
 key: production/database
 property: password

Enable Encryption at Rest

Ensure etcd encrypts secrets at rest:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
 - resources:
 - secrets
 providers:
 - aescbc:
 keys:
 - name: key1
 secret: <base64-encoded-32-byte-key>
 - identity: {}

Audit Logging

Kubernetes audit logs record every request to the API server, providing a detailed trail of who did what and when.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
 # Log all requests to secrets at the Metadata level
 - level: Metadata
 resources:
 - group: ""
 resources: ["secrets"]
 # Log pod exec/attach at RequestResponse level
 - level: RequestResponse
 resources:
 - group: ""
 resources: ["pods/exec", "pods/attach"]
 # Log all write operations at Request level
 - level: Request
 verbs: ["create", "update", "patch", "delete"]
 # Log everything else at Metadata level
 - level: Metadata
 omitStages:
 - RequestReceived

Enable in the API server with:

1
2
3
4
5


--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100

Ship audit logs to your SIEM or log aggregation system (Loki, Elasticsearch) for analysis and alerting.

Image Policies with Kyverno

Kyverno is a policy engine for Kubernetes that validates, mutates, and generates resources based on policies. It is simpler to adopt than OPA Gatekeeper because policies are written as Kubernetes resources rather than Rego.

Require Image Digest and Trusted Registry

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: require-image-digest
spec:
 validationFailureAction: Enforce
 rules:
 - name: require-digest
 match:
 any:
 - resources:
 kinds:
 - Pod
 validate:
 message: "Images must use a digest (@sha256:...) instead of a tag"
 pattern:
 spec:
 containers:
 - image: "*@sha256:*"
 - name: require-trusted-registry
 match:
 any:
 - resources:
 kinds:
 - Pod
 validate:
 message: "Images must come from the trusted registry"
 pattern:
 spec:
 containers:
 - image: "myregistry.com/*"

This policy ensures that only images from your trusted registry with pinned digests are deployed, preventing both supply chain attacks and tag mutability issues.

CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark provides a comprehensive set of security recommendations. Run automated checks with kube-bench:

1
2
3
4
5


# Run CIS benchmark checks
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml

# View results
kubectl logs job/kube-bench

Address findings by priority: critical items first (API server auth, etcd encryption), then high (RBAC, network policies), then medium and low.

Hardening Checklist

Use this checklist as a starting point for securing your cluster:

API server: anonymous auth disabled, audit logging enabled
etcd: encrypted at rest, access restricted to API server only
RBAC: no unnecessary cluster-admin bindings, least privilege enforced
Pod Security: restricted PSS enforced on production namespaces
NetworkPolicies: default deny in all namespaces, explicit allow rules
Secrets: external secrets manager or sealed secrets, encryption at rest
Images: signed images, trusted registry enforcement, digest pinning
Nodes: automatic security updates, CIS-hardened OS
Audit: API server audit logs shipped to SIEM
Monitoring: alerts on RBAC changes, privileged pod creation, exec into pods
Supply chain: vulnerability scanning in CI/CD, admission-time image scanning
Network: TLS between all components, service mesh for mTLS between pods

Security hardening is not a one-time activity. Schedule quarterly reviews to reassess your posture, run CIS benchmarks, review RBAC bindings, and update policies as your cluster evolves.

DevSecOps maturity model

Sun, 08 Oct 2023 10:00:00 +0100

Why a Maturity Model Helps

Most teams know they should “shift security left,” but knowing where to start is the hard part. A maturity model gives you a structured way to assess your current state, identify gaps, and plan a realistic roadmap for improvement.

Without a model, security improvements tend to be reactive (triggered by incidents or audit findings rather than deliberate planning). A maturity model turns security from a fire drill into an engineering discipline with measurable progress.

The model described here has five levels. The goal is not to rush to the highest level but to make steady, sustainable progress. Each level builds on the previous one.

The Five Maturity Levels

Level 1: Ad-Hoc

At this level, security is an afterthought. There are no formal processes, and security activities happen sporadically if at all.

What it looks like:

No security testing in CI/CD pipelines.
Vulnerabilities discovered in production or by external parties.
No dedicated security tooling.
Developers have little to no security training.
Incident response is improvised.
Compliance is addressed manually before audits.

Typical tools: None specifically for security. Maybe a firewall and antivirus.

Level 2: Reactive

Security is recognized as important, but the approach is reactive. The team responds to vulnerabilities and incidents but doesn’t proactively prevent them.

What it looks like:

Basic static analysis (SAST) runs occasionally, but findings are not always addressed.
Dependency scanning is done manually or on an ad-hoc basis.
There’s some security documentation, but it’s outdated.
Incident response exists as a documented process, though it’s rarely practiced.
Security reviews happen late in the development cycle (right before release).

Typical tools: SonarQube (basic rules), OWASP Dependency-Check, manual penetration testing.

Level 3: Proactive

Security is integrated into the development workflow. The team actively seeks to prevent vulnerabilities rather than just reacting to them.

What it looks like:

SAST and DAST run automatically in CI/CD pipelines.
Dependency scanning with automated alerts for known vulnerabilities.
Container image scanning before deployment (Trivy, Grype).
Infrastructure as Code is scanned for misconfigurations (Checkov, tfsec).
Threat modeling is performed for new features and architecture changes.
Security champions exist within development teams.
Blameless postmortems are conducted after security incidents.
Regular security training for developers.

Typical tools: Semgrep, Trivy, Checkov, OWASP ZAP, HashiCorp Vault, Falco.

Level 4: Optimized

Security is deeply embedded in every stage of the software lifecycle. Metrics drive decisions, and the team continuously improves based on data.

What it looks like:

Security gates in pipelines that block deployment if critical issues are found.
Mean time to remediate (MTTR) is tracked and continuously reduced.
Software Bill of Materials (SBOM) generated for every release.
Signed artifacts and verified supply chain.
Automated compliance checks mapped to frameworks (SOC2, ISO 27001, PCI-DSS).
Runtime security monitoring with automated response (Falco + custom rules).
Regular red team exercises and chaos engineering for security.
Security metrics are part of engineering dashboards.

Typical tools: Sigstore/cosign, OPA/Gatekeeper, Kyverno, SIEM integration, automated compliance platforms.

Level 5: Innovative

Security is a competitive advantage. The team contributes to the broader security community and pushes the state of the art.

What it looks like:

Bug bounty programs actively managed.
Custom security tooling developed for organization-specific risks.
Machine learning applied to anomaly detection and threat hunting.
Security is a feature sold to customers (certifications, transparency reports).
Active participation in open-source security projects.
Zero-trust architecture fully implemented.
Policy as code governs all infrastructure and application security.

Typical tools: Custom-built platforms, eBPF-based security tools, advanced SIEM with ML, zero-trust service mesh.

Key Dimensions

A maturity model isn’t one-dimensional. Assess your organization across these dimensions, as progress is rarely uniform:

Code Security

Level	Practices
Ad-Hoc	No code scanning
Reactive	Occasional SAST, manual code reviews for security
Proactive	Automated SAST/DAST in CI, security-focused code review guidelines
Optimized	Custom rules for organization-specific patterns, MTTR tracked
Innovative	AI-assisted code review, automatic fix suggestions

Infrastructure Security

Level	Practices
Ad-Hoc	Manual server configuration, no hardening standards
Reactive	Basic hardening checklists, occasional audits
Proactive	IaC scanning, automated hardening, CIS benchmarks
Optimized	Policy as code (OPA), drift detection, automated remediation
Innovative	Self-healing infrastructure, zero-trust networking

Monitoring and Detection

Level	Practices
Ad-Hoc	No security monitoring
Reactive	Basic log collection, manual review after incidents
Proactive	Centralized logging, alerting on known patterns, runtime monitoring
Optimized	SIEM with correlation rules, automated response playbooks
Innovative	ML-based anomaly detection, threat hunting programs

Incident Response

Level	Practices
Ad-Hoc	No process, ad-hoc response
Reactive	Documented runbooks, rarely tested
Proactive	Regular tabletop exercises, blameless postmortems, on-call rotation
Optimized	Automated incident classification, SLA-driven response times
Innovative	Chaos engineering for security, automated containment

Compliance

Level	Practices
Ad-Hoc	Manual evidence collection before audits
Reactive	Spreadsheet-based tracking, periodic reviews
Proactive	Automated evidence collection, continuous monitoring
Optimized	Compliance as code, real-time dashboards, automated reporting
Innovative	Continuous certification, public transparency reports

Self-Assessment Checklist

Rate your organization on each item (Yes / Partial / No):

Build Phase:

SAST runs automatically on every pull request.
Dependency scanning alerts on known CVEs.
Container images are scanned before being pushed to a registry.
IaC templates are scanned for misconfigurations.
Secrets detection prevents credentials from being committed.

Deploy Phase:

Security gates can block deployment for critical findings.
Artifacts are signed and signatures are verified.
SBOM is generated for every release.
Infrastructure changes go through policy-as-code validation.

Run Phase:

Runtime security monitoring is active (Falco, Sysdig, etc.).
Centralized logging with security-relevant alerts.
Network segmentation limits blast radius.
Secrets are managed through a dedicated vault.

Culture and Process:

Developers receive regular security training.
Security champions are embedded in development teams.
Blameless postmortems are conducted after incidents.
Threat modeling is part of the design process for new features.
Security metrics are tracked and reviewed regularly.

Roadmap for Progression

Moving up the maturity levels doesn’t happen overnight. Here’s a practical roadmap:

From Ad-Hoc to Reactive (3-6 months)

Add a SAST tool to your CI pipeline (start with Semgrep - it has good defaults and is fast).
Enable dependency scanning (GitHub Dependabot, or trivy fs in CI).
Document your incident response process, even if it’s simple.
Run a single security training session for the team.

From Reactive to Proactive (6-12 months)

Add container image scanning and IaC scanning to pipelines.
Implement secrets detection in pre-commit hooks (gitleaks, detect-secrets).
Appoint security champions in each team.
Start threat modeling for major features.
Conduct your first blameless postmortem after an incident.
Deploy runtime monitoring (Falco).

From Proactive to Optimized (12-18 months)

Implement security gates that can block deployments.
Track MTTR and set reduction targets.
Generate SBOMs and sign artifacts.
Implement policy-as-code for infrastructure (OPA/Gatekeeper).
Map automated checks to compliance frameworks.
Integrate security metrics into engineering dashboards.

From Optimized to Innovative (18+ months)

Launch a bug bounty program.
Build custom security tooling for organization-specific risks.
Implement zero-trust architecture.
Run regular red team exercises.
Contribute to open-source security projects.

Cultural Aspects

Tools and processes are necessary but insufficient. Culture determines whether security practices actually stick.

Blameless Postmortems

When a security incident occurs, the instinct is often to find someone to blame. This drives people to hide mistakes and cover up near-misses. Blameless postmortems flip this around: they focus on systemic failures and process improvements rather than individual fault. The question changes from “who made this mistake?” to “what allowed this mistake to happen, and how do we prevent it?”

Security Champions

A security champion is a developer who takes on extra responsibility for security within their team. They are not full-time security engineers — they are developers who act as a bridge between the security team and the development team. Their role includes:

Reviewing security-relevant pull requests.
Staying current on security topics and sharing knowledge.
Participating in threat modeling sessions.
Being the first point of contact for security questions.

This model scales far better than having a central security team review everything.

Making Security Easy

If security practices are painful, people will find workarounds. The goal is to make security the easiest path:

Provide secure templates and starter projects.
Automate as much as possible so developers don’t have to remember manual steps.
Give fast feedback. A SAST scan that takes 30 minutes will be ignored; one that takes 30 seconds will be used.
Celebrate security improvements just as you celebrate feature delivery.

Conclusion

A DevSecOps maturity model is a compass, not a destination. The value comes from honest self-assessment, setting realistic goals, and making steady progress. Start where you are, pick the dimension where improvement will have the most impact, and build from there. Security is a team sport. The best security cultures are built incrementally, one practice at a time.

How I hardened my K3s homelab

Thu, 14 Sep 2023 00:00:00 +0000

Starting point

I have been running K3s on a couple of physical machines at home for a while. I originally set it up to learn Kubernetes without deploying something heavy, and over time I started using it for small projects: a few web apps, monitoring services, things like that. The setup worked fine but I had never sat down to think carefully about security.

A few months ago I decided to do that exercise. No rush, no generic checklist from the internet. Just look at what I had, understand what was wrong, and fix it.

What I found was not catastrophic, but there were quite a few things I did not like. I am writing this down because K3s has its own quirks compared to a standard Kubernetes cluster, and most hardening guides cover kubeadm or managed clusters, not home setups running on a single binary.

What makes K3s different

Before getting into what I did, it is worth understanding what makes K3s distinct from a security standpoint.

K3s packages everything in a single binary: API server, scheduler, controller manager, kubelet, kube-proxy, and containerd. Instead of etcd it uses SQLite by default. The default CNI is Flannel. The default ingress controller is Traefik. All of this makes the installation very simple, but it also means there are active components you might not need, and some design decisions favor ease of use over security.

The main configuration file lives at /etc/rancher/k3s/config.yaml. That is where most of what I am going to touch ends up.

The node before the cluster

Before touching anything Kubernetes-related, the operating system. The cluster runs on Debian, so I started there.

SSH

I had password authentication enabled. That is the first thing to turn off:

1
2
3
4


# /etc/ssh/sshd_config
PasswordAuthentication no
PermitRootLogin no
AllowUsers myuser

I also changed the default port. It does not stop someone determined from finding you, but it eliminates a lot of noise in the logs.

Firewall with nftables

The machine had its network interfaces completely open. With a home K3s cluster, the API server (port 6443) should not be reachable from outside your local network. My basic rules with nftables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# List active ruleset
nft list ruleset

# Restrictive default policy
nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }

# Allow loopback and established connections
nft add rule inet filter input iifname lo accept
nft add rule inet filter input ct state established,related accept

# SSH only from local network
nft add rule inet filter input ip saddr 192.168.1.0/24 tcp dport 22 accept

# K3s API server only from local network
nft add rule inet filter input ip saddr 192.168.1.0/24 tcp dport 6443 accept

# Kubernetes node ports
nft add rule inet filter input tcp dport 30000-32767 accept

# Traefik (if used as ingress)
nft add rule inet filter input tcp dport { 80, 443 } accept

# ICMP
nft add rule inet filter input icmp type echo-request accept

What surprised me when reviewing this: the API server was listening on all interfaces and was reachable directly from the WAN because the router had a port forward for another service that was dragging along some traffic. Small scare, nothing exploited, but not something I wanted to leave.

Kernel parameters

K3s with the --protect-kernel-defaults flag verifies that certain kernel parameters are set correctly. If they are not, startup fails with a clear message. Better to configure them beforehand:

1
2
3
4
5
6
7


# /etc/sysctl.d/90-k3s-hardening.conf
kernel.panic = 10
kernel.panic_on_oops = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512

1

sysctl --system

K3s configuration

With the node in order, onto the cluster.

Base configuration file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: "0600"
protect-kernel-defaults: true
secrets-encryption: true
kube-apiserver-arg:
 - "anonymous-auth=false"
 - "audit-log-path=/var/log/k3s/audit.log"
 - "audit-log-maxage=30"
 - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
kube-controller-manager-arg:
 - "terminated-pod-gc-threshold=10"
kubelet-arg:
 - "streaming-connection-idle-timeout=5m"
 - "protect-kernel-defaults=true"
 - "make-iptables-util-chains=true"

The three most important lines here:

secrets-encryption: true enables encryption at rest for Kubernetes secrets. In K3s this is a first-class flag, no need to configure EncryptionConfiguration manually like in kubeadm.
anonymous-auth=false removes anonymous access to the API server.
write-kubeconfig-mode: "0600" enforces restrictive permissions on the kubeconfig that K3s generates at /etc/rancher/k3s/k3s.yaml.

Audit policy

Without audit logs you have no idea what is happening in your cluster. This is a reasonable minimum:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# /etc/rancher/k3s/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
 - level: Metadata
 resources:
 - group: ""
 resources: ["secrets"]
 - level: RequestResponse
 resources:
 - group: ""
 resources: ["pods/exec", "pods/attach"]
 - level: Request
 verbs: ["create", "update", "patch", "delete"]
 - level: None
 users: ["system:kube-proxy"]
 verbs: ["watch"]
 resources:
 - group: ""
 resources: ["endpoints", "services", "services/status"]
 - level: Metadata
 omitStages:
 - RequestReceived

The logs go to /var/log/k3s/audit.log. In my case I collect them with Promtail and send them to Loki, so I can run queries from Grafana when I need to review something.

Kubeconfig

The kubeconfig K3s generates at /etc/rancher/k3s/k3s.yaml has admin credentials. A mistake I spotted in my own setup: I had that file copied to ~/.kube/config with 644 permissions. Any process running under my user could read it.

1
2


# Correct permissions
chmod 600 ~/.kube/config

If you have additional users who need cluster access, create ServiceAccounts or use client certificates with limited RBAC. Do not distribute the admin kubeconfig.

CNI: from Flannel to Cilium

Flannel is the K3s default. It works, but it does not support NetworkPolicies out of the box. That means all pods can communicate with each other without restriction.

I switched to Cilium. The process with K3s requires disabling Flannel first:

1
2
3


# /etc/rancher/k3s/config.yaml (add)
flannel-backend: none
disable-network-policy: true

Then install Cilium with Helm:

1
2
3
4
5
6
7
8
9


helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium \
 --namespace kube-system \
 --set operator.replicas=1 \
 --set ipam.mode=kubernetes \
 --set kubeProxyReplacement=strict \
 --set k8sServiceHost=192.168.1.10 \
 --set k8sServicePort=6443

With kubeProxyReplacement=strict, Cilium also replaces kube-proxy, which gives better performance with eBPF and better network traffic visibility.

After the switch I started applying NetworkPolicies. The first and most important one: default deny in all application namespaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: apps
spec:
 podSelector: {}
 policyTypes:
 - Ingress
 - Egress

From there, explicitly allowing only what is necessary.

Secrets: what to do when it is just K3s

In a corporate environment you would use Vault or External Secrets Operator backed by AWS Secrets Manager or similar. In a homelab that is overkill. My solution was simpler.

K3s’s encryption at rest already protects secrets in the database. For manifests in Git I use age to encrypt sensitive values before committing them:

1
2


# Encrypt a value
echo "my-real-password" | age -r $(cat ~/.config/age/recipient.txt) | base64

The encrypted value goes into the repository. Deploying it requires a manual decryption step. Less convenient than an automatic operator, but for a homelab it is good enough and does not add operational complexity.

Falco for runtime detection

Everything above is prevention. Falco is detection: it monitors system calls and generates alerts when something suspicious happens inside a container.

I installed it with Helm:

1
2
3
4
5


helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
 --namespace falco \
 --create-namespace \
 --set driver.kind=ebpf

The default rules already cover quite a bit: shell execution in containers, writes to system directories, privilege changes, unexpected network connections. I added some custom rules for my specific use case.

Alerts go to Loki as well. That way I have the API server audit logs and Falco alerts in the same place.

Pod Security Standards

With K3s >= 1.25 you can use Pod Security Admission directly. I applied the baseline level on application namespaces and restricted where possible:

1
2
3
4
5
6
7


apiVersion: v1
kind: Namespace
metadata:
 name: apps
 labels:
 pod-security.kubernetes.io/enforce: baseline
 pod-security.kubernetes.io/warn: restricted

The restricted level is strict: containers cannot run as root, the root filesystem must be read-only, and all Linux capabilities must be dropped. Some of the workloads I had deployed did not comply. I fixed them one by one before raising the enforce level.

What I left for later

Not everything is perfect. There are things I know I should do that I have not gotten to yet.

Rootless K3s: you can run K3s entirely without root, which reduces the impact of a potential privilege escalation. I have looked into it and it has some limitations with certain network features, but for my use case it should work.

Image verification with cosign: signing and verifying images before deploying them. I have a private registry for my own images and the third-party ones I use, but I am not verifying signatures yet.

CIS Benchmark: K3s has official documentation for the CIS K3s benchmark. I have reviewed it partially but have not run kube-bench systematically to see what is still pending.

What I learned

The default K3s installation is surprisingly reasonable for what it is: a distribution designed to be easy to deploy. But “reasonable” is not the same as “hardened.”

What surprised me most was how easy most of the changes turned out to be. K3s has first-class flags for many things that require manual configuration in kubeadm. The secrets-encryption, protect-kernel-defaults, the audit policy directly in the config file. It is not that hard if you sit down and read the documentation.

The biggest risk in a homelab is not that someone on the internet attacks you directly. It is that you have a bunch of services running on the same network as your personal life, and if one gets compromised, the blast radius can be larger than it seems. Worth taking seriously even if it is a toy environment.

Container security best practices

Fri, 14 Jul 2023 10:00:00 +0100

The Attack Surface of Containers

Containers give you process isolation, not security isolation. A misconfigured container can expose the host kernel, leak secrets, or become a pivot point for lateral movement. Understand the attack surface first:

Container images - Vulnerable libraries, hardcoded credentials, unnecessary tools
Container runtime - Vulnerabilities that allow breakouts
Orchestrator misconfigurations (Kubernetes RBAC, network policies) - Expose services, grant excessive permissions
Supply chain - Compromised base images or dependencies
Secrets - Baked into images or environment variables that any process can access

Security requires defense in depth across the entire lifecycle: build, ship, run.

Image security

Use minimal base images

Every additional package in a container image is a potential vulnerability. Prefer minimal base images:

1
2
3
4


# Prefer distroless or Alpine over full distributions
FROM gcr.io/distroless/static-debian11
# or
FROM alpine:3.18

Distroless images contain only your application and its runtime dependencies — no shell, no package manager, no unnecessary binaries. This dramatically reduces the attack surface.

Multi-stage builds

Multi-stage builds let you compile in one stage and copy only the final artifact to a minimal runtime image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Build stage
FROM golang:1.21 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server

# Runtime stage
FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/server /server
USER nonroot:nonroot
ENTRYPOINT ["/server"]

The build tools, source code, and intermediate files never make it into the final image.

Never run as root

Running containers as root means a container escape gives the attacker root on the host. Always specify a non-root user:

1
2
3


# Create a non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

In Kubernetes, enforce this with Pod Security Standards (more on this below).

Secure vs. insecure Dockerfile comparison

Here is a side-by-side comparison of common mistakes versus secure practices:

1
2
3
4
5
6
7
8


# INSECURE - Do NOT do this
FROM ubuntu:latest
RUN apt-get update && apt-get install -y curl wget vim
COPY . /app
RUN echo "DB_PASSWORD=supersecret" > /app/.env
EXPOSE 22 80 443
USER root
CMD ["python3", "/app/main.py"]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# SECURE - Follow this pattern
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

FROM python:3.11-slim
RUN groupadd -r appgroup && useradd -r -g appgroup appuser
WORKDIR /app
COPY --from=builder /root/.local /home/appuser/.local
COPY --chown=appuser:appgroup . .
ENV PATH=/home/appuser/.local/bin:$PATH
EXPOSE 8080
USER appuser
HEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost:8080/health || exit 1
CMD ["python3", "main.py"]

Key differences: minimal base image, multi-stage build, no secrets in the image, non-root user, only necessary ports exposed, health check included.

Image scanning with Trivy

Trivy is an open-source vulnerability scanner that checks container images, filesystems, and IaC configurations. Integrate it into your CI pipeline to catch vulnerabilities before deployment.

Basic image scan

1
2
3
4
5
6
7
8


# Scan an image for vulnerabilities
trivy image python:3.11-slim

# Scan with severity filter
trivy image --severity HIGH,CRITICAL myapp:latest

# Fail CI if critical vulnerabilities are found
trivy image --exit-code 1 --severity CRITICAL myapp:latest

Scanning in CI/CD

Add Trivy to your pipeline so that no image with critical vulnerabilities reaches production:

1
2
3
4
5
6
7
8


# Example GitHub Actions step
- name: Run Trivy vulnerability scanner
 uses: aquasecurity/trivy-action@master
 with:
 image-ref: 'myapp:${{ github.sha }}'
 format: 'table'
 exit-code: '1'
 severity: 'CRITICAL,HIGH'

Scanning IaC and filesystems

Trivy goes beyond container images:

1
2
3
4
5


# Scan Kubernetes manifests for misconfigurations
trivy config ./k8s-manifests/

# Scan a filesystem for secrets and vulnerabilities
trivy fs --security-checks vuln,secret ./

Runtime security with Falco

While scanning catches vulnerabilities at build time, Falco monitors containers at runtime. It uses kernel-level instrumentation to detect suspicious behavior:

Unexpected shell spawns inside containers.
Processes reading sensitive files (/etc/shadow, /etc/passwd).
Outbound network connections to unexpected destinations.
Privilege escalation attempts.

Example Falco rules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


- rule: Terminal shell in container
 desc: Detect a shell being spawned in a container
 condition: >
 spawned_process and container and
 proc.name in (bash, sh, zsh, dash)
 output: >
 Shell spawned in container
 (user=%user.name container=%container.name
 shell=%proc.name parent=%proc.pname)
 priority: WARNING

- rule: Read sensitive file in container
 desc: Detect reads of sensitive files
 condition: >
 open_read and container and
 fd.name in (/etc/shadow, /etc/passwd)
 output: >
 Sensitive file read in container
 (user=%user.name file=%fd.name container=%container.name)
 priority: ERROR

Deploy Falco as a DaemonSet in your Kubernetes cluster to get visibility into runtime behavior across all nodes.

Kubernetes security

Pod Security Standards

Kubernetes Pod Security Standards define three levels of restriction:

Privileged — Unrestricted (for system-level workloads only).
Baseline — Prevents known privilege escalations.
Restricted — Heavily restricted, following security best practices.

Apply them at the namespace level:

1
2
3
4
5
6
7
8


apiVersion: v1
kind: Namespace
metadata:
 name: production
 labels:
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/audit: restricted
 pod-security.kubernetes.io/warn: restricted

NetworkPolicies

By default, all pods in Kubernetes can communicate with each other. NetworkPolicies let you restrict traffic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: api-allow
 namespace: production
spec:
 podSelector:
 matchLabels:
 app: api
 policyTypes:
 - Ingress
 - Egress
 ingress:
 - from:
 - podSelector:
 matchLabels:
 app: frontend
 ports:
 - protocol: TCP
 port: 8080
 egress:
 - to:
 - podSelector:
 matchLabels:
 app: database
 ports:
 - protocol: TCP
 port: 5432

This policy ensures the API pod only receives traffic from the frontend and only sends traffic to the database.

RBAC

Follow the principle of least privilege. Avoid giving cluster-admin to service accounts. Create specific roles:

1
2
3
4
5
6
7
8
9


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 namespace: production
 name: deployment-manager
rules:
 - apiGroups: ["apps"]
 resources: ["deployments"]
 verbs: ["get", "list", "watch", "update", "patch"]

Regularly audit RBAC bindings with tools like kubectl-who-can or rbac-tool.

Secrets management

Never store secrets in container images, environment variables in plain Dockerfiles, or version control. Instead:

Use Kubernetes Secrets (encrypted at rest with KMS).
Use external secrets managers: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault.
Use the External Secrets Operator to sync secrets from external providers into Kubernetes.
Rotate secrets regularly and audit access.

Supply chain security

Signed images

Sign your container images with cosign (part of the Sigstore project) and verify signatures before deployment:

1
2
3
4
5


# Sign an image
cosign sign --key cosign.key myregistry/myapp:v1.0

# Verify a signature
cosign verify --key cosign.pub myregistry/myapp:v1.0

SBOM (Software Bill of Materials)

Generate an SBOM for every image so you can quickly check whether a newly disclosed CVE affects your running containers:

1
2
3
4
5


# Generate SBOM with Trivy
trivy image --format spdx-json --output sbom.json myapp:latest

# Or use syft
syft myapp:latest -o spdx-json > sbom.json

Container security checklist

Use this checklist to assess your container security posture:

Base images are minimal (distroless or Alpine).
Multi-stage builds are used to exclude build tools.
Containers run as non-root users.
Images are scanned for vulnerabilities in CI/CD.
No secrets are stored in images or plain environment variables.
Kubernetes Pod Security Standards are enforced.
NetworkPolicies restrict pod-to-pod communication.
RBAC follows least privilege.
Runtime security monitoring is in place (Falco or equivalent).
Images are signed and signatures are verified before deployment.
SBOMs are generated and stored for all production images.
Secrets are managed through an external secrets manager.
Image pull policies are set to Always for mutable tags.
Regular security audits and penetration tests are conducted.

Conclusion

Container security is an ongoing practice, not a one-time task. Start with basics: minimal images, non-root users, scanning. Then layer on runtime monitoring, network policies, supply chain verification, and automated enforcement. Each layer shrinks the blast radius and moves you closer to actual security.

CI/CD pipeline security: a shift-left approach

Mon, 20 Jun 2022 00:00:00 +0000

What is shift-left security?

Shift-left security means moving security practices earlier in the software development lifecycle. Rather than treating security as a final gate (or worse, an afterthought) before production, you embed security checks directly into your CI/CD pipelines. This catches vulnerabilities when they’re cheapest to fix: at development time.

The traditional “build it, then audit it” model doesn’t scale. Modern applications ship dozens of times a day. If your security review is a manual quarterly process, you’re deploying vulnerable code most of the time.

Threat vectors in CI/CD pipelines

Before hardening your pipeline, understand what you’re defending against. CI/CD systems are high-value targets because they have access to production credentials, source code, and build artifacts.

Common threat vectors include:

Compromised dependencies: A malicious package in your dependency tree (supply chain attack).
Leaked secrets: API keys, tokens, or passwords committed to source code or exposed in build logs.
Code vulnerabilities: SQL injection, XSS, insecure deserialization, and other OWASP Top 10 issues introduced in your application code.
Insecure pipeline configuration: Overly permissive runner access, unprotected branch rules, or missing approval gates.
Container image vulnerabilities: Base images with known CVEs that end up in production.
Build artifact tampering: Unsigned or unverified artifacts that attackers can replace.

Integrating SAST into your pipeline

Static Application Security Testing (SAST) analyzes source code without executing it. It catches vulnerabilities early, before code even runs.

Recommended tools

Semgrep: Fast, flexible, and supports custom rules. Works across many languages. My top recommendation for most teams.
Bandit: Python-specific. Excellent for Python projects because it understands Python-specific security patterns.
SonarQube: Broader scope (code quality and security combined). Heavier to set up but useful if you want a single dashboard.

Running Semgrep locally

1
2
3
4
5
6
7
8


# Install
pip install semgrep

# Run with default rulesets
semgrep --config auto .

# Run with OWASP Top 10 rules specifically
semgrep --config "p/owasp-top-ten" .

Running Bandit for Python projects

1
2


pip install bandit
bandit -r ./src -f json -o bandit-report.json

The key is to run these tools on every pull request, not just on the main branch. Developers should see findings before code is merged.

Integrating DAST into your pipeline

Dynamic Application Security Testing (DAST) tests the running application from the outside, simulating an attacker. It catches issues that SAST misses (misconfigurations, authentication flaws, and runtime vulnerabilities).

OWASP ZAP

OWASP ZAP is the standard open-source DAST tool. You can run it in your pipeline against a staging deployment:

1
2
3
4
5
6
7


# Pull the ZAP Docker image
docker pull ghcr.io/zaproxy/zaproxy:stable

# Run a baseline scan against a target URL
docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py \
 -t https://staging.example.com \
 -r zap-report.html

DAST is typically slower than SAST. Run it after deployment to a staging environment rather than on every commit. A nightly or per-release cadence works well for most teams.

Secret scanning

Leaked secrets are one of the most common and damaging security failures. Once a secret makes it into Git history, consider it compromised, even if you remove it in a later commit.

Tools

gitleaks: Scans Git repositories for secrets using regex and entropy-based detection.
trufflehog: Searches through Git history for high-entropy strings and known secret patterns.

Running gitleaks

1
2
3
4
5
6
7
8


# Install
brew install gitleaks # or download binary from GitHub releases

# Scan the current repo
gitleaks detect --source . --report-path gitleaks-report.json

# Scan in CI (detect only new leaks in the PR)
gitleaks detect --source . --log-opts="origin/main..HEAD"

Pre-commit hook

Block secrets before they even reach the remote:

1
2
3
4
5
6


# .pre-commit-config.yaml
repos:
 - repo: https://github.com/gitleaks/gitleaks
 rev: v8.16.1
 hooks:
 - id: gitleaks

Dependency scanning

Your application is only as secure as its weakest dependency. Supply chain attacks are increasing, so you need to scan your dependency tree regularly.

Tools

Trivy: Scans container images, filesystems, and Git repos for vulnerabilities. Fast and comprehensive.
Snyk: Commercial option with a generous free tier. Good developer experience.
OWASP Dependency-Check: Mature, open-source, supports multiple ecosystems.

1
2
3
4
5


# Scan a container image with Trivy
trivy image my-app:latest

# Scan the filesystem for vulnerable dependencies
trivy fs --severity HIGH,CRITICAL .

GitHub Actions workflow with security stages

Here is a practical GitHub Actions workflow that integrates all the security stages discussed above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87


name: Secure CI/CD Pipeline

on:
 pull_request:
 branches: [main]
 push:
 branches: [main]

jobs:
 secret-scan:
 name: Secret Scanning
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 with:
 fetch-depth: 0
 - name: Run gitleaks
 uses: gitleaks/gitleaks-action@v2
 env:
 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

 sast:
 name: Static Analysis
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Run Semgrep
 uses: returntocorp/semgrep-action@v1
 with:
 config: >-
 p/security-audit
 p/owasp-top-ten
 - name: Run Bandit (Python)
 run: |
 pip install bandit
 bandit -r ./src -f json -o bandit-report.json || true
 - name: Upload SAST reports
 uses: actions/upload-artifact@v4
 with:
 name: sast-reports
 path: "*.json"

 dependency-scan:
 name: Dependency Scanning
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Run Trivy filesystem scan
 uses: aquasecurity/trivy-action@master
 with:
 scan-type: fs
 severity: HIGH,CRITICAL
 exit-code: 1

 build-and-scan-image:
 name: Build & Scan Image
 needs: [secret-scan, sast, dependency-scan]
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Build Docker image
 run: docker build -t my-app:${{ github.sha }} .
 - name: Scan image with Trivy
 uses: aquasecurity/trivy-action@master
 with:
 image-ref: my-app:${{ github.sha }}
 severity: HIGH,CRITICAL
 exit-code: 1

 deploy-staging:
 name: Deploy to Staging
 needs: [build-and-scan-image]
 runs-on: ubuntu-latest
 environment: staging
 steps:
 - name: Deploy to staging
 run: echo "Deploy to staging environment"

 dast:
 name: Dynamic Analysis
 needs: [deploy-staging]
 runs-on: ubuntu-latest
 steps:
 - name: OWASP ZAP Baseline Scan
 uses: zaproxy/action-baseline@v0.9.0
 with:
 target: "https://staging.example.com"

Notice the deliberate ordering: secret scanning, SAST, and dependency scanning run in parallel (fast feedback). Image scanning runs after build. DAST runs after staging deployment.

Best practices checklist

Here is a checklist to assess the security maturity of your CI/CD pipeline:

Secret scanning runs on every PR and blocks merge on findings.
Pre-commit hooks prevent secrets from being committed locally.
SAST runs on every PR with rules covering OWASP Top 10.
Dependency scanning runs on every PR and flags HIGH/CRITICAL CVEs.
Container image scanning runs before pushing images to a registry.
DAST runs against staging on every release (or nightly at minimum).
Branch protection requires passing security checks before merge.
Least privilege is enforced on CI runner permissions and secrets access.
Signed commits and signed artifacts are enforced for production releases.
Build logs are reviewed to ensure secrets are not leaked in output.
Dependency pinning is used (exact versions, not ranges) to prevent supply chain attacks.
Regular rotation of all CI/CD secrets and service account keys.

Final thoughts

Shift-left security isn’t about buying a tool, it’s about changing the culture. Security checks in CI/CD should be fast, automated, and non-negotiable. Start with secret scanning (highest ROI), add SAST, then layer in dependency scanning and DAST.

The goal isn’t to catch everything in the pipeline. The goal is to raise the bar high enough that obvious issues never reach production, freeing your security team to focus on the subtle, high-impact threats that require human judgment.