Shift-left security: integrating Trivy and Semgrep in GitLab CI

Mon, 16 Mar 2026 00:00:00 +0000

Why these two tools and not others

The security tooling landscape for pipelines is huge, and it is easy to end up with a pipeline that spends twenty minutes just on scans. After trying several combinations, I stick with Trivy and Semgrep for a simple reason: they cover two distinct attack surfaces with minimal friction.

Semgrep analyzes your source code looking for dangerous patterns — SQL injections, insecure deserialization, hardcoded secrets. It does this fast and without needing to compile anything. Trivy, on the other hand, takes care of everything that is not your code: dependencies with known CVEs, outdated base images, problematic IaC configurations. Between the two you cover your own code and third-party code.

Both are open-source, run without an external server, and produce JSON output that you can easily parse in CI. No licenses or third-party dashboards needed to get started.

Pipeline structure

The idea is that security should not be an isolated stage at the end, but something that runs in parallel with the rest of your checks. Here is the general layout:

1
2
3
4
5
6
7
8
9


stages:
 - test
 - security
 - build
 - deploy

variables:
 TRIVY_SEVERITY: "HIGH,CRITICAL"
 SEMGREP_RULES: "p/owasp-top-ten p/security-audit"

The security stage runs at the same level as test. If any scan fails, the pipeline stops before building the image or deploying anything.

Setting up Semgrep

Basic job

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


semgrep:
 stage: security
 image: semgrep/semgrep:latest
 script:
 - semgrep ci --config "$SEMGREP_RULES" --json --output semgrep-results.json
 artifacts:
 paths:
 - semgrep-results.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

This runs Semgrep on every merge request and on every push to the main branch. Results are always saved as an artifact, even if the pipeline fails — you will want to review them later.

Custom rules

Generic rulesets are fine to start with, but as soon as you have project-specific patterns you want to catch, you will need custom rules. Create a .semgrep/ directory at the project root:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# .semgrep/no-env-secrets.yml
rules:
 - id: no-os-environ-secrets
 patterns:
 - pattern: os.environ[$KEY]
 - metavariable-regex:
 metavariable: $KEY
 regex: ".*(SECRET|PASSWORD|TOKEN|KEY).*"
 message: "Direct access to secrets from environment variables. Use the secrets manager."
 languages: [python]
 severity: WARNING

Then reference it in the pipeline:

1
2


variables:
 SEMGREP_RULES: "p/owasp-top-ten p/security-audit .semgrep/"

Handling false positives

There will be false positives. That is unavoidable. What matters is how you handle them. The worst reaction is to disable the entire rule or slap allow_failure: true on the job. Instead, use inline annotations:

1
2


# nosemgrep: python.lang.security.audit.hardcoded-password
TEST_PASSWORD = "dummy" # test fixture, not used in production

Every suppression should have a comment explaining why it is safe to ignore. No exceptions. If you cannot justify it, do not suppress it.

For broader suppressions, use .semgrepignore:

1
2
3
4


# Exclude test fixtures
tests/fixtures/
# Exclude auto-generated code
*_generated.py

Setting up Trivy

Dependency scanning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


trivy-fs:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy fs --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-fs.json .
 artifacts:
 paths:
 - trivy-fs.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Trivy examines the project’s lockfiles (package-lock.json, requirements.txt, go.sum, etc.) and cross-references versions against vulnerability databases. The --exit-code 1 flag makes the job fail if it finds anything HIGH or CRITICAL.

Container image scanning

If you build Docker images, scan them before pushing to the registry:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


trivy-image:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 needs:
 - job: build-image
 artifacts: true
 script:
 - trivy image --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-image.json "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
 artifacts:
 paths:
 - trivy-image.json
 when: always
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

This job depends on the image build (via needs) and only runs on the main branch, not on every MR. Scanning images is slower than scanning the filesystem, so reserve that for what actually gets deployed.

IaC scanning

One often overlooked advantage of Trivy is that it also analyzes infrastructure configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


trivy-iac:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy config --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-iac.json .
 artifacts:
 paths:
 - trivy-iac.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 changes:
 - "**/*.tf"
 - "**/Dockerfile"
 - "**/*.yml"
 - "**/*.yaml"

It catches Dockerfiles running as root, Terraform files with overly open security groups, and Kubernetes configs without resource limits. The changes block ensures it only runs when relevant files are modified, avoiding unnecessary scans.

Blocking policies

Do not start blocking everything from day one. That breeds frustration, creative workarounds, and eventually someone puts allow_failure: true on every security job.

Better to do it in phases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Phase 1: Report only
semgrep:
 allow_failure: true

# Phase 2: Block critical only
semgrep:
 script:
 - semgrep ci --config "$SEMGREP_RULES" --json --output semgrep-results.json
 - |
 CRITICAL=$(jq '[.results[] | select(.extra.severity == "ERROR")] | length' semgrep-results.json)
 if [ "$CRITICAL" -gt 0 ]; then
 echo "Blocked: $CRITICAL critical findings"
 exit 1
 fi
 allow_failure: false

# Phase 3: Block high + critical
# ... adjust the jq filter

Same goes for Trivy. The --severity flag already lets you control which levels block the pipeline. Start with CRITICAL only, and once the team has adapted, add HIGH.

The complete pipeline

Putting it all together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


stages:
 - test
 - security
 - build
 - deploy

variables:
 TRIVY_SEVERITY: "HIGH,CRITICAL"
 SEMGREP_RULES: "p/owasp-top-ten p/security-audit"

semgrep:
 stage: security
 image: semgrep/semgrep:latest
 script:
 - semgrep ci --config "$SEMGREP_RULES" --json --output semgrep-results.json
 - |
 CRITICAL=$(jq '[.results[] | select(.extra.severity == "ERROR")] | length' semgrep-results.json)
 echo "Critical findings: $CRITICAL"
 if [ "$CRITICAL" -gt 0 ]; then
 echo "Pipeline blocked"
 exit 1
 fi
 artifacts:
 paths:
 - semgrep-results.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

trivy-fs:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy fs --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-fs.json .
 artifacts:
 paths:
 - trivy-fs.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

trivy-config:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 script:
 - trivy config --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-iac.json .
 artifacts:
 paths:
 - trivy-iac.json
 when: always
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 changes:
 - "**/*.tf"
 - "**/Dockerfile"
 - "**/*.yml"

trivy-image:
 stage: security
 image:
 name: aquasec/trivy:latest
 entrypoint: [""]
 needs:
 - job: build
 artifacts: true
 script:
 - trivy image --severity "$TRIVY_SEVERITY" --exit-code 1 --format json --output trivy-image.json "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
 artifacts:
 paths:
 - trivy-image.json
 when: always
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Semgrep and the Trivy scans run in parallel within the security stage. If any of them fails, the pipeline stops before building or deploying.

Surfacing results in merge requests

JSON reports are fine for auditing, but developers need feedback visible directly in the MR. GitLab supports native security reports if you use official templates, but with external tools you can parse the JSON and comment on the MR:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


comment-results:
 stage: .post
 image: alpine:latest
 script:
 - apk add --no-cache curl jq
 - |
 SEMGREP_COUNT=$(jq '.results | length' semgrep-results.json 2>/dev/null || echo "0")
 TRIVY_COUNT=$(jq '.Results[]?.Vulnerabilities // [] | length' trivy-fs.json 2>/dev/null || echo "0")

 BODY="### Security summary\n- Semgrep: ${SEMGREP_COUNT} findings\n- Trivy: ${TRIVY_COUNT} vulnerabilities"

 curl --request POST \
 --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" \
 --header "Content-Type: application/json" \
 --data "{\"body\": \"$BODY\"}" \
 "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/notes"
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 allow_failure: true

Not the most elegant solution, but it works. If you are on GitLab Ultimate you get integrated security reports. If not, this gives enough visibility.

Cache and performance

Scans add time to the pipeline, and if that time is excessive the team will end up disabling them. A few things that help:

1
2
3
4
5
6
7
8


trivy-fs:
 variables:
 TRIVY_CACHE_DIR: ".trivycache/"
 cache:
 key: trivy-db
 paths:
 - .trivycache/
 # ...rest of the job

Caching Trivy’s vulnerability database avoids downloading it on every run. It is roughly 40MB downloaded from GitHub, and on shared runners that download can take a while.

For Semgrep, the binary itself is already fast, but if your repository is large, limit the paths:

1
2
3


semgrep:
 script:
 - semgrep ci --config "$SEMGREP_RULES" --include="src/" --include="app/" --json --output semgrep-results.json

No point scanning node_modules/, vendor/, or asset directories.

What I learned running this in practice

I have been using this setup in production for a while now, and some things only become clear with actual use.

Trivy flags a lot of CVEs in base images that have no fix available. If you do not filter with --ignore-unfixed, you will have constant noise. Better to add that flag and focus on what you can actually fix:

1

trivy image --ignore-unfixed --severity HIGH,CRITICAL my-image:latest

With Semgrep, community rulesets are a good starting point, but the rules that deliver the most value are the ones you write yourself, tailored to your project’s patterns. A single rule that catches unparameterized ORM queries is worth more than a hundred generic rules.

And the most important thing: do not try to plug every hole at once. Start with scans in report-only mode, review what comes up, tune the rules, and only then start blocking. If you block everything on day one, someone will have put allow_failure: true on every job by day two.

Container security best practices

Fri, 14 Jul 2023 10:00:00 +0100

The Attack Surface of Containers

Containers give you process isolation, not security isolation. A misconfigured container can expose the host kernel, leak secrets, or become a pivot point for lateral movement. Understand the attack surface first:

Container images - Vulnerable libraries, hardcoded credentials, unnecessary tools
Container runtime - Vulnerabilities that allow breakouts
Orchestrator misconfigurations (Kubernetes RBAC, network policies) - Expose services, grant excessive permissions
Supply chain - Compromised base images or dependencies
Secrets - Baked into images or environment variables that any process can access

Security requires defense in depth across the entire lifecycle: build, ship, run.

Image security

Use minimal base images

Every additional package in a container image is a potential vulnerability. Prefer minimal base images:

1
2
3
4


# Prefer distroless or Alpine over full distributions
FROM gcr.io/distroless/static-debian11
# or
FROM alpine:3.18

Distroless images contain only your application and its runtime dependencies — no shell, no package manager, no unnecessary binaries. This dramatically reduces the attack surface.

Multi-stage builds

Multi-stage builds let you compile in one stage and copy only the final artifact to a minimal runtime image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Build stage
FROM golang:1.21 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server

# Runtime stage
FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/server /server
USER nonroot:nonroot
ENTRYPOINT ["/server"]

The build tools, source code, and intermediate files never make it into the final image.

Never run as root

Running containers as root means a container escape gives the attacker root on the host. Always specify a non-root user:

1
2
3


# Create a non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

In Kubernetes, enforce this with Pod Security Standards (more on this below).

Secure vs. insecure Dockerfile comparison

Here is a side-by-side comparison of common mistakes versus secure practices:

1
2
3
4
5
6
7
8


# INSECURE - Do NOT do this
FROM ubuntu:latest
RUN apt-get update && apt-get install -y curl wget vim
COPY . /app
RUN echo "DB_PASSWORD=supersecret" > /app/.env
EXPOSE 22 80 443
USER root
CMD ["python3", "/app/main.py"]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# SECURE - Follow this pattern
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

FROM python:3.11-slim
RUN groupadd -r appgroup && useradd -r -g appgroup appuser
WORKDIR /app
COPY --from=builder /root/.local /home/appuser/.local
COPY --chown=appuser:appgroup . .
ENV PATH=/home/appuser/.local/bin:$PATH
EXPOSE 8080
USER appuser
HEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost:8080/health || exit 1
CMD ["python3", "main.py"]

Key differences: minimal base image, multi-stage build, no secrets in the image, non-root user, only necessary ports exposed, health check included.

Image scanning with Trivy

Trivy is an open-source vulnerability scanner that checks container images, filesystems, and IaC configurations. Integrate it into your CI pipeline to catch vulnerabilities before deployment.

Basic image scan

1
2
3
4
5
6
7
8


# Scan an image for vulnerabilities
trivy image python:3.11-slim

# Scan with severity filter
trivy image --severity HIGH,CRITICAL myapp:latest

# Fail CI if critical vulnerabilities are found
trivy image --exit-code 1 --severity CRITICAL myapp:latest

Scanning in CI/CD

Add Trivy to your pipeline so that no image with critical vulnerabilities reaches production:

1
2
3
4
5
6
7
8


# Example GitHub Actions step
- name: Run Trivy vulnerability scanner
 uses: aquasecurity/trivy-action@master
 with:
 image-ref: 'myapp:${{ github.sha }}'
 format: 'table'
 exit-code: '1'
 severity: 'CRITICAL,HIGH'

Scanning IaC and filesystems

Trivy goes beyond container images:

1
2
3
4
5


# Scan Kubernetes manifests for misconfigurations
trivy config ./k8s-manifests/

# Scan a filesystem for secrets and vulnerabilities
trivy fs --security-checks vuln,secret ./

Runtime security with Falco

While scanning catches vulnerabilities at build time, Falco monitors containers at runtime. It uses kernel-level instrumentation to detect suspicious behavior:

Unexpected shell spawns inside containers.
Processes reading sensitive files (/etc/shadow, /etc/passwd).
Outbound network connections to unexpected destinations.
Privilege escalation attempts.

Example Falco rules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


- rule: Terminal shell in container
 desc: Detect a shell being spawned in a container
 condition: >
 spawned_process and container and
 proc.name in (bash, sh, zsh, dash)
 output: >
 Shell spawned in container
 (user=%user.name container=%container.name
 shell=%proc.name parent=%proc.pname)
 priority: WARNING

- rule: Read sensitive file in container
 desc: Detect reads of sensitive files
 condition: >
 open_read and container and
 fd.name in (/etc/shadow, /etc/passwd)
 output: >
 Sensitive file read in container
 (user=%user.name file=%fd.name container=%container.name)
 priority: ERROR

Deploy Falco as a DaemonSet in your Kubernetes cluster to get visibility into runtime behavior across all nodes.

Kubernetes security

Pod Security Standards

Kubernetes Pod Security Standards define three levels of restriction:

Privileged — Unrestricted (for system-level workloads only).
Baseline — Prevents known privilege escalations.
Restricted — Heavily restricted, following security best practices.

Apply them at the namespace level:

1
2
3
4
5
6
7
8


apiVersion: v1
kind: Namespace
metadata:
 name: production
 labels:
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/audit: restricted
 pod-security.kubernetes.io/warn: restricted

NetworkPolicies

By default, all pods in Kubernetes can communicate with each other. NetworkPolicies let you restrict traffic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: api-allow
 namespace: production
spec:
 podSelector:
 matchLabels:
 app: api
 policyTypes:
 - Ingress
 - Egress
 ingress:
 - from:
 - podSelector:
 matchLabels:
 app: frontend
 ports:
 - protocol: TCP
 port: 8080
 egress:
 - to:
 - podSelector:
 matchLabels:
 app: database
 ports:
 - protocol: TCP
 port: 5432

This policy ensures the API pod only receives traffic from the frontend and only sends traffic to the database.

RBAC

Follow the principle of least privilege. Avoid giving cluster-admin to service accounts. Create specific roles:

1
2
3
4
5
6
7
8
9


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 namespace: production
 name: deployment-manager
rules:
 - apiGroups: ["apps"]
 resources: ["deployments"]
 verbs: ["get", "list", "watch", "update", "patch"]

Regularly audit RBAC bindings with tools like kubectl-who-can or rbac-tool.

Secrets management

Never store secrets in container images, environment variables in plain Dockerfiles, or version control. Instead:

Use Kubernetes Secrets (encrypted at rest with KMS).
Use external secrets managers: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault.
Use the External Secrets Operator to sync secrets from external providers into Kubernetes.
Rotate secrets regularly and audit access.

Supply chain security

Signed images

Sign your container images with cosign (part of the Sigstore project) and verify signatures before deployment:

1
2
3
4
5


# Sign an image
cosign sign --key cosign.key myregistry/myapp:v1.0

# Verify a signature
cosign verify --key cosign.pub myregistry/myapp:v1.0

SBOM (Software Bill of Materials)

Generate an SBOM for every image so you can quickly check whether a newly disclosed CVE affects your running containers:

1
2
3
4
5


# Generate SBOM with Trivy
trivy image --format spdx-json --output sbom.json myapp:latest

# Or use syft
syft myapp:latest -o spdx-json > sbom.json

Container security checklist

Use this checklist to assess your container security posture:

Base images are minimal (distroless or Alpine).
Multi-stage builds are used to exclude build tools.
Containers run as non-root users.
Images are scanned for vulnerabilities in CI/CD.
No secrets are stored in images or plain environment variables.
Kubernetes Pod Security Standards are enforced.
NetworkPolicies restrict pod-to-pod communication.
RBAC follows least privilege.
Runtime security monitoring is in place (Falco or equivalent).
Images are signed and signatures are verified before deployment.
SBOMs are generated and stored for all production images.
Secrets are managed through an external secrets manager.
Image pull policies are set to Always for mutable tags.
Regular security audits and penetration tests are conducted.

Conclusion

Container security is an ongoing practice, not a one-time task. Start with basics: minimal images, non-root users, scanning. Then layer on runtime monitoring, network policies, supply chain verification, and automated enforcement. Each layer shrinks the blast radius and moves you closer to actual security.

Trivy on Adur