Load testing shows how an application behaves when many requests arrive at once: how latency changes, whether errors appear, and what the ingress layer sees while it happens.
In this post I walk through a small lab for load testing an application on Azure Kubernetes Service (AKS) with Grafana k6. The target is go-httpbin behind a Traefik ingress controller. The cluster sends ingress metrics to Azure Monitor managed service for Prometheus and Azure Managed Grafana, so you can compare what k6 reports on the client with what Traefik reports on the server.
All the files are in my Code-Snippets repository. This post explains the design and the main commands, and the repository has the exact files.
What We Are Building
The demo deploys:
- An AKS cluster (two nodes, managed identity, Azure CNI Overlay, Standard Load Balancer) created with Terraform.
- Three replicas of go-httpbin behind a
ClusterIPService, plus a two-replica Traefik ingress controller. - A Terraform-owned static public IP and an A record in an existing Azure DNS zone.
- A Let’s Encrypt certificate issued by cert-manager using Azure DNS DNS-01 and workload identity, so no client secret is stored.
- Azure Monitor managed Prometheus and Azure Managed Grafana, with the official Traefik dashboard imported automatically.
- k6 scripts for smoke, load, spike, failure-path, stress, and SLO-breach tests.
- Optionally, the Grafana k6 Operator to run the same tests from inside the cluster.
The traffic path looks like this:
k6 (laptop or in-cluster runner)
-> HTTPS hostname -> Azure Load Balancer (static IP)
-> Traefik (2 replicas) -> target-app Service -> go-httpbin (3 replicas)
Traefik :9100 /metrics -> Azure Monitor metrics agent -> managed Prometheus -> Managed Grafanago-httpbin is a good target because it can return specific status codes, delays, payload sizes, and authentication challenges on demand. That lets one script exercise normal traffic, slow responses, and errors without writing a custom application.
Note: Only load test infrastructure you own or are authorized to test. Do not point these scripts at a public httpbin instance or someone else’s service.
Why Test Through the Ingress?
You can port-forward to a Service and run k6 against localhost, and that is fine for a functional check. It is not a representative load test, because the traffic bypasses the load balancer, TLS termination, and Traefik, and it produces no ingress metrics.
For more representative results, send traffic to the public HTTPS hostname. The requests then pass through the same load balancer, TLS termination, and ingress controller that real users would, and Traefik records the request rate, status codes, and latency shown in the Grafana dashboard.
Prerequisites
- An Azure subscription where you can create resource groups, AKS, managed Prometheus/Grafana, and role assignments.
- An existing, publicly delegated Azure DNS zone in the same subscription (the demo only reads it and manages a single A record).
- Azure CLI with the
amgextension, Terraform 1.6 or later,kubectl, Helm,curl, andenvsubst. - k6 for local execution.
- Quota for two
Standard_D2s_v5nodes, plus one temporary node during upgrades.
Deploy the Lab
Clone the repository, sign in, and copy the example variables file:
cd AKS-K6-Load-Testing
az login
az account set --subscription "<subscription-id>"
az extension add --name amg
cp terraform/terraform.tfvars.example terraform/terraform.tfvarsEdit terraform/terraform.tfvars to set your region, resource names, and DNS zone details. The demo defaults to my own domain, so change the zone and hostname values and update the BASE_URL in k8s/k6-test-run.yaml to match. Then run the deployment script:
bash deploy.shThe script does the following:
- Runs Terraform to create the resource group, AKS cluster, static IP, DNS record, identities, Prometheus workspace, and Managed Grafana.
- Installs go-httpbin, Traefik, and cert-manager.
- Waits for a valid certificate.
- Applies the Azure Monitor agent settings and an Azure-specific ServiceMonitor.
- Imports the Traefik dashboard into Grafana.
It is safe to rerun after a recoverable failure. When it finishes, verify the endpoint:
kubectl get ingress,certificate -n k6-demo
export BASE_URL="https://httpbin.<your-domain>"
curl --fail "$BASE_URL/status/200"Use an Existing Cluster Instead
If you already have a cluster, skip Terraform and deploy.sh and deploy only the backend:
kubectl config current-context # confirm this is the intended cluster
kubectl create namespace k6-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/target-app.yaml
kubectl rollout status deployment/target-app -n k6-demoThis installs a ClusterIP Service only: no ingress, TLS, or monitoring. For local functional tests, start a port-forward in one terminal and leave it running:
kubectl port-forward -n k6-demo service/target-app 8080:80In another terminal, set BASE_URL and run the smoke test:
export BASE_URL="http://localhost:8080"
curl --fail "$BASE_URL/status/200"
TEST_TYPE=smoke k6 run k6/load-test.jsFor the operator, set BASE_URL to http://target-app.k6-demo.svc.cluster.local. Do not run deploy.sh against a shared cluster, because it expects to own a new cluster through Terraform.
How Metrics Reach Grafana
Two details matter for the monitoring setup on AKS.
Azure Monitor uses its own ServiceMonitor API group. The managed Prometheus agent scrapes azmonitoring.coreos.com/v1 ServiceMonitors, not the Prometheus Operator’s monitoring.coreos.com/v1. So the Traefik chart’s own ServiceMonitor is disabled, and the demo applies an Azure-compatible one instead:
apiVersion: azmonitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: traefik-azure-monitor
namespace: traefik
spec:
jobLabel: azure-monitor-job
selector:
matchLabels:
azure-monitor-job: traefik
endpoints:
- port: metrics
path: /metrics
interval: 30s
metricRelabelings:
- sourceLabels: [__name__]
action: keep
regex: traefik_.+The keep relabeling means only traefik_* series are ingested from this target. I covered the same ServiceMonitor pattern in more detail in Monitoring Harbor with Azure Monitor and Azure Managed Grafana.
Traefik needs to label its metrics. The Helm values enable addEntryPointsLabels, addRoutersLabels, and addServicesLabels, and leave Traefik’s default latency buckets unchanged. The dashboard’s Apdex and SLO panels use the 0.3 and 1.2 second buckets, so keep the defaults.
The dashboard itself is Grafana.com dashboard 17347, imported with az grafana dashboard import and bound to the data source Managed Grafana creates for the linked Azure Monitor workspace. No API key is needed.
The k6 Test Profiles
One script, k6/load-test.js, supports four profiles selected with the TEST_TYPE environment variable. Always start with smoke.
| Profile | Traffic shape | Purpose |
|---|---|---|
smoke | 1 virtual user (VU) for 10s | Verify the script and the target work |
load | Ramp to 5 VUs over 30s, to 20 over 1m, hold 2m, ramp down 30s | Typical expected traffic |
spike | 1 to 5 VUs over 10s, to 50 over 10s, hold 30s, ramp down | Sudden surge |
failure | 2 VUs for 20s | Normal traffic plus errors, delays, and a deliberate timeout |
TEST_TYPE=smoke k6 run k6/load-test.js
TEST_TYPE=load k6 run k6/load-test.js
TEST_TYPE=spike k6 run k6/load-test.js
TEST_TYPE=failure k6 run k6/load-test.jsRun them one after another, not in parallel. Each iteration sends a mix of requests and then sleeps for one second, so these are closed-model tests: when the server slows down, each VU waits longer and the request rate drops.
What a Normal Iteration Does
Every iteration sends five requests and validates the response content, not only the status code:
GET /get?scenario=mixedmust return 200 and echo the requested URL.POST /postmust echo the JSON body.GET /bytes/1024and/bytes/65536must return exactly that many bytes.GET /basic-auth/demo/secretwith valid credentials must authenticate.
The demo:secret credentials are public test values. Never send real credentials or data to this demo, because httpbin deliberately echoes request data back.
Separating Traffic Categories with Tags
The script tags each request with a kind so that thresholds can be defined per category. This matters in the failure profile, which deliberately returns 4xx and 5xx responses and a timeout. A single global http_req_failed threshold would count those as failures, so each category gets its own threshold instead:
const thresholds = {
checks: ['rate>0.99'],
'http_req_failed{kind:normal}': ['rate<0.01'],
normal_latency: [`p(95)<${normalP95}`],
};
if (testType === 'failure') {
thresholds['http_req_failed{kind:expected_error}'] = ['rate<0.01'];
thresholds['http_req_duration{kind:delay}'] = ['p(95)<3000'];
thresholds['http_req_failed{kind:timeout}'] = ['rate>0.99'];
}Expected errors use http.expectedStatuses(404) (or 401, 500) so a correct error response doesn’t count as a failure, and the deliberate timeout request (/delay/2 with a 500 ms client timeout) must fail with error code 1050. The failure profile therefore passes only when the failures happen as intended.
Normal latency is recorded in a custom normal_latency trend that excludes the intentional delays, with a default p95 budget of 200 ms. If your laptop is far from the cluster, internet round trips can exceed that, so raise the budget explicitly instead of weakening the other thresholds:
TEST_TYPE=smoke NORMAL_P95_MS=500 k6 run k6/load-test.jsA threshold breach makes k6 exit with a nonzero code, which is what lets you drop the same script into a CI pipeline later. Capture it with echo $? right after a run.
Stress Testing with a Fixed Request Rate
VU-based profiles send fewer requests when the server slows down, because each VU waits for its response. To see where the system starts to degrade, k6/stress-test.js uses the ramping-arrival-rate executor, which starts new requests at a fixed rate whether or not earlier ones have finished.
| Stage | Duration | Target requests per second |
|---|---|---|
| Warm-up | 1m | 25% of PEAK_RPS |
| Ramp | 2m | 50% of PEAK_RPS |
| Ramp | 2m | PEAK_RPS |
| Sustain | 3m | PEAK_RPS |
| Ramp down | 1m | 0 |
PEAK_RPS defaults to 300. One of the thresholds stops the test:
thresholds: {
http_req_failed: [{ threshold: 'rate<0.05', abortOnFail: true, delayAbortEval: '1m' }],
http_req_duration: ['p(95)<500', 'p(99)<1500'],
checks: ['rate>0.95'],
dropped_iterations: ['rate<1'],
},With abortOnFail, k6 stops the test automatically once more than 5% of requests fail (evaluated after the first minute), so the test does not keep sending load to a system that is already failing. The dropped_iterations threshold shows when k6 could not start iterations on schedule because every VU was still waiting on slow responses.
The script requires BASE_URL and has no default, so it can’t hit the wrong target by accident:
BASE_URL="https://httpbin.<your-domain>" k6 run k6/stress-test.js
BASE_URL="https://httpbin.<your-domain>" PEAK_RPS=500 k6 run k6/stress-test.jsFrom a laptop, your own bandwidth may become the bottleneck before the cluster does. If dropped_iterations climbs while server-side latency in Grafana stays flat, the load generator is the limit, and it is better to run the test from inside the cluster.
Running k6 Inside the Cluster with the k6 Operator
The k6 Operator runs k6 as pods in AKS and can split a test across several runners. Install it with Helm:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install k6-operator grafana/k6-operator \
--namespace k6-operator-system \
--create-namespace --wait --timeout 5m
kubectl wait --for=condition=Established crd/testruns.k6.io --timeout=60sUpload the script as a ConfigMap, then create a TestRun:
kubectl create configmap k6-test-script \
--from-file=test.js=k6/load-test.js \
--namespace k6-demo \
--dry-run=client -o yaml | kubectl apply -f -The TestRun resource references that ConfigMap and passes the profile and target as environment variables:
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
name: distributed-load-test
namespace: k6-demo
spec:
parallelism: 4
runner:
env:
- name: TEST_TYPE
value: "load"
- name: BASE_URL
value: "https://httpbin.<your-domain>"
script:
configMap:
name: k6-test-script
file: test.jsApply it and watch progress:
kubectl apply -f k8s/k6-test-run.yaml
kubectl get testrun distributed-load-test -n k6-demo -wA few things to keep in mind with distributed runs:
-
VUs are split across runners. With
parallelism: 4and 20 VUs, each runner gets about five, not 20. The same applies to arrival-rate tests: with two runners, each sends half ofPEAK_RPS. -
Thresholds are evaluated per runner, so there is no single merged percentile. Read every runner pod’s logs.
-
kubectl applysucceeding only means the test was submitted. Check each runner’s logs and exit code to see whether thresholds passed:kubectl logs -n k6-demo <runner-pod-name> --all-containers=true kubectl get pod <runner-pod-name> -n k6-demo \ -o jsonpath='{range .status.containerStatuses[*]}{.name}{": exit="}{.state.terminated.exitCode}{"\n"}{end}' -
A completed TestRun does not rerun when reapplied. Delete it first with
kubectl delete testrun <name> -n k6-demo. Deleting an active run cancels it. -
Environment variables on your laptop are not passed to pods. Set them in
spec.runner.env.
The repository’s README includes a kubectl patch --local snippet that renders a named TestRun per profile without modifying the checked-in template.
On a two-node cluster, the runners share nodes with httpbin, Traefik, and the metrics agent. If pods stay Pending or the results look noisy, add capacity or use dedicated load-generator nodes, because otherwise the load generator competes with the system under test.
Reading the Grafana Dashboard
Open the Grafana endpoint from terraform -chdir=terraform output -raw grafana_endpoint, then go to Dashboards > Traefik Official Kubernetes Dashboard. Run the load profile against the HTTPS hostname, select Last 30 minutes, and set 30s refresh.
Allow a few minutes for data to appear. The agent scrapes every 30 seconds and ingestion adds some delay, so rates need at least two scrapes. A 10-second smoke test is a functional check and is too short to produce a useful throughput graph.
Things to look at:
- Request rate per entrypoint and service, which should follow the k6 ramp shape.
- HTTP status codes. The failure profile deliberately produces 4xx and 5xx responses, so they show up here even though k6 passes its checks.
- Latency and Apdex, which use Traefik’s
0.3and1.2second buckets. - Open connections and request/response bytes.
Server-side latency from Traefik and client-side latency from k6 are different measurements. normal_latency comes from response.timings.duration, like http_req_duration: it includes sending, waiting, and receiving, but excludes DNS lookup, TCP connection establishment, and the TLS handshake. k6 reports those connection timings separately, and normal_latency excludes intentional delays. A gap between the two can help show where the time is spent.
Populating the SLO Tiles
The dashboard’s Services failing SLO of 300ms and 1200ms tiles stay empty while requests are fast. k6/slo-breach-test.js populates them with low, steady traffic that does not stress the cluster: two requests per second to /delay/0.5 (breaches 300 ms only) and two per second to /delay/2 (breaches both) for six minutes, all returning HTTP 200:
BASE_URL="https://httpbin.<your-domain>" k6 run k6/slo-breach-test.jsRun it alone, select the k6-demo-target-app-80 service, and allow several minutes for ingestion. In a window with only this traffic, expect about 100% of requests above 300 ms and about 50% above 1200 ms.
If the Panels Are Empty
The dashboard filters out zero-violation series, so “No data” can mean nothing is violating rather than nothing is collected. To check collection, run up{job="traefik"} in Grafana Explore. A missing series points to discovery or ingestion, while a value of 0 means the target was found but can’t be scraped. Then verify the ServiceMonitor, the agent logs, and the raw metrics. In one terminal, start the port-forward and leave it running:
kubectl get servicemonitors.azmonitoring.coreos.com -n traefik
kubectl logs deployment/ama-metrics -n kube-system --tail=100
kubectl port-forward -n traefik service/traefik-metrics 9100:9100In another terminal, check the raw metrics:
curl --fail http://localhost:9100/metricsFor certificate problems, check kubectl describe certificate httpbin-tls -n k6-demo, public DNS delegation, and the cert-manager identity’s DNS Zone Contributor role.
Cost and Safety Cautions
- This lab costs money until you destroy it. Nodes, disks, the static IP, load balancing, DNS queries, outbound traffic, managed Prometheus ingestion and queries, and Standard Azure Managed Grafana are all billable.
- Stress tests generate egress. At peak, about 10% of stress requests return 64 KiB, so 300 requests per second sends roughly 2 MB/s outbound, which Azure bills as egress.
- The API is public. It has no access control beyond httpbin’s demo auth endpoints. Don’t use real data and don’t treat the lab as production.
- Protect cluster access. The control-plane endpoint is public and your kubeconfig grants access to the cluster. Don’t commit kubeconfig files, subscription IDs, Terraform state, or
terraform.tfvars, and use a secured remote backend for shared environments. - Don’t overwrite DNS records used by another demo. If the record already exists outside Terraform, import it or choose another subdomain.
- The default cluster is small. httpbin pods are limited to 250m CPU each, so under the stress test, rising latency, Traefik 5xx responses, or an automatic abort are expected results.
Clean Up
For the cluster created by this demo:
bash cleanup.shAfter you confirm, the script destroys the cluster, the node and demo resource groups, the static IP, the Prometheus and Grafana resources, identities and role assignments, and the demo A record. The existing DNS zone and its other records are not touched. It verifies the groups and record are gone before removing local state. Cached DNS answers may persist for the 300-second TTL.
If you used the existing-cluster path, do not run cleanup.sh. Remove only the demo resources, after checking your Kubernetes context:
kubectl delete testrun --all -n k6-demo --ignore-not-found
kubectl delete namespace k6-demo
helm uninstall k6-operator --namespace k6-operator-system
kubectl delete namespace k6-operator-systemOnly uninstall the operator if nothing else depends on it. Helm may leave cluster-scoped CRDs behind; don’t delete those on a shared cluster without checking who else uses them.
Next Steps
The Grafana dashboard in this lab shows Traefik’s server-side metrics only. k6 results live in the terminal summary and the operator runner logs. In a follow-up post, I will look at k6 dashboards and how to view the load test results in Grafana, so the client-side k6 metrics sit next to the ingress metrics.
Other ideas to try from here:
- Run the smoke profile in a CI pipeline and fail the build on a threshold breach.
- Add dedicated load-generator nodes so the runners don’t compete with the target.
- Replace go-httpbin with your own application and reuse the tagged-threshold pattern for your critical endpoints.
Key Takeaways
- Test through the ingress path rather than a port-forward if you want representative results.
- Start with smoke, then run load, spike, failure, and stress one at a time.
- Tag request categories in k6 so deliberate failures do not affect thresholds for normal traffic.
- Use an arrival-rate executor for stress tests, with
abortOnFailto stop the run when errors climb. - Distributed k6 runs split the load across runners and evaluate thresholds per runner.
- Compare k6’s client-side results with Traefik’s server-side metrics in Grafana, and destroy the lab when you are done.
