#ADR-0007: Stress Test Strategy

Status: Draft

Date: 2026-05-22

Deciders: TBD

Relates to:

• ADR-0000 (Friends & Family MVP) — stress testing is a prerequisite for MVP confidence
• Issue #85 (Observability) — monitoring infrastructure needed during test execution
• Issue #96 (Define stress test strategy) — parent work item

#Context

Themelio is approaching its Friends & Family MVP release. The platform comprises multiple microservices — a Next.js frontend, two Fastify-based root actors, a GitLab CE instance, Valkey for session storage, and Kubernetes for deployment orchestration. These services interact under concurrent user workloads in ways that are difficult to reason about from code review alone.

We have no empirical data on how the system behaves under load. Critical operations such as concurrent git push commands, parallel deployment requests, and bulk user provisioning have never been tested beyond single-user flows. Without this data, we risk shipping an MVP that degrades unpredictably under modest concurrent usage.

This ADR captures the strategic decisions around stress testing: which services to test, what types of tests to run, how to select tooling, and where to execute them. Detailed test scenarios and execution plans will follow in a Design Document (DD-0007) once the decisions here are accepted.

#Requirements

1. Identify the services and user-facing operations most likely to bottleneck under concurrent load.
2. Establish baseline performance metrics before stress testing begins.
3. Select a load testing tool that the team can adopt with minimal ramp-up.
4. Define quantitative pass/fail thresholds so results produce actionable verdicts, not just data.
5. Execute tests in an environment that cannot impact development or staging workflows.
6. Document findings in a format that informs capacity planning and architecture decisions.

#Decision Drivers

• Actionability — results must translate into concrete engineering work, not just dashboards
• Team accessibility — tooling and process should be approachable for the whole team, not only infrastructure specialists
• Safety — stress tests must not corrupt data or disrupt shared environments
• Realism — scenarios must reflect actual usage patterns, not synthetic micro-benchmarks
• Repeatability — tests must be scripted and version-controlled so they can be re-run after changes

#Scope

#Services under test

#Test types

• Load test — simulate expected peak concurrent usage to establish a performance baseline
• Stress test — push beyond expected capacity to identify breaking points and failure modes
• Soak test — sustain moderate load over hours to surface memory leaks, connection pool exhaustion, and gradual degradation

#Out of scope

• Chaos engineering and failure injection (e.g., pod kills, network partitions)
• Security and penetration testing
• Frontend rendering performance (Lighthouse, Core Web Vitals)
• Third-party service stress (Keycloak, Waldur API)

#Considered Options

#Tooling

#Option A: k6

A developer-centric load testing tool. Tests are written in JavaScript/TypeScript, executed from the CLI, and produce structured metrics. Open-source with an optional cloud dashboard.

Pros:

• Scripts in JavaScript — familiar to the entire team given the Node.js stack
• Built-in support for HTTP, WebSocket, and gRPC protocols
• Lightweight CLI execution, no JVM or Python runtime required
• Rich metrics output with Prometheus, Grafana, and JSON export
• Active community and extensive documentation

Cons:

• No native git protocol support — concurrent git push scenarios require shelling out or a custom extension
• Browser-level testing (k6 browser) is experimental
• Single-machine execution by default; distributed mode requires k6-operator on Kubernetes

#Option B: Locust

A Python-based load testing framework where user behavior is defined as Python classes. Includes a built-in web UI for real-time monitoring.

Pros:

• Full Python ecosystem available — can invoke subprocess for git operations natively
• Built-in web dashboard for real-time test monitoring
• Distributed execution via worker nodes is straightforward
• Flexible — any Python-callable operation can be a test scenario

Cons:

• Python runtime dependency; the team primarily works in TypeScript
• Single-threaded per worker due to Python GIL — high concurrency requires multiple worker processes
• Metrics export to Grafana/Prometheus requires additional plugins

#Option C: Gatling

A Scala/Java-based load testing tool with a DSL for defining scenarios. Strong in HTTP protocol testing with detailed HTML reports.

Pros:

• Excellent HTML reports out of the box
• High performance due to JVM async I/O (Akka/Netty)
• Mature tool with strong enterprise adoption

Cons:

• Scala DSL has a steep learning curve for a team without JVM experience
• JVM dependency adds operational overhead
• Less flexible for non-HTTP protocols (git operations)

#Environment

#Option A: Dedicated stress-test namespace in existing cluster

Create an isolated Kubernetes namespace with its own GitLab instance, Valkey, and root actors. Share the underlying cluster but enforce resource quotas.

Pros:

• Lower infrastructure cost — reuses existing cluster capacity
• Faster to provision than a full separate cluster
• Namespace-level resource quotas prevent runaway tests from starving other workloads

Cons:

• Noisy-neighbor risk — stress tests may impact node-level resources (CPU, disk I/O) shared with other namespaces
• Network policies required to ensure test traffic doesn't leak
• Shared etcd and control plane could become a bottleneck under extreme load

#Option B: Dedicated cluster for stress testing

Provision a separate Kubernetes cluster solely for stress test execution. Torn down after test campaigns to control cost.

Pros:

• Complete isolation — zero risk of impacting development or staging
• Can be sized to match production topology for realistic results
• No contention on control plane or shared storage

Cons:

• Higher infrastructure cost and provisioning lead time
• Requires maintaining a separate deployment pipeline or ArgoCD instance
• Cluster lifecycle management adds operational overhead

#Option C: Reuse staging with scheduling safeguards

Run stress tests on the staging environment during off-hours windows, with automated pre/post checks and rollback.

Pros:

• No additional infrastructure cost
• Tests run against a production-like environment by definition
• Existing monitoring and alerting already in place

Cons:

• Risk of data corruption in a shared environment
• Scheduling constraints limit test flexibility
• Post-test cleanup may not fully restore environment state

#Decision

TBD — pending resolution in team kickoff meeting. The meeting should produce decisions on:

1. Tooling: which load testing tool to adopt
2. Environment: where stress tests will execute
3. Priority ordering: which services and scenarios to test first

#Consequences

TBD — will be filled in once a decision is made.

#Test data strategy (to be refined in DD-0007)

Regardless of tooling choice, the following test data concerns must be addressed:

• Synthetic users: bulk-provision GitLab accounts via gitlab-root-actor with predictable naming (e.g., stress-user-001 through stress-user-100)
• Template repositories: create repos of varying sizes — small (< 1 MB), medium (10-50 MB), large with LFS (500 MB+)
• Token management: generate and manage impersonation tokens for concurrent authenticated operations
• Teardown: all test data must be automatically cleaned up after test runs

#Success criteria (to be refined in DD-0007)

Each scenario must define quantitative thresholds. Proposed starting points for the kickoff discussion:

#Monitoring requirements

Capture at minimum during all test runs:

• System metrics: CPU, memory, disk I/O, network per pod and node
• Application metrics: request rate, latency histograms (p50/p95/p99), error codes
• GitLab internals: Sidekiq queue depth, Gitaly gRPC latency, PostgreSQL active connections
• Kubernetes: pod restarts, HPA scaling events, resource pressure signals
• Valkey: connected clients, memory usage, command latency

This overlaps with the observability work in issue #85 — the stress test campaign will both depend on and validate that instrumentation.

#Risks

#Open Questions

#Q1: Git protocol testing approach

None of the evaluated tools natively support the git protocol. How will we simulate concurrent git push and git clone operations? Options include wrapping git CLI calls in the load tool's scripting layer, using a custom test harness, or testing the GitLab API endpoints that back git operations instead.

#Q2: Production topology fidelity

How closely must the test environment mirror production? Specifically: do we need matching node counts, CPU/memory per pod, and storage backend (Ceph)? Or is a scaled-down environment with proportionally adjusted thresholds acceptable for the MVP phase?

#Q3: Coordination with observability work

Issue #85 covers logging and observability improvements. Should we block stress test execution on #85 completion, or instrument incrementally as we build test scenarios?

#References

• k6 documentation
• Locust documentation
• Gatling documentation
• Google SRE Book — Chapter 29: Dealing with Overload
• ISTQB Performance Testing Syllabus

Page source

Source snapshot · Revision f928834c8.