# torchio 4.6.2: documentation > Break your containers on purpose: burn CPU, hold memory, fill disk and more, on demand over HTTP. # torchio **Break your containers on purpose.** torchio is a small server, a single static binary written in Rust, that burns CPU, holds memory, fills disk, answers slowly or returns any HTTP status, all on demand over HTTP. Use it to see how your setup behaves under pressure: autoscaling, resource limits, OOM kills, probes, timeouts and retries. *Torchio* is the Italian word for the press that squeezes wine and oil, as in *mettere sotto torchio*: to put under pressure. Put your containers under the press. It began in 2019 as `high-cpu-node`, when it could only burn CPU. This is the documentation of torchio 4.6.2. The changes of each release are in the [changelog](https://torchio.altr.dev/changelog.html). Source code and issues: https://github.com/altrdev/torchio Using it from an AI agent: see [For AI agents](https://torchio.altr.dev/agents.html). > **Warning:** this server is meant to consume resources on purpose. Don't expose it to the public internet. --- # Install Pick one way to get torchio. It is a single static binary for Linux and macOS (`amd64` and `arm64`), and an image of about 1.4 MB. ## Docker ``` docker pull altrdev/torchio ``` The image holds the binary and nothing else. The tags and the rest are in [Docker](https://torchio.altr.dev/docker.html). ## Homebrew For macOS and Linux: ``` brew install altrdev/tap/torchio ``` It installs the shell completions too. If Tab does nothing in zsh, see [Shell completion](https://torchio.altr.dev/shell-completion.html). ## Install script A script that downloads the binary of the latest release for your system (macOS or Linux, `amd64` or `arm64`), checks its checksum and installs it: ``` curl -fsSL https://torchio.altr.dev/install | sh ``` The same script is also at `https://raw.githubusercontent.com/altrdev/torchio/main/install.sh`, which is the address to use if the site is down. Either way the binary comes from the GitHub release. It installs in `/usr/local/bin` (with `sudo` if needed, or in `~/.local/bin` when there is none). `TORCHIO_VERSION=v4.0.0` picks a release and `TORCHIO_INSTALL_DIR` the folder, for example `curl -fsSL ... | TORCHIO_INSTALL_DIR=$HOME/bin sh`. ## Release binary Take the binary of your system by hand from the [releases](https://github.com/altrdev/torchio/releases): `torchio-linux-amd64`, `torchio-linux-arm64`, `torchio-macos-amd64` or `torchio-macos-arm64`. `SHA256SUMS` is there to check it (`shasum -a 256 -c` on macOS): ``` curl -LO https://github.com/altrdev/torchio/releases/latest/download/torchio-linux-amd64 curl -LO https://github.com/altrdev/torchio/releases/latest/download/SHA256SUMS grep ' torchio-linux-amd64$' SHA256SUMS | sha256sum -c chmod +x torchio-linux-amd64 ``` ## From source Build it with cargo (Rust 1.88 or later): ``` cargo install --git https://github.com/altrdev/torchio ``` --- # Try it in 30 seconds Start the server with the image or with the binary (see [Install](https://torchio.altr.dev/install.html) if you have neither yet): ``` docker run --name torchio -p 3000:3000 altrdev/torchio # with Docker torchio serve # with the binary ``` `torchio` with no command does the same as `torchio serve`: it is what the Docker image runs. The server stays in the foreground, so Ctrl+C stops it; use a second terminal tab for `torchio watch`. In a second terminal, watch what it does: ``` docker exec -it torchio torchio watch # with Docker torchio watch # on your local machine, press m to start a memory test ``` In a third one, put it under load, and see the bars move: ``` torchio memory 200 --hold 20 torchio disk 100 --hold 20 torchio stress 44 ``` The same thing with `curl`: `curl "http://127.0.0.1:3000/memory?mb=200&hold=20"`. See [Starting a test](https://torchio.altr.dev/starting-a-test.html#starting-a-test). --- # What you can test ## Memory limits and OOM kills Give the container a memory limit and ask for more than that: ``` docker run -d --name oom -m 256m --memory-swap 256m -e MAX_MEMORY_MB=1024 -p 3000:3000 altrdev/torchio curl "http://127.0.0.1:3000/memory?mb=400" docker inspect -f '{{.State.OOMKilled}}' oom ``` The container should be killed, and the last command prints `true`. `MAX_MEMORY_MB` is raised above the limit on purpose: its default (`512`) is a safety cap, so with it you can't exhaust a larger container. `--memory-swap` has the same value as `-m` so the container can't spill into swap, which on a host that has some (Docker Desktop does) would delay the kill or avoid it. ## Autoscaling A core stays busy only while a request is running, so send at least as many concurrent requests as workers (one per CPU by default): ``` hey -z 60s -c 8 "http://127.0.0.1:3000/stress?number=40" ``` To hold the CPU at a level for a while, without sending requests, use `/cpu`. A CPU based autoscaler reacts to a percentage, which this gives you directly: ``` curl "http://127.0.0.1:3000/cpu?percent=80&hold=300" ``` More about it in [Load testing](https://torchio.altr.dev/load-testing.html#load-testing). ## Probes, timeouts and retries ``` curl -i http://127.0.0.1:3000/status/503 curl --max-time 2 "http://127.0.0.1:3000/delay?ms=5000" curl -i "http://127.0.0.1:3000/flaky?rate=30" ``` `/status/:code` answers with any status from `200` to `599`, and `/delay` waits without blocking the server, so the rest keeps being served while a slow request is pending. `/flaky` fails only some of the requests, to see how retries, circuit breakers and outlier detection behave. ## Disk and volumes ``` curl "http://127.0.0.1:3000/disk?mb=500&hold=60" ``` If the write fails (disk full, read-only filesystem, directory not writable) the answer is a `507`. Point `DISK_DIR` to a mounted volume to test it. ## Leaks and slow starts ``` curl "http://127.0.0.1:3000/leak?rate=10&mb=400&hold=60" docker run -p 3000:3000 -e START_DELAY_SECONDS=30 altrdev/torchio ``` `/leak` grows the memory by itself, a few MB each second, so you can watch a memory alert or an autoscaler react before the OOM kill. `START_DELAY_SECONDS` keeps the port closed for a while after the start, to test startup probes, rollouts and `depends_on`. ## Readiness, restarts and descriptors ``` curl "http://127.0.0.1:3000/ready/off?seconds=60" curl -i http://127.0.0.1:3000/ready curl "http://127.0.0.1:3000/crash?code=1" curl "http://127.0.0.1:3000/block?ms=10000" curl "http://127.0.0.1:3000/fd?count=500&hold=30" ``` `/ready/off` makes `/ready` answer `503` for a while while `/` keeps answering `200`, so you can see an instance leave a Service or a load balancer without being restarted. `/crash` ends the process with the exit code you choose, to watch restarts and backoff. `/block` freezes every thread of the server for a while, so a liveness probe times out on an instance that is alive. `/oom` (with `ALLOW_OOM=1`) allocates until the kernel kills the container, to see `OOMKilled` and the exit code `137`. `/fd` holds file descriptors open, to see what a process does when it runs out of them. ## Metrics and dashboards [`/info`](https://torchio.altr.dev/endpoints.html#info) is a quick look at what the server holds and what is left of each budget, and [`/metrics`](https://torchio.altr.dev/endpoints.html#metrics) has the same figures for Prometheus. --- # Endpoints The server listens on port 3000. Most of the endpoints that start a test are also a command, so you can skip `curl`: the [Reference](https://torchio.altr.dev/reference.html) names the command of each one, and [Starting a test](https://torchio.altr.dev/starting-a-test.html) has them all. | Endpoint | Parameters | Description | |-----------------|--------------------|-------------------------------------------------------------| | `/` | | Health check, returns `Status: OK` | | `/stress` | `number` | Runs a Fibonacci calculation to load a CPU | | `/cpu` | `percent`, `hold`, `workers` | Keeps the workers busy for a share of the time, to test CPU based autoscaling | | `/delay` | `ms` | Answers after a delay, to test timeouts and slow backends | | `/status/:code` | `code` (in the path) | Answers with the given HTTP status, to test probes and retries | | `/flaky` | `rate`, `code`, `ms` | Fails a share of the requests, to test retries and circuit breakers | | `/memory` | `mb`, `hold` | Allocates memory and holds it, to test memory limits and OOM kills | | `/leak` | `rate`, `mb`, `hold` | Memory that grows by itself up to a ceiling, to test alerts and leaks | | `/disk` | `mb`, `hold` | Writes a file and keeps it, to test disk limits and volumes | | `/ready` | | Readiness check: `200`, or `503` after `/ready/off` | | `/ready/off` | `seconds` | Makes `/ready` answer `503`, to test readiness probes | | `/ready/on` | | Makes `/ready` answer `200` again | | `/crash` | `code`, `after` | Ends the process, to test restarts and backoff | | `/block` | `ms` | Freezes every thread of the server, to test liveness probes and timeouts | | `/oom` | | Allocates until the kernel kills the process, only with `ALLOW_OOM=1` | | `/payload` | `mb` | Sends or takes a large body, to test body size limits and timeouts of proxies | | `/scenario` | | Runs a list of steps one after the other, [see Scenarios](https://torchio.altr.dev/scenarios.html) | | `/fd` | `count`, `hold` | Opens file descriptors and holds them, to test `EMFILE` | | `/connections` | `count`, `hold` | Opens TCP connections to the server and holds them, to test connection limits | | `/log` | `lines`, `rate` | Writes lines to stdout at a rate, to test log collection and rotation | | `/echo` | | The request as the server sees it, to see what proxies add or remove | | `/ui` | | Control panel in the browser, only with `UI=1` | | `/info` | | JSON with what the server is doing and the remaining budgets | | `/metrics` | | The same figures in the Prometheus text format | | `/help` | | Every endpoint with its parameters, defaults and limits | | `/config` | | The effective configuration, without the token | Every query parameter is optional. A value that is not a number falls back to its default; a number outside the allowed range gets a `400`. All the limits are configurable, see [Configuration](https://torchio.altr.dev/configuration.html#configuration). ## `/stress` Runs a recursive Fibonacci calculation in a worker thread, so it keeps one core busy for the whole request. | Parameter | Default | Allowed range | |-----------|---------|--------------------| | `number` | `45` | `0..MAX_NUMBER` | The calculation time grows exponentially with `number`. ``` curl "http://127.0.0.1:3000/stress?number=40" ``` ## `/cpu` Use [`/stress`](#stress) when the load must come from requests, so that response times show the pressure and concurrent requests queue up; use `/cpu` when you only need the CPU held at a level, as a percentage, for a while. Keeps the workers busy for `percent` of the time during `hold` seconds, instead of running one calculation. With one worker for each CPU (the default), `percent` is the load of the machine, which is what a CPU based autoscaler looks at. `workers` loads only the first ones and leaves the others idle, for an uneven load across the cores. A new request replaces the previous one, and `percent=0` stops it. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `percent` | `50` | `0..100` | | `hold` | `10` | `0..MAX_HOLD_SECONDS` | | `workers` | all of them | `1..WORKERS` | The answer comes right away. `/stress` keeps working while `/cpu` runs, and what is left is in `/info` (`stress.burn`, with how many workers are loaded) and in `/metrics` (`torchio_cpu_burn_percent` and `torchio_cpu_burn_workers`). The default number of workers is the number of CPUs the process can use, so a CPU limit of the container counts (a limit of 2 CPUs gives 2 workers); set `WORKERS` to choose another. ``` curl "http://127.0.0.1:3000/cpu?percent=70&hold=120" curl "http://127.0.0.1:3000/cpu?percent=90&workers=1" # one core at 90%, the others idle ``` ## `/delay` Answers after `ms` milliseconds. The wait does not block the server, so other requests are served normally. | Parameter | Default | Allowed range | |-----------|---------|--------------------| | `ms` | `1000` | `0..MAX_DELAY_MS` | ``` curl "http://127.0.0.1:3000/delay?ms=3000" ``` ## `/status/:code` Answers with the HTTP status you ask for. Codes must be integers from `200` to `599`, otherwise you get a `400`. ``` curl -i http://127.0.0.1:3000/status/503 ``` ## `/flaky` Answers with `code` for `rate` percent of the requests and with `200` for the rest, to test retries, circuit breakers and outlier detection. Each request is decided on its own, so the share is only exact over many requests; `rate=0` never fails and `rate=100` always does. A failing answer waits `ms` milliseconds first, like [`/delay`](#delay), and a request that succeeds never waits. | Parameter | Default | Allowed range | |-----------|---------|-------------------| | `rate` | `50` | `0..100` | | `code` | `503` | `400..599` | | `ms` | `0` | `0..MAX_DELAY_MS` | ``` curl -i "http://127.0.0.1:3000/flaky?rate=30" curl -i "http://127.0.0.1:3000/flaky?rate=100&code=429&ms=500" ``` The answers are counted in [`/metrics`](#metrics) by status as usual, so `torchio_http_requests_total{route="/flaky",code="503"}` is the number of failures. Like `/status/:code` and `/delay`, it changes nothing in the server, so [`TOKEN`](https://torchio.altr.dev/token.html#token) does not protect it. ## `/memory` Allocates `mb` megabytes and holds them for `hold` seconds, then releases them. It answers as soon as the memory is allocated, so you can chain requests. | Parameter | Default | Allowed range | |-----------|---------|-------------------------| | `mb` | `100` | `1..MAX_MEMORY_MB` | | `hold` | `10` | `0..MAX_HOLD_SECONDS` | `MAX_MEMORY_MB` is also the total that can be held at the same time: a request that would exceed it together with the memory already held gets a `429`. `hold=0` releases the memory right away. Every page is written, so the memory counts as used (RSS) as soon as the request is answered, and it goes back to the OS when it is released. The server itself needs only a few MB on top of what you allocate. To test an OOM kill, raise `MAX_MEMORY_MB` above the memory limit of your container. With the default (`512`) you can't exhaust a larger one. ``` curl "http://127.0.0.1:3000/memory?mb=200&hold=30" ``` ## `/leak` Allocates memory in steps of `rate` MB each second up to `mb` MB, keeps it for `hold` seconds after the last step and then releases it. It answers right away. It is a leak you can follow from the outside: `/info`, `/metrics` and `watch` show the memory growing. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `rate` | `10` | `1..MAX_MEMORY_MB` | | `mb` | `100` | `1..MAX_MEMORY_MB` | | `hold` | `10` | `0..MAX_HOLD_SECONDS` | It takes its memory from the same budget as [`/memory`](#memory) (`MAX_MEMORY_MB`) a step at a time. If the budget is already full the first request gets a `429`, and a leak that fills it while growing stops there and is held until `hold` ends. ``` curl "http://127.0.0.1:3000/leak?rate=20&mb=500&hold=60" ``` ## `/disk` Writes `mb` megabytes to a file in `DISK_DIR` and deletes it after `hold` seconds. It answers once the data is on disk. | Parameter | Default | Allowed range | |-----------|---------|-------------------------| | `mb` | `100` | `1..MAX_DISK_MB` | | `hold` | `10` | `0..MAX_HOLD_SECONDS` | `MAX_DISK_MB` is the total that can be held at the same time, with the same `400` and `429` rules as `/memory`. If the write fails (disk full, read-only filesystem, directory not writable) the answer is a `507`. Files are removed on shutdown, and leftovers from a previous run are removed at startup. ``` curl "http://127.0.0.1:3000/disk?mb=200&hold=30" ``` ## `/ready` A readiness check that you can switch. It answers `200` (`Ready`) until you call `/ready/off`, then `503` (`Not ready`). The liveness check `/` is not affected, so a container that is not ready stays alive. | Endpoint | Parameter | Default | Allowed range | Effect | |--------------|-----------|---------|-----------------------|-----------------------------------------------------------| | `/ready/off` | `seconds` | `30` | `0..MAX_HOLD_SECONDS` | `/ready` answers `503` for that long, `0` until `/ready/on` | | `/ready/on` | | | | `/ready` answers `200` again | The state is in `/info` (`ready`) and in `/metrics` (`torchio_ready`). Calling `/ready/off` again restarts the countdown. ``` curl "http://127.0.0.1:3000/ready/off?seconds=60" curl -i http://127.0.0.1:3000/ready ``` ## `/crash` Answers, then ends the whole process with the exit code you choose. Use it to test restart policies, backoff and `CrashLoopBackOff`. Nothing is cleaned up on the way out, like in a real crash; the files of `/disk` left behind are removed at the next start. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `code` | `1` | `0..255` | | `after` | `0` | `0..MAX_HOLD_SECONDS` seconds | ``` curl "http://127.0.0.1:3000/crash?code=137&after=5" ``` Remember that anyone who can reach the server can stop it. As with the rest, don't expose it to the internet. ## `/block` Freezes the server: after answering, every thread of the runtime goes to sleep for `ms`, so the process is alive but accepts nothing and answers nothing until the time is over. Use it to test liveness probes, client timeouts and what a load balancer does with an instance that hangs. `/stress` and `/cpu` run in threads of their own and don't count. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `ms` | `5000` | `1..MAX_BLOCK_MS` | The answer is sent first, so the request that starts the block returns at once; the next ones wait. A new `/block` can't start while one is going on, it is answered with a `409`. `/info` doesn't answer during the block either, but it counts them afterwards (`block`), and the log says when it starts and when the server answers again. ``` curl "http://127.0.0.1:3000/block?ms=10000" curl -m 2 http://127.0.0.1:3000/ # times out for the next 10 seconds ``` ## `/oom` Allocates memory without any budget, a chunk of 64 MB at a time, until the kernel kills the process, to test what a container does when it is killed for memory: the `OOMKilled` status, the exit code `137` and the restart policy. `/memory` and `/leak` stop at `MAX_MEMORY_MB` and never get there. It needs `ALLOW_OOM=1` (otherwise the answer is a `403` and nothing is allocated) and, when `TOKEN` is set, the token. The answer is sent first, and a warning in the log says what is about to happen and which limit the container has. ``` docker run -p 3000:3000 --memory 256m -e ALLOW_OOM=1 altrdev/torchio:latest curl http://127.0.0.1:3000/oom docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' ``` Use it only in a container with a memory limit. Without one it fills the memory of the machine, and the kernel may kill another process before this one. ## `/payload` `GET` answers with `mb` MB of data and `POST` takes an upload and throws it away, to load the network and to test the limits on the size of a body and the timeouts of a proxy, an ingress or a load balancer. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `mb` | `10` | `1..MAX_PAYLOAD_MB` | - **`GET`** streams the data a chunk at a time, so nothing is held in memory, and announces the length (`Content-Length`), so a client or a proxy can tell a body that was cut short. The bytes don't compress, so a proxy that compresses doesn't make the test smaller. - **`POST`** reads the body as it arrives, without keeping it, and answers with `{ "bytes", "seconds", "mbPerSecond" }`. It takes no parameter: a body over `MAX_PAYLOAD_MB` is answered with a `413`, which is also what a proxy with a limit on the size does. A connection that breaks halfway is a `400` that says how much had arrived. Both need the token when `TOKEN` is set, since they use the bandwidth of the instance. `torchio payload get` and `torchio payload post` do both from the command line and report the speed, see [Starting a test](https://torchio.altr.dev/starting-a-test.html#commands-that-send-or-read-a-body). ``` curl -o /dev/null -w "%{size_download} bytes in %{time_total} s\n" "http://127.0.0.1:3000/payload?mb=100" head -c 50000000 /dev/zero | curl -X POST --data-binary @- http://127.0.0.1:3000/payload ``` ## `/scenario` Runs a list of steps one after the other, so a test can be repeated without calling the endpoints by hand. `POST` starts the scenario in the body, `GET` shows where it is and `DELETE` stops it. All three need the token when `TOKEN` is set. ``` curl -X POST http://127.0.0.1:3000/scenario -d '[{"cpu": 80, "seconds": 60}, {"ready": "off", "seconds": 20}]' curl http://127.0.0.1:3000/scenario curl -X DELETE http://127.0.0.1:3000/scenario ``` The format of the steps, the checks, the answers and examples are in [Scenarios](https://torchio.altr.dev/scenarios.html). ## `/fd` Opens `count` file descriptors and closes them after `hold` seconds. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `count` | `100` | `1..MAX_FD` | | `hold` | `10` | `0..MAX_HOLD_SECONDS` | `MAX_FD` is the total that can be held at the same time, with the same `400` and `429` rules as `/memory`. The limit of the process (`ulimit -n`) is usually lower than `MAX_FD` in a container that sets one: when it is reached the answer is a `503` that says how many could be opened, and those stay open until `hold` ends. While they are open the server may fail to accept new connections or to open files, which is the situation being tested. ``` curl "http://127.0.0.1:3000/fd?count=2000&hold=30" ``` ## `/connections` Opens `count` TCP connections to the server itself and closes them after `hold` seconds. It is the same shape as [`/fd`](#fd), for what a load balancer, a proxy or a connection limit sees: a server with many open connections, which stay idle. The connections are also closed when the process ends, so a shutdown with open sockets is easy to try. | Parameter | Default | Allowed range | |-----------|---------|-----------------------| | `count` | `10` | `1..MAX_CONNECTIONS` | | `hold` | `10` | `0..MAX_HOLD_SECONDS` | `MAX_CONNECTIONS` is the total that can be held at the same time, with the same `400` and `429` rules as `/fd`. Each connection takes a descriptor on the side that opens it and another on the side that accepts it, both in this process, so the limit of the process (`ulimit -n`) is reached at half of it: the answer is then a `503` that says how many could be opened, and those stay open until `hold` ends. The answer comes when the connections are open. ``` curl "http://127.0.0.1:3000/connections?count=200&hold=60" ``` ## `/log` Writes `lines` lines (default 1000) to stdout at `rate` lines each second (default 100), then stops, to test log collection pipelines, rotation and how fast the logs of a container fill a disk. Each line has a timestamp and its number: `2026-10-18T14:00:00.000Z torchio-log line 7 of 1000`. A new request replaces the one still running; the lines already written stay. `MAX_LOG_LINES` and `MAX_LOG_RATE` set the limits. While it writes, [`/info`](#info) has `log` (`{ "sent", "lines", "rate" }`, `null` otherwise), and `watch` and the control panel show it. ## `/echo` Returns the request as this server sees it: method, URL, protocol, `hostname`, client address (`ip`, the `ips` chain and `remoteAddress`, the peer of the connection) and all the headers. Call it through an ingress controller, a proxy or a load balancer to see what they add, remove or rewrite (`Host`, `X-Forwarded-For`, `X-Forwarded-Proto`). `Authorization`, `Proxy-Authorization` and `Cookie` are shown as `[masked]`. By default the `X-Forwarded-*` headers are not trusted: `ip` is the peer and `protocol` is `http`. Set `TRUST_PROXY` to let them decide: `true` trusts every proxy, a number trusts that many hops from the server, and a list of addresses, subnets (`10.0.0.0/8`) and the names `loopback`, `linklocal` and `uniquelocal` trusts those. The client is then the first address, walking back from the connection through `X-Forwarded-For`, that is not a trusted proxy, and `ip`, `ips`, `protocol`, `secure` and `hostname` follow the proxy. ## `/info` Returns a JSON document with what the server is doing right now. It takes no parameters and doesn't use up any budget, so you can poll it while you test, for example with `watch -n1 curl -s http://127.0.0.1:3000/info`. ```json { "name": "torchio", "version": "4.0.0", "uptimeSeconds": 321, "auth": false, "stress": { "workers": 4, "inFlight": 2, "perWorker": [1, 1, 0, 0], "burn": null }, "delay": { "inFlight": 3 }, "memory": { "heldMb": 200, "limitMb": 512, "processRssMb": 310, "containerLimitMb": 256 }, "disk": { "heldMb": 100, "limitMb": 1024, "dir": "/tmp", "freeMb": 8120 }, "fd": { "held": 0, "limit": 10000, "processLimit": 1048576, "secondsLeft": null }, "ready": { "ready": true, "secondsLeft": null }, "log": null, "limits": { "maxNumber": 50, "maxDelayMs": 60000, "maxHoldSeconds": 300, "maxLogLines": 100000, "maxLogRate": 10000 } } ``` | Field | Meaning | |-------------------------------|-------------------------------------------------------------------------| | `stress.inFlight`, `perWorker`| `/stress` requests being calculated, in total and for each worker | | `stress.burn` | What `/cpu` is doing, `{ "percent", "secondsLeft", "workers" }` (`workers` is how many are loaded), or `null` when it is not running | | `delay.inFlight` | `/delay` requests still waiting | | `memory.heldMb`, `limitMb` | Memory held by `/memory` and the total it may hold (`MAX_MEMORY_MB`); the difference is what a new request can still allocate | | `memory.processRssMb` | Memory the whole process uses, the figure that counts for an OOM kill | | `memory.containerLimitMb` | Memory limit of the container, `null` when there is none | | `memory.secondsLeft`, `disk.secondsLeft`, `fd.secondsLeft`, `connections.secondsLeft` | Seconds until the last hold of that resource lets go, `null` when nothing is held | | `disk.heldMb`, `limitMb` | Same as for memory, for `/disk` (`MAX_DISK_MB`) | | `disk.freeMb` | Free space in `DISK_DIR`, `null` if the directory is not usable | | `fd.held`, `limit`, `processLimit` | File descriptors held by `/fd`, the total it may hold (`MAX_FD`) and the limit of the process itself (`ulimit -n`, `null` when it has none) | | `connections.held`, `limit` | Connections held by `/connections` and the total it may hold (`MAX_CONNECTIONS`) | | `log` | What `/log` is writing, `{ "sent", "lines", "rate" }`, or `null` when it is not running | | `shutdown` | `{ "draining", "graceSeconds", "drainSeconds", "openRequests", "openConnections" }`: whether SIGTERM started a [drain](https://torchio.altr.dev/shutdown.html#shutdown), for how long (`null` if not), and what is open now, the request for `/info` and its connection included | | `block` | `{ "count", "lastMs", "msLeft" }`: the blocks of `/block` so far, the length of the last one and what is left of the one running | | `scenario` | Where [`/scenario`](https://torchio.altr.dev/scenarios.html) is: `{ "state": "idle" }` when none has run, else its state, step, steps and, while it runs, `secondsLeft` and `remainingSeconds` | | `ready.ready`, `secondsLeft` | What `/ready` answers, and the seconds left before it is ready again (`null` if nothing is counting down) | ## `/metrics` The figures of [`/info`](#info) in the Prometheus text format, plus a counter of the requests answered and a histogram of the time they took. It takes no parameters and doesn't use up any budget. | Metric | Type | Meaning | |---------------------------------------------|---------|-------------------------------------------------------------| | `torchio_info{version}` | gauge | Always `1`, the version is in the label | | `torchio_start_time_seconds` | gauge | Start time, in Unix time | | `torchio_workers` | gauge | Worker threads for `/stress` | | `torchio_stress_requests_in_flight` | gauge | `/stress` requests being calculated | | `torchio_worker_requests_in_flight{worker}` | gauge | The same, for each worker | | `torchio_cpu_burn_percent` | gauge | Share of the time each worker is kept busy by `/cpu`, `0` when it is not running | | `torchio_cpu_burn_workers` | gauge | Workers `/cpu` keeps busy, `0` when it is not running | | `torchio_delay_requests_in_flight` | gauge | `/delay` requests still waiting | | `torchio_memory_held_bytes` | gauge | Memory held by `/memory` | | `torchio_memory_limit_bytes` | gauge | Total memory `/memory` may hold (`MAX_MEMORY_MB`) | | `process_resident_memory_bytes` | gauge | Resident memory of the whole process | | `torchio_container_memory_limit_bytes` | gauge | Memory limit of the container, only present when there is one | | `torchio_disk_held_bytes` | gauge | Disk space held by `/disk` | | `torchio_disk_limit_bytes` | gauge | Total disk space `/disk` may hold (`MAX_DISK_MB`) | | `torchio_disk_free_bytes` | gauge | Free space in `DISK_DIR`, only present when it is usable | | `torchio_fd_held` | gauge | File descriptors held by `/fd` | | `torchio_fd_limit` | gauge | Total descriptors `/fd` may hold (`MAX_FD`) | | `torchio_connections_held` | gauge | TCP connections held by `/connections` | | `torchio_connections_limit` | gauge | Total connections `/connections` may hold (`MAX_CONNECTIONS`) | | `torchio_ready` | gauge | `1` while `/ready` answers `200`, `0` while it answers `503` | | `torchio_draining` | gauge | `1` after SIGTERM while the server [drains](https://torchio.altr.dev/shutdown.html#shutdown) | | `torchio_open_requests` | gauge | Requests being answered, the one for `/metrics` included | | `torchio_open_connections` | gauge | Connections open, the one for `/metrics` included | | `torchio_http_requests_total{route,code}` | counter | Requests answered, by route pattern (such as `/status/:code`) and status code. A request the client abandons before the answer is not counted | | `torchio_http_request_duration_seconds{route,code}` | histogram | Time taken to answer, in seconds, with the same labels (`_bucket{le}`, `_sum`, `_count`). Buckets go from 5 ms to 60 s, which is the default `MAX_DELAY_MS`. It includes the time of `/delay`, and the requests to `/metrics` and `/info` themselves | To collect them with Prometheus: ```yaml scrape_configs: - job_name: torchio static_configs: - targets: ["torchio:3000"] ``` Some queries to start from: - `torchio_memory_held_bytes / torchio_memory_limit_bytes` is how much of the `/memory` budget is used. - `process_resident_memory_bytes / torchio_container_memory_limit_bytes` is how close the process is to an OOM kill. - `rate(torchio_http_requests_total{code=~"4..|5.."}[1m])` is the rate of rejected and failed requests (`429`, `507`, `/status/...`). - `histogram_quantile(0.95, sum by (le, route) (rate(torchio_http_request_duration_seconds_bucket[5m])))` is the 95th percentile of the response time of each route. ## `/help` Lists every endpoint with what it does, whether `TOKEN` protects it and its parameters with the default and the allowed range, so you can use the image without reading this page. It answers JSON, or plain text when asked by a browser: ``` curl -s localhost:3000/help | jq '.endpoints[] | select(.path == "/cpu")' ``` ```json { "path": "/cpu", "auth": true, "params": { "percent": { "fallback": 50, "min": 0, "max": 100 }, "hold": { "fallback": 10, "min": 0, "max": 300 } }, "description": "Keeps the workers busy for a share of the time, percent=0 stops it" } ``` The limits are the ones of this instance (`MAX_*`). The list is built from the same definitions the endpoints read their parameters from (`src/endpoints.rs`), so it can't go out of date. `/ui` is listed only with `UI=1`. ## `/config` Returns the effective configuration as JSON: port, number of workers, defaults and limits, `DISK_DIR`, `UI`, `START_DELAY_SECONDS`. The token is never shown, only whether there is one (`auth`). Like `/info`, it is always open. --- # Reference Every endpoint with its parameters, as `/help` lists them, with the defaults and limits of a server with no settings. The ones marked *token* need it when [`TOKEN`](https://torchio.altr.dev/token.html) is set. The page [Endpoints](https://torchio.altr.dev/endpoints.html) says what each one does. | Endpoint | Token | Command | What it does | |----------|-------|---------|--------------| | [`/`](#root) | | | Health check, returns "Status: OK" | | [`/stress`](#stress) | yes | `stress` | Runs a Fibonacci calculation in a worker, to keep a core busy | | [`/cpu`](#cpu) | yes | `cpu` | Keeps the workers (or the first ones, with workers) busy for a share of the time, percent=0 stops it | | [`/delay`](#delay) | | `delay` | Answers after a delay, without blocking the server | | [`/status/:code`](#statuscode) | | `status` | Answers with the given HTTP status | | [`/flaky`](#flaky) | | `flaky` | Fails rate percent of the requests with code, and answers 200 to the rest | | [`/memory`](#memory) | yes | `memory` | Allocates memory (MB) and holds it | | [`/leak`](#leak) | yes | `leak` | Memory that grows by itself, rate MB each second up to mb MB | | [`/disk`](#disk) | yes | `disk` | Writes a file (MB) in DISK_DIR and keeps it | | [`/fd`](#fd) | yes | `fd` | Opens file descriptors and holds them | | [`/connections`](#connections) | yes | `connections` | Opens TCP connections to the server itself and holds them | | [`/ready`](#ready) | | | Readiness check: 200, or 503 after /ready/off | | [`/ready/off`](#readyoff) | yes | `ready` | Makes /ready answer 503 for seconds (0 until /ready/on) | | [`/ready/on`](#readyon) | yes | `ready` | Makes /ready answer 200 again | | [`/crash`](#crash) | yes | `crash` | Ends the process with the exit code, after the given seconds | | [`/block`](#block) | yes | `block` | Freezes every thread of the server for ms, so it accepts and answers nothing | | [`/oom`](#oom) | yes | | Allocates memory until the kernel kills the process, needs ALLOW_OOM=1 | | [`/payload`](#payload) | yes | `payload` | GET answers with mb MB of data, streamed; POST takes an upload, discards it and reports its size and time | | [`/scenario`](#scenario) | yes | `scenario` | POST a list of steps to run one after the other, GET shows where it is, DELETE stops it | | [`/log`](#log) | yes | `log` | Writes lines to stdout at rate lines/s, then stops | | [`/echo`](#echo) | | | The request as the server sees it: headers, client address, protocol (credentials masked) | | [`/ui`](#ui) | | | Control panel in the browser | | [`/info`](#info) | | | JSON with what the server is doing and the remaining budgets | | [`/metrics`](#metrics) | | | The same figures in the Prometheus text format | | [`/help`](#help) | | | This list | | [`/config`](#config) | | | The effective configuration, without the token | ## The root, `/` {#root} Health check, returns "Status: OK". No parameters. ## `/stress` Runs a Fibonacci calculation in a worker, to keep a core busy. Command: `torchio stress [number]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `number` | query | `45` | `0..50` | ## `/cpu` Keeps the workers (or the first ones, with workers) busy for a share of the time, percent=0 stops it. Command: `torchio cpu [percent] [hold] [workers]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `percent` | query | `50` | `0..100` | | `hold` | query | `10` | `0..300` | | `workers` | query | `4` | `1..WORKERS` | ## `/delay` Answers after a delay, without blocking the server. Command: `torchio delay [ms]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `ms` | query | `1000` | `0..60000` | ## `/status/:code` Answers with the given HTTP status. Command: `torchio status `, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `code` | path | required | `200..599` | ## `/flaky` Fails rate percent of the requests with code, and answers 200 to the rest. Command: `torchio flaky [rate] [code] [ms]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `rate` | query | `50` | `0..100` | | `code` | query | `503` | `400..599` | | `ms` | query | `0` | `0..60000` | ## `/memory` Allocates memory (MB) and holds it. Command: `torchio memory [mb] [hold]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `mb` | query | `100` | `1..512` | | `hold` | query | `10` | `0..300` | ## `/leak` Memory that grows by itself, rate MB each second up to mb MB. Command: `torchio leak [rate] [mb] [hold]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `rate` | query | `10` | `1..512` | | `mb` | query | `100` | `1..512` | | `hold` | query | `10` | `0..300` | ## `/disk` Writes a file (MB) in DISK_DIR and keeps it. Command: `torchio disk [mb] [hold]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `mb` | query | `100` | `1..1024` | | `hold` | query | `10` | `0..300` | ## `/fd` Opens file descriptors and holds them. Command: `torchio fd [count] [hold]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `count` | query | `100` | `1..10000` | | `hold` | query | `10` | `0..300` | ## `/connections` Opens TCP connections to the server itself and holds them. Command: `torchio connections [count] [hold]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `count` | query | `10` | `1..500` | | `hold` | query | `10` | `0..300` | ## `/ready` Readiness check: 200, or 503 after /ready/off. No parameters. ## `/ready/off` Makes /ready answer 503 for seconds (0 until /ready/on). Command: `torchio ready off [seconds]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `seconds` | query | `30` | `0..300` | ## `/ready/on` Makes /ready answer 200 again. Command: `torchio ready on`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). No parameters. ## `/crash` Ends the process with the exit code, after the given seconds. Command: `torchio crash [code] [after]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `code` | query | `1` | `0..255` | | `after` | query | `0` | `0..300` | ## `/block` Freezes every thread of the server for ms, so it accepts and answers nothing. Command: `torchio block [ms]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `ms` | query | `5000` | `1..30000` | ## `/oom` Allocates memory until the kernel kills the process, needs ALLOW_OOM=1. No parameters. ## `/payload` GET answers with mb MB of data, streamed; POST takes an upload, discards it and reports its size and time. Command: `torchio payload get [mb] | post `, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `mb` | query | `10` | `1..1024` | ## `/scenario` POST a list of steps to run one after the other, GET shows where it is, DELETE stops it. Command: `torchio scenario [--wait] | status | stop`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). No parameters. ## `/log` Writes lines to stdout at rate lines/s, then stops. Command: `torchio log [lines] [rate]`, see [Starting a test](https://torchio.altr.dev/starting-a-test.html). | Parameter | In | Default | Range | |-----------|----|---------|-------| | `lines` | query | `1000` | `1..100000` | | `rate` | query | `100` | `1..10000` | ## `/echo` The request as the server sees it: headers, client address, protocol (credentials masked). No parameters. ## `/ui` Control panel in the browser. No parameters. ## `/info` JSON with what the server is doing and the remaining budgets. No parameters. ## `/metrics` The same figures in the Prometheus text format. No parameters. ## `/help` This list. No parameters. ## `/config` The effective configuration, without the token. No parameters. --- # Starting a test Every endpoint that changes the state of the server is also a command, so you don't need `curl`. It calls a running server (`--url`, by default `http://127.0.0.1:$PORT` or port `3000`) and prints its answer: ``` torchio cpu 80 --hold 60 # 80% of every worker for a minute torchio memory 512 # hold 512 MB for the default time torchio leak --rate 5 --mb 1000 torchio ready off 30 # not ready for 30 seconds torchio status 503 torchio crash 137 --after 5 ``` | Command | Endpoint | |----------------------------------|--------------------------| | `torchio stress [number]` | [`/stress`](https://torchio.altr.dev/endpoints.html#stress) | | `torchio cpu [percent] [hold] [workers]` | [`/cpu`](https://torchio.altr.dev/endpoints.html#cpu) | | `torchio delay [ms]` | [`/delay`](https://torchio.altr.dev/endpoints.html#delay) | | `torchio status ` | [`/status/:code`](https://torchio.altr.dev/endpoints.html#statuscode) | | `torchio flaky [rate] [code] [ms]` | [`/flaky`](https://torchio.altr.dev/endpoints.html#flaky) | | `torchio memory [mb] [hold]` | [`/memory`](https://torchio.altr.dev/endpoints.html#memory) | | `torchio leak [rate] [mb] [hold]`| [`/leak`](https://torchio.altr.dev/endpoints.html#leak) | | `torchio disk [mb] [hold]` | [`/disk`](https://torchio.altr.dev/endpoints.html#disk) | | `torchio fd [count] [hold]` | [`/fd`](https://torchio.altr.dev/endpoints.html#fd) | | `torchio connections [count] [hold]` | [`/connections`](https://torchio.altr.dev/endpoints.html#connections) | | `torchio log [lines] [rate]` | [`/log`](https://torchio.altr.dev/endpoints.html#log) | | `torchio crash [code] [after]` | [`/crash`](https://torchio.altr.dev/endpoints.html#crash) | | `torchio block [ms]` | [`/block`](https://torchio.altr.dev/endpoints.html#block) | | `torchio ready on \| off [seconds]` | [`/ready`](https://torchio.altr.dev/endpoints.html#ready) | | `torchio scenario [--wait]`, `status`, `stop` | [`/scenario`](https://torchio.altr.dev/scenarios.html) | | `torchio payload get [mb]`, `post ` | [`/payload`](https://torchio.altr.dev/endpoints.html#payload) | The values after the name are the parameters of the endpoint, in the order of the table; any of them can also be given by name, as `--hold 60` or `--hold=60`. The limits are the ones of the server: a value it refuses is printed with its `400` and the command exits with `1`. With [`TOKEN`](https://torchio.altr.dev/token.html#token), pass `--token ` or set `TOKEN`. `/stress`, `/delay` and `/flaky` with `ms` answer when they finish, so the command waits for them. Like `watch`, it speaks plain HTTP only. ## Commands that send or read a body `scenario` and `payload` don't fit the table above: they take a file, or send and read a body, and not only numbers. They take `--url` and `--token` (or `TOKEN`) like the others. ### `torchio scenario` ``` torchio scenario ramp.json # start the scenario in the file torchio scenario ramp.json --wait # and follow it until it ends cat ramp.json | torchio scenario - # from stdin torchio scenario status # where it is torchio scenario stop # end it ``` It prints what the server answers, written out. A mistake in the file is the `400` of the server, which names the step (`step 2: memory must be between 1 and 512`), and the command exits with `1`. With `--wait` the command stays until the scenario ends and prints each step as it starts (on the error output); the exit status is `0` if the scenario **finished** and `1` if it failed, was stopped or the server could not be reached for good. That makes it the line to put in a pipeline: `torchio scenario ramp.json --wait && ./check-the-metrics.sh`. The words `status` and `stop` are reserved: for a file with that name write `./status`. The steps are explained in [Scenarios](https://torchio.altr.dev/scenarios.html). ### `torchio payload` ``` torchio payload get 100 # download 100 MB, throw them away torchio payload post backup.tar # upload a file head -c 50000000 /dev/zero | torchio payload post - ``` `get` reads the body as it arrives, without keeping it, and prints how much came, how long it took and the speed (`Received 100.0 MB in 2.31 s (43.3 MB/s)`). If the server announced a length and the body ends before it, because a proxy or a timeout cut it, that is an error with `1`: it is the thing the command is for. `post` sends a file, or what is piped in, and prints what the server counted. A body over `MAX_PAYLOAD_MB` is a `413`, and a proxy with a smaller limit of its own may close the connection before answering, which the command says too (`The connection was closed while the body was being sent`). The limits and what each endpoint does are in [`/payload`](https://torchio.altr.dev/endpoints.html#payload). --- # Scenarios A test is often more than one request: load the CPU for a minute, then fill the memory, then take the instance out of the Service. A scenario is that list written once, in JSON, and run by the server by itself, one step after the other. You can repeat it, keep it in the repository next to the manifests and run it from a pipeline (`torchio scenario file.json --wait`), without a script that calls the endpoints in order. ``` curl -X POST http://127.0.0.1:3000/scenario -d '[ {"cpu": 80, "seconds": 60}, {"memory": 500, "seconds": 30}, {"ready": "off", "seconds": 20} ]' ``` The server answers `202` with the scenario and its first step already running. The three steps above take 110 seconds: the CPU at 80% for a minute, then 500 MB held for 30 seconds, then `/ready` answering `503` for 20. ## How a step is written A step is an object with **one action** and **`seconds`**, how long it lasts. When the seconds are over the next step starts. | Action | Value | What it does | Allowed range | |---------------|------------------------|-------------------------------------------------------------------------|-------------------------| | `cpu` | percent | Keeps the workers busy for that share of the time, like [`/cpu`](https://torchio.altr.dev/endpoints.html#cpu) | `0..100` | | `memory` | MB | Holds that much memory, like [`/memory`](https://torchio.altr.dev/endpoints.html#memory) | `1..MAX_MEMORY_MB` | | `disk` | MB | Holds a file of that size in `DISK_DIR`, like [`/disk`](https://torchio.altr.dev/endpoints.html#disk) | `1..MAX_DISK_MB` | | `fd` | count | Holds that many descriptors, like [`/fd`](https://torchio.altr.dev/endpoints.html#fd) | `1..MAX_FD` | | `connections` | count | Holds that many connections to the server, like [`/connections`](https://torchio.altr.dev/endpoints.html#connections) | `1..MAX_CONNECTIONS` | | `ready` | `"off"` | Makes `/ready` answer `503`, like [`/ready/off`](https://torchio.altr.dev/endpoints.html#ready) | only `"off"` | | `block` | ms | Freezes the server, like [`/block`](https://torchio.altr.dev/endpoints.html#block) | `1..MAX_BLOCK_MS` | | *(none)* | | A pause: nothing happens for `seconds` | | - **`seconds`** is needed in every step, from `1` to `MAX_HOLD_SECONDS`. The only exception is `block`, which lasts as long as its `ms` says (rounded up to the second) and takes no `seconds`. - **`workers`** goes with `cpu` and loads only the first workers, from `1` to the number of workers. Without it every worker is loaded: `{"cpu": 80, "workers": 1, "seconds": 60}`. - A scenario has at most **100 steps**. The limits are the ones of the endpoints: a step can't ask for more than the endpoint would give. A scenario is made of what the server can already do, with the same budgets. ### What is not a step - [`/crash`](https://torchio.altr.dev/endpoints.html#crash) and [`/oom`](https://torchio.altr.dev/endpoints.html#oom) end the instance, and nothing is left to run the next step. - [`/delay`](https://torchio.altr.dev/endpoints.html#delay), [`/status/:code`](https://torchio.altr.dev/endpoints.html#statuscode) and [`/flaky`](https://torchio.altr.dev/endpoints.html#flaky) act on the requests that come, not on the server, so there is no time in them to put in a step. Run them from your client while the scenario goes. - [`/leak`](https://torchio.altr.dev/endpoints.html#leak), [`/log`](https://torchio.altr.dev/endpoints.html#log) and [`/payload`](https://torchio.altr.dev/endpoints.html#payload) are not steps: they take several values or need a client on the other side. ## Checked before it starts The whole scenario is checked when it arrives, before anything runs. A mistake is a `400` that names the step and nothing starts, so you never find out at minute ten that step six was wrong: ``` curl -X POST http://127.0.0.1:3000/scenario -d '[{"cpu": 50, "seconds": 5}, {"memory": 9000, "seconds": 5}]' step 2: memory must be between 1 and 512 ``` | Message | Why | |--------------------------------------------------|-------------------------------------------------------| | `step 2: memory must be between 1 and 512` | A value outside the limit of its endpoint | | `step 1: seconds is needed` | A step with an action and no duration | | `step 1: unknown key 'crash'` | A key that is not an action, `seconds` or `workers` | | `step 1: more than one action (cpu and memory)` | A step does one thing; put the other in the next step, or run them one after the other with a pause | | `step 1: workers goes with cpu` | `workers` on another action | | `step 1: nothing to do, name an action or give seconds for a pause` | An empty step | | `The body must be a list of steps` | The body is not a JSON list | ## Running, reading, stopping | Request | What it does | |---------------------|-------------------------------------------------------------------------------| | `POST /scenario` | Starts the scenario in the body: `202`, or `400` (see above), or `409` if one is running | | `GET /scenario` | Where it is: the state, the step, the seconds left and the plan with each step's state | | `DELETE /scenario` | Stops it: `200`, or `404` if none is running | Only one scenario runs at a time. A second `POST` while one is running is a `409` that says at which step it is; stop it first with `DELETE`, or wait for the end. ``` curl http://127.0.0.1:3000/scenario ``` ```json { "state": "running", "step": 2, "steps": 3, "secondsLeft": 18, "remainingSeconds": 38, "now": { "memory": 500, "seconds": 30 }, "plan": [ { "cpu": 80, "workers": 8, "seconds": 60, "state": "done" }, { "memory": 500, "seconds": 30, "state": "running" }, { "ready": "off", "seconds": 20, "state": "pending" } ] } ``` The `state` of the scenario is `idle` (none has run), `running`, `finished`, `stopped` or `failed`, and the state of each step in the plan is `done`, `running`, `pending`, `stopped` or `failed`. `secondsLeft` is what remains of the current step and `remainingSeconds` of the whole scenario. The last scenario stays readable after it ends, until the next one starts. `/info` has the same figures (`scenario`, without the plan) and the log says each step when it starts: ``` Scenario started: 3 steps Scenario step 1 of 3: cpu 80% of 8 workers for 60 s Scenario step 2 of 3: memory 500 MB for 30 s ``` ### When a step can't run The limits are checked at the start, but the budgets are shared with everyone else: if another request holds the memory when a step asks for it, the step can't have it. The scenario then stops there with the state `failed` and the reason in `error` (`step 2 (memory): 300 MB already held, limit is 512 MB`, the same words the endpoint answers with a `429`), and the steps after it don't run. Opening fewer descriptors or connections than asked, because the process hit its own limit, is not a failure: it is what `/fd` and `/connections` are for, and the step goes on (the log says how many were opened). ### Stopping `DELETE /scenario` ends the sequence at once. The CPU load and the readiness switch of the current step are undone right away. What a step holds for a time (memory, disk, descriptors, connections) is given back when its seconds are over, as it would be after the same request on its own, so the budget is free again at most `seconds` later. A `block` can't be stopped once it has started, and while it lasts the server doesn't answer, `GET /scenario` included. The scenario's own clock is frozen with the rest, so the step after the block starts when the server wakes up. ## Examples **A ramp of CPU**, to watch an autoscaler add replicas and then remove them: ``` [ {"cpu": 30, "seconds": 120}, {"cpu": 60, "seconds": 120}, {"cpu": 90, "seconds": 300}, {"cpu": 0, "seconds": 300} ] ``` The last step, with `cpu` at `0`, is the cool-down: it keeps the load off for five minutes, which is the time a scale-down is judged on. **An instance that leaves the Service and comes back**, to test a readiness probe and what a load balancer does with it: ``` [ {"seconds": 30}, {"ready": "off", "seconds": 60}, {"seconds": 30} ] ``` **Every budget in turn**, to see what each limit does to the instance one resource at a time, with a pause so the previous one is given back: ``` [ {"memory": 400, "seconds": 60}, {"disk": 500, "seconds": 60}, {"fd": 5000, "seconds": 30}, {"connections": 200, "seconds": 30} ] ``` Each step starts when the previous one ends, so the resources are not held together. To hold two at the same time, run the second with its own endpoint while the scenario goes: the budgets are shared, so that works. **A hang in the middle of a test**, to see a liveness probe time out: ``` [ {"cpu": 50, "seconds": 20}, {"block": 15000}, {"cpu": 50, "seconds": 20} ] ``` ## From the command line ``` torchio scenario ramp.json --wait ``` starts the scenario in the file and stays until it ends, with the exit status `0` only if it finished: the line for a pipeline that loads an instance and then checks it. `torchio scenario status` and `torchio scenario stop` do the other two requests. See [Starting a test](https://torchio.altr.dev/starting-a-test.html#commands-that-send-or-read-a-body). ## Seeing it - [`torchio watch`](https://torchio.altr.dev/watch.html) has a line for the scenario (`step 2 of 3, 18 s left in it, 38 s in all`, or why it failed) next to the other activity. - The [control panel](https://torchio.altr.dev/control-panel.html) has a Scenario card: a box for the JSON, with an example, and the buttons to start and stop it. The step and the time left are in its header. - `GET /scenario` and [`/info`](https://torchio.altr.dev/endpoints.html#info) for scripts. ## The token All three requests need [`TOKEN`](https://torchio.altr.dev/token.html#token) when it is set, `GET` included, since it shows the whole plan. [`/info`](https://torchio.altr.dev/endpoints.html#info) stays open and shows where the scenario is. --- # Shell completion `torchio completions ` prints a script that makes the shell complete the commands (`torchio me` gives `memory`), their options (`--hold`, `--url`…) and the words `on` and `off` of `ready`. With Homebrew the completions are installed with the binary: open a new shell. Otherwise: ``` # bash: in ~/.bashrc eval "$(torchio completions bash)" # zsh: in ~/.zshrc, after compinit eval "$(torchio completions zsh)" # fish torchio completions fish > ~/.config/fish/completions/torchio.fish ``` If Tab does nothing in zsh after `brew install`, there are two usual causes: - The folder of the completions, `$(brew --prefix)/share/zsh/site-functions`, is not in your `fpath`. `brew shellenv` adds it, so run it before `compinit` or before Oh My Zsh (which calls `compinit`): `eval "$(brew shellenv)"`. See [Homebrew's page on shell completion](https://docs.brew.sh/Shell-Completion). - `compinit` ignores the folders that the group can write to, and Oh My Zsh calls it with `-i`, so it does not say so. If `ls -ld $(brew --prefix)/share/zsh/site-functions` shows `drwxrwxr-x`, remove the group's write permission from that folder and its two parents, and not from the whole prefix: `chmod go-w "$(brew --prefix)/share" "$(brew --prefix)/share/zsh" "$(brew --prefix)/share/zsh/site-functions"`. Then open a new shell, and delete `~/.zcompdump*` if it still does not complete. The scripts are generated from the commands of the binary, so they always match the version you have. On the command line of [`watch`](https://torchio.altr.dev/watch.html#keys), Tab completes the name of a command. --- # For AI agents An agent that runs tests with torchio has three things to read, from the least to the most work. ## The server describes itself `GET /help` lists every endpoint with its parameters, defaults and the limits of that instance, as JSON, and `GET /info` says what the server holds right now and what is left of each budget. They come from the same definitions the endpoints use, so they are right for the version and the settings that are running. See [`/help`](https://torchio.altr.dev/endpoints.html#help) and [`/info`](https://torchio.altr.dev/endpoints.html#info). ## llms.txt The site follows the [llms.txt](https://llmstxt.org) convention: - is an index of the pages, each with a line about it. - is the whole documentation in one file, without the changelog. Both are written from the pages of the book when the site is built, so they match the version that is published. ## A skill `skills/torchio/SKILL.md` is a skill that tells an agent how to put an instance under pressure: look at `/help` and `/info` first, which tests take the instance away and need a yes (`/crash`, `/oom`, `/block`), the budgets and their `429`, scenarios, and how to watch the result. It is also at . To use it with Claude Code, put it in a folder of skills: ``` mkdir -p ~/.claude/skills/torchio curl -fsSL https://torchio.altr.dev/SKILL.md -o ~/.claude/skills/torchio/SKILL.md ``` Use `.claude/skills/torchio/` inside a project to share it with the people who work on that project. Other agents that read `SKILL.md` files take it the same way, from the folder where they look for skills. --- # Watching a server `torchio watch` shows what a server is doing in the terminal, refreshing every second with bars and a short history of the last samples (workers, memory, process memory, disk and file descriptors), plus whether `/ready` is answering `200` or `503`, a line for each `/cpu`, `/memory`, `/disk`, `/fd` and `/connections` that is running with the seconds it has left (`/memory: 100 MB, 18 s left`), and a line for the [scenario](https://torchio.altr.dev/scenarios.html) while one runs or after it ends. It is a command of the same binary, reads [`/info`](https://torchio.altr.dev/endpoints.html#info) and works against any running instance: ``` torchio watch torchio watch --url http://my-host:3000 --interval 2 ``` Inside a container, where the server is on `localhost`: ``` docker exec -it torchio watch ``` | Option | Default | Description | |---------------|----------------------------|----------------------------------------------------------| | `--url` | `http://127.0.0.1:$PORT` or port `3000` | Server to watch | | `--interval` | `1` | Seconds between samples | | `--token` | `$TOKEN` | Token for the commands that need it | | `--read-only` | | Only look: no keys, even on a terminal | | `--once` | | Print one sample and exit; the status is `1` if the server can't be reached | ## Keys On a terminal, `watch` also starts the tests, so you can load a server and see the bars move in the same window. A key opens a command line with the command already written, you add the values and press Enter: | Key | Command | Key | Command | |-----|---------|-----|---------| | `c` | `cpu` | `g` | `log` | | `m` | `memory`| `s` | `stress`| | `l` | `leak` | `r` | `ready` | | `d` | `disk` | `x` | `crash` | | `f` | `fd` | `:` | any command, like `status 503` | The line shows the parameters of the command while you type (`cpu [percent] [hold]`), Enter with only the name uses the defaults, Esc cancels and `q` or Ctrl+C quits. The answer of the server, or its error, stays under the keys. The request runs in the background, so the view keeps refreshing while `/stress` or `/delay` wait. The commands are the ones of [Starting a test](https://torchio.altr.dev/starting-a-test.html#starting-a-test). The keys are on only when both the input and the output are a terminal. `--read-only` turns them off, which is the way to watch a shared instance without a key that can [`/crash`](https://torchio.altr.dev/endpoints.html#crash) it. With `--once`, or with the output in a pipe or a file, `watch` never takes keys. It speaks plain HTTP: to watch a server behind HTTPS, reach it with a port-forward. `watch` needs a server of version 4.5.0 or newer, and says so with the version of the server when it is older. A newer server is fine. The history only covers the time the command has been running. On a terminal the bars are smooth and shift from green to red as they get close to their limit (blue where there is no limit to compare with), the history is brighter when it is higher, and what is happening (`/stress`, `/cpu`, `/log`) lights up while the rest stays dim. Readiness is a badge, green or red. Set `NO_COLOR=1` for plain text, which is also what you get when the output is not a terminal. --- # Control panel `torchio` also has a page in the browser that does the job of `watch` and lets you start the tests with sliders and buttons instead of `curl`. It is off by default, turn it on with `UI=1` and open `/ui`: ``` docker run -p 3000:3000 -e UI=1 altrdev/torchio:latest ``` The gauges and their history (the last 60 seconds) come from [`/info`](https://torchio.altr.dev/endpoints.html#info) and [`/metrics`](https://torchio.altr.dev/endpoints.html#metrics), refreshed every second, and the page shows whether `/ready` is answering `200` or `503`. Each card makes the request you see written under its sliders, and the answer goes in the log at the bottom, so it is also a way to learn the API. The history lives in the page: reloading it starts again. Three cards do more than send one request. **Status** asks for any HTTP status. **Flaky** sends a batch of 20 requests to [`/flaky`](https://torchio.altr.dev/endpoints.html#flaky) and counts how many failed, since the failures come from the requests and not from the server. **Payload** downloads or uploads a body (generated, or a file you pick) and reports the speed, and if the body is cut short before its `Content-Length`: the browser counts what it gets and what it sends, so what is measured is the road between the browser and the server, proxies and ingresses included. An upload from the page is at most 200 MB, because the browser keeps the body in memory; the server's limit is `MAX_PAYLOAD_MB`. [`/oom`](https://torchio.altr.dev/endpoints.html#oom) has no card on purpose. A card runs a [scenario](https://torchio.altr.dev/scenarios.html): you write the steps in JSON, start it and stop it, and the step it is in and the time left show in the card and at the top of the page. The page makes [`/crash`](https://torchio.altr.dev/endpoints.html#crash) a button, which is why it is off by default (and the button asks to be clicked twice). It needs no extra dependency and no build, it is one static file. --- # Configuration Environment variables: | Variable | Default | Description | |--------------------|-----------------|---------------------------------------------------| | `PORT` | `3000` | Port the server listens on | | `WORKERS` | CPUs of the process | Number of worker threads for `/stress` and `/cpu` | | `MAX_NUMBER` | `50` | Highest `number` accepted by `/stress` | | `MAX_DELAY_MS` | `60000` | Highest `ms` accepted by `/delay` | | `MAX_MEMORY_MB` | `512` | Total memory `/memory` can hold at once | | `MAX_DISK_MB` | `1024` | Total disk space `/disk` can hold at once | | `UI` | off | `1` turns on the control panel at `/ui` | | `START_DELAY_SECONDS` | `0` | Seconds the port stays closed after the start | | `SHUTDOWN_GRACE_SECONDS` | `0` | Seconds SIGTERM drains for before the process ends, see [Shutdown](https://torchio.altr.dev/shutdown.html#shutdown) | | `MAX_FD` | `10000` | Total descriptors `/fd` can hold at once | | `MAX_CONNECTIONS` | `500` | Total connections `/connections` can hold at once | | `MAX_BLOCK_MS` | `30000` | Highest `ms` accepted by `/block` | | `ALLOW_OOM` | off | `1` lets `/oom` run: it allocates until the kernel kills the process | | `MAX_PAYLOAD_MB` | `1024` | Highest `mb` accepted by `/payload`, and the biggest upload it takes | | `MAX_LOG_LINES` | `100000` | Highest `lines` accepted by `/log` | | `MAX_LOG_RATE` | `10000` | Highest `rate` accepted by `/log` | | `TRUST_PROXY` | off | Trusts `X-Forwarded-*` (`true`, a number of hops, or addresses and subnets), see `/echo` | | `TOKEN` | none | Makes the endpoints that change the state need `Authorization: Bearer ` | | `MAX_HOLD_SECONDS` | `300` | Highest `hold`, `seconds` and `after` accepted by the endpoints that take them | | `DISK_DIR` | system temp dir | Directory where `/disk` writes its files | | `LOG_FORMAT` | `text` | `json` writes one JSON object a line instead of plain text, see [Logs](https://torchio.altr.dev/logs.html#logs) | A number must be a positive integer. A value that is not one is reported at startup and replaced by the default. ``` docker run -p 3000:3000 -e WORKERS=2 -e MAX_MEMORY_MB=2048 altrdev/torchio:latest ``` --- # Token Anyone who can reach the port can call `/crash` or `/disk`. Set `TOKEN` to ask for a password on the endpoints that change the state of the instance (`/stress`, `/cpu`, `/memory`, `/leak`, `/disk`, `/fd`, `/connections`, `/log`, `/ready/off`, `/ready/on`, `/block`, `/oom`, `/payload`, `/scenario` and `/crash`): ``` docker run -p 3000:3000 -e TOKEN=secret altrdev/torchio:latest curl -H "Authorization: Bearer secret" localhost:3000/crash ``` Without the header, or with a wrong token, they answer `401`. `/`, `/ready`, `/info`, `/metrics`, `/help`, `/config`, `/echo`, `/delay`, `/status/:code` and `/flaky` stay open, so probes, Prometheus and `watch` work as before. [`/info`](https://torchio.altr.dev/endpoints.html#info) tells whether a token is needed (`auth`). The control panel asks for it the first time a server answers `401` and keeps it for that tab. Without `TOKEN` nothing is checked. It is a shared password, not a login: without HTTPS it travels in clear, so on the internet put a proxy with TLS in front. --- # Logs By default the server writes plain text: what it is doing at the start, the settings it replaced and the errors, plus the lines of [`/log`](https://torchio.altr.dev/endpoints.html#log). It writes no line for each request. `LOG_FORMAT=json` writes one JSON object a line, for a log aggregator that would otherwise have to parse the text. Every line has `timestamp` (ISO 8601, UTC), `level` (`info`, `warn` or `error`) and `message`; warnings and errors go to stderr, the rest to stdout. The lines of `/log` have the same fields. In this format every request also has a line, with `method`, `path`, `status` and `duration_ms` (the query is left out, because it can hold a token): ```json {"timestamp":"2026-10-19T12:34:56.789Z","level":"info","message":"GET /status/418 418","method":"GET","path":"/status/418","status":418,"duration_ms":0.215} ``` The effective format is `logFormat` in [`/config`](https://torchio.altr.dev/endpoints.html#config). --- # Shutdown By default SIGTERM and SIGINT end the process right away, without waiting for the requests in progress, and release what the server holds (the files of `/disk`). To see what an orchestrator or a load balancer does during a rollout, set `SHUTDOWN_GRACE_SECONDS`: the server then **drains** first. - It closes the port, so a new connection is refused, and keeps serving the requests and the connections that are already open. - It writes what is going on every second: `Draining, 3 s: 2 requests and 5 connections open`. - It ends when nothing is open (`Drained after 2.4 s`) or when the grace period is over, whichever comes first, and releases what it holds then. A second signal ends it at once. - While it drains, [`/info`](https://torchio.altr.dev/endpoints.html#info) (`shutdown`) and [`/metrics`](https://torchio.altr.dev/endpoints.html#metrics) (`torchio_draining`, `torchio_open_requests`, `torchio_open_connections`) say so, for a client that still has a connection open: it is served as before, and a connection that stays idle keeps the drain waiting. ``` docker run -p 3000:3000 -e SHUTDOWN_GRACE_SECONDS=30 altrdev/torchio ``` [`/delay`](https://torchio.altr.dev/endpoints.html#delay) and [`/connections`](https://torchio.altr.dev/endpoints.html#connections) make something to wait for: `/delay?ms=10000` is a request in progress, and `/connections?count=50&hold=60` leaves 50 connections open that the drain waits for until the grace period is over. In Kubernetes, `terminationGracePeriodSeconds` must be longer than `SHUTDOWN_GRACE_SECONDS`, or the kubelet ends the container first. --- # Docker The image is built for `linux/amd64` and `linux/arm64`. It holds the static binary and an empty `/tmp`, about 1.4 MB in all, and nothing else: no shell and no package manager, so `docker exec` runs `torchio` itself (`torchio watch`) rather than a shell. It runs as the non-root user `65532` and has a `HEALTHCHECK` on `/`, which runs `torchio healthcheck`. With `START_DELAY_SECONDS` the container is unhealthy until the port opens, so give the healthcheck a longer `--start-period`. Because of the non-root user, a directory you use as `DISK_DIR` (for example a mounted volume) must be writable by it. Each release is tagged `X.Y.Z` (the exact version), `X.Y` and `X` (the latest patch of the minor and of the major), `vX.Y.Z` and `latest`. `latest`, `X.Y` and `X` move with each release, so pin the exact version where a change must not come by itself. The [tags on Docker Hub](https://hub.docker.com/r/altrdev/torchio/tags) of the first releases only have the `v`. The binary alone, for Linux and macOS on `amd64` and `arm64`, is attached to each [release](https://github.com/altrdev/torchio/releases) with its `SHA256SUMS`. --- # Examples ## Docker Compose ```yaml services: stress: image: altrdev/torchio:latest ports: - "3000:3000" environment: WORKERS: 2 MAX_MEMORY_MB: 1024 deploy: resources: limits: cpus: "1" memory: 512M ``` ## Kubernetes The [`deploy`](https://github.com/altrdev/torchio/tree/main/deploy) folder has what you need to run torchio in a cluster, with a memory and CPU limit, a liveness probe on `/` and a readiness probe on `/ready`, and a container that runs as non-root with a read-only filesystem: ``` kubectl apply -k deploy/kubernetes ``` applies the Deployment and the Service as they are in [`deploy/kubernetes`](https://github.com/altrdev/torchio/tree/main/deploy/kubernetes). Or install the Helm chart in [`deploy/helm/torchio`](https://github.com/altrdev/torchio/tree/main/deploy/helm/torchio), where the image tag, the replicas, `WORKERS`, `UI`, the `TOKEN` (kept in a Secret), any other variable and the resources are values: ``` helm install torchio deploy/helm/torchio --set replicaCount=3 --set config.workers=2 ``` They are a starting point: copy and change them, or leave them as they are and override what you need (in a kustomization of your own, or with `-f my-values.yaml` for the chart; the `env` value takes any variable the chart has no value for). Both use the image of the version they sit next to, `altrdev/torchio:X.Y.Z`, and CI fails when it is not the one of `Cargo.toml`. The memory budget of the manifests (`MAX_MEMORY_MB`) is above the container limit, so `/memory` can trigger an OOM kill. Some things to try, after `kubectl port-forward deploy/torchio 3000:3000`: - `/stress?number=45` on several terminals keeps the container at its CPU limit, which you can watch with `kubectl top pod` or use to trigger a CPU-based autoscaler. - `/memory?mb=400` goes above the `256Mi` limit, so the container should be OOM killed and restarted. - `/ready/off?seconds=60` takes the pod out of the Service endpoints for a minute without restarting it (the readiness probe now checks `/ready`; the liveness probe still checks `/`). - `/crash?code=1` restarts the container, and calling it again each time shows the backoff growing. - `/status/503` is a quick way to check how a probe or a retry policy reacts to errors. --- # How it works `/stress` requests are spread round-robin over a pool of worker threads (one for each CPU by default), so concurrent requests can saturate multiple cores. The other endpoints run on an async runtime (tokio), so a slow request never blocks the rest. The memory of `/memory` and `/leak` comes straight from the OS and goes back as soon as it is released. --- # Load testing To keep several cores busy you need concurrent requests, at least as many as `WORKERS`. With [hey](https://github.com/rakyll/hey): ``` hey -n 100 -c 8 "http://127.0.0.1:3000/stress?number=40" ``` With [k6](https://k6.io), save this as `stress.js`: ```js import http from 'k6/http'; export const options = { vus: 8, duration: '60s' }; export default function () { http.get('http://127.0.0.1:3000/stress?number=40'); } ``` and run `k6 run stress.js`. With plain `curl` (8 parallel requests): ``` for i in $(seq 8); do curl -s "http://127.0.0.1:3000/stress?number=40" & done; wait ``` Lower or raise `number` to tune how long each request keeps a core busy. --- # Development ``` cargo run # starts the server cargo run -- watch # terminal view of a running server cargo test # every test: unit, router and integration cargo clippy --all-targets -- -D warnings cargo fmt ``` ``` src/ main.rs the program: runs the server or one of the commands lib.rs the crate, with a line on each module config.rs environment variables, validated endpoints.rs the parameters and limits of every endpoint, shared by the routes and /help app.rs what the routes share, and the router actions.rs what the endpoints that change the state do and say, shared by the routes and the scenarios server.rs listens, and ends the process scenario_parse.rs reads a scenario from the body of a request, checked against the limits routes/ one file for each endpoint: reads the parameters, asks resources, writes the answer resources/ what the server holds or does on purpose, without knowing about HTTP budget.rs the total that /memory, /disk and /fd share between requests memory.rs memory taken from the OS (/memory, /leak) disk.rs the files of /disk fd.rs the descriptors of /fd connections.rs the connections of /connections opened.rs what /fd and /connections share once they have opened what was asked counted.rs the guard that counts what is going on while it lives scenario.rs the steps of a scenario and where a running one is splitmix.rs the generator of random numbers of /flaky and /payload workers.rs the pool of worker threads for /stress and /cpu ready.rs the readiness switch shutdown.rs the drain after SIGTERM, and what is open log.rs the lines of /log delay.rs the requests of /delay still waiting flaky.rs the coin of /flaky: which requests fail http/ what serves requests and is not specific to an endpoint params.rs the integer query parameters auth.rs TOKEN trust.rs TRUST_PROXY accept.rs JSON or text for /help counters.rs requests answered, for /metrics open.rs the listener that counts connections and stops accepting, and the count of requests cli/ the commands besides the server watch.rs torchio watch common.rs the options, the token and the usage that the commands share terminal.rs the screen sequences, NO_COLOR and one frame of the view view.rs what watch draws info.rs what /info says, as watch reads it http.rs the small HTTP client of the commands healthcheck.rs torchio healthcheck logging.rs LOG_FORMAT: the text and the JSON lines, and the access log snapshot.rs the figures behind /info and /metrics system.rs RSS, the memory limit of the container, free space, names of errors units.rs units of size assets/ui.html the control panel, a single static page tests/ integration tests, by area deploy/ Kubernetes manifests and the Helm chart book/ this documentation, with mdBook (`SUMMARY.md` is the list of pages); `llms.py` writes llms.txt and `sitemap.py` writes sitemap.xml from it skills/torchio/ the skill that tells an agent how to run a test, published at /SKILL.md examples/ the reference of the endpoints, written from the catalog Dockerfile builds the binary for both platforms with cargo-zigbuild, into an image of one file ``` The tests come in three kinds, and `cargo test` runs them all: - **Unit tests** live in the module they test (`#[cfg(test)]`): the budgets, the parsing of parameters and `TRUST_PROXY`, the view of `watch`, the worker pool, the timers (with a paused clock). - **Router tests** (`tests/api.rs`) call the router without a socket: every endpoint of the catalog is routed and protected as it says, every parameter is rejected one step outside its range, CORS, methods, content negotiation, and the shape of `/info`, `/config` and `/metrics`. - **Integration tests** (`tests/*.rs`, with what they share in `tests/common`) start the real binary on a free port with small limits and call it over HTTP: `probes` (`/`, `/status`, `/ready`, `/delay`), `resources` (`/memory`, `/leak`, `/disk`, `/fd`), `load` (`/stress`, `/cpu`), `payload` (`/payload`), `scenario` (`/scenario`), `observability` (`/info`, `/metrics`, `/log`, `/echo`, `/help`, `/config`), `lifecycle` (configuration, start, signals, `/crash`), `auth` (`TOKEN`), `ui` and `cli` (`watch`, `healthcheck`). CI runs them on Linux and macOS with the format check, `clippy` and the documentation, builds the image for both platforms, runs it with a memory limit and a CPU limit, and validates `/metrics` with `promtool`. ## The documentation site The pages are in `book/src`, written in Markdown, and `book/book.toml` and `book/theme` set the title and the colours. The images are in `docs/`, where the README takes them from, and the second command below copies them next to the pages. The reference of the endpoints is not written by hand: `cargo run --example endpoints_reference > book/src/reference.md` writes it from the catalog that `/help` is built from, so it is always right. With [mdBook](https://rust-lang.github.io/mdBook/) installed: ``` cargo run --example endpoints_reference > book/src/reference.md mkdir -p book/src/images && cp docs/* book/src/images/ mdbook serve book --open python3 book/check-links.py book/book # after mdbook build book ``` The version, in a small label next to the title and on the first page, is written by the workflow from `Cargo.toml`; on your machine they say it is the development version. The `Documentation` workflow builds the site for a pull request, to catch a mistake before the merge. It publishes to GitHub Pages when a release is tagged, so the site describes the version that can be installed, and when it is run by hand (*Actions*, *Documentation*, *Run workflow*) to publish a correction of the pages in between. ## Releasing `scripts/release.sh X.Y.Z` writes the version in `Cargo.toml`, `Cargo.lock`, the image tag of `deploy/kubernetes/deployment.yaml` and the `version` and `appVersion` of the Helm chart, and dates the `Unreleased` section of the changelog. It changes nothing else: review the diff, commit it as `chore: release X.Y.Z`, merge it and push the tag `vX.Y.Z`, which publishes the image, the binaries and the documentation. CI fails when one of these places is left behind. --- # Security and license This tool is meant to consume resources, so run it only in environments you control and don't expose it to the internet. If you find a security problem, please open an issue at https://github.com/altrdev/torchio/issues. ## License MIT ---