2019-03-30 02:40:11 +00:00
|
|
|
+++
|
|
|
|
title = "Debugging"
|
|
|
|
weight = 120
|
|
|
|
+++
|
|
|
|
|
|
|
|
To enable debug and system call logging, add the `runtimeArgs` below to your
|
|
|
|
[Docker](../docker/) configuration (`/etc/docker/daemon.json`):
|
|
|
|
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"runtimes": {
|
|
|
|
"runsc": {
|
|
|
|
"path": "/usr/local/bin/runsc",
|
|
|
|
"runtimeArgs": [
|
|
|
|
"--debug-log=/tmp/runsc/",
|
|
|
|
"--debug",
|
|
|
|
"--strace"
|
|
|
|
]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
2019-04-03 19:18:46 +00:00
|
|
|
> Note: the last `/` in `--debug-log` is needed to interpret it as a directory.
|
|
|
|
> Then each `runsc` command executed will create a separate log file.
|
|
|
|
> Otherwise, log messages from all commands will be appended to the same file.
|
|
|
|
|
2019-03-30 02:40:11 +00:00
|
|
|
You may also want to pass `--log-packets` to troubleshoot network problems. Then
|
|
|
|
restart the Docker daemon:
|
|
|
|
|
|
|
|
```bash
|
|
|
|
sudo systemctl restart docker
|
|
|
|
```
|
|
|
|
|
|
|
|
Run your container again, and inspect the files under `/tmp/runsc`. The log file
|
2019-04-03 19:18:46 +00:00
|
|
|
ending with `.boot` will contain the strace logs from your application, which can
|
|
|
|
be useful for identifying missing or broken system calls in gVisor. If you are
|
|
|
|
having problems starting the container, the log file ending with `.create` may
|
|
|
|
have the reason for the failure.
|
2019-04-04 23:51:50 +00:00
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
## Stack traces
|
2019-04-04 23:51:50 +00:00
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
The command `runsc debug --stacks` collects stack traces while the sandbox is
|
|
|
|
running which can be useful to troubleshoot issues or just to learn more about
|
|
|
|
gVisor. It connects to the sandbox process, collects a stack dump, and writes
|
|
|
|
it to the console. For example:
|
2019-04-04 23:51:50 +00:00
|
|
|
|
|
|
|
```bash
|
2019-05-28 10:53:18 +00:00
|
|
|
docker run --runtime=runsc --rm -d alpine sh -c "while true; do echo running; sleep 1; done"
|
2019-04-04 23:51:50 +00:00
|
|
|
63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
|
|
|
|
|
|
|
|
sudo runsc --root /var/run/docker/runtime-runsc/moby debug --stacks 63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
|
|
|
|
```
|
|
|
|
|
2019-04-03 19:18:46 +00:00
|
|
|
> Note: `--root` variable is provided by docker and is normally set to
|
2019-04-04 23:51:50 +00:00
|
|
|
> `/var/run/docker/runtime-[runtime-name]/moby`. If in doubt, `--root` is logged to
|
|
|
|
> `runsc` logs.
|
|
|
|
|
2019-07-30 01:35:27 +00:00
|
|
|
## Debugger
|
|
|
|
|
|
|
|
You can debug gVisor like any other Golang program. If you're running with Docker,
|
|
|
|
you'll need to find the sandbox PID and attach the debugger as root. Other than
|
|
|
|
that, it's business as usual.
|
|
|
|
|
|
|
|
```bash
|
|
|
|
# Start the container you want to debug.
|
|
|
|
docker run --runtime=runsc --rm --name=test -d alpine sleep 1000
|
|
|
|
|
|
|
|
# Find the sandbox PID.
|
|
|
|
docker inspect test | grep Pid | head -n 1
|
|
|
|
|
|
|
|
# Attach your favorite debugger.
|
|
|
|
sudo dlv attach <PID>
|
|
|
|
|
|
|
|
# Set a breakpoint and resume.
|
|
|
|
break mm.MemoryManager.MMap
|
|
|
|
continue
|
|
|
|
```
|
|
|
|
|
|
|
|
> Note: if the debugger cannot find symbols, rebuild runsc in debug mode:
|
|
|
|
> `bazel build -c dbg //runsc:runsc`
|
|
|
|
|
2019-04-04 23:51:50 +00:00
|
|
|
## Profiling
|
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
`runsc` integrates with Go profiling tools and gives you easy commands to profile
|
2019-04-05 05:02:37 +00:00
|
|
|
CPU and heap usage. First you need to enable `--profile` in the command line options
|
2019-04-04 23:51:50 +00:00
|
|
|
before starting the container:
|
|
|
|
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"runtimes": {
|
|
|
|
"runsc-prof": {
|
|
|
|
"path": "/usr/local/bin/runsc",
|
|
|
|
"runtimeArgs": [
|
|
|
|
"--profile"
|
|
|
|
]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
> Note: Enabling profiling loosens the seccomp protection added to the sandbox,
|
2019-04-05 05:02:37 +00:00
|
|
|
> and should not be run in production under normal circumstances.
|
2019-04-04 23:51:50 +00:00
|
|
|
|
|
|
|
Then restart docker to refresh the runtime options. While the container is running,
|
|
|
|
execute `runsc debug` to collect profile information and save to a file. Here are
|
|
|
|
the options available:
|
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
* **--profile-heap:** Generates heap profile to the speficied file.
|
|
|
|
* **--profile-cpu:** Enables CPU profiler, waits for `--profile-delay` seconds
|
2019-04-04 23:51:50 +00:00
|
|
|
and generates CPU profile to the speficied file.
|
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
For example:
|
2019-04-04 23:51:50 +00:00
|
|
|
|
|
|
|
```bash
|
2019-05-28 10:53:18 +00:00
|
|
|
docker run --runtime=runsc-prof --rm -d alpine sh -c "while true; do echo running; sleep 1; done"
|
2019-04-04 23:51:50 +00:00
|
|
|
63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
|
|
|
|
|
|
|
|
sudo runsc --root /var/run/docker/runtime-runsc-prof/moby debug --profile-heap=/tmp/heap.prof 63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
|
|
|
|
sudo runsc --root /var/run/docker/runtime-runsc-prof/moby debug --profile-cpu=/tmp/cpu.prof --profile-delay=30 63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
|
|
|
|
```
|
|
|
|
|
2019-04-05 05:31:51 +00:00
|
|
|
The resulting files can be opened using `go tool pprof` or [pprof]. The examples
|
|
|
|
below create image file (`.svg`) with the heap profile and writes the top
|
|
|
|
functions using CPU to the console:
|
2019-04-04 23:51:50 +00:00
|
|
|
|
|
|
|
```bash
|
|
|
|
go tool pprof -svg /usr/local/bin/runsc /tmp/heap.prof
|
|
|
|
go tool pprof -top /usr/local/bin/runsc /tmp/cpu.prof
|
|
|
|
```
|
2019-04-05 05:31:51 +00:00
|
|
|
|
|
|
|
[pprof]: https://github.com/google/pprof/blob/master/doc/README.md
|