gvisor/test/runtimes

gVisor Runtime Tests

App Engine uses gvisor to sandbox application containers. The runtime tests aim to test runsc compatibility with these standard runtimes. The test itself runs the language-defined tests inside the sandboxed standard runtime container.

Note: Ruby runtime is currently in beta mode and so we do not run tests for it yet.

Testing Locally

To run runtime tests individually from a given runtime, use the following table.

Language	Version	Download Image	Run Test(s)
Go	1.12	make -C images load-runtimes_go1.12	If the test name ends with .go, it is an on-disk test: docker run --runtime=runsc -it gvisor.dev/images/runtimes/go1.12 ( cd /usr/local/go/test ; go run run.go -v -- <TEST_NAME>... ) Otherwise it is a tool test: docker run --runtime=runsc -it gvisor.dev/images/runtimes/go1.12 go tool dist test -v -no-rebuild ^TEST1$|^TEST2$...
Java	11	make -C images load-runtimes_java11	docker run --runtime=runsc -it gvisor.dev/images/runtimes/java11 jtreg -agentvm -dir:/root/test/jdk -noreport -timeoutFactor:20 -verbose:summary <TEST_NAME>...
NodeJS	12.4.0	make -C images load-runtimes_nodejs12.4.0	docker run --runtime=runsc -it gvisor.dev/images/runtimes/nodejs12.4.0 python tools/test.py --timeout=180 <TEST_NAME>...
Php	7.3.6	make -C images load-runtimes_php7.3.6	docker run --runtime=runsc -it gvisor.dev/images/runtimes/php7.3.6 make test "TESTS=<TEST_NAME>..."
Python	3.7.3	make -C images load-runtimes_python3.7.3	docker run --runtime=runsc -it gvisor.dev/images/runtimes/python3.7.3 ./python -m test <TEST_NAME>...

To run an entire runtime test locally, use the following table.

Note: java runtime test take 1+ hours with 16 cores.

Language	Version	Running the test suite
Go	1.12	make go1.12-runtime-tests{_vfs2}
Java	11	make java11-runtime-tests{_vfs2}
NodeJS	12.4.0	make nodejs12.4.0-runtime-tests{_vfs2}
Php	7.3.6	make php7.3.6-runtime-tests{_vfs2}
Python	3.7.3	make python3.7.3-runtime-tests{_vfs2}

Clean Up

Sometimes when runtime tests fail or when the testing container itself crashes unexpectedly, the containers are not removed or sometimes do not even exit. This can cause some docker commands like docker system prune to hang forever.

Here are some helpful commands (should be executed in order):

docker ps -a  # Lists all docker processes; useful when investigating hanging containers.
docker kill $(docker ps -a -q)  # Kills all running containers.
docker rm $(docker ps -a -q)  # Removes all exited containers.
docker system prune  # Remove unused data.

Testing Infrastructure

There are 3 components to this tests infrastructure:

• runner - This is the test entrypoint. This is the binary is invoked by bazel test. The runner spawns the target runtime container using runsc and then copies over the proctor binary into the container.
• proctor - This binary acts as our agent inside the container which communicates with the runner and actually executes tests.
• exclude - Holds a CSV file for each language runtime containing the full path of tests that should be excluded from running along with a reason for exclusion.