Commit Graph

93 Commits

Author SHA1 Message Date
Zach Koopmans fd84cddab0 Internal change
PiperOrigin-RevId: 300410856
2020-03-11 14:46:17 -07:00
Michael Pratt 5ee9bbb15d Upgrade Kythe
PiperOrigin-RevId: 300409401
2020-03-11 14:39:21 -07:00
Brad Burlage ededa90d07 Internal change.
PiperOrigin-RevId: 296972565
2020-02-24 14:22:32 -08:00
gVisor bot 3a20eccf8b Internal change.
PiperOrigin-RevId: 296055452
2020-02-19 14:18:11 -08:00
gVisor bot ddc02e489c Internal change.
PiperOrigin-RevId: 296039442
2020-02-19 13:12:31 -08:00
gVisor bot 90d0851c0e Internal change.
PiperOrigin-RevId: 296016675
2020-02-19 11:36:20 -08:00
gVisor bot 8dae8a10f0 Internal change.
PiperOrigin-RevId: 295857142
2020-02-18 17:04:58 -08:00
gVisor bot cf1e50a809 Minor runtime test fixes.
* Allow scripts/common.sh to be sourced from outside the scripts/ directory
* Fix passing empty args to Bazel, which causes the tool to exit with a failure
  even if the command succeeds.

PiperOrigin-RevId: 294785456
2020-02-12 16:27:07 -08:00
Brad Burlage bc504d5202 Fix build_file in runtimes_tests.
PiperOrigin-RevId: 294286242
2020-02-10 13:10:06 -08:00
Eyal Soha 37abbbc547 Add packetdrill tests to presubmit and CI testing
PiperOrigin-RevId: 293409718
2020-02-05 11:18:02 -08:00
Brad Burlage 6cd7901d7d Add 1 Kokoro job per runtime test.
PiperOrigin-RevId: 293019326
2020-02-03 15:56:57 -08:00
gVisor bot bc3a24d627 Internal change.
PiperOrigin-RevId: 292587459
2020-01-31 13:19:42 -08:00
Brad Burlage 24cfbf4b98 Fix corpus_name to match our ingestion config[1].
PiperOrigin-RevId: 291412676
2020-01-24 11:45:44 -08:00
Brad Burlage 14d2ed1ad7 Fix kythe build.
* Pass --auth_credentials now that we're using RBE
* Fix kzips not being uploaded to the root of the GCS bucket

PiperOrigin-RevId: 291241757
2020-01-23 14:14:24 -08:00
Brad Burlage 98e83c444f Try running kythe build on RBE.
Also add our RBE project/instance to the --config=remote defaults.

PiperOrigin-RevId: 291201426
2020-01-23 11:01:38 -08:00
Adin Scannell cb3906ae00 Add tools for generating images.
This formalizes the adhoc scripts previously in kokoro. The image targets can
be used by e.g. benchmarks in order to automated image prepation.

PiperOrigin-RevId: 290982744
2020-01-22 10:39:45 -08:00
Nicolas Lacasse a944fcd946 Install Bazel 2.0.0 on kokoro images.
PiperOrigin-RevId: 290850738
2020-01-21 18:23:59 -08:00
Fabricio Voznika 290908fa8a Configure issue reviver to run with Kokoro
PiperOrigin-RevId: 288921032
2020-01-09 10:17:54 -08:00
Zach Koopmans e548ce1805 Add python3-pip as dependency for Kokoro VM images.
bm-tools requires python3 and pip3 in order to run
tests. Add pip3 so that dependencies correctly install
for Kokoro runs with bazel.

PiperOrigin-RevId: 286923840
2019-12-23 11:49:29 -08:00
Kevin Krakauer be2754a4b9 Add iptables testing framework.
It would be preferrable to test iptables via syscall tests, but there are some
problems with that approach:

* We're limited to loopback-only, as syscall tests involve only a single
  container. Other link interfaces (e.g. fdbased) should be tested.
* We'd have to shell out to call iptables anyways, as the iptables syscall
  interface itself is too large and complex to work with alone.
* Running the Linux/native version of the syscall test will require root, which
  is a pain to configure, is inherently unsafe, and could leave host iptables
  misconfigured.

Using the go_test target allows there to be no new test runner.

PiperOrigin-RevId: 285274275
2019-12-12 14:42:11 -08:00
Brad Burlage e690651c67 Run kythe build with -std=c++17
We seem to be getting some compiler errors when using the Kythe
extractors.bazelrc.

I'm not able to reproduce this on my local machine, so I'm hoping copying the
cxxopt from gvisor.dev/pr/1350 will fix the build.

PiperOrigin-RevId: 285054258
2019-12-11 14:48:11 -08:00
Brad Burlage a0aa784ecf Remove pyenv calls but log the python 3 version in use.
Apparently our Kokoro VM images don't have pyenv -- I previously tested this on
the Kokoro QA shared pool.

PiperOrigin-RevId: 284855160
2019-12-10 14:55:35 -08:00
Brad Burlage 87337e92e3 Add Kokoro configs for publishing Kythe xrefs.
PiperOrigin-RevId: 284835614
2019-12-10 13:25:27 -08:00
Fabricio Voznika 3c2e2f7d12 Update Kokoro image to install Golang 1.13
PiperOrigin-RevId: 284308422
2019-12-06 20:13:09 -08:00
Adin Scannell b6a00aa375 Use a GitHub credential for tagging a release.
PiperOrigin-RevId: 281617882
2019-11-20 15:39:34 -08:00
Andrei Vagin 14f4461f93 kokoro: update images to install zip
PiperOrigin-RevId: 279406266
2019-11-08 15:45:16 -08:00
Andrei Vagin 50d6236111 Update kokoro images to install junitparser
junitparser will be used to merge junit xml files.

PiperOrigin-RevId: 279387305
2019-11-08 14:08:54 -08:00
Adin Scannell e904823833 Fix repository build scripts.
This fixes a number of issues with the repository build process:

 * Fix the overall structure of the repository.
 * Fix the debian package description.
 * Fix the broken version number for packages.
 * Update the digest algorithm used for signing the release.

I've validated that installation works from a separate staging bucket.

Updates #852

PiperOrigin-RevId: 278716914
2019-11-05 15:16:04 -08:00
Andrei Vagin 493334f8b5 kokoro: run KVM syscall tests
We don't know how stable they are, so let's start with warning.

PiperOrigin-RevId: 278484186
2019-11-04 16:00:34 -08:00
Brad Burlage df125c9869 Add Kokoro config for new runtime tests
PiperOrigin-RevId: 277607217
2019-10-30 16:16:15 -07:00
Andrei Vagin 8720bd643e netstack/tcp: software segmentation offload
Right now, we send each tcp packet separately, we call one system
call per-packet. This patch allows to generate multiple tcp packets
and send them by sendmmsg.

The arguable part of this CL is a way how to handle multiple headers.
This CL adds the next field to the Prepandable buffer.

Nginx test results:

Server Software:        nginx/1.15.9
Server Hostname:        10.138.0.2
Server Port:            8080

Document Path:          /10m.txt
Document Length:        10485760 bytes

w/o gso:
Concurrency Level:      5
Time taken for tests:   5.491 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      1048600200 bytes
HTML transferred:       1048576000 bytes
Requests per second:    18.21 [#/sec] (mean)
Time per request:       274.525 [ms] (mean)
Time per request:       54.905 [ms] (mean, across all concurrent requests)
Transfer rate:          186508.03 [Kbytes/sec] received

sw-gso:

Concurrency Level:      5
Time taken for tests:   3.852 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      1048600200 bytes
HTML transferred:       1048576000 bytes
Requests per second:    25.96 [#/sec] (mean)
Time per request:       192.576 [ms] (mean)
Time per request:       38.515 [ms] (mean, across all concurrent requests)
Transfer rate:          265874.92 [Kbytes/sec] received

w/o gso:
$ ./tcp_benchmark --client --duration 15  --ideal
[SUM]  0.0-15.1 sec  2.20 GBytes  1.25 Gbits/sec

software gso:
$ tcp_benchmark --client --duration 15  --ideal --gso $((1<<16)) --swgso
[SUM]  0.0-15.1 sec  3.99 GBytes  2.26 Gbits/sec

PiperOrigin-RevId: 276112677
2019-10-22 11:55:56 -07:00
Fabricio Voznika 739f53fc17 Add runsc logs to kokoro artifacts
PiperOrigin-RevId: 272286122
2019-10-01 13:51:10 -07:00
Adin Scannell 0c4d080631 Ensure runsc is uploaded.
One would reasonably assume that a field named "regex" would expect
a regular expression. However, in this case, one would be wrong.

The "regex" field actually requires "FileSet" [1] syntax.

?\_(?)_/?

[1] http://ant.apache.org/manual/Types/fileset.html

PiperOrigin-RevId: 271917356
2019-09-29 23:49:34 -07:00
Andrei Vagin 7582385f05 kokoro: don't pass KOKORO_REPO_KEY in presubmit jobs
We don't want to upload packages from the presubmit jobs.

This will fix the error:
[11:01:34][ERROR] Cannot inject environment variables into
                  the build without allowed_env_vars regexes.

PiperOrigin-RevId: 271622996
2019-09-27 12:23:51 -07:00
Brad Burlage e6cf60bc0e Fix runsc checksum artifact name.
scripts/build.sh uses sha512sum, not sha256sum.

PiperOrigin-RevId: 270713604
2019-09-23 10:58:18 -07:00
Adin Scannell 191297ecbe Fix artifact pattern for repositories.
PiperOrigin-RevId: 269468986
2019-09-16 18:41:13 -07:00
Adin Scannell 2bbf73d9ed Remove stale configurations.
PiperOrigin-RevId: 268947847
2019-09-13 11:39:55 -07:00
Adin Scannell 1e6bdd5855 Update key environment variables.
PiperOrigin-RevId: 268604220
2019-09-11 21:04:42 -07:00
Adin Scannell c06ef5131f Fix authorization for continuous integration.
The credentials must be explicitly refreshed for pushing to
the repository on the Go branch.

PiperOrigin-RevId: 268589817
2019-09-11 18:49:53 -07:00
Nicolas Lacasse f52dd451ed Bump bazel, rules_go, gazelle, and go toolchain to latest versions.
PiperOrigin-RevId: 268486127
2019-09-11 10:19:50 -07:00
Adin Scannell 849c57314f Fix minor Kokoro issues.
A recent Kokoro change pointed to go_tests.cfg (in line with the
other configurations), which unfortunately broke the presubmits.

This change also enabled the KVM tests, which were still using a
remote execution strategy.

This fixes both of these issues and allows presubmits to pass.

One additional test was caught with this case, which seems to
have been broken. It's unclear why this was not being caught.

PiperOrigin-RevId: 268166291
2019-09-10 00:38:52 -07:00
Adin Scannell 67a2ab1438 Impose order on test scripts.
The simple test script has gotten out of control. Shard this script into
different pieces and attempt to impose order on overall test structure. This
change helps lay some of the foundations for future improvements.

 * The runsc/test directories are moved into just test/.
 * The runsc/test/testutil package is split into logical pieces.
 * The scripts/ directory contains new top-level targets.
 * Each test is now responsible for building targets it requires.
 * The install functionality is moved into `runsc` itself for simplicity.
 * The existing kokoro run_tests.sh file now just calls all (can be split).

After this change is merged,  I will create multiple distinct workflows for
Kokoro, one for each of the scripts currently targeted by `run_tests.sh` today,
which should dramatically reduce the time-to-run for the Kokoro tests, and
provides a better foundation for further improvements to the infrastructure.

PiperOrigin-RevId: 267081397
2019-09-03 22:02:43 -07:00
Adin Scannell 3ec0b64d8c Update image to install docker and fix permissions
PiperOrigin-RevId: 266467361
2019-08-30 15:27:47 -07:00
Adin Scannell 599136b25e Internal change.
PiperOrigin-RevId: 266073190
2019-08-29 13:59:33 -07:00
Adin Scannell f048c8be1e Fix permissions on .ssh files
PiperOrigin-RevId: 266066839
2019-08-28 22:36:16 -07:00
Adin Scannell 06ae36185e Fix kokoro image build scripts
The /tmpfs directory needs to be writable, and kokoro needs passwordless sudo
access to execute the test scripts.

PiperOrigin-RevId: 266063723
2019-08-28 22:07:12 -07:00
Adin Scannell 784f48a78d kokoro: Add scripts to rebuild images.
These scripts generated the following images:

gvisor-kokoro-testing/image-a53bac71541a209e (ubuntu 18.04)
gvisor-kokoro-testing/image-f5b20c5fbd23f448 (ubuntu 16.04)

Any modifications to these scripts should generate new images.

PiperOrigin-RevId: 265843929
2019-08-27 23:29:02 -07:00
Adin Scannell 7436ea247b Fix Kokoro revision and 'go get usage'
As a convenience for debugging, also factor the scripts such that
can be run without Kokoro. In the future, this may be used to add
additional presubmit hooks that run without Kokoro.

PiperOrigin-RevId: 251474868
2019-06-04 11:07:27 -07:00
Adin Scannell 216da0b733 Add tooling for Go-compatible branch.
The WORKSPACE go_repositories can be generated from a standard go.mod file. Add
the necessary gazelle hooks to do so, and include a test that sanity checks
there are no changes. This go.mod file will be used in a subsequent commit to
generate a go gettable branch of the repository.

This commit also adds a tools/go_branch.sh script, which given an existing go
branch in the repository, will add an additional synthetic change to the branch
bringing it up-to-date with HEAD.

As a final step, a cloudbuild script is included, which can be used to automate
the process for every change pushed to the repository. This may be used after
an initial go branch is pushed, but this is manual process.

PiperOrigin-RevId: 251095016
2019-06-01 23:10:43 -07:00
Adin Scannell 132bf68de4 Switch to new dedicated RBE project.
PiperOrigin-RevId: 250970783
2019-05-31 15:43:30 -07:00