Commit Graph

76 Commits

Author SHA1 Message Date
Adin Scannell 30794512d3 Add basic microbenchmarks.
PiperOrigin-RevId: 296104390
2020-02-19 18:21:54 -08:00
gVisor bot 660cfdff3f Handle situations where go-marshal generates an empty test file.
This can happen due to conditional compilation, where a subset of the
source files contain no marshallable types. go-marshal is still
required to write an output file in these cases, since bazel defines
the output package before calling go-marshal.

PiperOrigin-RevId: 296074321
2020-02-19 15:42:19 -08:00
gVisor bot 737a3d072e go-marshal: Stop complaining about files with no +marshal types.
Since we tag entire packages as marshallable, due to conditional
compiling for different architectures we can end up with sets of
source files that don't contain any marshallable types. It's safe to
silently ignore this scenario.

PiperOrigin-RevId: 295831871
2020-02-18 15:09:17 -08:00
gVisor bot 5cc0bbbafb Ensure Marshallable.SizeBytes() always works on a typed nil pointer.
This lets go-marshal replace various calls to binary.Size() throughout
the sentry without requiring concrete objects.

PiperOrigin-RevId: 295299965
2020-02-15 00:01:02 -08:00
gVisor bot 3d32ad1367 Generate implementation of io.WriterTo via go-marshal.
PiperOrigin-RevId: 295269654
2020-02-14 18:32:49 -08:00
gVisor bot 48d9aa7ab3 Add a minimal binary target for escape analysis on go-marshal.
Note that this is not an automated test.

PiperOrigin-RevId: 295238672
2020-02-14 15:21:03 -08:00
gVisor bot b2e86906ea Fix various issues related to enabling go-marshal.
- Add missing build tags to files in the abi package.

- Add the marshal package as a sentry dependency, allowed by deps_test.

- Fix an issue with our top-level go_library BUILD rule, which
  incorrectly shadows the variable containing the input set of source
  files. This caused the expansion for the go_marshal clause to
  silently omit input files.

- Fix formatting when copying build tags to gomarshal-generated files.

- Fix a bug with import statement collision detection in go-marshal.

PiperOrigin-RevId: 295112284
2020-02-14 03:27:34 -08:00
gVisor bot 336f758d59 Ensure the marshalled object doesn't escape.
Add new Marshallable interface methods CopyIn/CopyOut, which can be directly
called on the marshalled object, avoiding an interface indirection. Such
indirections are problematic because they always cause the marshalled object to
escape.

PiperOrigin-RevId: 295028010
2020-02-13 16:32:23 -08:00
gVisor bot 3ad6d30563 Call py_requirement with named argument for optional kwarg.
PiperOrigin-RevId: 294930818
2020-02-13 09:21:39 -08:00
gVisor bot 9be46e55c2 Stateify: register types with full package names
This is to avoid conflicts with types that share the same
[short] package and type names, e.g. proc.smapsData exist
in pkg/sentry/fs/proc and pkg/sentry/fsimpl/proc.

Updates #1663

PiperOrigin-RevId: 294485146
2020-02-11 11:41:55 -08:00
Adin Scannell dc5a8e52d7 Rename build to builddefs and minor build clean-up.
The name 'bazel' also doesn't work because bazel will treat it specially.

Fixes #1807

PiperOrigin-RevId: 294321221
2020-02-10 15:45:17 -08:00
Brad Burlage 20840bfec0 Move x86 state definition to its own file.
PiperOrigin-RevId: 294271541
2020-02-10 12:00:46 -08:00
Adin Scannell 0efa8168c7 Update visibility.
PiperOrigin-RevId: 294265019
2020-02-10 11:30:21 -08:00
Zach Koopmans 16561e461e Add logic to run from baked images.
Change adds the following:
- logic to run from "baked images". See [GVISOR_DIR]/tools/images
- installers which install modified files from a workspace. This
allows users to run benchmarks while modifying runsc.
- removes the --preemptible tag from built GCE instances. Preemptible
instances are much more likely to be preempted on startup, which
manifests for the user as a failed benchmark. I don't currently have
a way to detect if a VM has been preempted that will work for this
change.
https://cloud.google.com/compute/docs/instances/preemptible#preemption_process
https://cloud.google.com/compute/docs/instances/preemptible#preemption_selection

PiperOrigin-RevId: 293697949
2020-02-06 16:02:48 -08:00
Adin Scannell 1b6a12a768 Add notes to relevant tests.
These were out-of-band notes that can help provide additional context
and simplify automated imports.

PiperOrigin-RevId: 293525915
2020-02-05 22:46:35 -08:00
Adin Scannell 95ce8bb4c7 Automatically propagate tags for stateify and marshal.
Note that files will need to be appropriately segmented in order for the
mechanism to work, in suffixes implying special tags. This only needs to happen
for cases where marshal or state structures are defined, which should be rare
and mostly architecture specific.

PiperOrigin-RevId: 293231579
2020-02-04 14:37:39 -08:00
Adin Scannell 14959250fe Simplify testing link rules.
PiperOrigin-RevId: 292458933
2020-01-30 17:49:17 -08:00
Adin Scannell 0e2f1b7abd Update package locations.
Because the abi will depend on the core types for marshalling (usermem,
context, safemem, safecopy), these need to be flattened from the sentry
directory. These packages contain no sentry-specific details.

PiperOrigin-RevId: 291811289
2020-01-27 15:31:32 -08:00
Adin Scannell 90ec596166 Fix licenses.
The preferred Copyright holder is "The gVisor Authors".

PiperOrigin-RevId: 291786657
2020-01-27 13:23:57 -08:00
Adin Scannell d29e59af9f Standardize on tools directory.
PiperOrigin-RevId: 291745021
2020-01-27 12:21:00 -08:00
Adin Scannell cb3906ae00 Add tools for generating images.
This formalizes the adhoc scripts previously in kokoro. The image targets can
be used by e.g. benchmarks in order to automated image prepation.

PiperOrigin-RevId: 290982744
2020-01-22 10:39:45 -08:00
Fabricio Voznika 1c24201467 Github bug reviver
For everyone's joy, this is a tool that reopens issues that
have been closed, but are still referenced by TODOs in the
code. The idea is to run it in Kokoro nightly. Kokoro changes
are coming up next.

PiperOrigin-RevId: 288789560
2020-01-08 16:06:40 -08:00
gVisor bot d1528df715 Merge pull request #1026 from lubinszARM:pr_savable
PiperOrigin-RevId: 286667216
2019-12-20 18:07:09 -08:00
Adin Scannell d92dc065fd Fix typo in go_branch.sh script.
With the ticks, the command `master` is actually be run and the output included
(which is nothing). This is confusing, as we actually mean to say "master" in
the description of the Go branch.

PiperOrigin-RevId: 282426081
2019-11-25 13:53:27 -08:00
Adin Scannell c0f89eba6e Import and structure cleanup.
PiperOrigin-RevId: 281795269
2019-11-21 11:41:30 -08:00
Adin Scannell 9601943685 release: fix tag script
The tag script, when not run interactively, will fail without a provided commit
message (since it now uses annotated tags). For now, use a trivial message. In
the future, this could be extended to provide automated release notes.

PiperOrigin-RevId: 281112651
2019-11-18 11:22:38 -08:00
Adin Scannell e904823833 Fix repository build scripts.
This fixes a number of issues with the repository build process:

 * Fix the overall structure of the repository.
 * Fix the debian package description.
 * Fix the broken version number for packages.
 * Update the digest algorithm used for signing the release.

I've validated that installation works from a separate staging bucket.

Updates #852

PiperOrigin-RevId: 278716914
2019-11-05 15:16:04 -08:00
Adin Scannell fe2e0764ac Add LICENSE and AUTHORS to the go branch.
Also, construct the README directly so that edits can be made.

PiperOrigin-RevId: 277782095
2019-10-31 12:53:27 -07:00
Bin Lu e9d43f9022 Add tools/go_stateify to support Arm64
2 jobs were finished in this patch:
  1, add a new attribute to specify the target platform.
  2, check the source files that whether we can build this file for target platform by checking file name and build tags.
Signed-off-by: Bin Lu <bin.lu@arm.com>
2019-10-28 09:44:19 +00:00
Kevin Krakauer 2a82d5ad68 Reorder BUILD license and load functions in gvisor.
PiperOrigin-RevId: 275139066
2019-10-16 16:40:30 -07:00
Adin Scannell 461123ea35 Move the component into the repository structure.
The RELEASE file must be at the top-level for the signed
repository to work correctly.

PiperOrigin-RevId: 269897109
2019-09-18 14:57:12 -07:00
Michael Pratt df5d377521 Remove go_test from go_stateify and go_marshal
They are no-ops, so the standard rule works fine.

PiperOrigin-RevId: 268776264
2019-09-12 15:10:17 -07:00
gVisor bot 78cfbbda4b Merge pull request #843 from nlacasse:version
PiperOrigin-RevId: 268772451
2019-09-12 14:55:24 -07:00
Adin Scannell 574eda8880 Update repository directory structure.
Currently it will not work with apt out of the box, as we
require the dists/ prefix, along with a distribution name.

This tweaks the overall structure to allow for the same URL
prefix to be used for all repositories, and enables multiple
architectures.

Fixes #852

PiperOrigin-RevId: 268756104
2019-09-12 13:44:18 -07:00
Nicolas Lacasse c1a8275b47 Fix `runsc --version` and add a test.
We need to include the `--stamp` flag in `tools/workspace_status.sh` for
the version to be picked up by the linker. Not sure why.

Also changes the VERSION string to STABLE_VERSION, which will cause the
program to be re-linked if the string changes.

Fixes #830
2019-09-10 15:41:47 -07:00
Adin Scannell 849c57314f Fix minor Kokoro issues.
A recent Kokoro change pointed to go_tests.cfg (in line with the
other configurations), which unfortunately broke the presubmits.

This change also enabled the KVM tests, which were still using a
remote execution strategy.

This fixes both of these issues and allows presubmits to pass.

One additional test was caught with this case, which seems to
have been broken. It's unclear why this was not being caught.

PiperOrigin-RevId: 268166291
2019-09-10 00:38:52 -07:00
Rahat Mahmood 3733b9b893 go_marshal: Implement automatic generation of ABI marshalling code.
This CL implements go_marshal, a code generation utility for
automatically serializing and deserializing ABI structs.

The go_marshal tool automatically generates implementations of the new
marshal interface. Unlike binary.Marshal/Unmarshal, the generated
interface implementations use no runtime reflection, and translates to
a single memcpy for most structs. See go_marshal/README.md for
details.

PiperOrigin-RevId: 268065475
2019-09-09 13:36:39 -07:00
Adin Scannell 1a0a940587 Fix repository build scripts
This has the following fixes:

* Packages are passed to the tools/make_repository.sh command.
* All matching tags are built, for commits with multiple.
* The binary path is generated by the build command.
* Output from signing the repository is supressed.
* Allow a release author.

Change-Id: I2d08954ba76e35612f352be99d5bb99080f80892
2019-09-06 14:28:12 -07:00
Adin Scannell 67a2ab1438 Impose order on test scripts.
The simple test script has gotten out of control. Shard this script into
different pieces and attempt to impose order on overall test structure. This
change helps lay some of the foundations for future improvements.

 * The runsc/test directories are moved into just test/.
 * The runsc/test/testutil package is split into logical pieces.
 * The scripts/ directory contains new top-level targets.
 * Each test is now responsible for building targets it requires.
 * The install functionality is moved into `runsc` itself for simplicity.
 * The existing kokoro run_tests.sh file now just calls all (can be split).

After this change is merged,  I will create multiple distinct workflows for
Kokoro, one for each of the scripts currently targeted by `run_tests.sh` today,
which should dramatically reduce the time-to-run for the Kokoro tests, and
provides a better foundation for further improvements to the infrastructure.

PiperOrigin-RevId: 267081397
2019-09-03 22:02:43 -07:00
Adin Scannell 784f48a78d kokoro: Add scripts to rebuild images.
These scripts generated the following images:

gvisor-kokoro-testing/image-a53bac71541a209e (ubuntu 18.04)
gvisor-kokoro-testing/image-f5b20c5fbd23f448 (ubuntu 16.04)

Any modifications to these scripts should generate new images.

PiperOrigin-RevId: 265843929
2019-08-27 23:29:02 -07:00
Adin Scannell b4cdaef4a1 Don't lose test output.
PiperOrigin-RevId: 265606322
2019-08-26 21:36:34 -07:00
Nicolas Lacasse fa3d0e6f63 Bump Bazel to v0.28.0
The new version has a change in behavior when using a custom platform:
* Old behavior: rules that don't require a toolchain used host_platform, no
  matter what execution platforms are specified.
* New behavior: rules that don't require a toolchain use standard platform
  resolution that starts with execution platforms.

As part of this change, we cannot use the "extra_exectution_platforms" flag
provided by the default bazelrc. I got rid of the default bazelrc file, and
made our custom .bazelrc as minimal as possible.

PiperOrigin-RevId: 263176802
2019-08-13 11:21:55 -07:00
Fabricio Voznika 2762692621 Add debug symbols to published runsc binary
This allows published binary to be debugged if needed.

PiperOrigin-RevId: 260228367
2019-07-26 15:47:34 -07:00
gVisor bot 8e2ea9c1e0 Merge pull request #504 from matthyx:master
PiperOrigin-RevId: 258654826
2019-07-17 15:32:59 -07:00
Matthias Bertschy 239d7c6fdf go_generics: treat the Sel part of an ast.SelectorExpr 2019-07-12 08:09:48 +02:00
Andrei Vagin a018b229b5 kokoro: use bazel 2.27.1
The latest version 2.28.0 doesn't work:
./runsc/linux_amd64_pure_stripped/runsc: operation not permitted, want 0

PiperOrigin-RevId: 257663312
2019-07-11 12:53:19 -07:00
Adin Scannell dea3cb92f2 build: add nogo for static validation
PiperOrigin-RevId: 257297820
2019-07-09 16:44:06 -07:00
Michael Pratt 5b41ba5d0e Fix various spelling issues in the documentation
Addresses obvious typos, in the documentation only.

COPYBARA_INTEGRATE_REVIEW=https://github.com/google/gvisor/pull/443 from Pixep:fix/documentation-spelling 4d0688164eafaf0b3010e5f4824b35d1e7176d65
PiperOrigin-RevId: 255477779
2019-06-27 14:25:50 -07:00
Andrei Vagin 03ae91c662 gvisor: lockless read access for task credentials
Credentials are immutable and even before these changes we could read them
without locks, but we needed to take a task lock to get a credential object
from a task object.

It is possible to avoid this lock, if we will guarantee that a credential
object will not be changed after setting it on a task.

PiperOrigin-RevId: 254989492
2019-06-25 09:52:49 -07:00
Andrei Vagin 3d1e44a677 gvisor/kokoro: don't modify tests names in the BUILD file
PiperOrigin-RevId: 253746380
2019-06-18 01:41:29 -07:00