Add parser and formatting for golang benchmarks for docker benchmarks.
Change adds a library for printing and parsing Test parameters and metrics.
Benchmarks use the library to print parameters in the Benchmark title
(e.g. the name field in b.Run()), and to report CustomMetrics. Parser
uses the library to parse printed data from benchmark output and
put it into BigQuery structs.
PiperOrigin-RevId: 336365628
This change also adds support to go_stateify for detecting an appropriate
receiver name, avoiding a large number of false positives.
PiperOrigin-RevId: 335994587
Previously, the go-marshal-generated tests did not respect build
tags. This can cause the test to unbuildable under some build
configurations, as the original types the tests refer to may not be
defined.
This CL copies the build tags from the input files to the test,
similar to the generated library; however test packages have an
additional constraint. A test package cannot be totally empty
(i.e. have no test/example/benchmark defined), otherwise the go
compiler returns an error. To ensure the generated test package always
contains a testable entity under all build configurations, we now emit
an extra test file with no build tags that contains a single no-op
example.
PiperOrigin-RevId: 334496821
When nogo checks are violated, they will automatically posted
as annotations on the specific GitHub commit. This allows us
to ensure analysis & style rules and have them called out.
PiperOrigin-RevId: 334447285
CopyContext is a better name for the interface because from
go-marshal's perspective, the interface has nothing to do with a
task. A kernel.Task happens to implement the interface, but so can
other things like MemoryManager and IO sequences.
PiperOrigin-RevId: 331959678
This change makes the following fixes:
- When creating a test repo.key, create a secret keyring as other workflows
also use secret keyrings only.
- We should not be using both --keyring and --secret-keyring options. Just use
--secret-keyring.
- Pass homedir to all gpg commands. dpkg-sig takes an arg -g which stands for
gpgopts. So we need to pass the homedir there too.
PiperOrigin-RevId: 330443280
This immediately revealed an escape analysis violation (!), where
the sync.Map was being used in a context that escapes were not
allowed. This is a relatively minor fix and is included.
PiperOrigin-RevId: 328611237
Due to how marshallable interface implementation was generated, we could not
marshal a struct whose named started with W because there was a naming
collision with parameter (w io.Writer) and type (w *StuctName).
Used "writer" as parameter name to avoid collision.
PiperOrigin-RevId: 328343930
Remove the old benchmark-tools directory, including
imports in the WORKSPACE file and associated bazel rules.
The new Golang benchmark-tools can be found at //test/benchmarks
and it is functionally equivalent, excepting syscall_test
which can be found in //test/perf/linux.
PiperOrigin-RevId: 325529075
This required minor fixes to the bazel wrapper. The "dut_platform" is
also changed to "native" to line-up with the system call tests and
remove the hard-coded "linux" and "netstack" strings.
As it happens
gvisor/tools/bazel.mk:88
useradd --uid $(UID) --non-unique --no-create-home \
adds the user-id to /var/log/lastlog which happens to be a sparse file except
Go's tar support can't handle sparse files so it actually tries to allocate
the file to seek to the end causing the VM to run out of disk space.
See:
https://github.com/moby/moby/issues/5419#issuecomment-193876183
The fix is to pass -l to useradd to prevent it from trying to add to lastlog.
Fixes#3397
PiperOrigin-RevId: 323492591
The bazel server was being started as the wrong user, leading to issues
where the container would suddenly exit during a build.
We can also simplify the waiting logic by starting the container in two
separate steps: those that must complete first, then the asynchronous bit.
PiperOrigin-RevId: 323391161
Using "--no-cache-dir" flag in pip install ,make sure dowloaded packages
by pip don't cached on system . This is a best practise which make sure
to fetch ftom repo instead of using local cached one . Further , in case
of Docker Containers , by restricing caching , we can reduce image size.
In term of stats , it depends upon the number of python packages
multiplied by their respective size . e.g for heavy packages with a lot
of dependencies it reduce a lot by don't caching pip packages.
Further , more detail information can be found at
https://medium.com/sciforce/strategies-of-docker-images-optimization-2ca9cc5719b6
- All Marshal* and Unmarshal* methods now require buffers to be correctly sized
- Only the Copy{In/Out} variants can handle smaller buffers (or address spaces)
PiperOrigin-RevId: 322953881
For some reason these tests were broken when run via the bazel docker
container. The mechanism used was a bit crazy (self-extracting bundle),
so convert them to use straight-forward starlark rules. This has the
added advantaged that they are now independent tests.
Fixed the following error:
tools/bazel.mk:119: *** Destination not provided.. Stop.
The issue was that we were running all make commands in a subshell so variables
like $T were not passed on.
The other issue was related to credentials. The test passed locally but not on
kokoro, the only difference in the workflow was the credentials bit.
Also fixed up some other tiny issues I came across, were not blockers.
PiperOrigin-RevId: 322782457
- Fixes some html validation issues.
- Fixes links on security basics blog post.
- Adds rel=noopener to links with target=_blank and adds a check to
htmlproofer.
- Add favicon check to htmlproofer.
Fixes#3286Fixes#3284
PiperOrigin-RevId: 321892602
`set -e` does not catch errors in bash command substituions like
`echo $(bad cmd)` so bazel thinks it succeeded and cached the result.
PiperOrigin-RevId: 321412327
This rule isn't restored when the Kokoro VM is restarted, so it's not doing
anything. And the problem it was meant to solved is instead addressed by
https://github.com/google/gvisor/pull/3207.
PiperOrigin-RevId: 321026846
The go.mod dependency tree for the shim was somehow contradictory. After
resolving these issues (e.g. explicitly imported k8s 1.14, pulling a
specific dbus version), and adding all dependencies, the shim can now be
build as part of the regular bazel tree.
As part of this process, minor cleanup was done in all the source files:
headers were standardized (and include "The gVisor Authors" in addition
to the "The containerd Authors" if originally derived from containerd
sources), and comments were cleaned up to meet coding standards.
This change makes the containerd installation dynamic, so that multiple
versions can be tested, and drops the static installer for the VM image
itself.
This change also updates test/root/crictl_test.go and related utilities,
so that the containerd tests can be run on any version (and in cases
where it applies, they can be run on both v1 and v2 as parameterized
tests).
We need docker IPv6 for ip6tables tests. Docker has spotty IPv6 support, so just
enabling IPv6 in daemon.json breaks the Ruby image test (AFAICT it breaks
anything that tries to use IPv6 to reach the internet). An ip6tables NAT rule is
added to handle this.
We could make these changes as part of scripts/, but these would overwrite
global values and possibly break users' systems.
PiperOrigin-RevId: 318508209
Previously, it was not possible to encode/decode an object graph which
contained a pointer to a field within another type. This was because the
encoder was previously unable to disambiguate a pointer to an object and a
pointer within the object.
This CL remedies this by constructing an address map tracking the full memory
range object occupy. The encoded Refvalue message has been extended to allow
references to children objects within another object. Because the encoding
process may learn about object structure over time, we cannot encode any
objects under the entire graph has been generated.
This CL also updates the state package to use standard interfaces intead of
reflection-based dispatch in order to improve performance overall. This
includes a custom wire protocol to significantly reduce the number of
allocations and take advantage of structure packing.
As part of these changes, there are a small number of minor changes in other
places of the code base:
* The lists used during encoding are changed to use intrusive lists with the
objectEncodeState directly, which required that the ilist Len() method is
updated to work properly with the ElementMapper mechanism.
* A bug is fixed in the list code wherein Remove() called on an element that is
already removed can corrupt the list (removing the element if there's only a
single element). Now the behavior is correct.
* Standard error wrapping is introduced.
* Compressio was updated to implement the new wire.Reader and wire.Writer
inteface methods directly. The lack of a ReadByte and WriteByte caused issues
not due to interface dispatch, but because underlying slices for a Read or
Write call through an interface would always escape to the heap!
* Statify has been updated to support the new APIs.
See README.md for a description of how the new mechanism works.
PiperOrigin-RevId: 318010298
This change fixes the apt repository structure to avoid emiting warnings on
Ubuntu 18.04 (and potentially other versions). This requires a slight refactor
of the repository generation scripts, since we can no longer copy the same
release files for different "suites".
This should avoid the warning by setting the suite to the distribution:
https://github.com/Debian/apt/blob/master/apt-pkg/metaindex.cc#L75
This change also moves over to the standardized Makefile entrypoint, which
makes settings clearer and enables local testing.
PiperOrigin-RevId: 313817017
The --groups command will fail if the group is not defined in
/etc/groups, even though it is specified by GID.
By coincidence, the group happens to be there for many installations of
Ubuntu 18.04 (which uses 999 for the Docker group), but it is strangely
absent in others (which use 130).
Fixes#2765
This change was derived from a change by:
Reapor-Yurnero <reapor.yurnero@gmail.com>
And has been modified by:
Adin Scannell <ascannell@google.com>
(The original change author is preserved for the commit.)
This change implements gap tracking in the segment set by adding additional
information in each node, and using that information to speed up gap finding
from a linear scan to a O(log(n)) walk of the tree.
This gap tracking is optional, and will default to off except for segment
instances that set gapTracking equal to 1 in their const lists.
PiperOrigin-RevId: 312621607
Without this fix, make exits with zero code when bazel build failed:
$ make run TARGETS="--abra --kadabra"
ERROR: Unrecognized option: --abra
$ echo $?
0
Signed-off-by: Andrei Vagin <avagin@gmail.com>
This change two does things:
1) Name the container based on the canonical directory path.
2) Allow the container to exit after bazel itself has exited.
The first is necessary to support multiple working directories,
while the second one allows these instances to clean up properly.
PiperOrigin-RevId: 310460748
This adapts the merged website repository to use the image and bazel
build framework. It explicitly avoids the container_image rules provided
by bazel, opting instead to build with direct docker commands when
necessary.
The relevant build commands are incorporated into the top-level
Makefile.
This change moves all Docker images to a standard location, and abstracts the
build process so that they can be maintained in an automated fashion. This also
allows the images to be architecture-independent.
All images will now be referred to by the test framework via the canonical
`gvisor.dev/images/<name>`, where `<name>` is a function of the path within the
source tree.
In a subsequent change, continuous integration will be added so that the images
will always be correct and available locally.
In the end, using `bazel` for Docker containers is simply not possible. Given
that we already have the need to use `make` with the base container (for
Docker), we extend this approach to get more flexibility.
This change also adds a self-documenting and powerful Makefile that is intended
to replace the collection of scripts in scripts. Canonical (self-documenting)
targets can be added here for targets that understand which images need to be
loaded and/or built.
PiperOrigin-RevId: 308322438
This change adds a layer of abstraction around the internal Docker APIs,
and eliminates all direct dependencies on Dockerfiles in the infrastructure.
A subsequent change will automated the generation of local images (with
efficient caching). Note that this change drops the use of bazel container
rules, as that experiment does not seem to be viable.
PiperOrigin-RevId: 308095430
Even though the default build option is to use 1.14, we want to be
want to keep the ability to target different Go versions for testing
and in case the new release has bugs.
PiperOrigin-RevId: 308078876
See tools/nogo/README.md.
The checkescape tool is able to perform recursive escape analysis, using the
actual generated binary to confirm the results produced by the compiler itself.
As an initial use case, this replaces the manual escape analysis tests used for
go_marshal, and validates that the CopyIn and CopyOut paths will not require
any allocation or stack splits.
Updates #2243
PiperOrigin-RevId: 307532986
This change renames the tools/images directory to tools/vm for clarity, and
adds a functional vm_test. Sharding is also added to the same test, and some
documentation added around key flags & variables to describe how they work.
Subsequent changes will add vm_tests for other cases, such as the runtime tests.
PiperOrigin-RevId: 307492245