test: Add macros for doing memory-usage tests of varying precision, based on platform by jmarantz · Pull Request #7243 · envoyproxy/envoy

jmarantz · 2019-06-12T02:57:54Z

Description:
Around June 5, 2019, some of the existing tests that measured exact memory consumption across a series of operations started diverging based on platform. This PR adds new EXPECT-style macros to help tests deal with this. EXPECT_MEMORY_EQ only runs in 'release' builds on CI, where we expect stable, consistent behavior. EXPECT_MEMORY_LE runs on any build with tcmalloc. On other platforms those EXPECT macros print 'info' messages indicating that they are skipped.
Risk Level: low
Testing: //test/... and a few platform variants
Docs Changes:
Release Notes:
Fixes: #7216

…y tests. Signed-off-by: Joshua Marantz <jmarantz@google.com>

Signed-off-by: Joshua Marantz <jmarantz@google.com>

ci/do_ci.sh

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz · 2019-06-13T03:16:36Z

/retest

repokitteh-read-only · 2019-06-13T03:16:40Z

🔨 rebuilding ci/circleci: docker (failed build)

🐱

Caused by: a #7243 (comment) was created by @jmarantz.

see: more, trace.

jmarantz · 2019-06-13T14:56:20Z

/retest

repokitteh-read-only · 2019-06-13T14:56:24Z

🔨 rebuilding ci/circleci: docker (failed build)

🐱

Caused by: a #7243 (comment) was created by @jmarantz.

see: more, trace.

htuch

/wait

ci/do_ci.sh

test/common/stats/stat_test_utility.h

test/common/stats/symbol_table_impl_test.cc

test/common/stats/stat_test_utility.h

jmarantz · 2019-06-13T19:00:28Z

btw this doesn't work yet because we need to restore (and fix) the old but not-totally-functional code that allocated memory and verified we could detect it.

…memory-usage at all. Signed-off-by: Joshua Marantz <jmarantz@google.com>

Signed-off-by: Joshua Marantz <jmarantz@google.com>

ci/do_ci.sh

…rflow behavior. Signed-off-by: Joshua Marantz <jmarantz@google.com>

Signed-off-by: Joshua Marantz <jmarantz@google.com>

lizan · 2019-06-15T07:48:12Z

Q: How do we run in Bazel CI? Do they use our scripts and will thus do exact testing?

No, the setting for bazel CI just invoke bazel without going through this script. So they are OK.
https://github.com/envoyproxy/envoy/blob/master/.bazelci/presubmit.yml

Signed-off-by: Joshua Marantz <jmarantz@google.com>

mattklein123

Generally LGTM, will defer to @lizan as build expert!

test/integration/stats_integration_test.cc

test/common/stats/stat_test_utility.cc

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz · 2019-06-17T23:14:50Z

/azp

azure-pipelines · 2019-06-17T23:14:54Z

Supported commands help: Get descriptions, examples and documentation about supported commands Example: help "command_name" list: List all pipelines for this repository using a comment. Example: "list" run: Run all pipelines or a specific pipeline for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify a pipeline to run. Example: "run" or "run pipeline_name" where: Report back the Azure DevOps orgs that are related to this repository and org Example: "where" See additional documentation.

jmarantz · 2019-06-17T23:48:05Z

/azp list

azure-pipelines · 2019-06-17T23:48:11Z

CI/CD Pipelines for this repository: envoy-macos envoy-linux

jmarantz · 2019-06-17T23:52:11Z

/azp run envoy-macos

azure-pipelines · 2019-06-17T23:52:21Z

Azure Pipelines successfully started running 1 pipeline(s).

jmarantz · 2019-06-18T03:36:26Z

/azp run envoy-macos

azure-pipelines · 2019-06-18T03:36:36Z

Azure Pipelines successfully started running 1 pipeline(s).

jmarantz · 2019-06-18T11:01:45Z

/azp run envoy-macos

azure-pipelines · 2019-06-18T11:01:56Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz · 2019-06-18T14:59:36Z

/azp run envoy-linux

azure-pipelines · 2019-06-18T14:59:51Z

Azure Pipelines successfully started running 1 pipeline(s).

comments addressed, 2 other senior approvers approved, htuch OOO

jmarantz added 5 commits June 11, 2019 22:43

add macros and helper functions for doing approximate and exact memor…

32df03d

…y tests. Signed-off-by: Joshua Marantz <jmarantz@google.com>

spelling fix attempt

078db33

Signed-off-by: Joshua Marantz <jmarantz@google.com>

spelling

f5d713d

Signed-off-by: Joshua Marantz <jmarantz@google.com>

spelling

309024d

Signed-off-by: Joshua Marantz <jmarantz@google.com>

spelling

01e31b1

Signed-off-by: Joshua Marantz <jmarantz@google.com>

mattklein123 self-assigned this Jun 12, 2019

jmarantz changed the title ~~test: Exact memtest in ci~~ WiP test: Exact memtest in ci Jun 12, 2019

jmarantz added 3 commits June 12, 2019 08:23

fix errors, add comments.

5023a62

Signed-off-by: Joshua Marantz <jmarantz@google.com>

tweak comments.

59097ce

Signed-off-by: Joshua Marantz <jmarantz@google.com>

inject memory exact comparison bug to make sure 'release' fails.

f0227fc

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz commented Jun 12, 2019

View reviewed changes

ci/do_ci.sh Outdated Show resolved Hide resolved

mattklein123 assigned lizan and htuch and unassigned mattklein123 Jun 12, 2019

jmarantz added 2 commits June 12, 2019 20:17

Enable memory-tests for optimized builds.

5152360

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Tweak comments and correct canonical sizes.

6a9fabd

Signed-off-by: Joshua Marantz <jmarantz@google.com>

htuch previously requested changes Jun 13, 2019

View reviewed changes

ci/do_ci.sh Outdated Show resolved Hide resolved

test/common/stats/stat_test_utility.h Show resolved Hide resolved

test/common/stats/symbol_table_impl_test.cc Show resolved Hide resolved

repokitteh-read-only bot added the waiting label Jun 13, 2019

lizan reviewed Jun 13, 2019

View reviewed changes

test/common/stats/stat_test_utility.h Show resolved Hide resolved

Review nits; and attempt to fix MacOS by verifying that we can check …

d916445

…memory-usage at all. Signed-off-by: Joshua Marantz <jmarantz@google.com>

repokitteh-read-only bot removed the waiting label Jun 13, 2019

jmarantz added 3 commits June 14, 2019 00:07

Merge branch 'master' into exact-memtest-in-ci

fbed0e1

Signed-off-by: Joshua Marantz <jmarantz@google.com>

tweak comments.

641f5f0

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Merge branch 'master' into exact-memtest-in-ci

a81d441

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz marked this pull request as ready for review June 14, 2019 13:12

lizan reviewed Jun 14, 2019

View reviewed changes

ci/do_ci.sh Outdated Show resolved Hide resolved

jmarantz added 2 commits June 14, 2019 20:24

address review comments, and be more explicit about unsigned int unde…

821fe30

…rflow behavior. Signed-off-by: Joshua Marantz <jmarantz@google.com>

do the builds with ${MEMORY_TEST_EXACT_ARGS}.

35cea04

Signed-off-by: Joshua Marantz <jmarantz@google.com>

repokitteh-read-only bot removed the waiting label Jun 15, 2019

Merge branch 'master' into exact-memtest-in-ci

dff4838

Signed-off-by: Joshua Marantz <jmarantz@google.com>

mattklein123 previously approved these changes Jun 17, 2019

View reviewed changes

lizan reviewed Jun 17, 2019

View reviewed changes

test/integration/stats_integration_test.cc Outdated Show resolved Hide resolved

test/common/stats/stat_test_utility.cc Outdated Show resolved Hide resolved

jmarantz added 2 commits June 17, 2019 18:32

Merge branch 'master' into exact-memtest-in-ci

5ab8926

Signed-off-by: Joshua Marantz <jmarantz@google.com>

Cleanup history, add 'volatile' keyword.

4927c8f

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz dismissed mattklein123’s stale review via 4927c8f June 17, 2019 22:48

update date in history line.

b374f37

Signed-off-by: Joshua Marantz <jmarantz@google.com>

lizan approved these changes Jun 18, 2019

View reviewed changes

Merge branch 'master' into exact-memtest-in-ci

065fd91

Signed-off-by: Joshua Marantz <jmarantz@google.com>

jmarantz merged commit 4fa3e9f into envoyproxy:master Jun 18, 2019

jmarantz deleted the exact-memtest-in-ci branch June 18, 2019 15:56

PiotrSikora mentioned this pull request Jun 18, 2019

bazel: avoid invalidating analysis cache between build and tests. #7294

Merged

Conversation

jmarantz commented Jun 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jmarantz commented Jun 13, 2019

Uh oh!

repokitteh-read-only bot commented Jun 13, 2019

Uh oh!

jmarantz commented Jun 13, 2019

Uh oh!

repokitteh-read-only bot commented Jun 13, 2019

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmarantz commented Jun 13, 2019

Uh oh!

Uh oh!

lizan commented Jun 15, 2019

Uh oh!

mattklein123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jmarantz commented Jun 17, 2019

Uh oh!

azure-pipelines bot commented Jun 17, 2019

Uh oh!

jmarantz commented Jun 17, 2019

Uh oh!

azure-pipelines bot commented Jun 17, 2019

Uh oh!

jmarantz commented Jun 17, 2019

Uh oh!

azure-pipelines bot commented Jun 17, 2019

Uh oh!

jmarantz commented Jun 18, 2019

Uh oh!

azure-pipelines bot commented Jun 18, 2019

Uh oh!

jmarantz commented Jun 18, 2019

Uh oh!

azure-pipelines bot commented Jun 18, 2019

Uh oh!

jmarantz commented Jun 18, 2019

Uh oh!

azure-pipelines bot commented Jun 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jmarantz commented Jun 12, 2019 •

edited

Loading