Skip to content

test(node): New Node integration test runner#10117

Merged
AbhiPrasad merged 27 commits intogetsentry:developfrom
timfish:test/e2e-runner
Jan 11, 2024
Merged

test(node): New Node integration test runner#10117
AbhiPrasad merged 27 commits intogetsentry:developfrom
timfish:test/e2e-runner

Conversation

@timfish
Copy link
Copy Markdown
Collaborator

@timfish timfish commented Jan 9, 2024 •
edited
Loading

To aid in adding tests for #9907

Adds a new Node integration test runner that runs the scenario in its own Node process and reads the envelope back from stdout.

I have converted all the Anr and LocalVariables tests to use this new format.

How to use

First, in your test scenario, you need to set the transport to the loggingTransport from @sentry-internal/node-integration-tests. This will cause envelopes to be logged to stdout as a single line of JSON:

import * as Sentry from '@sentry/node';
import { loggingTransport } from '@sentry-internal/node-integration-tests';

Sentry.init({
  dsn: 'https://public@dsn.ingest.sentry.io/1337',
  transport: loggingTransport,
});

Then in your test, use the createRunner function to create your test. The createRunner args are the path segments to the scenario. If you pass a typescript entry file, -r ts-node/register gets added to the node args.

Every expect call adds an expected envelope item. Currently you can expect event, transaction and session but it's easy to add extra type-safe expectations. You must add an expect for every envelope item that will be sent and they must be in the correct order or an error is thrown and the test fails. The expect event/transaction/session can be a callback or an object that will be tested against the received envelope.

If you don't care about specific types of envelope item, you can ignore them:

createRunner(__dirname, 'basic.js')
  .ignore('session', 'sessions')
  .expect(//...

If you pass done to start, done will be called when all the envelope items have matched the expected and done will be passed any errors that occur which ends the test without waiting for the timeout.

Example

Here is an Anr test that ensures we get the expected session envelope followed by the expected event envelope:

import { assertSentryEvent, assertSentrySession, createRunner } from '../../utils/runner';

const EXPECTED_ANR_EVENT = {
  // Ensure we have context
  contexts: {
    trace: {
      span_id: expect.any(String),
      trace_id: expect.any(String),
    },
    device: {
      arch: expect.any(String),
    },
    app: {
      app_start_time: expect.any(String),
    },
    os: {
      name: expect.any(String),
    },
    culture: {
      timezone: expect.any(String),
    },
  },
  // and an exception that is our ANR
  exception: {
    values: [
      {
        type: 'ApplicationNotResponding',
        value: 'Application Not Responding for at least 200 ms',
        mechanism: { type: 'ANR' },
        stacktrace: {
          frames: expect.arrayContaining([
            {
              colno: expect.any(Number),
              lineno: expect.any(Number),
              filename: expect.any(String),
              function: '?',
              in_app: true,
            },
            {
              colno: expect.any(Number),
              lineno: expect.any(Number),
              filename: expect.any(String),
              function: 'longWork',
              in_app: true,
            },
          ]),
        },
      },
    ],
  },
};

test('Anr with session', done => {
  createRunner(__dirname, 'basic-session.js')
    .expect({ session: { status: 'abnormal', abnormal_mechanism: 'anr_foreground' } })
    .expect({ event:  EXPECTED_ANR_EVENT })
    .start(done);
});

It's also possible to pass flags to node via the withFlags() function:

  test('With --inspect',done => {
    createRunner(__dirname, 'basic.mjs')
      .withFlags('--inspect')
      .expect({ event: EXPECTED_ANR_EVENT })
      .start(done);
  });

@timfish timfish changed the title test(node): New ee2e test runner test(node): New Node e2e test runner Jan 9, 2024
@mydea
Copy link
Copy Markdown
Member

mydea commented Jan 9, 2024

this sounds nice! Does that work with OpenTelemetry, I wonder? So with node-experimental? Would be a great way to write tests for integrations etc...!

@timfish
Copy link
Copy Markdown
Collaborator Author

timfish commented Jan 9, 2024 •
edited
Loading

Does that work with OpenTelemetry, I wonder? So with node-experimental?

Yep, that's partly what its for:

  test('otel with loader', done => {
    createRunner(__dirname, 'express-otel.mjs')
      .withFlags('--experimental-loader=@opentelemetry/instrumentation/hook.mjs')
      .expect({ transaction: EXPRESS_TRANSACTION })
      .start(done);
  });

@timfish
Copy link
Copy Markdown
Collaborator Author

timfish commented Jan 9, 2024 •
edited
Loading

I've added server request support and modified some of the express tracing tests to use the new runner.

To support server requests, make the following changes to your test scenario:

image

It can then be tested by calling makeRequest:

test('should create and send transactions for Express routes and spans for middlewares.', done => {
  createRunner(__dirname, 'server.ts')
    .expect({
      transaction: {
        contexts: {
          trace: {
            span_id: expect.any(String),
            trace_id: expect.any(String),
            data: {
              url: '/test/express',
              'http.response.status_code': 200,
            },
            op: 'http.server',
            status: 'ok',
            tags: {
              'http.status_code': '200',
            },
          },
        },
        spans: [
          expect.objectContaining({
            description: 'corsMiddleware',
            op: 'middleware.express.use',
          }),
        ],
      },
    })
    .start(done)
    .makeRequest('get', '/test/express');
});

timfish
timfish commented Jan 9, 2024
View reviewed changes
dev-packages/node-integration-tests/package.json
"type-check": "tsc",
"pretest": "run-s --silent prisma:init prisma:init:new",
"test": "ts-node ./utils/run-tests.ts",
"jest": "jest --config ./jest.config.js",
Copy link
Copy Markdown
Collaborator Author

@timfish timfish Jan 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this simply because it allows you to call yarn jest -t "name-of-test" --verbose and get debug output very quickly from a single test. yarn test -t "name-of-test" runs through the existing custom test runner code and takes ages...

Hopefully we can migrate to just using the jest runner!

@timfish timfish marked this pull request as ready for review January 10, 2024 10:40
@lforst lforst changed the title test(node): New Node e2e test runner test(node): New Node integration test runner Jan 10, 2024
@lforst lforst self-requested a review January 11, 2024 11:29
lforst
lforst approved these changes Jan 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@lforst lforst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@AbhiPrasad AbhiPrasad merged commit 98979d8 into getsentry:develop Jan 11, 2024
@timfish timfish deleted the test/e2e-runner branch January 23, 2024 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mydea mydea Awaiting requested review from mydea

@onurtemizkan onurtemizkan Awaiting requested review from onurtemizkan

2 more reviewers

@lforst lforst lforst approved these changes

@AbhiPrasad AbhiPrasad AbhiPrasad approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@timfish @mydea @lforst @AbhiPrasad