Skip to content

Commit fe7bebb

Browse files
mergify[bot]mergify[bot]
authored
Merge branch 'master' into conroy/kafka
2 parents 810b3d2 + fcf981b commit fe7bebb

52 files changed

Lines changed: 4561 additions & 1300 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎INTEGRATION_TESTS.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ on what type of changes require integrations tests and how you should write inte
1010
- [New L2 Constructs](#new-l2-constructs)
1111
- [Existing L2 Constructs](#existing-l2-constructs)
1212
- [Assertions](#assertions)
13+
- [Running Integration Tests](#running-integration-tests)
1314

1415
## What are CDK Integration Tests
1516

@@ -223,3 +224,40 @@ to deploy the Lambda Function _and_ then rerun the assertions to ensure that the
223224

224225
### Assertions
225226
...Coming soon...
227+
228+
## Running Integration Tests
229+
230+
Most of the time you will only need to run integration tests for an individual module (i.e. `aws-lambda`). Other times you may need to run tests across multiple modules.
231+
In this case I would recommend running from the root directory like below.
232+
233+
_Run snapshot tests only_
234+
```bash
235+
yarn integ-runner --directory packages/@aws-cdk
236+
```
237+
238+
_Run snapshot tests and then re-run integration tests for failed snapshots_
239+
```bash
240+
yarn integ-runner --directory packages/@aws-cdk --update-on-failed
241+
```
242+
243+
One benefit of running from the root directory like this is that it will only collect tests from "built" modules. If you have built the entire
244+
repo it will run all integration tests, but if you have only built a couple modules it will only run tests from those.
245+
246+
### Running large numbers of Tests
247+
248+
If you need to re-run a large number of tests you can run them in parallel like this.
249+
250+
```bash
251+
yarn integ-runner --directory packages/@aws-cdk --update-on-failed \
252+
--parallel-regions us-east-1 \
253+
--parallel-regions us-east-2 \
254+
--parallel-regions us-west-2 \
255+
--parallel-regions eu-west-1 \
256+
--profiles profile1 \
257+
--profiles profile2 \
258+
--profiles profile3 \
259+
--verbose
260+
```
261+
262+
When using both `--parallel-regions` and `--profiles` it will execute (regions*profiles) tests in parallel (in this example 12)
263+
If you want to execute more than 16 tests in parallel you can pass a higher value to `--max-workers`.

‎packages/@aws-cdk/aws-lambda/lib/function.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ export interface FunctionOptions extends EventInvokeConfigOptions {
9696
readonly memorySize?: number;
9797

9898
/**
99-
* The size of the function’s /tmp directory in MB.
99+
* The size of the function’s /tmp directory in MiB.
100100
*
101101
* @default 512 MiB
102102
*/

‎packages/@aws-cdk/aws-logs/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,6 +306,23 @@ const pattern = logs.FilterPattern.spaceDelimited('time', 'component', '...', 'r
306306
.whereNumber('result_code', '!=', 200);
307307
```
308308

309+
## Logs Insights Query Definition
310+
311+
Creates a query definition for CloudWatch Logs Insights.
312+
313+
Example:
314+
315+
```ts
316+
new logs.QueryDefinition(this, 'QueryDefinition', {
317+
queryDefinitionName: 'MyQuery',
318+
queryString: new logs.QueryString({
319+
fields: ['@timestamp', '@message'],
320+
sort: '@timestamp desc',
321+
limit: 20,
322+
}),
323+
});
324+
```
325+
309326
## Notes
310327

311328
Be aware that Log Group ARNs will always have the string `:*` appended to

‎packages/@aws-cdk/aws-logs/lib/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ export * from './pattern';
66
export * from './subscription-filter';
77
export * from './log-retention';
88
export * from './policy';
9+
export * from './query-definition';
910

1011
// AWS::Logs CloudFormation Resources:
1112
export * from './logs.generated';

‎packages/@aws-cdk/aws-logs/lib/query-definition.ts‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
import { Resource } from '@aws-cdk/core';
2+
import { Construct } from 'constructs';
3+
import { CfnQueryDefinition } from '.';
4+
import { ILogGroup } from './log-group';
5+
6+
7+
/**
8+
* Properties for a QueryString
9+
*/
10+
export interface QueryStringProps {
11+
/**
12+
* Retrieves the specified fields from log events for display.
13+
*
14+
* @default - no fields in QueryString
15+
*/
16+
readonly fields?: string[];
17+
18+
/**
19+
* Extracts data from a log field and creates one or more ephemeral fields that you can process further in the query.
20+
*
21+
* @default - no parse in QueryString
22+
*/
23+
readonly parse?: string;
24+
25+
/**
26+
* Filters the results of a query that's based on one or more conditions.
27+
*
28+
* @default - no filter in QueryString
29+
*/
30+
readonly filter?: string;
31+
32+
/**
33+
* Uses log field values to calculate aggregate statistics.
34+
*
35+
* @default - no stats in QueryString
36+
*/
37+
readonly stats?: string;
38+
39+
/**
40+
* Sorts the retrieved log events.
41+
*
42+
* @default - no sort in QueryString
43+
*/
44+
readonly sort?: string;
45+
46+
/**
47+
* Specifies the number of log events returned by the query.
48+
*
49+
* @default - no limit in QueryString
50+
*/
51+
readonly limit?: Number;
52+
53+
/**
54+
* Specifies which fields to display in the query results.
55+
*
56+
* @default - no display in QueryString
57+
*/
58+
readonly display?: string;
59+
}
60+
61+
interface QueryStringMap {
62+
readonly fields?: string,
63+
readonly parse?: string,
64+
readonly filter?: string,
65+
readonly stats?: string,
66+
readonly sort?: string,
67+
readonly limit?: Number,
68+
readonly display?: string,
69+
}
70+
71+
/**
72+
* Define a QueryString
73+
*/
74+
export class QueryString {
75+
private readonly fields?: string[];
76+
private readonly parse?: string;
77+
private readonly filter?: string;
78+
private readonly stats?: string;
79+
private readonly sort?: string;
80+
private readonly limit?: Number;
81+
private readonly display?: string;
82+
83+
constructor(props: QueryStringProps = {}) {
84+
this.fields = props.fields;
85+
this.parse = props.parse;
86+
this.filter = props.filter;
87+
this.stats = props.stats;
88+
this.sort = props.sort;
89+
this.limit = props.limit;
90+
this.display = props.display;
91+
}
92+
93+
/**
94+
* String representation of this QueryString.
95+
*/
96+
public toString(): string {
97+
return noUndef({
98+
fields: this.fields !== undefined ? this.fields.join(', ') : this.fields,
99+
parse: this.parse,
100+
filter: this.filter,
101+
stats: this.stats,
102+
sort: this.sort,
103+
limit: this.limit,
104+
display: this.display,
105+
}).join(' | ');
106+
}
107+
}
108+
109+
function noUndef(x: QueryStringMap): string[] {
110+
const ret: string[] = [];
111+
for (const [key, value] of Object.entries(x)) {
112+
if (value !== undefined) {
113+
ret.push(`${key} ${value}`);
114+
}
115+
}
116+
return ret;
117+
}
118+
119+
/**
120+
* Properties for a QueryDefinition
121+
*/
122+
export interface QueryDefinitionProps {
123+
/**
124+
* Name of the query definition.
125+
*/
126+
readonly queryDefinitionName: string;
127+
128+
/**
129+
* The query string to use for this query definition.
130+
*/
131+
readonly queryString: QueryString;
132+
133+
/**
134+
* Specify certain log groups for the query definition.
135+
*
136+
* @default - no specified log groups
137+
*/
138+
readonly logGroups?: ILogGroup[];
139+
}
140+
141+
/**
142+
* Define a query definition for CloudWatch Logs Insights
143+
*/
144+
export class QueryDefinition extends Resource {
145+
/**
146+
* The ID of the query definition.
147+
*
148+
* @attribute
149+
*/
150+
public readonly queryDefinitionId: string;
151+
152+
constructor(scope: Construct, id: string, props: QueryDefinitionProps) {
153+
super(scope, id, {
154+
physicalName: props.queryDefinitionName,
155+
});
156+
157+
const queryDefinition = new CfnQueryDefinition(this, 'Resource', {
158+
name: props.queryDefinitionName,
159+
queryString: props.queryString.toString(),
160+
logGroupNames: typeof props.logGroups === 'undefined' ? [] : props.logGroups.flatMap(logGroup => logGroup.logGroupName),
161+
});
162+
163+
this.queryDefinitionId = queryDefinition.attrQueryDefinitionId;
164+
}
165+
}

‎packages/@aws-cdk/aws-logs/test/integ.save-logs-insights-query-definition.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { App, RemovalPolicy, Stack, StackProps } from '@aws-cdk/core';
2+
import { LogGroup, QueryDefinition, QueryString } from '../lib';
3+
4+
class LogsInsightsQueryDefinitionIntegStack extends Stack {
5+
constructor(scope: App, id: string, props?: StackProps) {
6+
super(scope, id, props);
7+
8+
const logGroup = new LogGroup(this, 'LogGroup', {
9+
removalPolicy: RemovalPolicy.DESTROY,
10+
});
11+
12+
new QueryDefinition(this, 'QueryDefinition', {
13+
queryDefinitionName: 'QueryDefinition',
14+
queryString: new QueryString({
15+
fields: ['@timestamp', '@message'],
16+
parse: '@message "[*] *" as loggingType, loggingMessage',
17+
filter: 'loggingType = "ERROR"',
18+
sort: '@timestamp desc',
19+
limit: 20,
20+
display: 'loggingMessage',
21+
}),
22+
logGroups: [logGroup],
23+
});
24+
}
25+
}
26+
27+
const app = new App();
28+
new LogsInsightsQueryDefinitionIntegStack(app, 'aws-cdk-logs-querydefinition-integ');
29+
app.synth();

‎packages/@aws-cdk/aws-logs/test/query-definition.test.ts‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
import { Template } from '@aws-cdk/assertions';
2+
import { Stack } from '@aws-cdk/core';
3+
import { LogGroup, QueryDefinition, QueryString } from '../lib';
4+
5+
describe('query definition', () => {
6+
test('create a query definition', () => {
7+
// GIVEN
8+
const stack = new Stack();
9+
10+
// WHEN
11+
new QueryDefinition(stack, 'QueryDefinition', {
12+
queryDefinitionName: 'MyQuery',
13+
queryString: new QueryString({
14+
fields: ['@timestamp', '@message'],
15+
sort: '@timestamp desc',
16+
limit: 20,
17+
}),
18+
});
19+
20+
// THEN
21+
Template.fromStack(stack).hasResourceProperties('AWS::Logs::QueryDefinition', {
22+
Name: 'MyQuery',
23+
QueryString: 'fields @timestamp, @message | sort @timestamp desc | limit 20',
24+
});
25+
});
26+
27+
test('create a query definition against certain log groups', () => {
28+
// GIVEN
29+
const stack = new Stack();
30+
31+
// WHEN
32+
const logGroup = new LogGroup(stack, 'MyLogGroup');
33+
34+
new QueryDefinition(stack, 'QueryDefinition', {
35+
queryDefinitionName: 'MyQuery',
36+
queryString: new QueryString({
37+
fields: ['@timestamp', '@message'],
38+
sort: '@timestamp desc',
39+
limit: 20,
40+
}),
41+
logGroups: [logGroup],
42+
});
43+
44+
// THEN
45+
Template.fromStack(stack).hasResourceProperties('AWS::Logs::QueryDefinition', {
46+
Name: 'MyQuery',
47+
QueryString: 'fields @timestamp, @message | sort @timestamp desc | limit 20',
48+
LogGroupNames: [{ Ref: 'MyLogGroup5C0DAD85' }],
49+
});
50+
});
51+
52+
test('create a query definition with all commands', () => {
53+
// GIVEN
54+
const stack = new Stack();
55+
56+
// WHEN
57+
const logGroup = new LogGroup(stack, 'MyLogGroup');
58+
59+
new QueryDefinition(stack, 'QueryDefinition', {
60+
queryDefinitionName: 'MyQuery',
61+
queryString: new QueryString({
62+
fields: ['@timestamp', '@message'],
63+
parse: '@message "[*] *" as loggingType, loggingMessage',
64+
filter: 'loggingType = "ERROR"',
65+
sort: '@timestamp desc',
66+
limit: 20,
67+
display: 'loggingMessage',
68+
}),
69+
logGroups: [logGroup],
70+
});
71+
72+
// THEN
73+
Template.fromStack(stack).hasResourceProperties('AWS::Logs::QueryDefinition', {
74+
Name: 'MyQuery',
75+
QueryString: 'fields @timestamp, @message | parse @message "[*] *" as loggingType, loggingMessage | filter loggingType = "ERROR" | sort @timestamp desc | limit 20 | display loggingMessage',
76+
});
77+
});
78+
});

‎packages/@aws-cdk/aws-logs/test/save-logs-insights-query-definition.integ.snapshot/aws-cdk-logs-querydefinition-integ.template.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"Resources": {
3+
"LogGroupF5B46931": {
4+
"Type": "AWS::Logs::LogGroup",
5+
"Properties": {
6+
"RetentionInDays": 731
7+
},
8+
"UpdateReplacePolicy": "Delete",
9+
"DeletionPolicy": "Delete"
10+
},
11+
"QueryDefinition4190BC36": {
12+
"Type": "AWS::Logs::QueryDefinition",
13+
"Properties": {
14+
"Name": "QueryDefinition",
15+
"QueryString": "fields @timestamp, @message | parse @message \"[*] *\" as loggingType, loggingMessage | filter loggingType = \"ERROR\" | sort @timestamp desc | limit 20 | display loggingMessage",
16+
"LogGroupNames": [
17+
{
18+
"Ref": "LogGroupF5B46931"
19+
}
20+
]
21+
}
22+
}
23+
}
24+
}

‎packages/@aws-cdk/aws-logs/test/save-logs-insights-query-definition.integ.snapshot/cdk.out‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"version":"17.0.0"}

0 commit comments

Comments
 (0)