Skip to content

(cli): ECS hotswap breaks Firelens configuration #21692

@tmokmss

Description

@tmokmss

Describe the bug

When I hotswap ECS service with the below Firelens configuration, the deployment fails because it seems options.Name is implicitly converted to options.name during hotswap.

      logging: LogDrivers.firelens({
        options: {
          Name: 'cloudwatch',
          region: Stack.of(this).region,
          log_group_name: logGroup.logGroupName,
          log_stream_prefix: 'prefix',
        },
      }),

Expected Behavior

Hotswap deployment successes just like CFn deployment.

Current Behavior

After hotswap, the fluentbit container failed to start with the following stoppedReason:

Stopped reason InternalError: unable to generate fireLens config file: unable to generate fireLens config content: unable to generate fluent config output section: unable to apply log options of container Main to fireLens config: missing output key Name which is required for fireLens configuration of type fluentbit

Reproduction Steps

  1. deploy the stack below
  2. change fromRegistry('nginx') to fromRegistry('nginx:latest') to trigger hotswap
  3. run cdk deploy --hotswap
  4. hotswap gets stuck. Checking management console, you can see the error above.
import * as cdk from 'aws-cdk-lib';
import { SubnetType, Vpc } from 'aws-cdk-lib/aws-ec2';
import { Cluster } from 'aws-cdk-lib/aws-ecs';
import { RemovalPolicy, Stack } from 'aws-cdk-lib';
import {
  ContainerImage,
  FargateService,
  FargateTaskDefinition,
  FirelensLogRouterType,
  LogDrivers,
} from 'aws-cdk-lib/aws-ecs';
import { LogGroup, RetentionDays } from 'aws-cdk-lib/aws-logs';
import { Construct } from 'constructs';

export class SampleStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = new Vpc(this, 'Vpc', {
      subnetConfiguration: [
        {
          cidrMask: 20,
          name: 'public',
          subnetType: SubnetType.PUBLIC,
        },
      ],
    });

    const cluster = new Cluster(this, 'Cluster', {
      vpc,
    });

    const logGroup = new LogGroup(this, 'ExecutionLog', {
      retention: RetentionDays.SIX_MONTHS,
      removalPolicy: RemovalPolicy.DESTROY,
    });

    const taskDefinition = new FargateTaskDefinition(this, 'TaskDef', {
      memoryLimitMiB: 512,
      cpu: 256,
    });

    taskDefinition.addContainer('Main', {
      image: ContainerImage.fromRegistry('nginx'),
      logging: LogDrivers.firelens({
        options: {
          Name: 'cloudwatch',
          region: Stack.of(this).region,
          log_group_name: logGroup.logGroupName,
          log_stream_prefix: 'main',
        },
      }),
    });

    taskDefinition.addFirelensLogRouter('Firelens', {
      firelensConfig: {
        type: FirelensLogRouterType.FLUENTBIT,
      },
      image: ContainerImage.fromRegistry('amazon/aws-for-fluent-bit'),
      logging: LogDrivers.awsLogs({
        streamPrefix: 'firelens',
        logGroup,
      }),
    });

    const service = new FargateService(this, 'Default', {
      cluster,
      taskDefinition,
      assignPublicIp: true,
    });
  }
}

Possible Solution

This line seems to be the root cause of this problem.

const lowercasedTaskDef = transformObjectKeys(this.taskDefinitionResource, lowerCaseFirstCharacter);
const registerTaskDefResponse = await sdk.ecs().registerTaskDefinition(lowercasedTaskDef).promise();

There are several properties in the RegisterTaskDefinition request that take arbitrary string as a property key, such as dockerLabels , logConfiguration.options, or dockerVolumeConfiguration.labels in addition to firelensConfiguration.options. We must not lowercase those property keys selectively.

Additional Information/Context

No response

CDK CLI Version

2.38.1

Framework Version

2.38.1

Node.js Version

16.15.0

OS

macOS

Language

Typescript

Language Version

No response

Other information

No response

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.needs-triageThis issue or PR still needs to be triaged.package/toolsRelated to AWS CDK Tools or CLI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions