Dynamic source ami by ChrisLundquist · Pull Request #3817 · hashicorp/packer

ChrisLundquist · 2016-08-21T00:42:18Z

Example packer file:

{
"builders":[{
  "type": "amazon-ebs",
  "access_key": "YOUR KEY",
  "secret_key": "YOUR SECRET",
  "region": "us-west-2",
  "source_ami_filter": {
    "filters": {
      "virtualization-type": "hvm",
      "name": "*ubuntu-xenial-16.04-amd64-server-*",
      "root-device-type": "ebs"
    },
    "owners": ["099720109477"],
    "most_recent": true
  },
  "instance_type": "t2.micro",
  "ssh_username": "ubuntu",
  "ami_name": "packer-quick-start {{timestamp}}",
  "subnet_id": "YOUR SUBNET",
  "vpc_id": "YOUR VPC"
}]
}

This adds a new option, source_ami_filter which will use the AWS API to find an image at build time. This allows users to hit a moving target, while keeping the configuration in packer, rather than in a bash script.

This is desirable as it allows one to get the latest kernels, packages, and security fixes.

This effectively makes the source_ami a simple option. One could specify it inside source_ami_filters and achieve the same result.

Closes #2756

ChrisLundquist · 2016-08-21T00:45:23Z

builder/amazon/common/step_run_source_instance.go

 	}

 	ui.Say("Launching a source AWS instance...")
-	imageResp, err := ec2conn.DescribeImages(&ec2.DescribeImagesInput{


Previously we made the DescribeImages call twice, rather than using our state bag. This now uses the result in the state bag.

rickard-von-essen · 2016-08-21T05:54:15Z

This should also cover amazon-instance and amazon-chroot builders.

lorengordon · 2016-08-21T11:18:40Z

builder/amazon/common/run_config.go

-	if c.SourceAmi == "" {
-		errs = append(errs, errors.New("A source_ami must be specified"))
+	if c.SourceAmi == "" && c.SourceAmiFilter.Empty() {
+		errs = append(errs, errors.New("A source_ami or dynamic_source_ami must be specified"))


Param is now source_ami_filter...

Fixed, good catch!

Hashfyre · 2016-09-02T09:19:11Z

We currently use source AMI passed from ENV to achieve this, having an inbuilt aws-ami-filter would be amazing for our use-case.

{
    ...
    "source_ami": "{{env `PHP_5_AMI_ID`}}"
 },

ChrisLundquist · 2016-09-26T21:46:35Z

This there anything else I can help with / do here? :)

rickard-von-essen · 2016-10-01T21:36:00Z

website/source/docs/builders/amazon-ebs.html.md

+    ```
+    This selects the most recent Ubuntu 16.04 HVM EBS AMI from Canonical.
+
+-   `source_ami_filter.filters` (map of strings) - filters used to select a `source_ami`.


This should be indented and only say filters as key, see ami_block_device_mappings. The same goes for owners, and most_recent.

rickard-von-essen · 2016-10-01T21:38:18Z

website/source/docs/builders/amazon-ebs.html.md

+    This is helpful to limit the AMIs to a trusted third party, or to your own account.
+
+-   `source_ami_filter.most_recent` (bool) - Selects the newest created image when true.
+    This is most useful for selecting a daily distro build.


This should state that it fail unless exactly one AMI is found, like in Terraform.

rickard-von-essen · 2016-10-01T21:42:22Z

This looks great! But I would also like to have this on amazon-instance and amazon-chroot and the docs comments fixed.

ChrisLundquist · 2016-10-01T22:30:26Z

@rickard-von-essen 0e26376 should achieve the doc enhancements you pointed out.

rickard-von-essen · 2016-10-01T22:39:20Z

👍

Maybe you are working on this but, the chroot builder need to have Prepare updated, see https://github.com/mitchellh/packer/blob/master/builder/amazon/chroot/builder.go#L152 and both chroot and instance builder needs an additional param to StepSourceAMIInfo.

ChrisLundquist · 2016-10-01T22:44:24Z

Good catch. I spaced on that part since it has been a while ( I thought it was already part of the common flow). I'll fix that up too.

ChrisLundquist · 2016-10-01T23:06:30Z

I've updated the other two amazon builders. I'm not as familiar with them so I didn't run a complete usage test for them.

One wrinkle was that the chroot builder doesn't use common/run_config so I had to shoe horn it into chroot/config.go

I've also checked the "Allow editors from maintainers." check box. Feel free to push any clean up / fixes to this branch. Thank you for taking a look at this!

rickard-von-essen · 2016-10-01T23:06:47Z

Super! I'll try to do some testing tomorrow.

mwhooker

This is great. Besides a few comments, this lgtm. I want to wait for either @rickard-von-essen or myself to test these before merging, but looks super useful

mwhooker · 2016-10-08T01:15:54Z

builder/amazon/common/step_run_source_instance.go

-
-	if len(imageResp.Images) != 1 {
-		state.Put("error", fmt.Errorf("The source AMI '%s' could not be found.", s.SourceAMI))
+	image, ok := state.Get("source_image").(*ec2.Image)


I think it's ok to let go panic here. Existing practice is

image := state.Get("source_image").(*ec2.Image)

Yeah, I hit a panic when writing this and realized I should handle the error.

As a new comer to golang seeing panics were a little overwhelming.
My intention here was to provide a fairly easy debug point in the future.

This shouldn't ever happen since "we" are the ones putting the object in the shared state. I think I hit this because I had an ec2.Image instead of *ec2.Image.

It seems like since we have the above, there isn't too much to gain by removing it.
Perhaps we could improve the error message though?

I guess, to phrase it differently,
Yeah, it probably is fine, since this should get hit. If we have it though, is it worth removing?

nah probably not worth removing

mwhooker · 2016-10-08T01:24:10Z

website/source/docs/builders/amazon-chroot.html.md

+    }
+    ```
+    This selects the most recent Ubuntu 16.04 HVM EBS AMI from Canonical.
+    NOTE: This will fail unless *exactly* one AMI is returned.


won't this succeed even if more than one image is returned since most_recent is true?

Yeah, let me clarify the docs to say that most_recent will work.

jrnt30 · 2016-10-19T13:27:13Z

@mwhooker If this is in an acceptable state, would it be possible to get this into the v0.11 release this month then?

mwhooker · 2016-10-19T18:25:55Z

builder/amazon/common/step_source_ami_info.go

+func (a imageSort) Len() int      { return len(a) }
+func (a imageSort) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
+func (a imageSort) Less(i, j int) bool {
+	itime, _ := time.Parse(time.RFC3339, *a[i].CreationDate)


also not a huge issue, but RFC3339 times are lexicographically sortable

5.1. Ordering If date and time components are ordered from least precise to most precise, then a useful property is achieved. Assuming that the time zones of the dates and times are the same (e.g., all in UTC), expressed using the same string (e.g., all "Z" or all "+00:00"), and all times have the same number of fractional second digits, then the date and time strings may be sorted as strings (e.g., using the strcmp() function in C) and a time-ordered sequence will result. The presence of optional punctuation would violate this characteristic.

https://www.ietf.org/rfc/rfc3339.txt

Yeah, This was direct from terraform, so we'll be consistent :)

https://github.com/hashicorp/terraform/blob/1b2f553d52c6fb25fd65d5b4b81ffd4a2bd264f4/builtin/providers/aws/data_source_aws_ami.go#L299-L314

mwhooker · 2016-10-19T18:34:53Z

@jrnt30 it's possible, but probably not. there's a lot I need to get done for 0.11 and I'm not sure i'll have time to test this.

If you want to help by testing the PR, I might feel more confident merging it in for 0.11, however.

I'm mostly looking to test that it's backwards compatible, ie existing builders work, and to test a few of the edge cases, like how it behaves if the filters return a huge number of images with mostRecent = true.

rickard-von-essen · 2016-10-22T20:08:43Z

@mwhooker I have done some basic testing. It returns the ami I expect and bails when I have "use_latest": false and there is more than one ami.

I haven't found any performance issues when doing stupid things like:

      "source_ami_filter": {
        "filters": {
          "name": "*x86_64*",
          "virtualization-type": "hvm",
          "root-device-type": "ebs"
        },
        "most_recent": [true|false]
      },

rickard-von-essen · 2016-10-22T20:09:23Z

@mwhooker If you don't want to do any more testing LGTM.

mwhooker · 2016-10-25T18:24:00Z

Lgtm. Feel free to merge in, or I can when I get back next week
On Sat, Oct 22, 2016 at 13:09 Rickard von Essen notifications@github.com
wrote:

@mwhooker https://github.com/mwhooker If you don't want to do any more
testing LGTM.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3817 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA9AtVKLGkbU_yEMkpktO3ImObdfAyRks5q2m1-gaJpZM4JpMmh
.

ChrisLundquist · 2016-10-25T19:30:01Z

Thank you all!

mwhooker · 2016-10-27T15:37:40Z

0.11 is already out. Should make it in 0.11.1 which will be next release
On Wed, Oct 26, 2016 at 19:42 Marat Bakeev notifications@github.com wrote:

Will this PR make it into v0.11?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3817 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA9AlyUTtDpb3-tijuzRC6KpqAN0c5Dks5q3-VigaJpZM4JpMmh
.

mwhooker · 2016-10-28T14:50:22Z

Actually I misspoke before. 0.12 is the next release this will be in
On Wed, Oct 26, 2016 at 21:14 Matthew Hooker mwhooker@gmail.com wrote:

0.11 is already out. Should make it in 0.11.1 which will be next release
On Wed, Oct 26, 2016 at 19:42 Marat Bakeev notifications@github.com
wrote:

Will this PR make it into v0.11?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3817 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA9AlyUTtDpb3-tijuzRC6KpqAN0c5Dks5q3-VigaJpZM4JpMmh
.

ghost · 2020-03-31T02:20:58Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

ChrisLundquist added 6 commits August 20, 2016 18:58

[dynamic-source-ami] proof of concept

ec1abb9

[dynamic-source-ami] make most_recent work

0e248f1

[dynamic-source-ami] rename to source_ami_filter

95afaa5

[dynamic-source-ami] add some docs

3d6cf56

[dynamic-source-ami] fix lint error

57e7a1a

[dynamic-source-ami] add some very basic tests

b4d2070

ChrisLundquist reviewed Aug 21, 2016
View reviewed changes

[dynamic-source-ami] fix old key name in error message

3cc8d35

rickard-von-essen added enhancement builder/amazon labels Aug 21, 2016

rickard-von-essen added the pr/needs-review label Aug 21, 2016

lorengordon reviewed Aug 21, 2016
View reviewed changes

Merge remote-tracking branch 'origin/master' into dynamic-source-ami

defdd1e

rickard-von-essen self-assigned this Sep 27, 2016

rickard-von-essen suggested changes Oct 1, 2016

View reviewed changes

rickard-von-essen added stage/waiting-reply and removed pr/needs-review labels Oct 1, 2016

ChrisLundquist added 2 commits October 1, 2016 22:20

Merge remote-tracking branch 'origin/master' into dynamic-source-ami

1b4895c

[dynamic-source-ami] minor doc improvements

0e26376

rickard-von-essen added pr/needs-review and removed stage/waiting-reply labels Oct 1, 2016

rickard-von-essen approved these changes Oct 1, 2016

View reviewed changes

mwhooker suggested changes Oct 8, 2016

View reviewed changes

mwhooker added the stage/waiting-reply label Oct 10, 2016

ChrisLundquist added 3 commits October 14, 2016 11:51

Update amazon-chroot.html.md

77a7728

Update amazon-ebs.html.md

7edc3f9

Update amazon-instance.html.md

e15c227

mwhooker approved these changes Oct 14, 2016

View reviewed changes

mwhooker removed the stage/waiting-reply label Oct 14, 2016

mwhooker added this to the v0.12 milestone Oct 14, 2016

mwhooker reviewed Oct 19, 2016

View reviewed changes

rickard-von-essen added pr/ready-for-merge and removed pr/needs-review labels Oct 22, 2016

rickard-von-essen assigned mwhooker and unassigned rickard-von-essen Oct 22, 2016

rickard-von-essen merged commit d16d5d9 into hashicorp:master Oct 25, 2016

ChrisLundquist deleted the dynamic-source-ami branch July 7, 2018 06:11

ghost locked and limited conversation to collaborators Mar 31, 2020

Conversation

ChrisLundquist commented Aug 21, 2016 • edited by rickard-von-essen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickard-von-essen commented Aug 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hashfyre commented Sep 2, 2016

Uh oh!

ChrisLundquist commented Sep 26, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickard-von-essen Oct 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rickard-von-essen commented Oct 1, 2016

Uh oh!

ChrisLundquist commented Oct 1, 2016

Uh oh!

rickard-von-essen commented Oct 1, 2016

Uh oh!

ChrisLundquist commented Oct 1, 2016

Uh oh!

ChrisLundquist commented Oct 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickard-von-essen commented Oct 1, 2016

Uh oh!

mwhooker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrnt30 commented Oct 19, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChrisLundquist Oct 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwhooker commented Oct 19, 2016

Uh oh!

rickard-von-essen commented Oct 22, 2016

Uh oh!

rickard-von-essen commented Oct 22, 2016

Uh oh!

mwhooker commented Oct 25, 2016

Uh oh!

ChrisLundquist commented Oct 25, 2016

Uh oh!

mwhooker commented Oct 27, 2016

Uh oh!

mwhooker commented Oct 28, 2016

Uh oh!

ghost commented Mar 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ChrisLundquist commented Aug 21, 2016 •

edited by rickard-von-essen

Loading

rickard-von-essen Oct 1, 2016 •

edited

Loading

ChrisLundquist commented Oct 1, 2016 •

edited

Loading

ChrisLundquist Oct 19, 2016 •

edited

Loading