Conversation
ruflin
left a comment
There was a problem hiding this comment.
Idea LGTM.
One thing we probably will need later is to also be able to build a custom config file.
dev-tools/mage/crossbuild.go
Outdated
There was a problem hiding this comment.
I first stumbled over this line of code and was thinking "why is there a list of x-pack options"? I now get that it either has 1 or 2 entries for with and without x-pack. I wonder if a better name for the slice would be buildOptions or similar??
dev-tools/mage/crossbuild.go
Outdated
There was a problem hiding this comment.
What about seeting this line above b.Xpack check and then only have an if that overwrites it? Would be more similar to what you do with buildCmd. Outcome is the same.
dev-tools/mage/clean.go
Outdated
There was a problem hiding this comment.
What about .exe for Windows.
dev-tools/mage/crossbuild.go
Outdated
There was a problem hiding this comment.
In order to omit any XPack knowledge from the CrossBuild function and the crossBuildParams struct, I suggest creating a separate build target like crossBuildXPack in the filebeat/magefile.go that invokes a new function mage.CrossBuildXPack() that is doing CrossBuild(WithTarget("buildXPack"), InDir("x-pack", BeatName)).
And then add crossBuildXPack as a dependency of package.
There was a problem hiding this comment.
Thank you for your input @andrewkroh, I pushed a new change, let me know if this is looking better now. Once we are settled with how this should work I will extend it to all beats + tackle testing
dev-tools/mage/crossbuild.go
Outdated
There was a problem hiding this comment.
If you do add some kind of target/work dir parameter to GolangCrossBuilder then I think you can use filepath.Rel here to get a relative path from the current working directory to the target dir.
|
@exekias we were discussing offline on this one. it would be good if this approach allows community beats to be able to bundle their own custom processors/outputs into their custom beat. this is to overcome the short comings of lack of go-plugins in Windows. |
filebeat/magefile.go
Outdated
There was a problem hiding this comment.
comment on exported function CrossBuildXPack should be of the form "CrossBuildXPack ..."
andrewkroh
left a comment
There was a problem hiding this comment.
I happy with the current approach. It looks sufficiently simple.
x-pack/filebeat/cmd/root.go
Outdated
There was a problem hiding this comment.
You are the lucky person who adds the first Elastic licensed Go code to the project. This code needs to have the Elastic license header and we need to figure out a way to enforce these checks.
My preferred way would be to make https://github.com/elastic/go-licenser handle more license types.
There was a problem hiding this comment.
And I assume that the top-level make check fails because these files do not have Apache headers (I didn't check). That can be addressed by merging elastic/go-licenser#14 and using the new -exclude flag.
There was a problem hiding this comment.
I opened elastic/go-licenser#15 to handle this, will update this PR once that's merged
There was a problem hiding this comment.
Personally I use beatsfmt. This one searches for a .go_license_header file relative to the source file to be formatted.
dev-tools/mage/crossbuild.go
Outdated
There was a problem hiding this comment.
How about:
// CrossBuildXPack executes the 'golangCrossBuild' target in the Beat's
// associated x-pack directory to produce a version of the Beat that contains
// Elastic licensed content.
dev-tools/mage/crossbuild.go
Outdated
There was a problem hiding this comment.
I think it's important to document that the directory is automatically suffixed with the current sub-directory path relative to the repo root. I wasn't expecting this.
I was expecting to the see it used like InDir(filepath.Join("x-pack", BeatName)). Or make the signature be InDir(path ...string) and automatically invoke filepath.Join(path...) in the implementation.
There was a problem hiding this comment.
I can do that, but I don't have the BeatName in CrossBuild, it's just using the current subidr. In filebeat/magefile.go I could pass InDir("filebeat") for oss build and InDir("x-pack", "filebeat") in the xpack. WDYT?
There was a problem hiding this comment.
I think there is a variable named BeatName that is defined in settings.go that you can use. Does that work?
There was a problem hiding this comment.
Got it, I updated this again, I think I finally understood what you meant :)
1c206cd to
fbb5709
Compare
|
@andrewkroh I just saw APM is not copying our |
|
Then it seems fine just make sure APM is aware. |
|
@simitt is working on introducing our default |
Makefile
Outdated
There was a problem hiding this comment.
No need to exclude vendor or .git its in the default from go licenser.
https://github.com/elastic/go-licenser/blob/02324788cd84244b695b57aec2a4c388ea2d8fc8/main.go#L92
filebeat/magefile.go
Outdated
There was a problem hiding this comment.
This magefile change is missing for the other beats.
|
I have updated this PR with what we were missing from our discussion. |
|
I have fixed all the targets and paths and running |
|
Failures will be fixed in #7882 |
|
Rebased from master. |
|
The failure is not related to this change. |
|
We discussed in the @elastic/apm-server team that it would be great to stick with copying the Thus, keeping beats only changes in a separate definition, like suggested by @exekias would be great. |
|
@simitt I had a quick chat with @andrewkroh about this, we came to the conclusion that the current requirements is temporary and someday you have an x-pack folder for apm and you will build a xpack binary. But since the packing is done on our side we agree that we need to provide a migration path and allow futures changes to go back to you. After looking at it, merging yaml or filtering was not an option for us. We have decided to add a new method |
|
Sounds like a straight forward change on our side, thank you!. |
dev-tools/mage/pkgspecs.go
Outdated
dev-tools/mage/build.go
Outdated
There was a problem hiding this comment.
How come this was changed? Won't this be chowning files unnecessarily?
There was a problem hiding this comment.
I've changed the logic of DockerChown to not recursively walk up the path, instead it walk down the path, using only params.outputDir will change the permission bellow that directory instead. The other implementation had issues with using absolute paths.
There was a problem hiding this comment.
I will revert it back, it will work with both logic.
|
@andrewkroh I just updated the PR with your reviews comment. |
|
I've restarted the build, the |
This commit implements the necessary logic to build licensed beats, before that commit, the OSS and the License binaries were exactly the same. Changes: - Added `mage.CrossBuildXPack()` to build the elastic licensed beat. x-pack/beatname - Changed the packages.yml to include the artifact from the x-pack folder. - Added `mage.UseElasticBeatPackaging()` to build the packages with the new binaries. - Added `mage.UseElasticBeatWithoutXPackPackaging()` allow to keep the previous behavior. - `make check` and `make fmt` will use the right license for the x-pack folder. Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
|
@andrewkroh I've looked at the two failures I don't think they are related to this change since other PR like #7954 have the same issue, maybe hiccups with the release manager? I haven't investigated a lot. |
|
Updating APM Server to these changes worked smoothly, thanks to everyone involved! |
This commit implements the necessary logic to build licensed beats, before that commit, the OSS and the License binaries were exactly the same. Changes: - Added `mage.CrossBuildXPack()` to build the elastic licensed beat. x-pack/beatname - Changed the packages.yml to include the artifact from the x-pack folder. - Added `mage.UseElasticBeatPackaging()` to build the packages with the new binaries. - Added `mage.UseElasticBeatWithoutXPackPackaging()` allow to keep the previous behavior. - `make check` and `make fmt` will use the right license for the x-pack folder. Co-authored-by: Carlos Pérez-Aradros Herce <exekias@gmail.com> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> (cherry picked from commit bcb0531)
This commit implements the necessary logic to build licensed beats, before that commit, the OSS and the License binaries were exactly the same. Changes: - Added `mage.CrossBuildXPack()` to build the elastic licensed beat. x-pack/beatname - Changed the packages.yml to include the artifact from the x-pack folder. - Added `mage.UseElasticBeatPackaging()` to build the packages with the new binaries. - Added `mage.UseElasticBeatWithoutXPackPackaging()` allow to keep the previous behavior. - `make check` and `make fmt` will use the right license for the x-pack folder. Co-authored-by: Carlos Pérez-Aradros Herce <exekias@gmail.com> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> (cherry picked from commit bcb0531)
This change brings with-xpack binaries building by doing this:
Introduce
x-pack/<beatname>/main.goand itscmdpackage. These take oss beats RootCmd and inject X-Pack features on it.The existing
magefile.gogains awareness of this and uses it to build and package the x-pack version.I believe this is the minimal change we would need to just have different binaries. If modules are introduced they will require more packaging changes to take them into account.