Hi Ben. It would be very useful to have a test brave-compose.yml file and a test script in test folder. Could you please add them. Thank you!

Hi Ben. It would be very useful to have a test brave-compose.yml file and a test script in test folder. Could you please add them. Thank you!

I am still working on a reproducible script and brave-compose.yml to show an example of the compose command in action, will add them when possible.

Just as a quick update: I've added support for dependencies between services in the compose file - the services will be topologically ordered to ensure that each service is built after the services it depends on. Tests for the topological ordering pass, so it appears to work as expected - let me know if I missed any cases.

Switching build/deploy directory

A further update: I realized that the majority of service Bravefiles will be written from an individual per-component basis and will rely on copying resources into containers from the local dir of that service. When composing these individual components together from a higher directory the relative paths would be incorrect.

To address this, brave compose now by default switches into the directory containing the Bravefile of the service if that is provided when building and deploying an image. This should be the most common case and so should prioritised. If for some reason that needs to be overriden, the new "context" field allows users to define a different working directory for building/deploying on a per-service basis.

Next steps

While working on reproducible examples for brave compose I ran into the following blocking issues:

• Difficulties detaching from a server process if starting that service in Postdeploy Run command. Without detaching, deploy will stop on the first server started. But attempting to start a process in the background that keeps running was hard.
• Passing environment variables into containers before running commands was also a problem.

The first problem is much more severe, as it essentially blocks the use of Compose for spinning up multiple servers. The second is less serious but could probably be addressed along with the first. The primary issue seems to be with how the LXD client (lxc) is sending commands currently in bravetools. I have some ideas on how to make the above cases work with a few additions/changes to command execution.

Hi Ben. It would be very useful to have a test brave-compose.yml file and a test script in test folder. Could you please add them. Thank you!

I've added an example of using brave compose to build and deploy three simple python web services to the examples folder. The three services communicate with each other and are exposed on bravetools host through the API. The "brave-compose.yml" file used in the example is minimal - most of the settings are drawn from the Bravefiles.

Changes since I last checked in:

• Test for compose added to host_api_test.go
• Docs for brave compose added
• Switching to brave-compose.yaml directory before build/deploy to make relative bravefile paths work
• Loading of all Bravefiles happens on compose file load; this prevents errors from incorrect bravefile paths interrupting a deployment as the error will be caught up front.

I'm happy with the current state of things and think it's ready to merge into master.

Potential future additions might be worth considering later, such as brave compose sub-commands that help manage deployments. For example:

• brave compose up
• brave compose down
• brave compose remove

These additions belong in a separate pull request however - the functionality present in the current PR (and its size) is enough to merge it as it is.

Updates:

Skip build phase for existing images

Recent additions to bravetools now raise an error if an image already exists. I've updated brave compose to handle this in what I think is the best way for the tool.

If a service's "build" field is set to true and the image already exists, compose will skip building the image and continue with the rest of the compose operations (deploy etc.). This makes it easy to leave the build flag set to true in the compose file regardless of the state of the system, with it acting as a "build if image doesn't exist" flag. This seems better than the alternative (requiring manually tweaking the compose file to turn off build after first building the images).

"base" field within compose file

This one is speculative, but thought it was worth considering. Sometimes several images might share the same base dependencies, and it would be nice to be able to reuse the same base image as a starting point. It is already possible to first build a base image and have other builds import that local image, but it's a hassle to do and relies on the existing state of the system.

The base field makes this kind of thing possible within a compose file. A service with "base" set to true is not deployed, only built. In addition, since it is primarily a way of accelerating builds, by default a "base" image will automatically be cleaned up after a build - but if you want to keep it, you may set the "build" field to true. Services that depend on the base image can import it as a local image in their Bravefiles - in the compose file they can also express their dependency on the base image.

If every image depending on a base image already exists, the base image build will also be skipped as it is not required.

An example using the base idea to reuse a base image between 3 services has been added to the examples folder - in this very basic case it speeds up the build by 25% on my machine (saving ~1 minute) but as cost of building the base image increases the bigger the time savings will be as you'll only need to build it once instead of multiple times (offsetting the fixed cost to importing/exporting a new image).

Let me know if you think this is worth adding.

This is where I think that compose shouldn't be about "build", but rather, about "create a new unit based on image and then stand up the system". It feels like having image build and compose declared in one place mixes different concepts. Here's what I thought originally:

1. define and build an image. Image can be a) a simple OS, b) prebuilt packages, c) custom built application
   Images are stored in the local or remote repository. Image can have different versions.

2. Use image from repository as-it-is or modify it as needed for deployment. This is where "post-deploy" declarations are coming in. This can be declared in "compose" to build a custom system.

3. There's option "publish" which can export newly customised and deployed unit as a brand new image that can be used in the future as "a base".

Last thought - having various flags, in my opinion, would bring unnecessary complexity and might lead to more complex logic in the Bravetools code.

What do you think?

Thanks for you thoughts @adrozdovbering, it is good to have some feedback. I appreciate the vision of bravetools handling multiple remotes for both image storage and for deploy and also see it as something we need to work towards.

It would also be good to keep the number of flags low if possible! I felt that the "build" and "base" flags required only minimal changes to the code, are not too much to remember, and are justified by their contributions.

Build

I agree that primarily compose should focus more on deploy over build - this is why only "Service" fields are allowed in the compose file. Builds are only allowed when a Bravefile for that service has been provided, keeping the logic of the build process separate. Seeing as the Bravefile is present, it makes sense to be able to (optionally) use it if needed to facilitate the deployment of a service.

In my view having this option to build and deploy using a single command is in-line with the existing Bravefile structure of having build and deploy defined together. It fits in nicely with the idea of compose operating on a higher level, aiding with the process of managing deployments of multiple services by building on top of the lower level commands. It enables scaling up to build a large number services before deploying them, a proceses that otherwise requires manual work (or a script).

Base

The workflow mentioned regarding using build with postdeploy commands, followed by publish and then import and build is an interesting idea - I had not considered doing this! It is quite a different way of managing multi-stage builds. In my opinion it is quite a manual process, plus the order of builds matters and must be considered by the user. The intention is less obvious just from reading the Bravefile.

The strength of handling multi-stage builds in the brave-compose file is that the dependencies between images and their descendants are formally expressed in a declarative way and clearly visible in a single file at a glance - there's no need to read multiple Bravefiles to work out the ordering. In addition, since the order of dependencies is resolved by Bravetools, building a images that depend upon others is more reproducible, removing the possibility of manual errors or missing local images.

Alternatives?

I think the value provided by having these functions in brave compose is clear, but the best way to expose them is still a question. It may be that the idea I mentioned before of brave compose subcommands (like brave compose build?) may be another way of doing this.

But in a way I like the behaviour of the existing set up - it makes deployments truly reproducible by eliminating the need to consider the state of the server before running the command.