Unfortunately I think we have to deprecate this first and warn about it, before removing it.

Yes, I think this is the same change we made a while back, and it broke tons of Dockerfiles. We then reverted that change, knowing it's ugly, but probably something we'll have to accept being something we can't change.

The (probably confusing) part is that comment lines here are Dockerfile comments, not shell comments

I'm curious how this same change broke tons of Dockerfiles. My Dockerfiles work just fine with or without comments between continuation lines right now. It's just that the build command started complaining today (when I updated to Docker for Mac's latest build) about having supposedly blank lines between continuations, which are in fact non-blank lines which just have a comment.

It's just that the build command started complaining today (when I updated to Docker for Mac's latest build) about having supposedly blank lines between continuations, which are in fact non-blank lines which just have a comment.

Hm, that's something that should be fixed, probably requires detecting if it was an empty file, or a comment line.

I'm curious how this same change broke tons of Dockerfiles. My Dockerfiles work just fine with or without comments between continuation lines right now.

It's kinda horrid, but the use case is to allow blank lines in RUN statements. Basically after the first line-continuation character, empty lines, comments are allowed. See #24725 and #29005 (I summarised some things in this comment: #29005 (comment))

Why would it be horrid to allow blank lines in RUN statements? It is a really good way of reducing layers in an image and keep readability at the same time.

Example:

RUN set -ex ; \  
  apk --no-cache --virtual .setup_dependencies add \
    build-base \
    autoconf \
    pcre-dev \
    libxml2-dev \
    libpng-dev \
    cyrus-sasl-dev \
    openssl; \

  curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer --version=${COMPOSER_VERSION}; \
  chmod +x /usr/local/bin/composer; \

  docker-php-ext-install bcmath pdo_mysql; 

I think the expected behaviour here is what is supported by a shell. Ignoring blank lines and allowing comment lines in continuations does not match the behaviour of a shell.

Why would it be horrid to allow blank lines in RUN statements?

@baptiste-ls multiple reasons; first of all it breaks the expectation that instructions inside a RUN command follow shell rules. Second; take the following Dockerfiles. Both are valid;

FROM busybox
RUN echo "foo" \

# install the bar package and dependencies
# TODO disabled; the bar package is broken
&& echo "bar" \

# install the baz package 
&& echo "baz"

# Now run some configuration
RUN echo "something else"

FROM busybox
RUN echo "foo" \

# install the bar package and dependencies
# TODO disabled; the bar package is broken
#&& echo "bar" \

# install the baz package 
#&& echo "baz"

# Now run some configuration
RUN echo "something else"

While the first executes echo "something else", the second doesn't

@dnephin yes, but changing that was a breaking change; we can possibly reconsider when/if the here doc proposal is there, but even then, it's gonna be hard.

I do think we should look at the errors for comment-lines, because those should be valid, and not produce a warning

allowing comment lines in continuations does not match the behaviour of a shell.

Untrue. The following script prints 'hi' without error or warning in bash 3.2.25 and 4.4.0, as well as zsh 4.2.6 and 5.0.8:

#!/bin/bash
true && \
    #comment
    echo 'hi'

That's a fair point. I didn't realize that worked until I re-read the description in this PR.

However this doesn't work:

echo hi \
    # comment
    morethingshere

So to support the shell behaviour correctly we'd actually have to fully parse the line as a shell command instead of just shlex the line.

I would guess this broken example is more common. If you want to comment between commands you could just as easily run with bash -e and not bother with && on every line, which would also let you provide comments without the need to support comments on continuation lines.

If you want to comment between commands

This is something I do frequently in my docker files to explain why I do some operations in multi-line/mutli-command RUN statments.

I also use full line comments (sometimes multi-line comments) to describe sections of a large multi-line RUN command all the time. Please accept this PR and don't deprecate the usage.

Given how our instructions work and are parsed, we cannot reasonably support comments only in the same way the shell does (which is the reason for our divergence from the shell in the first place).

In a normal shell script, if you want a comment in a long block of code, you can simply add it between statements and everything works. In our case, the continuations are not for the shell, but for the Dockerfile instruction, and the only way we can add comments in the middle of a long set of instructions is to ignore them completely when parsing the Dockerfile (because if we parse them as part of the Dockerfile parsing, we do not include any newlines, so the rest of the Dockerfile instruction ends up becoming a shell comment).

For example, in shell, one would write something like this:

set -e

stuff
stuff
# comment
stuff
stuff
# more
# comment
stuff stuff

In a Dockerfile, we're instead forced to do something like this:

RUN set -e; \
	\
	stuff; \
	stuff; \
# comment
	stuff; \
	stuff; \
# more
# comment
	stuff stuff

Which is fine, and still really readable (and fairly easy to write). The flip side is that this actually simply Dockerfile syntax sugar (not shell syntax) for the following:

RUN set -e; 		stuff; 	stuff; 	stuff; 	stuff; 	stuff stuff

(including my tab characters, but not any newlines or comments)

If we included the comments in that parsing (assuming I added \ on the end of all my comment lines), it would instead become:

RUN set -e; 		stuff; 	stuff; # comment 	stuff; 	stuff; # more # comment	stuff stuff

This makes it very clear that everything after that first # comment then gets commented out (since shell comments are entirely line-based, and we literally cannot introduce a newline in our RUN line unless we use the JSON array syntax, and then it's \n instead of something readable like an actual newline character).

Now, there's some discussion in #34423 about making a much easier syntax that would allow for actual newlines in a RUN instruction, thus making comments being passed to the shell not only permissible, but desirable. That entire discussion is orthogonal IMO, however.

The point stands that these comment lines are the only good way we currently have to put comments in the middle of a long RUN instruction, and thus are very important for writing complex but readable Dockerfiles.

As for empty lines, I think the requirement of adding \ is totally sane, as @thaJeztah outlined in his option C over in #29005 (comment) the last time we discussed this change. 😄

I opened #35004 as alternative solution for comment-only lines

I fully support #35004, as it solves this problem much more effectively than mine. I'll close this PR.

Thanks @coredumperror !