Skip to content

Create RUBYCOMMONDIR directory and .RUBYCOMMONDIR.time file in advance.#2669

Closed
junaruga wants to merge 2 commits into
ruby:masterfrom
junaruga:feature/travis-ci-arm64-make-install-error
Closed

Create RUBYCOMMONDIR directory and .RUBYCOMMONDIR.time file in advance.#2669
junaruga wants to merge 2 commits into
ruby:masterfrom
junaruga:feature/travis-ci-arm64-make-install-error

Conversation

@junaruga

Copy link
Copy Markdown
Member

It's to fix the failure in Travis arm64.
This PR is pointed out from https://bugs.ruby-lang.org/issues/16234 .
I tested it 3 times on my forked repository's Travis. And it was okay.

Create RUBYCOMMONDIR (.ext/common) .ext/.timestamp/.RUBYCOMMONDIR.time
file in common.mk before building extensions.
This prevents following error in make install.

Because $(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time file can be touched more than
2 times by each ext/**/Makefile $(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time task
in case of make -jN.

$ make -jN

ext/**/Makefile

$(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time:
    $(Q) $(MAKEDIRS) $(@D) $(RUBYLIBDIR)
    $(Q) $(TOUCH) $@

This causes "Permission defined" error like https://travis-ci.org/ruby/ruby/jobs/606916890#L2412 .

$ make install
...
Prerequisite '../../.ext/.timestamp/.RUBYCOMMONDIR.time' is newer than target '../../.ext/common/json.rb'.
...
cp ../../../ext/json/lib/json.rb ../../.ext/common
cp: cannot create regular file '../../.ext/common/json.rb': Permission denied
  => ../../.ext/common/json.rb already exists as permission 444 (-r--r--r--).
...

The template/Makefile.in TOUCH definition comes from lib/mkmf.rb TOUCH.

@junaruga

Copy link
Copy Markdown
Member Author

Travis is okay.
https://travis-ci.org/ruby/ruby/builds/610461332

…vance.

Create `RUBYCOMMONDIR` (`.ext/common`) `.ext/.timestamp/.RUBYCOMMONDIR.time`
file in `common.mk` before building extensions.
This prevents following error in `make install`.

Because `$(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time` file can be touched more than
2 times by each `ext/**/Makefile` `$(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time` task
in case of `make -jN`.

```
$ make -jN
```

`ext/**/Makefile`

```
$(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time:
    $(Q) $(MAKEDIRS) $(@d) $(RUBYLIBDIR)
    $(Q) $(TOUCH) $@
```

This causes "Permission defined" error like https://travis-ci.org/ruby/ruby/jobs/606916890#L2412 .

```
$ make install
...
Prerequisite '../../.ext/.timestamp/.RUBYCOMMONDIR.time' is newer than target '../../.ext/common/json.rb'.
...
cp ../../../ext/json/lib/json.rb ../../.ext/common
cp: cannot create regular file '../../.ext/common/json.rb': Permission denied
  => ../../.ext/common/json.rb already exists as permission 444 (-r--r--r--).
...
```

The `template/Makefile.in` `TOUCH` definition comes from `lib/mkmf.rb` `TOUCH`.
@junaruga junaruga force-pushed the feature/travis-ci-arm64-make-install-error branch from 0f6ea0c to 3bd3b6a Compare November 11, 2019 17:55
@junaruga

Copy link
Copy Markdown
Member Author

Rebsaed on the latest master branch.

@k0kubun

k0kubun commented Nov 11, 2019

Copy link
Copy Markdown
Member

I believe it's Travis-specific issue. Do you have any other ideas to fix the issue without touching the build system (common.mk, Makefile.in)?

@junaruga

junaruga commented Nov 11, 2019

Copy link
Copy Markdown
Member Author

I believe it's Travis-specific issue. Do you have any other ideas to fix the issue without touching the build system (common.mk, Makefile.in)?

Why do you think so?

I think the reason of the error is

$ make -jN
  => ext/foo/Makefile executed.

For example. In ext/json/Makefile, following part is executed.

$(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time:
    $(Q) $(MAKEDIRS) $(@D) $(RUBYLIBDIR)
    $(Q) $(TOUCH) $@

Then

$ make install

In ext/json/Makefile, following part is executed wrongly, but the part should not be executed in make install in Travis.

$(RUBYLIBDIR)/json.rb: $(srcdir)/lib/json.rb $(TIMESTAMP_DIR)/.RUBYCOMMONDIR.time
    $(Q) $(COPY) $(srcdir)/lib/json.rb $(@D)

So, this issue can be happened when running make -jN in multi CPU cores environment.
I assume as it is -j33 in arm64, it can happen easily.

But if you want to fix the issue without touching the build system (common.mk, Makefile.in), maybe just adding following code fixes it.

$ git diff
diff --git a/.travis.yml b/.travis.yml
index 8b72afaaae..f30c270df7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -453,6 +453,10 @@ before_script:
   - rm -rf ~/config_2nd
   - mv ../config_2nd ~
   - chmod u-w ..
+  - |-
+    if [ "$(uname -m)" = aarch64 ]; then
+      mkdir -p .ext/common .ext/.timestamp && touch .ext/.timestamp/.RUBYCOMMONDIR.time
+    fi
   - $SETARCH make -s $JOBS

Or run make COPY='cp -f' install instead of make install in case of aarch64 case to pass the make install step forcely.

@nobu

nobu commented Nov 12, 2019

Copy link
Copy Markdown
Member

Thank you for the report.
But this doesn't seem to fix the issue enough.

@matzbot matzbot closed this in fb02e31 Nov 12, 2019
@junaruga

junaruga commented Nov 12, 2019

Copy link
Copy Markdown
Member Author

Thank you for your fixing.

Seeing your commit: fb02e31 .

Now I understand that there has already been the similar logic for .ext/$(arch) directory in common.mk.

$(TIMESTAMPDIR)/$(arch)/.time:
    $(Q)$(MAKEDIRS) $(@D) $(EXTOUT)/$(arch)
    @$(NULLCMD) > $@

And my modification was not at a proper place and was not the proper way in it.

Seeing https://travis-ci.org/ruby/ruby/builds after the commit, arm64-linux case is 100% success 4 times.

I think that it's time to remove arm64 from allow_failures again adapting this commit.
212f4d4

@junaruga

Copy link
Copy Markdown
Member Author

I think that it's time to remove arm64 from allow_failures again adapting this commit.

I sent PR for that.
#2672

@junaruga junaruga deleted the feature/travis-ci-arm64-make-install-error branch December 3, 2019 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants