@wuisawesome can you help me take a look at the following error: https://travis-ci.com/github/ray-project/ray/jobs/456604099

It seems to be a checkstyle error but not sure what is triggering it.

@acxz that test is flakey, we can safely just ignore it.

Got it, let's see what the CI says about the numpy version bump and then I believe all the tests will be passing.

Changing the object size is a bit weird, as in python38 80 GBs was just fine, however, with py39 we need a slight bit more. I tried both 96 and 82 and changing those values allows the JAVA_TESTS run to pass, but it breaks quite a few others, change a very small amount seems to do the trick, but now its not a nice round number :(.

Do you have any thoughts on this? Here is the particular run: https://travis-ci.com/github/ray-project/ray/jobs/454107657 If @kfstorm could comment as well that would be appreciated as he is the one who wrote the test that fails without the +100 fix in this commit: f1098ab

@acxz The Java test failure you saw was discussed here and the root cause was explained here. It has been fixed by #10762.

I see that the CI link you provided is 6 days ago. I suggest you revert Java code changes and see the latest CI results.

@kfstorm thanks for commenting! I'll do just that.

looks like the linux wheels are building just fine now, bumping the numpy version did the trick

@wuisawesome can help you me take a look at the macos test: https://travis-ci.com/github/ray-project/ray/jobs/457112601?
It seems like it is not able to find setproctitle, but I'm not sure why. This one is the last nonflakey test left to resolve.

Hi @acxz, can we try print out sys.path before

 File "python/ray/_raylet.pyx", line 21, in init ray._raylet
    import setproctitle

@simon-mo sure,

this is what we get: https://travis-ci.com/github/ray-project/ray/jobs/458830682#L5713

['/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/ray/thirdparty_files', '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/ray/pickle5_files', '/Users/travis/build/ray-project/ray/python/ray/tests', '/Library/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages']

seems like ../ray/thirdparty_files is in our sys.path

After digging into this, I found out that the latest setproctitle from pypi has built wheels unlike the last version. This means that we can just install it from pypi and specify it ourselves in the install_requires instead of having to build it ourselves and listing it under thirdparty_files. This resolves the error for the macos wheels.

Although it seems that the windows build is still trying to build them and is failing in doing so.

After digging into this, I found out that the latest setproctitle from pypi has built wheels unlike the last version. This means that we can just install it from pypi and specify it ourselves in the install_requires instead of having to build it ourselves and listing it under thirdparty_files. This resolves the error for the macos wheels.

Although it seems that the windows build is still trying to build them and is failing in doing so.

@amogkam do you have any ideas about resolving this error on the windows build? Its the same error you resolved in #12415 by pinning the setproctitle. We can't do that for py39 so if you have any ideas on how to properly resolve it that would be great.

Looks like python wheels are coming soon for windows: dvarrazzo/py-setproctitle#90
Not on pypi yet tho, but we can wait for it.

Hi @acxz you know more about this than I do :). I don't know what the windows error actually means, I just pinned setproctitle and it resolved the issue so I didn't look into this further.

One thing you can try is just using a different codepath for windows vs. mac/linux? In install-dependencies.sh you can do if [ "${OSTYPE}" = msys ]; to check for windows and in setup.py you can do if sys.platform == "win32". But maybe this workaround might break even more things...

Also, why doesn't pinning setproctitle and building from source work for py39?

One thing you can try is just using a different codepath for windows vs. mac/linux? In install-dependencies.sh you can do if [ "${OSTYPE}" = msys ]; to check for windows and in setup.py you can do if sys.platform == "win32". But maybe this workaround might break even more things...

Yeah that might break more things, I'll try to wait and look into this again once the setproctitle wheels are out on pip.

Also, why doesn't pinning setproctitle and building from source work for py39?

Thats a good question, which I don't completely know the answer to. Basically for some reason, when we built setproctitle from source for macos py39, ray couldn't find it when we tried to import it. Every other case worked: windows-{36,37,38,39}, macos-{36,37,38}, and linux-{36,37,38,39}
I spent some time debugging it but couldn't make head way and I thought maybe something weird is happening with older versions of setproctitle on 39, so I decided to just use the latest version. And I still got the same issue. I then realized that setproctitle has wheels now, so I decided to use the wheels instead of building it ourselves. But they don't have windows wheels out yet, its merged but not on pypi.

Got it, thanks for the clarification! So it sounds like we just want to wait for the setproctitle windows wheels on pypi. Did they say when this was going to happen?

Also one thing I'm confused about: it looks like there's wheels only for linux and even mac doesn't have wheels yet? https://github.com/dvarrazzo/py-setproctitle/blob/master/.github/workflows/packages.yml, dvarrazzo/py-setproctitle#47 (comment)

Got it, thanks for the clarification! So it sounds like we just want to wait for the setproctitle windows wheels on pypi. Did they say when this was going to happen?

Also one thing I'm confused about: it looks like there's wheels only for linux and even mac doesn't have wheels yet? https://github.com/dvarrazzo/py-setproctitle/blob/master/.github/workflows/packages.yml, dvarrazzo/py-setproctitle#47 (comment)

I think it's because they need funding for CI/building wheels for mac: dvarrazzo/py-setproctitle#47 (comment)

Did they say when this was going to happen?

Nope, but it'll be whenever they sync their github repo version 1.2.2 to pypi (which is still on 1.2.1)

Also one thing I'm confused about: it looks like there's wheels only for linux and even mac doesn't have wheels yet?

whoa yeah you are right, welp now I'm even more confused. Could there be something wrong with our macos tests then?
I wonder how they passed: https://travis-ci.com/github/ray-project/ray/jobs/461527463?

Yeah I'm not sure what's going on...this is weird.

One thing we can do is just leave it as before with setproctitle v1.1.10 and building from source, and then just debug why this didn't work on mac with python 3.9. If you have a mac this might be easier to reproduce/debug instead of windows issues.

One thing we can do is just leave it as before with setproctitle v1.1.10 and building from source, and then just debug why this didn't work on mac with python 3.9. If you have a mac this might be easier to reproduce/debug instead of windows issues.

I have neither...

I am of the opinion if both versions of a library are breaking, we should put effort in fixing the later version instead of the older unsupported version. Besides py39 support for setproctitle was only added in 1.2, so we cannot use the older 1.1.10 for py39 anyway.

We should fix the windows error.

Now ofc debugging the source build with latest setproctitle with py39 is another equivalent path forward, but if the wheels work (even though we don't know why, the tests still pass [potentially issues with the macos test should be opened in another issue/PR]) and if windows wheels are coming soon, I would rather just wait.

Are there official plans yet as to when Python 3.9 support will be merged to master?

not that I know of until our dependencies get on python 3.9 themselves, specifically setproctitle.
My personal goal is to get this merged in for the next release (2.0).

actually it seems like setproctitle released python 3.9 builds (https://pypi.org/project/setproctitle/#files)!

I'll clean this PR up and hopefully the windows tests pass through this time

If we can get this to work on Linux (even without OS X or Windows support), I think that's a good enough/useful starting point to merge.

(slight nit: releases in the near future will still be 1.X, we just changed the nightly to 2.0.0.dev0 so that the url doesn't have to change every release, I'll make an announcement once I publish to 1.2.0 wheels to PyPI).

(if it doesn't work on all of them, we'll only put the working ones in the table, then merge this)

sounds like a plan

@wuisawesome not really sure about the buildkite tests (seems like a new one that has been added since I created this PR). But the rest of the tests pass!

Can you take another look at this and take the appropriate action?

@wuisawesome we discussed a little offline, but introducing the dependency on setproctitle will require compilation in some cases, which will make pip install ray flaky. So I think we should continue to ship setproctitle with Ray in those cases (unless we have a better solution).

So I think we should continue to ship setproctitle with Ray in those cases

With #14538 we are building a py39 compat setproctitle with Ray so there is no need for using the wheels provided by pypi anymore. As such this PR has been greatly simplified.

Looks like the mac wheel build is failing. Let me try pulling in latest master and see if that helps, but we may have to bump some timeouts/limits? cc @simon-mo

I'm going to merge this. @wuisawesome can you monitor the build? the timeout should go away after the bazel cache is propogated.

@acxz thanks so much for all the work you put into this and your patience here. I think many members of the community appreciate it :)

Yes, this is really great!

Thank you everyone for your reviews and suggestions!