Add missing startup flags in runtime/start event by sywhang · Pull Request #55151 · dotnet/runtime

sywhang · 2021-07-04T20:20:42Z

Add missing startup flags in Runtime/Start event.

This should fix #52042.

josalem · 2021-07-06T15:37:23Z

At first I thought this might be a profiler error, but it looks like all the test logs indicate those passed.

@davmason could you validate that? one said:

Profiler detach succeeded
PROFILER TEST PASSES
Profiler did not set the callback, test will fail.

Looking at the Helix logs, I see some python errors:

+ /home/helixbot/.azdo-env/bin/python -c import azure.devops
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'azure'
+ /home/helixbot/.azdo-env/bin/python -m pip install azure-devops==5.0.0b9
WARNING: The directory '/home/helixbot/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting azure-devops==5.0.0b9
  Downloading azure_devops-5.0.0b9-py3-none-any.whl (1.3 MB)
Collecting msrest<0.7.0,>=0.6.0
  Downloading msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting requests-oauthlib>=0.5.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
  Downloading isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests~=2.16
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib/python3.6/http/client.py", line 459, in read
    n = self.readinto(b)
  File "/usr/lib/python3.6/http/client.py", line 503, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 319, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 128, in resolve
    requirements, max_rounds=try_to_avoid_resolution_too_deep
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 473, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 367, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_criteria_to_update(candidate)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 203, in _get_criteria_to_update
    name, crit = self._merge_into_criterion(r, parent=candidate)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _merge_into_criterion
    if not criterion.candidates:
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/resolvelib/structs.py", line 139, in __bool__
    return bool(self._sequence)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in __bool__
    return any(self)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 129, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 33, in _iter_built
    candidate = func()
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 205, in _make_candidate_from_link
    version=version,
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 312, in __init__
    version=version,
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 151, in __init__
    self.dist = self._prepare()
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 234, in _prepare
    dist = self._prepare_distribution()
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 318, in _prepare_distribution
    self._ireq, parallel_builds=True
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 552, in _prepare_linked_requirement
    self.download_dir, hashes
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 243, in unpack_url
    hashes=hashes,
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 102, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/network/download.py", line 157, in __call__
    for chunk in chunks:
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/cli/progress_bars.py", line 152, in iter
    for x in it:
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_internal/network/utils.py", line 86, in response_chunks
    decode_content=False,
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/helixbot/.azdo-env/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
pip._vendor.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
+ /home/helixbot/.azdo-env/bin/python -c import future
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'future'
+ /home/helixbot/.azdo-env/bin/python -m pip install future==0.17.1
WARNING: The directory '/home/helixbot/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting future==0.17.1
  Downloading future-0.17.1.tar.gz (829 kB)
Building wheels for collected packages: future
  Building wheel for future (setup.py): started
  Building wheel for future (setup.py): finished with status 'done'
  Created wheel for future: filename=future-0.17.1-py3-none-any.whl size=488741 sha256=c204ff47dd9c6fc6fc0a16b250728b6c530dc26e7f8b769f886f64c31a06e9bb
  Stored in directory: /tmp/pip-ephem-wheel-cache-lgt_h3g8/wheels/c4/f0/e2/8e4ecc9e1b12a428a7657ba683576d3e79d0a75982f63e8fd2

@dotnet/runtime-infrastructure it looks like there were some permissions issues while pip was installing packages. Is this ignorable? I'm going to rerun the job and see if it happens again.

ViktorHofer · 2021-07-06T15:39:08Z

cc @dotnet/dnceng

MattGal · 2021-07-06T15:47:27Z

cc @dotnet/dnceng

The actual error here is not that warning, rather the Azure Devops test result API hanging up on us:

pip._vendor.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')"

Please consider including correlation id with such comments, it saves time.

MattGal · 2021-07-06T15:55:30Z

I took a look, and it's not exactly what I thought, rather a network problem talking to pypi.org not AzDO. However, it's still a piece that is being re-written to be more reliable and should be phased out quite soon (like, next-promoted arcade dependency flow soon).

From the console log it's indeed passing the test, then failing to install the dependencies for the reporter (thing which is being phased out).
The momentary pip outage breaks reporting, and since we don't know whether we reported or not (due to the exit-code coercion runtime does to make 1s into 0s), we fail.

@ChadNedzlek 's changes to use the new built-in AzDO reporting are on their way, until then you'll need to retry when pypi has an outage (and perhaps updating your image to ubuntu-18.04-helix-arm32v7-20210628192820-97d8652 would help as well)

add missing startup flags in runtime/start event

ef5c064

ghost added the area-Tracing-coreclr label Jul 4, 2021

josalem approved these changes Jul 6, 2021

View reviewed changes

josalem added this to the 6.0.0 milestone Jul 6, 2021

josalem merged commit 86b7f92 into dotnet:main Jul 6, 2021

sywhang deleted the 52042 branch July 6, 2021 22:23

ghost locked as resolved and limited conversation to collaborators Aug 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missing startup flags in runtime/start event#55151

Add missing startup flags in runtime/start event#55151
josalem merged 1 commit intodotnet:mainfrom
sywhang:52042

sywhang commented Jul 4, 2021

Uh oh!

josalem commented Jul 6, 2021

Uh oh!

ViktorHofer commented Jul 6, 2021

Uh oh!

MattGal commented Jul 6, 2021

Uh oh!

MattGal commented Jul 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sywhang commented Jul 4, 2021

Uh oh!

josalem commented Jul 6, 2021

Uh oh!

ViktorHofer commented Jul 6, 2021

Uh oh!

MattGal commented Jul 6, 2021

Uh oh!

MattGal commented Jul 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants