Skip to content

new GetGCMemoryInfo API#37879

Merged
Maoni0 merged 4 commits intodotnet:masterfrom
Maoni0:GetMemoryInfo
Jun 18, 2020
Merged

new GetGCMemoryInfo API#37879
Maoni0 merged 4 commits intodotnet:masterfrom
Maoni0:GetMemoryInfo

Conversation

@Maoni0
Copy link
Member

@Maoni0 Maoni0 commented Jun 15, 2020

Most of the changes inside GC are the following -

All the info is updated at the end of a GC with the exception of BGC which needs special care. We maintain 2 BGC info
data structures and one could be in the process of getting updated and if the user requests either BGC info or the last
GC info which is also a BGC, we would return the one that's complete. See comments in code for a more detailed explanation.

Fill in the after size data for all generations instead of just the affected generations.

Timing info recording -

  • Changed the GC time recording from ms to us. Previously we did the timing analysis with events which also carry their
    own high res timestamps so in GC using ms is good enough. But now since we are returning the timing info we want to
    record it in us instead of ms.
  • Consolidated when we get end_gc_time. It was not including all of the GC work before (eg, decommit eph seg). And moving
    it to do_post_gc makes it easier for getting the GC info.
  • Not including the EE restart time for blocking GCs - the reason being restart takes very little time and by the time we
    get to the actual RestartEE the GC settings could be changed and it's not worth the effort to wait till RestartEE at which
    point we have to spend more effort to figure out which GC to attribute it to.
  • Maintain the running % Pause time in GC info.

@Dotnet-GitSync-Bot
Copy link
Collaborator

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, to please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

@ghost
Copy link

ghost commented Jun 15, 2020

Tagging subscribers to this area: @Maoni0
Notify danmosemsft if you want to be subscribed.

@Maoni0 Maoni0 requested review from PeterSolMS and jkotas June 15, 2020 01:45
@stephentoub
Copy link
Member

Did you accidentally miss including a file containing GCMemoryInfoData?

@Maoni0
Copy link
Member Author

Maoni0 commented Jun 15, 2020

I see it in src/coreclr/src/System.Private.CoreLib/src/System/GC.cs, is that not what you are seeing?

@jkotas
Copy link
Member

jkotas commented Jun 15, 2020

Large number of build legs is failing with:

src/libraries/System.Private.CoreLib/src/System/GCMemoryInfo.cs(20,26): error CS0246: The type or namespace name 'GCMemoryInfoData' could not be found (are you missing a using directive or an assembly reference?)

After closer look, these build errors are in Mono build. So you will need to do something to make the Mono build green.

@PeterSolMS
Copy link
Contributor

I wonder about this:

size_t GetHighPrecisionTimeStamp()
{
int64_t ts = GCToOSInterface::QueryPerformanceCounter();

return (size_t)(ts / qpf_us);

}

qpf_us being an int64_t, there could be a round-off error on some systems. It's probably more precise and efficient to use a double qpf_interval_us = 1e6 / qpf and do return (size_t)(ts * qpf_interval) instead.

@PeterSolMS
Copy link
Contributor

I think the doc comment here needs a few words more:

    /// <summary>
    /// The index of this GC.
    /// </summary>
    public long Index => _data._index;

suggestion:

    /// The index of this GC. Indices start at 1 and increase by 1 for each GC, so the third GC in the execution of the program will have index 3.

But perhaps you can think of something better...

@Maoni0
Copy link
Member Author

Maoni0 commented Jun 16, 2020

@jkotas right, it was just mono build failures; I have since moved the definition of GCMemoryInfoData from src\coreclr\src\System.Private.CoreLib\src\System\GC.cs to src\libraries\System.Private.CoreLib\src\System\GCMemoryInfo.cs, hopefully that addresses the issue. too bad this kind of failures do not show up when we build the coreclr version.

@PeterSolMS I've added details to the comment of Index and also changed the type of qpf_us to double and the return type of GetHighPrecisionTimeStamp to uint64_t.

@Maoni0 Maoni0 force-pushed the GetMemoryInfo branch 8 times, most recently from c17b3c4 to 629ab79 Compare June 16, 2020 03:39
@Maoni0 Maoni0 force-pushed the GetMemoryInfo branch 2 times, most recently from a34d27c to 36f0609 Compare June 16, 2020 06:18
@Maoni0
Copy link
Member Author

Maoni0 commented Jun 16, 2020

@marek-safar, what's the process of making it work for mono for this PR? obviously mono's implementation is not there yet.

@Maoni0
Copy link
Member Author

Maoni0 commented Jun 16, 2020

@PeterSolMS thanks so much for finding those!!! I'll make the corresponding changes a bit later today.

@Maoni0 Maoni0 force-pushed the GetMemoryInfo branch 2 times, most recently from 08e650a to 317192e Compare June 16, 2020 23:22
@jkotas
Copy link
Member

jkotas commented Jun 16, 2020

documenting the new GetGCMemoryInfo API

You should just include the docs under triple-slash comments in the doc. The doc generation tool will use to populate the docs for newly added APIs.

@Maoni0
Copy link
Member Author

Maoni0 commented Jun 16, 2020

okie; I didn't know that was all there is for API doc... I thought we'd have sections that you don't see as code comments like "examples of using this API" and etc.

@stephentoub
Copy link
Member

I thought we'd have sections that you don't see as code comments like "examples of using this API" and etc.

We (and others) can always submit PRs to the docs repo to augment the content further after they're initially populated. We just use the doc comments to seed the minimum so we at least have basic docs for summary, parameters, etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this overload used to work, but it's now being changed to throw?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, because there isn't a mono implementation yet. I did this based on @marek-safar's suggestion but I dunno if I misunderstood him.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Maoni0 I think @marek-safar's suggestion originated from the assumption that this is a new API that we don't yet support and we should just throw exception for now. However, this API turned out to be an extension on the existing GCMemoryInfo, and, as Stephen pointed out, we already support part of it and I don't think we should regress what is already implemented.

We used to just leave fields that we don't yet support uninitialized in the GCMemoryInfo (we might not even support every single field right now) and I remember coreclr was doing the same at some point, when comparing between Windows and Linux, for example. My opinion is that we should leave the fields from the new API with the default value for now and just initialize the fields from the netcore 3.0 version. So instead of throwing, we could just create a new GCMemoryInfoData with just these 5 fields initialized and we'll add later support in the mono runtime for the rest of the api.

Copy link
Member Author

@Maoni0 Maoni0 Jun 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see; makes sense! does this look good to you? I'm throwing PNSE in the overload but changed the old API the way you suggested.

@GSPP
Copy link

GSPP commented Jun 17, 2020

if there's no GC of the kind that happened, then by design you'd get back all 0's including the index. we can mention this in the doc.

That's nasty... Maybe return a nullable value instead? Or bool return and out parameter?

@marek-safar marek-safar requested a review from BrzVlad June 17, 2020 17:50
@Maoni0
Copy link
Member Author

Maoni0 commented Jun 17, 2020

I believe I've addressed all your concerns, @jkotas and @stephentoub, please take a look at the latest commit. the only thing I'm waiting on is the review from mono folks. @BrzVlad I have no experience doing this; just let me know if I did something wrong.

Maoni0 added 4 commits June 17, 2020 17:53
Most of the changes inside GC are the following -

All the info is updated at the end of a GC with the exception of BGC which needs special care. We maintain 2 BGC info
data structures and one could be in the process of getting updated and if the user requests either BGC info or the last
GC info which is also a BGC, we would return the one that's complete. See comments in code for a more detailed explanation.

Fill in the after size data for all generations instead of just the affected generations.

Timing info recording -

+ Changed the GC time recording from ms to us. Previously we did the timing analysis with events which also carry their
  own high res timestamps so in GC using ms is good enough. But now since we are returning the timing info we want to
  record it in us instead of ms.
+ Consolidated when we get end_gc_time. It was not including all of the GC work before (eg, decommit eph seg). And moving
  it to do_post_gc makes it easier for getting the GC info.
+ Not including the EE restart time for blocking GCs - the reason being restart takes very little time and by the time we
  get to the actual RestartEE the GC settings could be changed and it's not worth the effort to wait till RestartEE at which
  point we have to spend more effort to figure out which GC to attribute it to.
+ Maintain the running % Pause time in GC info.
@Maoni0
Copy link
Member Author

Maoni0 commented Jun 18, 2020

this is good to go; I will wait till tomorrow noon and if I don't hear any objections by then I will merge.

thank you all for your reviews; really appreciate your help!

@Maoni0 Maoni0 merged commit 6a8fd0b into dotnet:master Jun 18, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants