Skip to content

Resource Tracker#741

Merged
pgbrodrick merged 19 commits into
isofit:devfrom
jammont:memory/tracker
Aug 27, 2025
Merged

Resource Tracker#741
pgbrodrick merged 19 commits into
isofit:devfrom
jammont:memory/tracker

Conversation

@jammont

@jammont jammont commented Aug 5, 2025

Copy link
Copy Markdown
Collaborator

Added a new module under debug called resource_tracker.py which implements some utilities to track a system's memory and CPU resources.

This'll be used by isoplots to plot the resources in real time and connect the log file to this to track important events, such as if memory spikes during certain operations, etc

@jammont jammont added the enhancement New feature or request label Aug 5, 2025
@pgbrodrick

pgbrodrick commented Aug 10, 2025

Copy link
Copy Markdown
Collaborator

Looks like a good start - running through some tests, a few notes:

  • cpu usage isn't actually tracked
  • mem_free isn't accurate
  • I'd suggest gathering timestamps, both to verify the sampling interval (in case anything stalls the tracker), and to independently provide per-PID resource net consumption. - realizing this is what 'ts' is

@pgbrodrick pgbrodrick mentioned this pull request Aug 10, 2025
@jammont jammont marked this pull request as ready for review August 13, 2025 06:27
@jammont

jammont commented Aug 13, 2025

Copy link
Copy Markdown
Collaborator Author

cpu usage isn't actually tracked

This is now fixed. CPU usage is defined as the percentage of use over the interval; eg. our default calculates the usage over the last two seconds

I also added some new CPU features:

  • cores parameter to be passed in which is used in the new cpu_avg summary stat
    • If the program uses less cores than the system total, this helps track the efficiency of the program itself
  • sys_cpu_per_core retrieves the CPU usage of every core
    • Summary stat sys_cpu to get the average of these

mem_free isn't accurate

This has been replaced with mem_avail. Previously, mem_free just reported the amount of memory that was completely unused and didn't account for memory that was actually available on the system. We care about availability, not unused


Some other changes:

  • The first row written to the jsonl contains information that doesn't change, such as cores, mem_unit, mem_total, polling_interval, etc -- may be useful to power users
  • The memory units is now configurable. Default is GB but could be changed to anything else, if that's ever desired
  • ts renamed to timestamp for clarity

@pgbrodrick

Copy link
Copy Markdown
Collaborator

Working great for me now, thanks for the improvements!

@pgbrodrick pgbrodrick merged commit d33010e into isofit:dev Aug 27, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants