By "stats" I mean the internal numbers gathered by the VM for performance and debugging. This has nothing to do with any statistics module.

Currently various parts of the VM gather stats: the GC, dicts, the bytecode interpreter, type lookup cache, etc.

These stats have various compile time flags to turn them on and off. They have differing ways to display the stats, and no unified way to gather stats across different runs.

For the specialization stats we dump stats, which we can parse to collate stats across runs.

We should:

1. Add a --with-pystats config flag (like with-pydebug) to turn on stat gathering at build time.
2. Convert the other stats to the format used by the specialization stats, so all stats can be parsed and collated.

New changeset 342b93f by Mark Shannon in branch 'main':
bpo-46072: Add --with-pystats configure option to simplify gathering of VM stats (GH-30116)
342b93f

Could you please add the new option to Doc/using/configure.rst ?

New changeset 4506bbe by Mark Shannon in branch 'main':
bpo-46072: Document --enable-stats option. (GH-30139)
4506bbe

I just noticed that you are using hard-coded paths with /tmp for the pystats directory. That's problematic and opens the possibility of a symlink race attack.

Could please add exclusive create to _Py_PrintSpecializationStats()? The will prevent symlink attacks. fopen() mode "x" is not generally available in all libcs. You have to combine open() and fdopen():

int flags = O_WRONLY | O_CREAT | O_EXCL;
#ifdef O_NOFOLLOW
flags |= O_NOFOLLOW;
#endif
#ifdef O_CLOEXEC
flags |= O_CLOEXEC;
#endif

int fd = open(path, flags);
if (fd >= 0) {
    FILE *fout = fdopen(fd, "w");
}

The --enable-stats option is for CPython development and shouldn't be turned on for a release version, so I'm not really concerned about people attacking their own machines.

New changeset efd6236 by Mark Shannon in branch 'main':
bpo-46072: Add top level stats struct (GH-30169)
efd6236

New changeset 90ab138 by Mark Shannon in branch 'main':
bpo-46072: Add simple stats for Python calls. (GH-30989)
90ab138

New changeset 48be46e by Mark Shannon in branch 'main':
bpo-46072: Add some object layout and allocation stats (GH-31051)
48be46e

New changeset 187930f by Mark Shannon in branch 'main':
bpo-46072: Add some frame stats. (GH-31060)
187930f

New changeset 062460e by Mark Shannon in branch 'main':
bpo-46072: Improve LOAD_METHOD stats (GH-31104)
062460e

New changeset 9c979d7 by Mark Shannon in branch 'main':
bpo-46072: Merge dxpairs into py_stats. (GH-31197)
9c979d7

New changeset f71a69a by Mark Shannon in branch 'main':
bpo-46072: Output stats as markdown with collapsible sections. (GH-31228)
f71a69a

New changeset 580cd9a by Brandt Bucher in branch 'main':
bpo-46072: Add detailed failure stats for BINARY_OP (GH-31289)
580cd9a

Python API should be provided? This allows us to have more accurate statistical information about the status.

For example, in pyperformance bm_hexiom:

import _opcode as stats
def main(loops, level):
    ...
    ...

    stats.init_specialization_stats() # Clear unrelated status information
    stats.enable_specialization_stats()
    for _ in range_it: # core loop of benchmark
        stream = io.StringIO()
        solve_file(board, strategy, order, stream)
        output = stream.getvalue()
        stream = None
    stats.disable_specialization_stats()
    print(stats.get_specialization_stats())# Print status information
    ...
    ...
    return dt

Then get stats dict:

{'load_attr': {'success': 55, 'failure': 4, 'hit': 72355, 'deferred': 168, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'load_global': {'success': 81, 'failure': 0, 'hit': 47503, 'deferred': 0, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'load_method': {'success': 36, 'failure': 0, 'hit': 36698, 'deferred': 0, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'binary_subscr': {'success': 47, 'failure': 48, 'hit': 78278, 'deferred': 2488, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'store_subscr': {'success': 9, 'failure': 0, 'hit': 3610, 'deferred': 0, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'store_attr': {'success': 9, 'failure': 0, 'hit': 671, 'deferred': 0, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'call': {'success': 54, 'failure': 9, 'hit': 36321, 'deferred': 1079, 'miss': 783, 'deopt': 14, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'binary_op': {'success': 43, 'failure': 4, 'hit': 11646, 'deferred': 16, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'compare_op': {'success': 37, 'failure': 480, 'hit': 24938, 'deferred': 30707, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 480, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'unpack_sequence': {'success': 2, 'failure': 0, 'hit': 122, 'deferred': 0, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)}, 'precall': {'success': 82, 'failure': 9, 'hit': 79148, 'deferred': 234, 'miss': 0, 'deopt': 0, 'failure_kinds': (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0)}}

This helps us to more precisely analyze the impact of optimization on benchmark.