Skip to content

Fix ZEO cache tracing wrt 2038 year#246

Merged
navytux merged 1 commit intomasterfrom
y/trace-2038
Jan 28, 2026
Merged

Fix ZEO cache tracing wrt 2038 year#246
navytux merged 1 commit intomasterfrom
y/trace-2038

Conversation

@navytux
Copy link
Contributor

@navytux navytux commented Jan 27, 2026

@bmwiedemann reports that ZEO tests fail in year 2038. From #245:

Failure in test cache_simul_properly_handles_load_miss_after_eviction_and_inval (ZEO.tests.test_cache)
Failed doctest test for ZEO.tests.test_cache.cache_simul_properly_handles_load_miss_after_eviction_and_inval
  File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/tests/test_cache.py", line 1015, in cache_simul_properly_handles_load_miss_after_eviction_and_inval

----------------------------------------------------------------------
File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/tests/test_cache.py", line 1021, in ZEO.tests.test_cache.cache_simul_properly_handles_load_miss_after_eviction_and_inval
Failed example:
    cache = ZEO.cache.ClientCache('cache', 1<<21)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.11/doctest.py", line 1355, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest ZEO.tests.test_cache.cache_simul_properly_handles_load_miss_after_eviction_and_inval[1]>", line 1, in <module>
        cache = ZEO.cache.ClientCache('cache', 1<<21)
      File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/cache.py", line 239, in __init__
        self._setup_trace(path)
      File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/cache.py", line 823, in _setup_trace
        _trace(0x00)
      File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/cache.py", line 814, in _trace
        pack(">iiH8s8s",
    struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Unfortunately as ZEO/scripts/cache_stats.py and ZEO/scripts/cache_simul.py show ZEO cache-trace files do not have header nor magic, so we cannot change format of time field from 32-bit integer to 64-bit integer without backward compatibility breakage. But there is a hack: we can change the format from 32-bit signed int to 32-bit unsigned int and gain support till 2106 in backward compatible manner:

In [11]: time.ctime((1 << 32) - 1)
Out[11]: 'Sun Feb  7 09:28:15 2106'

In [12]: time.ctime((1 << 31) - 1)
Out[12]: 'Tue Jan 19 06:14:07 2038'

-> Do that to keep binary compatibility and give us another 70 years to find a proper solution similarly to how mariadb did it.

/reported-and-helped-by @bmwiedemann
/fixes #245

  • I signed and returned the Zope Contributor Agreement, and received and accepted an invitation to join a team in the zopefoundation GitHub organization.
  • I verified there aren't any other open pull requests for the same change.
  • I followed the guidelines in Developer guidelines.
  • I successfully ran code quality checks on my changes locally.
  • I successfully ran tests on my changes locally.
  • If needed, I added new tests for my changes.
  • If needed, I added documentation for my changes.
  • I included a change log entry in my commits.

/cc @icemac, @d-maurer

Closes #245

@navytux navytux mentioned this pull request Jan 27, 2026
Closed
@coveralls
Copy link

coveralls commented Jan 27, 2026 •
edited
Loading

Pull Request Test Coverage Report for Build 21439552802

Details

  • 2 of 2 (100.0%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.02%) to 73.245%
Totals Coverage Status
Change from base Build 21429121949: 0.02%
Covered Lines: 7783
Relevant Lines: 10107
💛 - Coveralls

@navytux
Copy link
Contributor Author

navytux commented Jan 27, 2026

I'm not sure what happens with linting failure with

  File "/home/runner/.cache/pre-commit/repot439s7fz/py_env-python3.14/lib/python3.14/site-packages/pyupgrade/_main.py", line 297, in _fix_tokens
190
    tokenize.cookie_re.match(token.src)
191
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
192
TypeError: cannot use a bytes pattern on a string-like object

This seems unrelated to my changes.

All other QA checks passed ok.

@navytux
Copy link
Contributor Author

navytux commented Jan 27, 2026

@bmwiedemann do you confirm that this patch fixes your original issue?

@perrinjerome
Copy link
Contributor

perrinjerome commented Jan 27, 2026

I can confirm that running the test suite with faketime is enough to reproduce the problem and that it is fixed until year 2106 with this patch. I used a command like that:

faketime 2040-01-01 zope-testrunner -u --test-path=src ZEO.tests.test_cache -v

For the linter failure I think it's needed to run the "meta" tool to update the supported python versions, I tried to do this in #247

@navytux
Copy link
Contributor Author

navytux commented Jan 27, 2026

Thanks!

bmwiedemann
bmwiedemann approved these changes Jan 28, 2026
View reviewed changes
Copy link

@bmwiedemann bmwiedemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test was successful with the date set to 2042-03-01

Total: 731 tests, 0 failures, 0 errors and 1 skipped in 3 minutes 29.089 seconds.

Independent of this PR, you might want to reduce code-duplication to only have this pack format specified in a single place. => DRY

@navytux
Fix ZEO cache tracing wrt 2038 year
7ea0181 
@bmwiedemann reports that ZEO tests fail in year 2038. From
#245:

    Failure in test cache_simul_properly_handles_load_miss_after_eviction_and_inval (ZEO.tests.test_cache)
    Failed doctest test for ZEO.tests.test_cache.cache_simul_properly_handles_load_miss_after_eviction_and_inval
      File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/tests/test_cache.py", line 1015, in cache_simul_properly_handles_load_miss_after_eviction_and_inval

    ----------------------------------------------------------------------
    File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/tests/test_cache.py", line 1021, in ZEO.tests.test_cache.cache_simul_properly_handles_load_miss_after_eviction_and_inval
    Failed example:
        cache = ZEO.cache.ClientCache('cache', 1<<21)
    Exception raised:
        Traceback (most recent call last):
          File "/usr/lib64/python3.11/doctest.py", line 1355, in __run
            exec(compile(example.source, filename, "single",
          File "<doctest ZEO.tests.test_cache.cache_simul_properly_handles_load_miss_after_eviction_and_inval[1]>", line 1, in <module>
            cache = ZEO.cache.ClientCache('cache', 1<<21)
          File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/cache.py", line 239, in __init__
            self._setup_trace(path)
          File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/cache.py", line 823, in _setup_trace
            _trace(0x00)
          File "/home/abuild/rpmbuild/BUILD/python-ZEO-6.1-build/zeo-6.1/src/ZEO/cache.py", line 814, in _trace
            pack(">iiH8s8s",
        struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Unfortunately as ZEO/scripts/cache_stats.py and ZEO/scripts/cache_simul.py show
ZEO cache-trace files do not have header nor magic, so we cannot change format
of time field from 32-bit integer to 64-bit integer without backward
compatibility breakage. But there is a hack: we can change the format from
32-bit signed int to 32-bit unsigned int and gain support till 2106 in backward
compatible manner:

    In [11]: time.ctime((1 << 32) - 1)
    Out[11]: 'Sun Feb  7 09:28:15 2106'

    In [12]: time.ctime((1 << 31) - 1)
    Out[12]: 'Tue Jan 19 06:14:07 2038'

-> Do that to keep binary compatibility and give us another 70 years to find a
proper solution similarly to how mariadb did it.

/reported-and-helped-by @bmwiedemann
/reviewed-by @perrinjerome, @dataflake, @bmwiedemann
/reviewed-on #246
/fixes #245
@navytux navytux merged commit 5c9291a into master Jan 28, 2026
26 checks passed
@navytux navytux deleted the y/trace-2038 branch January 28, 2026 13:23
@navytux
Copy link
Contributor Author

navytux commented Jan 28, 2026

Thanks, everyone. I've rebased and merged the patch. For cache trace records format deduplication see #248.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@perrinjerome perrinjerome perrinjerome approved these changes

@dataflake dataflake dataflake approved these changes

+1 more reviewer

@bmwiedemann bmwiedemann bmwiedemann approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

tests fail in year 2038

5 participants

@navytux @coveralls @perrinjerome @bmwiedemann @dataflake