dateutil mishandles today's dates on NetBSD 9.0 etc. #1059
Description
#462 talks about a longstanding bug in dateutil. In 1995, tzcode 95f introduced a 64-bit extension to TZif files, designed to make them work after the year 2038. dateutil's TZif parser does not use this data, causing it to mishandle timestamps after the last explicit 32-bit transition in a TZif file, which means timestamps after 2038 in (say) Los Angeles are mishandled by dateutil. I'm sure you've been meaning to fix this before 2038 rolls around.
Something new has happened, though. A year ago, tzdb 2019b introduced the '-b slim' flag to the zic command, causing it to omit unnecessary data, the idea being to speed up the parsing of TZif files and make them smaller. Since the 32-bit data entries are always unnecessary, they're omitted. NetBSD 9.0, dated 2020-02-14, is using the '-b slim' flag to generate its TZif files, which means dateutil does not work even for today's timestamps on NetBSD. I assume other platforms will follow suit.
This change means it's time to boost the priority of fixing this bug.
To reproduce the problem, run the shell script datetime-slim-bug.sh (see below) in a writeable directory. It will create a slim TZif file 'Los_Angeles' and will use it to convert the time_t value 1592782632 into Los Angeles time. GNU 'date' handles this correctly, but datetime gets confused and thinks that Los Angeles is observing UTC. On my platform (Fedora 31 x86-64, Python 3.7.7, datetime 2.8.0) this script outputs:
2020-06-21 16:37:12-07:00
2020-06-21 23:37:12+00:00
The first line (from GNU 'date') is correct; the second line (from datetime) is wrong).
Here are the contents of datetime-slim-bug.sh:
#!/bin/sh
export LC_ALL=C
PWD=`pwd`
base64 -d >Los_Angeles <<'EOF'
VFppZjIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAEAAAAAAAAAVFppZjIA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB9AAAABQAAABT/////XgQawP////+epkig////
/5+7FZD/////oIYqoP////+hmveQ/////8uJGqD/////0iP0cP/////SYSYQ/////9b+dFz/////
2ICtkP/////a/sOQ/////9vAkBD/////3N6lkP/////dqayQ/////96+h5D/////34mOkP/////g
nmmQ/////+FpcJD/////4n5LkP/////jSVKQ/////+ReLZD/////5Sk0kP/////mR0oQ/////+cS
URD/////6CcsEP/////o8jMQ/////+oHDhD/////6tIVEP/////r5vAQ/////+yx9xD/////7cbS
EP/////ukdkQ/////++v7pD/////8HG7EP/////xj9CQ//////J/wZD/////82+ykP/////0X6OQ
//////VPlJD/////9j+FkP/////3L3aQ//////goohD/////+Q9YkP/////6CIQQ//////r4gyD/
////++hmEP/////82GUg//////3ISBD//////rhHIP//////qCoQAAAAAACYKSAAAAAAAYgMEAAA
AAACeAsgAAAAAANxKJAAAAAABGEnoAAAAAAFUQqQAAAAAAZBCaAAAAAABzDskAAAAAAHjUOgAAAA
AAkQzpAAAAAACa2/IAAAAAAK8LCQAAAAAAvgr6AAAAAADNnNEAAAAAANwJGgAAAAAA65rxAAAAAA
D6muIAAAAAAQmZEQAAAAABGJkCAAAAAAEnlzEAAAAAATaXIgAAAAABRZVRAAAAAAFUlUIAAAAAAW
OTcQAAAAABcpNiAAAAAAGCJTkAAAAAAZCRggAAAAABoCNZAAAAAAGvI0oAAAAAAb4heQAAAAABzS
FqAAAAAAHcH5kAAAAAAesfigAAAAAB+h25AAAAAAIHYrIAAAAAAhgb2QAAAAACJWDSAAAAAAI2ra
EAAAAAAkNe8gAAAAACVKvBAAAAAAJhXRIAAAAAAnKp4QAAAAACf+7aAAAAAAKQqAEAAAAAAp3s+g
AAAAACrqYhAAAAAAK76xoAAAAAAs036QAAAAAC2ek6AAAAAALrNgkAAAAAAvfnWgAAAAADCTQpAA
AAAAMWeSIAAAAAAycySQAAAAADNHdCAAAAAANFMGkAAAAAA1J1YgAAAAADYy6JAAAAAANwc4IAAA
AAA4HAUQAAAAADjnGiAAAAAAOfvnEAAAAAA6xvwgAAAAADvbyRAAAAAAPLAYoAAAAAA9u6sQAAAA
AD6P+qAAAAAAP5uNEAAAAABAb9ygAAAAAEGEqZAAAAAAQk++oAAAAABDZIuQAAAAAEQvoKAAAAAA
RURtkAAAAABF89MgAgECAQIDBAIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIB
AgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgECAQIBAgEC
AQIBAgECAQIBAgECAQIBAgECAQIBAgH//5EmAAD//52QAQT//4+AAAj//52QAQz//52QARBMTVQA
UERUAFBTVABQV1QAUFBUAApQU1Q4UERULE0zLjIuMCxNMTEuMS4wCg==
EOF
now=1592782632
TZ=$PWD/Los_Angeles date -d@$now +'%Y-%m-%d %H:%M:%S%:z'
python -c 'if 1:
import datetime, dateutil.tz
tz = dateutil.tz.gettz("'"$PWD"'/Los_Angeles");
print(datetime.datetime.fromtimestamp('$now', tz))
'