Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

https://issues.apache.org/jira/browse/ARROW-12980

Errors appear to be due to windows builds not finding tz database. See.

I think it would be good to write some tests in python as well, as currently the C++ tests are very hard to verify since we don't yet have the ability to parse strings localized in the timezone (I mean: the strings in the tests are interpreted as UTC and not the "Australia/Broken_Hill" timezone. And thus as a result, the expected values can't be read/verified from the strings).
(while in python we could create the localized input timestamps with pandas)

Given that, it might also make sense to first add a "localize" kernel for converting timestamps from naive to a certain timezone.

I think it would be good to write some tests in python as well, as currently the C++ tests are very hard to verify since we don't yet have the ability to parse strings localized in the timezone (I mean: the strings in the tests are interpreted as UTC and not the "Australia/Broken_Hill" timezone. And thus as a result, the expected values can't be read/verified from the strings).
(while in python we could create the localized input timestamps with pandas)

Indeed! I've used that approach to generate data in ARROW-11759 by pandas already and I'll just adopt that code here.
It did lead me to an odd issue? in pandas that I cant quite explain. We'll either need to address it or avoid it in tests ..

Given that, it might also make sense to first add a "localize" kernel for converting timestamps from naive to a certain timezone.

Localize kernel would be great. I suppose we'd need a scalar and a vector one depending if timezone is shared between rows or not? Vector version would apply here.

Also strptime kernel ignores timezones at the moment.

Localize kernel would be great. I suppose we'd need a scalar and a vector one depending if timezone is shared between rows or not?

I think a scalar kernel is the most important (with signature like localize(array[timestamp], string tz) -> array[timestamp, tz], where array can also be chunked array or scalar of course). Mixed timezones is a rather corner case (and in principle also could be done as a scalar kernel if the second tz argument is also an array). I think ARROW-5912 is more a bug in the python->arrow conversion code, and not necessarily related / fixable by a localize kernel.

I opened https://issues.apache.org/jira/browse/ARROW-13033 for this.

Another question: is the locate_zone configurable in some way to give some hints where to find the tz database?

The database is not always available on the system (as you noticed for windows). But depending on the application / binding language, it might already be available elsewhere. For example, in recent Python versions, there is now support for this in the standard library, automatically using the tzdata package if there is no system database. I am wondering if pyarrow should be able to instruct the kernel where to look for the database (https://www.python.org/dev/peps/pep-0615/#search-path-configuration in python).
Or does the vendored date.h also include functionalities to automatically download the data if not available on the system?

I opened https://issues.apache.org/jira/browse/ARROW-13033 for this.

Nice, I was also writing it just now :)
The strptime timzone ignoring issue is here.

Or does the vendored date.h also include functionalities to automatically download the data if not available on the system?

As per HowardHinnant's answer:

The library can be configured to not go to the internet for tzdata. tzdata can either be downloaded manually, and this lib can find it, or (on non-Windows systems), the OS-supplied tzdata can be used.

I suppose we could have the kernel try getting data online. If that fails try OS and as a final fallback use arrow bundled tz db (which we would have to add in this PR).

@jorisvandenbossche added the python tests. Will add some more for timezone naive and if needed make another PR :).

Added the timezone-naive test for component extractions.
I'll wait for the consensus to build on the timezone handling discussions before closing the PR and moving the python tests to a new PR.

I'll wait for the consensus to build on the timezone handling discussions before closing the PR and moving the python tests to a new PR.

I think there was no opposition for this one (i.e. that the field extraction should yield local hour/minute/etc), so I don't think there is a need to close the PR.

I think there was no opposition for this one (i.e. that the field extraction should yield local hour/minute/etc), so I don't think there is a need to close the PR.

Huh, I forget exactly what I was thinking there and you appear to be right.

I think the main remaining task is the tz db issue. I'll take a look at it tomorrow.

@pitrou Thanks for the review! And sorry for late reply.
Could you please review again and suggest if templating could be further improved.

@pitrou thanks for the review and the changes - those look much better!

Rebased again to pick up AppVeyor fix :-/

Thanks @pitrou @jorisvandenbossche and @kszucs !