Skip to content

DALEC daily to hourly raw file script naive datetime error #354

@oceancolorcoder

Description

@oceancolorcoder

@mabdelmegidjr Most daily files for the cruise process okay, but DALEC_12_2024-05-31_174230Z.TXT (on the Box with configs) has a problem with timezone naive comparison at L101:

Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-05-31_174230Z.TXT
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
... 0%
... 5%
... 10%
... 15%
... 20%
... 25%
... 30%
... 35%
... 40%
... 45%
... 50%
... 55%
... 60%
... 65%
... 70%
... 75%
... 80%
... 85%
... 90%
... 95%
Procesing: 100%
---- Writing Data to new files...
Traceback (most recent call last):
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/algorithms.py", line 1870, in safe_sort
    sorter = values.argsort()
             ^^^^^^^^^^^^^^^^
  File "pandas/_libs/tslibs/timestamps.pyx", line 380, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
TypeError: Cannot compare tz-naive and tz-aware timestamps

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/daurin/GitRepos/HyperCP/Source/raw_Data_Hourly_Cutter.py", line 130, in <module>
    split_csv_hourly(args.file, args.output)
  File "/Users/daurin/GitRepos/HyperCP/Source/raw_Data_Hourly_Cutter.py", line 101, in split_csv_hourly
    for (date, hour), group in df.groupby(["date_group", "hour_group"]):
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 788, in get_iterator
    splitter = self._get_splitter(data, axis=axis)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 799, in _get_splitter
    ids, _, ngroups = self.group_info
                      ^^^^^^^^^^^^^^^
  File "pandas/_libs/properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 946, in group_info
    comp_ids, obs_group_ids = self._get_compressed_codes()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 972, in _get_compressed_codes
    group_index = get_group_index(self.codes, self.shape, sort=True, xnull=True)
                                  ^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 897, in codes
    return [ping.codes for ping in self.groupings]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 897, in <listcomp>
    return [ping.codes for ping in self.groupings]
            ^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/grouper.py", line 621, in codes
    return self._codes_and_uniques[0]
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/groupby/grouper.py", line 692, in _codes_and_uniques
    codes, uniques = algorithms.factorize(  # type: ignore[assignment]
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/algorithms.py", line 832, in factorize
    uniques, codes = safe_sort(
                     ^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/algorithms.py", line 1883, in safe_sort
    ordered = _sort_mixed(values)
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/algorithms.py", line 1936, in _sort_mixed
    nums = np.sort(values[~str_pos & ~none_pos])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 1017, in sort
    a.sort(axis=axis, kind=kind, order=order)
  File "pandas/_libs/tslibs/timestamps.pyx", line 380, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
TypeError: Cannot compare tz-naive and tz-aware timestamps
Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-06-01_125333Z.TXT
(hypercp) daurin@gs616-daurinm1:~/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS$ 

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions