DALEC daily to hourly raw file script error

I tried batching my daily DALEC files with the python script provided by @mabdelmegidjr in /Source and while it sometimes works, at other times it encounters multiple errors parsing date strings in the files. Unclear if it skips the bad lines and moves on or drops the remaining data in the file, or ...? E.g. 2025-05-23 returned no hourly files.

```
(hypercp) daurin@gs616-daurinm1:~/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS$ ./batch_daily2hourly_raw.sh 
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
... 0%
... 5%
... 10%
... 15%
... 20%
... 25%
... 30%
... 35%
... 40%
... 45%
... 50%
... 55%
... 60%
... 65%
... 70%
... 75%
... 80%
... 85%
... 90%
... 95%
Procesing: 100%
---- Writing Data to new files...
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-19_2300.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_1500.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_1600.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_1700.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_1800.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_1900.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_2000.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_2100.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_2200.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-20_2300.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-21_0000.TXT
Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-05-19_232814Z.TXT
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
... 0%
... 5%
... 10%
... 15%
... 20%
... 25%
... 30%
... 35%
... 40%
... 45%
... 50%
... 55%
... 60%
... 65%
... 70%
... 75%
... 80%
... 85%
... 90%
Procesing: 100%
---- Writing Data to new files...
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-21_0000.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-21_1100.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-21_1200.TXT
Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-05-21_003340Z.TXT
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
Traceback (most recent call last):
  File "pandas/_libs/tslib.pyx", line 605, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslibs/parsing.pyx", line 289, in pandas._libs.tslibs.parsing.parse_datetime_string
ValueError: Given date string 970 not likely a datetime

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas/_libs/tslib.pyx", line 616, in pandas._libs.tslib.array_to_datetime
TypeError: invalid string coercion to datetime for "970" at position 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/daurin/GitRepos/HyperCP/Source/raw_Data_Hourly_Cutter.py", line 126, in <module>
    split_csv_hourly(args.file, args.output)
  File "/Users/daurin/GitRepos/HyperCP/Source/raw_Data_Hourly_Cutter.py", line 82, in split_csv_hourly
    timestamp = pd.to_datetime(parts[3])
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/tools/datetimes.py", line 1102, in to_datetime
    result = convert_listlike(np.array([arg]), format)[0]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/tools/datetimes.py", line 438, in _convert_listlike_datetimes
    result, tz_parsed = objects_to_datetime64ns(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py", line 2177, in objects_to_datetime64ns
    result, tz_parsed = tslib.array_to_datetime(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/tslib.pyx", line 427, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 683, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 829, in pandas._libs.tslib._array_to_datetime_object
  File "pandas/_libs/tslib.pyx", line 819, in pandas._libs.tslib._array_to_datetime_object
  File "pandas/_libs/tslibs/parsing.pyx", line 289, in pandas._libs.tslibs.parsing.parse_datetime_string
ValueError: Given date string 970 not likely a datetime present at position 0
Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-05-23_133246Z.TXT
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
... 0%
... 5%
... 10%
... 15%
... 20%
Traceback (most recent call last):
  File "pandas/_libs/tslib.pyx", line 605, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslibs/parsing.pyx", line 289, in pandas._libs.tslibs.parsing.parse_datetime_string
ValueError: Given date string -83.7405853 not likely a datetime

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas/_libs/tslib.pyx", line 616, in pandas._libs.tslib.array_to_datetime
TypeError: invalid string coercion to datetime for "-83.7405853" at position 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/daurin/GitRepos/HyperCP/Source/raw_Data_Hourly_Cutter.py", line 126, in <module>
    split_csv_hourly(args.file, args.output)
  File "/Users/daurin/GitRepos/HyperCP/Source/raw_Data_Hourly_Cutter.py", line 82, in split_csv_hourly
    timestamp = pd.to_datetime(parts[3])
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/tools/datetimes.py", line 1102, in to_datetime
    result = convert_listlike(np.array([arg]), format)[0]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/tools/datetimes.py", line 438, in _convert_listlike_datetimes
    result, tz_parsed = objects_to_datetime64ns(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/miniforge3/envs/hypercp/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py", line 2177, in objects_to_datetime64ns
    result, tz_parsed = tslib.array_to_datetime(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/tslib.pyx", line 427, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 683, in pandas._libs.tslib.array_to_datetime
  File "pandas/_libs/tslib.pyx", line 829, in pandas._libs.tslib._array_to_datetime_object
  File "pandas/_libs/tslib.pyx", line 819, in pandas._libs.tslib._array_to_datetime_object
  File "pandas/_libs/tslibs/parsing.pyx", line 289, in pandas._libs.tslibs.parsing.parse_datetime_string
ValueError: Given date string -83.7405853 not likely a datetime present at position 0
Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-05-24_132923Z.TXT
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
... 0%
... 5%
... 10%
... 15%
... 20%
... 25%
... 30%
... 35%
... 40%
... 45%
... 50%
... 55%
... 60%
... 65%
... 70%
... 75%
... 80%
... 85%
... 90%
... 95%
Procesing: 100%
---- Writing Data to new files...
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1300.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1400.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1500.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1600.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1700.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1800.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_1900.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_2000.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_2100.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_2200.TXT
Created: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/DALEC_12_2024-05-25_2300.TXT
Processing file: /Users/daurin/Projects/HyperPACE/field_data/DALEC/NF2405_VIIRS/raw/daily_files/DALEC_12_2024-05-25_135818Z.TXT
---- Initializing Data...
---- Reading Input File...
---- Processing Data...
... 0%
... 5%
... 10%
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DALEC daily to hourly raw file script error #353

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DALEC daily to hourly raw file script error #353

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions