🐛 Problem
Errors are seen in Data Factory relating to the ingestion of reservation recommendations:
ErrorCode=UserErrorInvalidDataValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'Net Savings' contains an invalid value '{"currency":"USD","value":114961.468965322154294}'. Cannot convert '{"currency":"USD","value":114961.468965322154294}' to type 'Decimal'.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.FormatException,Message=Input string was not in a correct format.,Source=mscorlib,'
ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'SKU' specified in column mapping cannot be found in 'billingProfiles/XXXX-XXXX-XXX-XXX/PRE-ReservationsRecommendations-Kits/202502121437/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/part0.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'
Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'finopshubenybzcya46q36'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/02/providers/microsoft.billing/billingaccounts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_2019-05-31/billingprofiles/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '66c1d54e-301f-008d-496f-7cf906000000'. TimeStamp: 'Tue, 11 Feb 2025 10:26:38 GMT'.
👣 Repro steps
TODO: Add repro steps below:
1.Create a data export from Azure Cost Management for reservation recommendations (MCA)
2. Wait for ingestion
3. Look at the monitor in Data Factory and examine the errors
🤔 Expected
Ingestion should happen without errors
🐛 Problem
Errors are seen in Data Factory relating to the ingestion of reservation recommendations:
ErrorCode=UserErrorInvalidDataValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'Net Savings' contains an invalid value '{"currency":"USD","value":114961.468965322154294}'. Cannot convert '{"currency":"USD","value":114961.468965322154294}' to type 'Decimal'.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.FormatException,Message=Input string was not in a correct format.,Source=mscorlib,'
ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'SKU' specified in column mapping cannot be found in 'billingProfiles/XXXX-XXXX-XXX-XXX/PRE-ReservationsRecommendations-Kits/202502121437/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/part0.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'
Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'finopshubenybzcya46q36'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/02/providers/microsoft.billing/billingaccounts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_2019-05-31/billingprofiles/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '66c1d54e-301f-008d-496f-7cf906000000'. TimeStamp: 'Tue, 11 Feb 2025 10:26:38 GMT'.
👣 Repro steps
TODO: Add repro steps below:
1.Create a data export from Azure Cost Management for reservation recommendations (MCA)
2. Wait for ingestion
3. Look at the monitor in Data Factory and examine the errors
🤔 Expected
Ingestion should happen without errors