🐛 Issue

During data ingestion, the Cost_Final_V1_0 update policy (function Costs_transform_v1_0) fails due to an "out of memory" exception.

👣 Steps to Reproduce

1. Setup

   • Configure the FinOps Toolkit with the default settings, using the smallest ADX SKU.

2. Prepare Data

   • Ensure price data has already been ingested.

3. Trigger Issue

   • Ingest 50 MB of cost data in Snappy-compressed Parquet format.

4. Verify Ingestion Failure

   • Run the command .show ingestion failures in the ingestion database.
   • Confirm the presence of an "out of memory" exception.

5. Verify Operation Failure

   • Run the command .show operations in the ingestion database.
   • Confirm that the Costs_transform_v1_0 update policy has failed.

🤔 Expected Outcome

The ingestion process should successfully complete without any errors.

• No "out of memory" exceptions should occur during the operation.
• The Costs_transform_v1_0 update policy should execute without failures, ensuring proper data transformation and ingestion.

ℹ️ Additional context

I suspect that the join in the Costs_transform_v1_0 function might be part of the problem. It could potentially be replaced with a lookup. Maybe other KQL optimizations apply such as materializing the price lookup before joining.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.