Skip to content

Hubs ADX ingestion out of memory fix#1216

Merged
flanakin merged 3 commits into
devfrom
flanakin/dev/hubs-outofmemory
Dec 27, 2024
Merged

Hubs ADX ingestion out of memory fix#1216
flanakin merged 3 commits into
devfrom
flanakin/dev/hubs-outofmemory

Conversation

@flanakin

@flanakin flanakin commented Dec 16, 2024

Copy link
Copy Markdown
Collaborator

🛠️ Description

Improve performance and memory consumption of the Costs_transform_v1_0() KQL function

Fixes #1188

📋 Checklist

🔬 How did you test this change?

  • 🤏 Lint tests
  • 🤞 PS -WhatIf / az validate
  • 👍 Manually deployed + verified
  • 💪 Unit tests
  • 🙌 Integration tests

🙋‍♀️ Do any of the following that apply?

  • 🚨 This is a breaking change.
  • 🤏 The change is less than 20 lines of code.

📑 Did you update docs/changelog.md?

  • ✅ Updated changelog (required for dev PRs)
  • ➡️ Will add log in a future PR (feature branch PRs only)
  • ❎ Log not needed (small/internal change)

📖 Did you update documentation?

  • ✅ Public docs in docs (required for dev)
  • ✅ Internal dev docs in src (required for dev)
  • ➡️ Will add docs in a future PR (feature branch PRs only)
  • ❎ Docs not needed (small/internal change)

@flanakin flanakin added this to the 2024-12 - December milestone Dec 16, 2024
@flanakin flanakin requested a review from MSBrett as a code owner December 16, 2024 00:17
@flanakin flanakin requested review from MSBrett and Copilot and removed request for MSBrett December 16, 2024 00:17
@microsoft-github-policy-service microsoft-github-policy-service Bot added Micro PR 🔬 Very small PR that should be especially easy for newcomers Needs: Review 👀 PR that is ready to be reviewed labels Dec 16, 2024
Copilot AI reviewed Dec 16, 2024
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

@flanakin flanakin requested a review from bwatts64 December 16, 2024 01:55
@flanakin flanakin added the Tool: FinOps hubs Data pipeline solution label Dec 16, 2024
@RolandKrummenacher

RolandKrummenacher commented Dec 20, 2024

Copy link
Copy Markdown
Collaborator

I was able to reproduce the out of memory exception in my test environment:

image

I've tried to ingest a single snappy.parquet file:

image

image

Cluster size is Dev(No SLA)_Standard_D11_v2

@RolandKrummenacher

RolandKrummenacher commented Dec 20, 2024

Copy link
Copy Markdown
Collaborator

By applying the fix in this PR, I was able to ingest the file successfully:

image

RolandKrummenacher
RolandKrummenacher approved these changes Dec 20, 2024
View reviewed changes

@RolandKrummenacher RolandKrummenacher left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments

@flanakin

flanakin commented Dec 27, 2024

Copy link
Copy Markdown
Collaborator Author

@allcontributors please add RolandKrummenacher for review

@allcontributors allcontributors Bot mentioned this pull request Dec 27, 2024
Merged
@allcontributors

allcontributors Bot commented Dec 27, 2024

Copy link
Copy Markdown
Contributor

@flanakin

I've put up a pull request to add @RolandKrummenacher! 🎉

@flanakin flanakin merged commit 9d17573 into dev Dec 27, 2024
@flanakin flanakin deleted the flanakin/dev/hubs-outofmemory branch December 27, 2024 12:52
@MSBrett MSBrett mentioned this pull request Apr 17, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@RolandKrummenacher RolandKrummenacher RolandKrummenacher approved these changes
@MSBrett MSBrett Awaiting requested review from MSBrett MSBrett is a code owner
@arthurclares arthurclares Awaiting requested review from arthurclares
@bwatts64 bwatts64 Awaiting requested review from bwatts64

Assignees

@arthurclares arthurclares

Labels

Micro PR 🔬 Very small PR that should be especially easy for newcomers Needs: Review 👀 PR that is ready to be reviewed Tool: FinOps hubs Data pipeline solution

Projects

None yet

Milestone

2025-02 - February

Development

Successfully merging this pull request may close these issues.

Cost_Final table update policy fails with out of memory exception.

4 participants

@flanakin @RolandKrummenacher @arthurclares