Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: Data-Centric-AI-Community/fg-data-profiling
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v4.18.0
Choose a base ref
...
head repository: Data-Centric-AI-Community/fg-data-profiling
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v4.18.1
Choose a head ref
  • 1 commit
  • 23 files changed
  • 1 contributor

Commits on Dec 30, 2025

  1. fix: Multiple Spark Enhancements (#1800) 

    * fix: update spark numeric stats calculations

In the Pandas implementation, the numeric stats like min/max/stddev/etc.
by default ignore null values.
This commit updates the spark implementation to more closely match that.

* fix: update spark null checks for describe_counts_spark method

Need to add the isnan() check because Pandas isnull check will count NaN as null,
 but Spark does not.

* fix: update Spark frequency counts

The previous calculation of counts was actually counting an already summarized
dataframe, so it wasn't capturing the correct counts for each instance of a value.

This is updated by summing the count value instead of performing a row count.

* fix: adding tests for issue 1429

* fix: edge case - completely null numeric field in Spark

Discovered this edge case with real data, and still need to fix the rendering of an
empty histogram.

* fix: add handling for spark DecimalType

This change addresses issue #1602.

Computations in the summarize process result in some floats when computing against
decimal columns.To solution this, we simply convert those types to a DoubleType
when performing those numeric operations.

* fix: add handling for spark correlations with no numeric fields

Assembling a vector column in Spark with no numeric columns results in features with
a NULL size, NULL indices, and an empty list of values.
This causes an exception to be raised when computing correlations.

The solution here is to avoid computing the correlation matrix when there are no
interval columns (numeric).

This change addresses issue #1722.

* fix: allow NoneType values to be string formatted

This change addresses issue #1723.
It implements a "N/A" string as the default when formatting NoneType values.

* fix: multiple Spark fixes to approach closer parity to Pandas profiles

Addresses handling of completely null numeric columns, and gracefully handling
empty correlation sets and plots.
    @MCBoarder289
    MCBoarder289 authored Dec 30, 2025
    Configuration menu
    Copy the full SHA
    45d76ea View commit details
    Browse the repository at this point in the history
Loading