O(1) Creates an empty dataframe

Checks if the dataframe is empty (has no columns).

Returns True if the dataframe has no columns, False otherwise. Note that a dataframe with columns but no rows is not considered null.

A record that contains information about how and what rows are grouped in the dataframe. This can only be used with aggregate.

For showing the dataframe as markdown in notebooks.

O(n) Convert a list to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> fromList [(1 :: Int), 2, 3, 4]
[1,2,3,4]

O(n) Converts a column to a list. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toList Int column
[1,2,3,4]
> toList Double column
exception: ...

Our representation of a column is a GADT that can store data based on the underlying data.

This allows us to pattern match on data kinds and limit some operations to only some kinds of vectors. E.g. operations for missing data only happen in an OptionalColumn.

O(n) Convert an unboxed vector to a column. This avoids the extra conversion if you already have the data in an unboxed vector.

Examples:

> import qualified Data.Vector.Unboxed as V
> fromUnboxedVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert a vector to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> import qualified Data.Vector as V
> fromVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

Checks if a column is of a given type values.

Checks if a column contains missing values.

Checks if a column contains numeric values.

Converts a column to a vector of a specific type.

This is a type-safe conversion that requires the column's element type to exactly match the requested type. You must specify the desired type via type applications.

Type Parameters

Examples

>>> toVector @Int @VU.Vector column
Right (unboxed vector of Ints)

>>> toVector @Text @VB.Vector column
Right (boxed vector of Text)

Returns

See also

Unwraps a value from an Any type.

Given a row gets the value associated with a field.

Examples

>>> map (rowValue (F.col @Int "age")) (toRowList df)
[25,30, ...]

Wraps a value into an Any type. This helps up represent rows as heterogenous lists.

Converts the entire dataframe to a list of rows.

Each row contains all columns in the dataframe, ordered by their column indices. The rows are returned in their natural order (from index 0 to n-1).

Examples

>>> toRowList df
[[("name", "Alice"), ("age", 25), ...], [("name", "Bob"), ("age", 30), ...], ...]

Performance note

Converts the dataframe to a vector of rows with only the specified columns.

Each row will contain only the columns named in the names parameter. This is useful when you only need a subset of columns or want to control the column order in the resulting rows.

Parameters

Examples

>>> toRowVector ["name", "age"] df
Vector of rows with only name and age fields

>>> toRowVector [] df  -- Empty column list
Vector of empty rows (one per dataframe row)

Adds a foldable collection to the dataframe. If the collection has less elements than the dataframe and the dataframe is not empty the collection is converted to type `Maybe a` filled with Nothing to match the size of the dataframe. Similarly, if the collection has more elements than what's currently in the dataframe, the other columns in the dataframe are change to `Maybe Type` and filled with Nothing.

Be careful not to insert infinite collections with this function as that will crash the program.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> D.insert "numbers" [(1 :: Int)..10] D.empty

--------
 numbers
--------
   Int
--------
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10

A left fold for dataframes that takes the dataframe as the last object. This makes it easier to chain operations.

Example

>>> df = D.fromNamedColumns [("x", D.fromList [1..100]), ("y", D.fromList [11..110])]
>>> D.fold D.dropLast [1..5] df

---------
 x  |  y
----|----
Int | Int
----|----
1   | 11
2   | 12
3   | 13
4   | 14
5   | 15
6   | 16
7   | 17
8   | 18
9   | 19
10  | 20
11  | 21
12  | 22
13  | 23
14  | 24
15  | 25
16  | 26
17  | 27
18  | 28
19  | 29
20  | 30

Showing 20 rows out of 85

O(n) Renames a single column.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> import qualified Data.Vector as V
>>> df = insertVector "numbers" (V.fromList [1..10]) D.empty
>>> D.rename "numbers" "others" df

-------
 others
-------
  Int
-------
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10

Retrieves a column as an unboxed vector of Double values.

Returns Left with a DataFrameException if the column cannot be converted to doubles. This may occur if the column contains non-numeric data.

O(1) Get DataFrame dimensions i.e. (rows, columns)

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> df = D.fromNamedColumns [("a", D.fromList [1..100]), ("b", D.fromList [1..100]), ("c", D.fromList [1..100])]
>>> D.dimensions df

(100, 3)

O(k) Get column names of the DataFrame in order of insertion.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> df = D.fromNamedColumns [("a", D.fromList [1..100]), ("b", D.fromList [1..100]), ("c", D.fromList [1..100])]
>>> D.columnNames df

["a", "b", "c"]

O(1) Get number of rows in a dataframe.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> df = D.fromNamedColumns [("a", D.fromList [1..100]), ("b", D.fromList [1..100]), ("c", D.fromList [1..100])]
>>> D.nRows df
100

O(1) Get number of columns in a dataframe.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> df = D.fromNamedColumns [("a", D.fromList [1..100]), ("b", D.fromList [1..100]), ("c", D.fromList [1..100])]
>>> D.nColumns df
3

Adds a vector to the dataframe. If the vector has less elements than the dataframe and the dataframe is not empty the vector is converted to type `Maybe a` filled with Nothing to match the size of the dataframe. Similarly, if the vector has more elements than what's currently in the dataframe, the other columns in the dataframe are change to `Maybe Type` and filled with Nothing.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> import qualified Data.Vector as V
>>> D.insertVector "numbers" (V.fromList [(1 :: Int)..10]) D.empty

--------
 numbers
--------
   Int
--------
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10

O(n) Add a column to the dataframe.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> D.insertColumn "numbers" (D.fromList [(1 :: Int)..10]) D.empty

--------
 numbers
--------
   Int
--------
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10

Adds a vector to the dataframe and pads it with a default value if it has less elements than the number of rows.

Example

>>> :set -XOverloadedStrings
>>> import qualified Data.Vector as V
>>> import qualified DataFrame as D
>>> df = D.fromNamedColumns [("x", D.fromList [(1 :: Int)..10])]
>>> D.insertVectorWithDefault 0 "numbers" (V.fromList [(1 :: Int),2,3]) df

-------------
 x  | numbers
----|--------
Int |   Int
----|--------
1   | 1
2   | 2
3   | 3
4   | 0
5   | 0
6   | 0
7   | 0
8   | 0
9   | 0
10  | 0

Adds a list to the dataframe and pads it with a default value if it has less elements than the number of rows.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> df = D.fromNamedColumns [("x", D.fromList [(1 :: Int)..10])]
>>> D.insertWithDefault 0 "numbers" [(1 :: Int),2,3] df

-------------
 x  | numbers
----|--------
Int |   Int
----|--------
1   | 1
2   | 2
3   | 3
4   | 0
5   | 0
6   | 0
7   | 0
8   | 0
9   | 0
10  | 0

O(n) Adds an unboxed vector to the dataframe.

Same as insertVector but takes an unboxed vector. If you insert a vector of numbers through insertVector it will either way be converted into an unboxed vector so this function saves that extra work/conversion.

O(n) Clones a column and places it under a new name in the dataframe.

Example

>>> :set -XOverloadedStrings
>>> import qualified Data.Vector as V
>>> df = insertVector "numbers" (V.fromList [1..10]) D.empty
>>> D.cloneColumn "numbers" "others" df

-----------------
 numbers | others
---------|-------
   Int   |  Int
---------|-------
 1       | 1
 2       | 2
 3       | 3
 4       | 4
 5       | 5
 6       | 6
 7       | 7
 8       | 8
 9       | 9
 10      | 10

O(n) Renames many columns.

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> import qualified Data.Vector as V
>>> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)
>>> df

-----------------
 numbers | others
---------|-------
   Int   |  Int
---------|-------
 1       | 11
 2       | 12
 3       | 13
 4       | 14
 5       | 15
 6       | 16
 7       | 17
 8       | 18
 9       | 19
 10      | 20

>>> D.renameMany [("numbers", "first_10"), ("others", "next_10")] df

-------------------
 first_10 | next_10
----------|--------
   Int    |   Int
----------|--------
 1        | 11
 2        | 12
 3        | 13
 4        | 14
 5        | 15
 6        | 16
 7        | 17
 8        | 18
 9        | 19
 10       | 20

O(n * k ^ 2) Returns the number of non-null columns in the dataframe and the type associated with each column.

Example

>>> import qualified Data.Vector as V
>>> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)
>>> D.describeColumns df

--------------------------------------------------------
 Column Name | # Non-null Values | # Null Values | Type
-------------|-------------------|---------------|-----
    Text     |        Int        |      Int      | Text
-------------|-------------------|---------------|-----
 others      | 10                | 0             | Int
 numbers     | 10                | 0             | Int

Creates a dataframe from a list of tuples with name and column.

Example

>>> df = D.fromNamedColumns [("numbers", D.fromList [1..10]), ("others", D.fromList [11..20])]
>>> df
-----------------
 numbers | others
---------|-------
   Int   |  Int
---------|-------
 1       | 11
 2       | 12
 3       | 13
 4       | 14
 5       | 15
 6       | 16
 7       | 17
 8       | 18
 9       | 19
 10      | 20

Create a dataframe from a list of columns. The column names are "0", "1"... etc. Useful for quick exploration but you should probably always rename the columns after or drop the ones you don't want.

Example

>>> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]
>>> df
-----------------
  0  |  1
-----|----
 Int | Int
-----|----
 1   | 11
 2   | 12
 3   | 13
 4   | 14
 5   | 15
 6   | 16
 7   | 17
 8   | 18
 9   | 19
 10  | 20

Create a dataframe from a list of column names and rows.

Example

>>> df = D.fromRows [A, B] [[D.toAny 1, D.toAny 11], [D.toAny 2, D.toAny 12], [D.toAny 3, D.toAny 13]]

>>> df

----------
  A  |  B
-----|----
 Int | Int
-----|----
 1   | 11
 2   | 12
 3   | 13

O (k * n) Counts the occurences of each value in a given column.

Example

>>> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

>>> D.valueCounts @Int "0" df

[(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1)]

Get a specific column as a vector.

You must specify the type via type applications.

Examples

>>> columnAsVector (F.col @Int "age") df
Right [25, 30, 35, ...]

>>> columnAsVector (F.col @Text "name") df
Right ["Alice", "Bob", "Charlie", ...]

O (k * n) Shows the proportions of each value in a given column.

Example

>>> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

>>> D.valueCounts @Int "0" df

[(1,0.1),(2,0.1),(3,0.1),(4,0.1),(5,0.1),(6,0.1),(7,0.1),(8,0.1),(9,0.1),(10,0.1)]

Returns a dataframe as a two dimensional vector of floats.

Converts all columns in the dataframe to float vectors and transposes them into a row-major matrix representation.

This is useful for handing data over into ML systems.

Returns Left with an error if any column cannot be converted to floats.

Returns a dataframe as a two dimensional vector of doubles.

Converts all columns in the dataframe to double vectors and transposes them into a row-major matrix representation.

This is useful for handing data over into ML systems.

Returns Left with an error if any column cannot be converted to doubles.

Returns a dataframe as a two dimensional vector of ints.

Converts all columns in the dataframe to int vectors and transposes them into a row-major matrix representation.

This is useful for handing data over into ML systems.

Returns Left with an error if any column cannot be converted to ints.

Retrieves a column as an unboxed vector of Int values.

Returns Left with a DataFrameException if the column cannot be converted to ints. This may occur if the column contains non-numeric data or values outside the Int range.

Retrieves a column as an unboxed vector of Float values.

Returns Left with a DataFrameException if the column cannot be converted to floats. This may occur if the column contains non-numeric data.

Get a specific column as a list.

You must specify the type via type applications.

Examples

>>> columnAsList @Int "age" df
[25, 30, 35, ...]

>>> columnAsList @Text "name" df
["Alice", "Bob", "Charlie", ...]

Throws

Construct a SchemaType for the given a.

Examples

>>> :set -XTypeApplications
>>> schemaType @T.Text == schemaType @T.Text
True

>>> show (schemaType @Double)
"Double"

STANDARD CONFIG TYPES

CSV read parameters.

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsv "./data/taxi.csv"

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsvWithOpts "./data/taxi.csv" (D.defaultReadOptions { dateFormat = "%d%-m%-Y" })

Read text file with specified delimiter into a dataframe.

Example

ghci> D.readSeparated (D.defaultReadOptions { columnSeparator = ';' }) "./data/taxi.txt"

Read TSV (tab separated) file from path and load it into a dataframe.

Example

ghci> D.readTsv "./data/taxi.tsv"

Read a parquet file from path and load it into a dataframe.

Example

ghci> D.readParquet "./data/mtcars.parquet"

Sample a dataframe. The double parameter must be between 0 and 1 (inclusive).

Example

ghci> import System.Random
ghci> D.sample (mkStdGen 137) 0.1 df

O(n * k) Filter rows by a given condition.

filter "x" even df

O(k * n) Take a range of rows of a DataFrame.

O(k * n) Take the first n rows of a DataFrame.

O(k * n) Drop the first n rows of a DataFrame.

O(n) Selects a number of columns in a given dataframe.

select ["name", "age"] df

O(n) select columns by column predicate name.

Criteria for selecting columns whose indices are in the given (inclusive) range.

selectBy [byIndexRange (0, 5)] df

Criteria for selecting a column by name.

selectBy [byName "Age"] df

equivalent to:

select ["Age"] df

Criteria for selecting columns whose name satisfies given predicate.

selectBy [byNameProperty (T.isPrefixOf "weight")] df

Criteria for selecting columns whose names are in the given lexicographic range (inclusive).

selectBy [byNameRange ("a", "c")] df

Criteria for selecting columns whose property satisfies given predicate.

selectBy [byProperty isNumeric] df

O(k) cuts the dataframe in a cube of size (a, b) where a is the length and b is the width.

cube (10, 5) df

O(k * n) Drop the last n rows of a DataFrame.

O(n) inverse of select

exclude ["Name"] df

O(n * k) removes all rows with Nothing from the dataframe.

filterAllJust df

O(n * k) keeps any row with a null value.

filterAllNothing df

O(k) a version of filter where the predicate comes first.

filterBy even "x" df

O(k) removes all rows with Nothing in a given column from the dataframe.

filterJust "col" df

O(k) returns all rows with Nothing in a give column.

filterNothing "col" df

O(k) filters the dataframe with a boolean expression.

filterWhere (F.col @Int x + F.col y F.> 5) df

Creates n folds of a dataframe.

Example

ghci> import System.Random
ghci> D.kFolds (mkStdGen 137) 5 df

Split a dataset into two. The first in the tuple gets a sample of p (0 <= p <= 1) and the second gets (1 - p). This is useful for creating test and train splits.

Example

ghci> import System.Random
ghci> D.randomSplit (mkStdGen 137) 0.9 df

O(k * n) Take the last n rows of a DataFrame.

O(k * n) groups the dataframe by the given rows aggregating the remaining rows into vector that should be reduced later.

Aggregate a grouped dataframe using the expressions given. All ungrouped columns will be dropped.

Filter out all non-unique values in a dataframe.

O(k log n) Sorts the dataframe by a given row.

sortBy Ascending ["Age"] df

Sort order taken as a parameter by the sortBy function.

Calculates the sum of a given column as a standalone value.

Calculates the Pearson's correlation coefficient between two given columns as a standalone value.

Show a frequency table for a categorical feaure.

Examples:

ghci> df <- D.readCsv "./data/housing.csv"

ghci> D.frequencies "ocean_proximity" df

---------------------------------------------------------------------
   Statistic    | <1H OCEAN | INLAND | ISLAND | NEAR BAY | NEAR OCEAN
----------------|-----------|--------|--------|----------|-----------
      Text      |    Any    |  Any   |  Any   |   Any    |    Any
----------------|-----------|--------|--------|----------|-----------
 Count          | 9136      | 6551   | 5      | 2290     | 2658
 Percentage (%) | 44.26%    | 31.74% | 0.02%  | 11.09%   | 12.88%

Calculates the nth percentile of a given column as a standalone value.

O(n) Impute missing values in a column using a derived scalar.

Given

• an expression f :: Expr b -> Expr b that, when interpreted over a non-nullable column, produces the same value in every row (for example a mean, median, or other aggregate), and
• a nullable column Expr (Maybe b)

this function:

1. Drops all Nothing values from the target column.
2. Interprets f on the remaining non-null values.
3. Checks that the resulting column contains a single repeated value.
4. Uses that value to impute all Nothings in the original column.

Throws

Example

>>> :set -XOverloadedStrings
>>> import qualified DataFrame as D
>>> let df =
...       D.fromNamedColumns
...         [ ("age", D.fromList [Just 10, Nothing, Just 20 :: Maybe Int]) ]
>>>
>>> -- Impute missing ages with the mean of the observed ages
>>> D.imputeWith F.mean "age" df
-- age
-- ----
-- 10
-- 15
-- 20

Calculates the inter-quartile range of a given column as a standalone value.

Calculates the mean of a given column as a standalone value.

Calculates the median of a given column as a standalone value.

Calculates the median of a given column (containing optional values) as a standalone value.

Calculates the nth percentile of a given column as a standalone value.

Calculates the skewness of a given column as a standalone value.

Calculates the standard deviation of a given column as a standalone value.

Descriptive statistics of the numeric columns.

Calculates the variance of a given column as a standalone value.