Ibis

Dynamic UDF Rewriting with Predicate Pushdowns

Hussain Sultan — Wed, 12 Feb 2025 00:00:00 GMT

Introduction

In an ideal world, deploying machine learning models within SQL queries would be as simple as calling a built-in function. Unfortunately, many ML predictions live inside User-Defined Functions (UDFs) that traditional SQL planners can’t modify, preventing optimizations like predicate pushdowns.

This blog post will showcase how you can prune decision tree models based on query filters by dynamically rewriting your expression using Ibis and quickgrove, an experimental GBDT inference library built in Rust. We’ll also show how LetSQL can simplify this pattern further and integrate seamlessly into your ML workflows.

ML models meet SQL

When you deploy machine learning models (like a gradient-boosted trees model from XGBoost) in a data warehouse, you typically wrap them in a UDF. Something like:

SELECT
    my_udf_predict(carat, depth, color, clarity, ...)
FROM diamonds
WHERE color_i < 1 AND clarity_vvs2 < 1

The challenge is that SQL planners don’t know what’s happening inside the UDF. Even if you filter color_i < 1, the full model, including skippable tree paths, are evaluated for every row. With tree-based models, entire branches might never be evaluated at all — so the ideal scenario is to prune those unnecessary branches before evaluating them.

Smart UDFs with Ibis

Ibis is known for letting you write engine-agnostic deferred expressions in Python without losing the power of underlying engines like Spark, DuckDB, or BigQuery. Meanwhile, quickgrove provides a mechanism to prune Gradient Boosted Decision Tree (GBDT) models based on known filter conditions.

Key Ideas:

Prune decision trees by removing branches that can never be reached, given the known filters
Rewrite expressions with the pruned model into the query plan to skip unnecessary computations

Understanding tree pruning

Tree Pruning

Take a simple example: a decision tree that splits on color_i < 1. If your query also has a predicate color < 1, any branches with feature color_i >= 1 will never be evaluated. By removing that branch, the tree becomes smaller and faster to evaluate—especially when you have hundreds of trees (as in many gradient-boosted models).

Reference: Check out the Raven optimizer paper. It demonstrates how you can prune nodes in query plans for tree-based inference, so we’re taking a similar approach here for forests (GBDTs) using Ibis.

Quickgrove: prunable GBDT models

Quickgrove is an experimental package that can load GBDT JSON models and provides a .prune(...) API to remove unreachable branches. For example:

#pip install quickgrove
import quickgrove

model = quickgrove.json_load("diamonds_model.json")  # Load an XGBoost model
model.prune([quickgrove.Feature("color_i") < 0.2]) # Prune based on known predicate

Once pruned, the model is leaner to evaluate. Note: The results heavily depend on model splits and interactions with predicate pushdowns.

Scalar PyArrow UDFs in Ibis

Please note that we are using our own modified DataFusion backend. The DataFusion backend and DuckDB backend behave differently: DuckDB expects a ChunkedArray while DataFusion UDFs expect ArrayRef. We are working on extending quickgrove to work with the DuckDB backend.

We’ll define a simple Ibis UDF that calls our model.predict_arrays under the hood:

import ibis
import ibis.expr.datatypes as dt

ibis.set_backend("datafusion")
@ibis.udf.scalar.pyarrow
def predict_gbdt(
    carat: dt.float64,
    depth: dt.float64,
    # ... other features ...
) -> dt.float32:
    array_list = [carat, depth, ...]
    return model.predict_arrays(array_list)

Currently, UDFs are opaque to Ibis. We need Ibis to teach Ibis how to rewrite a udf based on predicates it knows about.

Making Ibis UDFs predicate-aware

Here’s the general process:

Collect predicates from the user’s filter (e.g. x < 0.3).
Prune the model based on those predicates (removing unreachable tree branches).
Rewrite a new UDF that references the pruned model, preserving the rest of the query plan.

1. Collecting predicates

from ibis.expr.operations import Filter, Less, Field, Literal
from typing import List, Dict

def collect_predicates(filter_op: Filter) -> List[dict]:
    """Extract 'column < value' predicates from a Filter operation."""
    predicates = []
    for pred in filter_op.predicates:
        if isinstance(pred, Less) and isinstance(pred.left, Field):
            if isinstance(pred.right, Literal):
                predicates.append({
                    "column": pred.left.name,
                    "op": "Less",
                    "value": pred.right.value
                })
    return predicates

2. Pruning model and creating a new UDF

import functools
from ibis.expr.operations import ScalarUDF
from ibis.common.collections import FrozenDict

def create_pruned_udf(original_udf, model, predicates):
    """Create a new UDF using the pruned model based on the collected predicates."""
    from quickgrove import Feature

    # Prune the model
    pruned_model = model.prune([
        Feature(pred["column"]) < pred["value"]
        for pred in predicates
        if pred["op"] == "Less" and pred["value"] is not None
    ])
    # For simplicity, let’s assume we know the relevant features or keep them the same.

    def fn_from_arrays(*arrays):
        return pruned_model.predict_arrays(list(arrays))

    # Construct a dynamic UDF class
    meta = {
        "dtype": dt.float32,
        "__input_type__": "pyarrow",
        "__func__": property(lambda self: fn_from_arrays),
        "__config__": FrozenDict(volatility="immutable"),
        "__udf_namespace__": original_udf.__module_
        "__module__": original_udf.__module__,
        "__func_name__": original_udf.__name__ + "_pruned"
    }

    # Create a new ScalarUDF node type on the fly
    node = type(original_udf.__name__ + "_pruned", (ScalarUDF,), {**fields, **meta})

    @functools.wraps(fn_from_arrays)
    def construct(*args, **kwargs):
        return node(*args, **kwargs).to_expr()

    construct.fn = fn_from_arrays
    return construct

3. Rewriting the plan

Now we use an Ibis rewrite rule (or a custom function) to detect filters on the expression, prune the model, and produce a new project/filter node.

from ibis.expr.operations import Project

@replace(p.Filter)
def prune_gbdt_model(filter_op, original_udf, model):
    """Rewrite rule to prune GBDT model based on filter predicates."""

    predicates = collect_predicates(filter_op)
    if not predicates:
        # Nothing to prune if no relevant predicates
        return filter_op
    # in a real implementation you'd want to match on a ScalarUDF and ensure that the instance of the model type is
    # the one implemented with quickgrove
    pruned_udf, required_features = create_pruned_udf(original_udf, model, predicates)

    parent_op = filter_op.parent
    # Build a new projection with the pruned UDF
    new_values = {}
    for name, value in parent_op.values.items():
        # If it’s the column that calls the UDF, swap with pruned version
        if name == "prediction":
            # For brevity, assume we pass the same columns to the pruned UDF
            new_values[name] = pruned_udf(value.op().args[0], value.op().args[1])
        else:
            new_values[name] = value

    new_project = Project(parent_op.parent, new_values)

    # Re-add the filter conditions on top
    new_predicates = []
    for pred in filter_op.predicates:
        if isinstance(pred, Less) and isinstance(pred.left, Field):
            new_predicates.append(
                Less(Field(new_project, pred.left.name), pred.right)
            )
        else:
            new_predicates.append(pred)

    return Filter(parent=new_project, predicates=new_predicates)

Diff

For a query like the following:

expr = (
    t.mutate(prediction=predict_gbdt(t.carat, t.depth, ...))
    .filter(
        (t["clarity_vvs2"] < 1),
        (t["color_i"] < 1),
        (t["color_j"] < 1)
    )
    .select("prediction")
)

See the diff below:

Notice that with pruning we can drop some of the projections in the UDF e.g., color_i, color_j and clarity_vvs2. The underlying engine (e.g., DataFusion) may optimize this further when pulling data for UDFs. We cannot completely drop these from the query expression.

- predict_gbdt_3(
+ predict_gbdt_pruned(
    carat, depth, table, x, y, z,
    cut_good, cut_ideal, cut_premium, cut_very_good,
-   color_e, color_f, color_g, color_h, color_i, color_j,
+   color_e, color_f, color_g, color_h,
    clarity_if, clarity_si1, clarity_si2, clarity_vs1,
-   clarity_vs2, clarity_vvs1, clarity_vvs2
+   clarity_vs2, clarity_vvs1
)

Putting it all together

The complete example can be found here.

# 1. Load your dataset into Ibis
t = ibis.read_csv("diamonds_data.csv")

expr = (
    t.mutate(prediction=predict_gbdt(t.carat, t.depth, ...))
    .filter(
        (t["clarity_vvs2"] < 1),
        (t["color_i"] < 1),
        (t["color_j"] < 1)
    )
    .select("prediction")
)

# 3. Apply your custom optimization
optimized_expr = prune_gbdt_model(expr.op(), predict_gbdt, model)

# 4. Execute the optimized query
result = optimized_expr.to_expr().execute()

When this is done, the model inside predict_gbdt will be pruned based on the expression’s filter conditions. This can yield significant speedups on large datasets (see Table 1).

Performance impact

Here is the benchmark results ran on Apple M2 Mac Mini, 8 cores / 8GB Memory run with a model trained with 100 trees and depth 6 with following filter conditions:

_.carat < 1,
_.clarity_vvs2 < 1,
_.color_i < 1,
_.color_j < 1,

Benchmark results:

Table 1: Performance improvements

File Size	Regular (s)	Optimized (s)	Improvement
5M	0.82 ±0.02	0.67 ±0.02	18.0%
25M	4.16 ±0.01	3.46 ±0.05	16.7%
100M	16.80 ±0.17	14.07 ±0.11	16.3%

Key takeaway: As data volume grows, skipping unneeded tree branches can translate to real compute savings, albeit heavily dependent on how pertinent the filter conditions might be.

LetSQL: simplifying UDF rewriting

LetSQL makes advanced UDF rewriting and multi-engine pipelines much simpler. It builds on the same ideas we explored here but wraps them in a higher-level API.

Here’s a quick glimpse of how LetSQL might simplify the pattern:

# pip install letsql

import letsql as ls
from letsql.expr.ml import make_quickgrove_udf, rewrite_quickgrove_expression

model_path = "xgboost_model.json"
predict_udf = make_quickgrove_udf(model_path)

t = ls.memtable(df).mutate(pred=predict_udf.on_expr).filter(ls._.carat < 1)
optimized_t = rewrite_quickgrove_expression(t)

result = ls.execute(optimized_t)

The complete example can be found here. With LetSQL, you get a shorter, more declarative approach to the same optimization logic we manually coded with Ibis. It abstracts away the gritty parts of rewriting your query plan.

Best practices & considerations

Predicate Types: Currently, we demonstrated column < value logic. You can extend it to handle <=, >, BETWEEN, or even categorical splits.
Quickgrove only supports a handful of objective functions and most notably does not have categorical support yet. In theory, categorical variables make better candidates for pruning based on filter conditions. It only supports XGBoost format.
Model Format: XGBoost JSON is straightforward to parse. Other formats (e.g. LightGBM, scikit-learn trees) require similar logic or conversion steps.
Edge Cases: If the filter references columns not in the model features, or if multiple filters combine in more complex ways, your rewriting logic may need more robust parsing.
When to Use: This approach is beneficial when queries often filter on the same columns your trees split on. For purely adhoc queries or rarely used filters, the overhead of rewriting might outweigh the benefit.

Conclusion

Combining Ibis with a prune-friendly framework like quickgrove lets you optimize large-scale ML inference inside ML workflows. By pushing filter predicates down into your decision trees, you speed up queries significantly.

With LetSQL, you can streamline this entire process—especially if you’re looking for an out-of-the-box solution that integrates with multiple engines along with batteries included features like caching and aggregate/window UDFs. For the next steps, consider experimenting with more complex models, exploring different tree pruning strategies, or even extending this pattern to other ML models beyond GBDTs.

Try it out: Explore the Ibis documentation to learn how to build custom UDFs.
Dive deeper: Check out quickgrove or read the Raven optimizer paper.
Experiment with LetSQL: If you need a polished solution for dynamic ML UDF rewriting, LetSQL may be just the ticket.

Resources

Raven Paper: End-to-end Optimization of Machine Learning Prediction Queries
Ibis + Torch: Ibis Project Blog Post
Multi-Engine Data Stack with Ibis

Does Ibis understand SQL?

Deepyaman Datta — Thu, 06 Feb 2025 00:00:00 GMT

Last month, an insightful article on the dbt Developer Blog on what SQL comprehension really means came across my LinkedIn feed. The big deal about SDF is that it, unlike dbt, actually understands SQL. As an Ibis user and contributor, several of the concepts covered in the post were familiar—in fact, I first learned about Ibis because the product I was working on required an intermediate representation that could be compiled to Flink SQL code. In that case, as a dataframe library that interfaces with databases, does Ibis also understand SQL?

Tl;dr

Ibis doesn’t understand SQL per se, but it does understand what you’re trying to do. Ibis, much like SQL, defines a standardized interface for working with databases. Because Ibis understands queries expressed through this user interface, it also provides users with some of the unique capabilities SDF offers, including the ability to execute said logic on the backend of the user’s choice.

How does Ibis work?

To answer the question of whether Ibis understands SQL, we first need to understand the internals of Ibis. Specifically, how is the code that users write with Ibis eventually executed on a SQL backend?

Building an expression

Ibis provides a dataframe API for writing expressions. A follow-up article on the dbt Developer Blog on the key technologies behind SQL comprehension used the following SQL query in illustrating what the parser and compiler do:

select x as u from t where x > 0

In SQL, the binder adds type information to the syntax tree produced by the parser. This order of operations differs from the way Ibis works; in Ibis, Nodes— the core operations that can be applied to expressions, such as ibis.expr.operations.Add and ibis.expr.operations.WindowFunction—must be applied to Expr objects containing data type and shape information.

The Table is one of the core Ibis data structures, analogous to a SQL table. It’s also an Expr subclass. We begin by manually defining a Table with our desired schema here, but one can also construct a table from an existing database table, file, or in-memory data representation:

import ibis

t = ibis.table(dict(x="int32", y="float", z="string"), name="t")

Next, we apply a filter and rename the x column as in the SQL query above:

expr = t.filter(t.x > 0).select(t.x.name("u"))

In SQL, the parser translates the query into a syntax tree, but in Ibis, expressions are inherently represented as a tree of Expr and Node objects. Ibis enables users to visualize() this intermediate representation for any expression:

from ibis.expr.visualize import to_graph

to_graph(expr)

In non-interactive mode, pretty-printing the expression yields an equivalent textual representation:

expr

r0 := UnboundTable: t
  x int32
  y float64
  z string

r1 := Filter[r0]
  r0.x > 0

Project[r1]
  u: r1.x

Look at that! Unsurprisingly, the resulting repr() matches the generated logical plan from the SQL comprehension technologies article:

Note that the abstract syntax tree is an artifact of the parsing step and has no direct Ibis analog.

Compiling the expression

In the past, Ibis would compile expressions to SQL using its own SQL generation logic. However, with the completion of “the big refactor” in Ibis 9.0, Ibis fully transitioned to producing SQLGlot expressions under the hood.

We can see the intermediate SQLGlot representation of our expression using the to_sqlglot() method on an Ibis backend compiler implementation:

from ibis.backends.sql.compilers.duckdb import compiler

query = compiler.to_sqlglot(expr)
query

Select(
  expressions=[
    Alias(
      this=Column(
        this=Identifier(this='x', quoted=True),
        table=Identifier(this='t0', quoted=True)),
      alias=Identifier(this='u', quoted=True))],
  from=From(
    this=Table(
      this=Identifier(this='t', quoted=True),
      alias=Identifier(this='t0', quoted=True))),
  where=Where(
    this=GT(
      this=Column(
        this=Identifier(this='x', quoted=True),
        table=Identifier(this='t0', quoted=True)),
      expression=Literal(this=0, is_string=False))))

Ibis delegates SQL generation to SQLGlot, essentially calling the sql() method on the above query:

query.sql()

'SELECT "t0"."x" AS "u" FROM "t" AS "t0" WHERE "t0"."x" > 0'

Ibis also provides a top-level to_sql() method, so most users don’t need to be aware of SQLGlot or the inner workings of Ibis expression compilation—unless they want to:

ibis.to_sql(expr)

SELECT
  "t0"."x" AS "u"
FROM "t" AS "t0"
WHERE
  "t0"."x" > 0

Why SQLGlot?

SQLGlot is a no-dependency SQL parser, transpiler, optimizer, and engine. It’s a widely-used open-source project that powers the SQL comprehension and generation capabilities of tools like SQLMesh, Apache Superset, and Dagster. In fact, SQLGlot is also the engine behind the column-level lineage feature available in dbt Cloud!

The specifics of why Ibis chose SQLGlot are beyond the scope of this article, but you can learn more about the reasoning from a GitHub discussion on moving the SQL backends from SQLAlchemy to SQLGlot.

Executing the compiled expression

Execution is the most straightforward part of the process. For most Ibis-supported SQL backends, the execute() method uses the database connection associated with the backend instance—usually managed by the underlying Python client library—to submit and fetch results for the compiled query. Last but not least, Ibis massages the returned data into the desired format (e.g. a pandas DataFrame for easy consumption). Because this final processing step falls outside the expression-understanding process, we’ll end our journey through the Ibis execution flow here.

Note that the database still performs all of the activities covered by the SQL comprehension levels; the database is completely oblivious to whatever Ibis did prior to providing the raw SQL to execute.

So, does Ibis understand SQL?

Ibis’s expressive dataframe API lets users avoid writing handcrafted SQL queries in most situations, but there are still cases where you may need to use SQL strings in your Ibis code. Table.sql() and Backend.sql() work very similarly to each other. To explore both of these options, we’ll need to initialize a backend connection (DuckDB works perfectly for this purpose) and register our table:

con = ibis.duckdb.connect()
t = con.create_table("t", t)

Now we can use the Table.sql() method to handle our SQL query:

expr = t.sql("select x as u from t where x > 0")
expr

r0 := DatabaseTable: memory.main.t
  x int32
  y float64
  z string

SQLStringView[r0]
  query:
    select x as u from t where x > 0
  schema:
    u int32

Note that the expression tree doesn’t have the same level of detail as the one we built using the dataframe API in the first section. Instead, a SQLStringView node encapsulates the query. Ibis only understands the output schema for the operation, which it uses to validate any downstream operations.

The main difference between Table.sql() and Backend.sql() is that Backend.sql() can only refer to tables that already exist in the database. We see that reflected in the expression tree repr(); the DatabaseTable node is not present in the Backend.sql() case, and a SQLQueryResult node (that does not contain a reference to another Ibis relation) replaces the SQLStringView node:

expr = con.sql("select x as u from t where x > 0")
expr

SQLQueryResult
  query:
    select x as u from t where x > 0
  schema:
    u int32

That said, the intermediate SQLGlot representations are identical for both alternatives:

con.compiler.to_sqlglot(expr)

Select(
  expressions=[
    Alias(
      this=Column(
        this=Identifier(this=x, quoted=False)),
      alias=Identifier(this=u, quoted=False))],
  from=From(
    this=Table(
      this=Identifier(this=t, quoted=False))),
  where=Where(
    this=GT(
      this=Column(
        this=Identifier(this=x, quoted=False)),
      expression=Literal(this=0, is_string=False))))

In fact, the resulting SQLGlot expression is only slightly, nonfunctionally different from the one we got from the dataframe API. Similarly, the compiled SQL is identical except for the table alias:

ibis.to_sql(expr)

SELECT
  x AS u
FROM t
WHERE
  x > 0

In short, while Ibis doesn’t understand the inner structure of SQL queries passed to Table.sql() or Backend.sql(), it still compiles to SQLGlot expressions under the hood, which means SQLGlot’s capabilities (like producing column-level lineage) are still applicable.

The third option for writing SQL strings in Ibis, Backend.raw_sql(), is opaque and only exists for situations where the user needs to run arbitrary SQL code. Ibis simply executes the code and returns the associated cursor; it does not attempt to understand the SQL.

Querying Amazon Athena from the comfort of your Python interpreter

Anja Boskovic — Tue, 04 Feb 2025 00:00:00 GMT

Have you ever wanted to harness the power of AWS Athena, but found yourself tangled up in Presto SQL syntax? Good news! Ibis now supports Amazon Athena as its newest backend, bringing you the familiar comfort of DataFrame operations while tapping into AWS’s robust data lake architecture.

Why?

There’s even more to love about this integration. Athena’s pay-per-query pricing model means that users pay for each query they run. With Ibis’ query optimisation before execution, you can potentially reduce costs without needing to agonise over query efficiency. Plus, since Athena can query data directly from S3, this new backend lets you analyse your data lake contents with beloved Python libraries like PyArrow and pandas without the hassle of downloading or moving massive datasets.

Installation Prerequisites

Make sure you have an IAM account and that your credentials are in an expected location in your local environment.

Additionally, using the same account and region that you are using for Athena, you will need to create an S3 bucket where Athena can dump query results. This bucket will be set to s3_staging_dir in the connection call to the Athena backend.

Note

If you are not able to query Athena through awscli, your queries will similarly not work on Ibis. Please note that AWS charges will apply for queries to Athena executed in following this tutorial.

Installation

Install Ibis with the dependencies needed to work with AWS Athena:

$ pip install 'ibis-framework[athena]'

Data

We are going to be creating some sample ecological data about ibis behaviour. The data will contain multiple columns with information about species, location, weather, group size, behaviour, and location temperature.

import pandas as pd
import numpy as np


def create_observations(n: int, seed: int = 42) -> pd.DataFrame:
    ibis_species = ["Sacred Ibis", "Scarlet Ibis", "Glossy Ibis", "White Ibis"]
    locations = ["Wetland", "Grassland", "Coastal"]
    behaviors = ["Feeding", "Nesting", "Flying"]
    weather_conditions = ["Sunny", "Rainy"]

    np.random.seed(seed)  # For reproducibility

    return pd.DataFrame(
        {
            "observation_date": np.full(n, np.datetime64("2024-01-01"))
                + np.random.randint(0, 365, size=n).astype("timedelta64[D]"),
            "species": np.random.choice(ibis_species, size=n),
            "location": np.random.choice(locations, size=n),
            "group_size": np.random.randint(1, 20, size=n),
            "behavior": np.random.choice(behaviors, size=n),
            "weather": np.random.choice(weather_conditions, size=n),
            "temperature_c": np.random.normal(25, 5, size=n)  # Mean 25°C, std 5°C
        }
    )


ibis_observations = create_observations(1000)

Demo

Let’s start by opening a connection to AWS Athena with Ibis, using the S3 bucket we created to store query results.

from ibis.interactive import *

con = ibis.athena.connect(
    s3_staging_dir="s3://aws-athena-query-results-ibis-testing",
    region_name="us-east-2",
)

Let’s create some data using our ibis_observations pandas DataFrame.

con.create_database("mydatabase", force=True)
con.drop_table("ibis_observations", force=True)
con.create_table("ibis_observations", obj=ibis_observations, database="mydatabase")
con.list_tables(database="mydatabase")

['ibis_observations']

And we can grab information about table schemas to help us out with our queries:

con.get_schema("ibis_observations", database="mydatabase")

ibis.Schema {
  observation_date  timestamp
  species           string
  location          string
  group_size        int64
  behavior          string
  weather           string
  temperature_c     float64
}

And now we are able to grab the table, and make some Ibis queries! Like what is the average group size by species?

t = con.table("ibis_observations", database="mydatabase")

# Average group size by species
t.group_by("species").aggregate(avg_group=t.group_size.mean())

┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ species      ┃ avg_group ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ string       │ float64   │
├──────────────┼───────────┤
│ Glossy Ibis  │ 10.003906 │
│ Sacred Ibis  │ 10.030075 │
│ Scarlet Ibis │  9.673307 │
│ White Ibis   │ 10.259912 │
└──────────────┴───────────┘

And ibis does all the work on generating the Presto SQL that Athena can understand:

How about most common behaviour during rainy weather?

(
    t.filter(t.weather == "Rainy")
    .group_by("behavior")
    .aggregate(count=lambda t: t.count())
    .order_by(ibis.desc("count"))
)

┏━━━━━━━━━━┳━━━━━━━┓
┃ behavior ┃ count ┃
┡━━━━━━━━━━╇━━━━━━━┩
│ string   │ int64 │
├──────────┼───────┤
│ Feeding  │   172 │
│ Nesting  │   168 │
│ Flying   │   166 │
└──────────┴───────┘

Temperature effects on behaviour?

t.group_by("behavior").aggregate(avg_temp=t.temperature_c.mean()).order_by("avg_temp")

┏━━━━━━━━━━┳━━━━━━━━━━━┓
┃ behavior ┃ avg_temp  ┃
┡━━━━━━━━━━╇━━━━━━━━━━━┩
│ string   │ float64   │
├──────────┼───────────┤
│ Feeding  │ 25.006455 │
│ Flying   │ 25.116323 │
│ Nesting  │ 25.133050 │
└──────────┴───────────┘

Now that we’re nearing the end of this demo, I wanted to show you that you can also delete tables and databases using ibis:

con.drop_table("ibis_observations", database="mydatabase")
con.drop_database("mydatabase")
con.disconnect()

You wouldn’t need to fiddle with Athena’s SDK!

How does this all work?

Under the hood, AWS Athena runs on a version of Trino (formerly known as Presto SQL). Instead of writing a completely new SQL compiler for Athena, we were able to leverage Ibis’ existing Trino compiler with some careful adjustments.

This provides significant benefits in code efficiency - the Athena backend implementation required only about 40 lines of unique code.

There are some nuances to note: since Athena runs on an older version of Trino, not all of Trino’s newest features are available. For a detailed comparison of supported features across different backends, please check out the Ibis backend support matrix.

If you’re new here, welcome. Here are some resources to learn more about Ibis:

Chat with us on Zulip:

Ibis Zulip Chat

Classification metrics on the backend

Tyler White — Thu, 05 Dec 2024 00:00:00 GMT

A review of binary classification models, metrics used to evaluate them, and corresponding metric calculations with Ibis.

We’re going explore common classification metrics such as accuracy, precision, recall, and F1 score, demonstrating how to compute each one using Ibis. In this example, we’ll use DuckDB, the default Ibis backend, but we could use this same code to execute against another backend such as Postgres or Snowflake. This capability is useful as it offers an easy and performant way to evaluate model performance without extracting data from the source system.

Classification models

In machine learning, classification entails categorizing data into different groups. Binary classification, which is what we’ll be covering in this post, specifically involves sorting data into only two distinct groups. For example, a model could differentiate between whether or not an email is spam.

Model evaluation

It’s important to validate the performance of the model to ensure it makes correct predictions consistently and doesn’t only perform well on the data it was trained on. These metrics help us understand not just the errors the model makes, but also the types of errors. For example, we might want to know if the model is more likely to predict a positive outcome when the actual outcome is negative.

The easiest way to break down how this works is to look at a confusion matrix.

Confusion matrix

A confusion matrix is a table used to describe the performance of a classification model on a set of data for which the true values are known. As binary classification only involves two categories, the confusion matrix only contains true positives, false positives, false negatives, and true negatives.

Here’s a breakdown of the terms with examples.

True Positives (TP): Correctly predicted positive examples.

We guessed it was a spam email, and it was. This email is going straight to the junk folder.

False Positives (FP): Incorrectly predicted as positive.

We guessed it was a spam email, but it actually wasn’t. Hopefully, the recipient doesn’t miss anything important as this email is going to the junk folder.

False Negatives (FN): Incorrectly predicted as negative.

We didn’t guess it was a spam email, but it really was. Hopefully, the recipient doesn’t click any links!

True Negatives (TN): Correctly predicted negative examples.

We guessed it was not a spam email, and it actually was not. The recipient can read this email as intended.

Building a confusion matrix

Sample dataset

Let’s create a sample dataset that includes twelve rows with two columns: actual and prediction. The actual column contains the true values, and the prediction column contains the model’s predictions.

from ibis.interactive import *

t = ibis.memtable(
    {
        "id": range(1, 13),
        "actual": [1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1],
        "prediction": [1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1],
    }
)

t

┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ id    ┃ actual ┃ prediction ┃
┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ int64 │ int64  │ int64      │
├───────┼────────┼────────────┤
│     1 │      1 │          1 │
│     2 │      0 │          0 │
│     3 │      0 │          0 │
│     4 │      1 │          1 │
│     5 │      1 │          0 │
│     6 │      0 │          0 │
│     7 │      0 │          1 │
│     8 │      0 │          1 │
│     9 │      1 │          0 │
│    10 │      1 │          1 │
│     … │      … │          … │
└───────┴────────┴────────────┘

We can use the case function to create a new column that categorizes the outcomes.

case_expr = ibis.cases(
    ((_.actual == 0) & (_.prediction == 0), "TN"),
    ((_.actual == 0) & (_.prediction == 1), "FP"),
    ((_.actual == 1) & (_.prediction == 0), "FN"),
    ((_.actual == 1) & (_.prediction == 1), "TP"),
)

t = t.mutate(outcome=case_expr)

t

┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ id    ┃ actual ┃ prediction ┃ outcome ┃
┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ int64 │ int64  │ int64      │ string  │
├───────┼────────┼────────────┼─────────┤
│     1 │      1 │          1 │ TP      │
│     2 │      0 │          0 │ TN      │
│     3 │      0 │          0 │ TN      │
│     4 │      1 │          1 │ TP      │
│     5 │      1 │          0 │ FN      │
│     6 │      0 │          0 │ TN      │
│     7 │      0 │          1 │ FP      │
│     8 │      0 │          1 │ FP      │
│     9 │      1 │          0 │ FN      │
│    10 │      1 │          1 │ TP      │
│     … │      … │          … │ …       │
└───────┴────────┴────────────┴─────────┘

To create the confusion matrix, we’ll group by the outcome, count the occurrences, and use pivot_wider. Widening our data makes it possible to perform column-wise operations on the table expression for metric calculations.

cm = (
    t.group_by("outcome")
    .agg(counted=_.count())
    .pivot_wider(names_from="outcome", values_from="counted")
    .select("TP", "FP", "FN", "TN")
)

cm

┏━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃ TP    ┃ FP    ┃ FN    ┃ TN    ┃
┡━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━┩
│ int64 │ int64 │ int64 │ int64 │
├───────┼───────┼───────┼───────┤
│     4 │     2 │     3 │     3 │
└───────┴───────┴───────┴───────┘

We can plot the confusion matrix to visualize the results.

import matplotlib.pyplot as plt
import seaborn as sns

data = cm.to_pyarrow().to_pydict()

plt.figure(figsize=(6, 4))
sns.heatmap(
    [[data["TP"][0], data["FP"][0]], [data["FN"][0], data["TN"][0]]],
    annot=True,
    fmt="d",
    cmap="Blues",
    cbar=False,
    xticklabels=["Predicted Positive", "Predicted Negative"],
    yticklabels=["Actual Positive", "Actual Negative"],
)
plt.title("Confusion Matrix")
plt.show()

Now that we’ve built a confusion matrix, we’re able to more easily calculate a few common classification metrics.

Metrics

Here are the metrics we’ll calculate as well as a brief description of each.

Accuracy: The proportion of correct predictions out of all predictions made. This measures the overall effectiveness of the model across all classes.
Precision: The proportion of true positive predictions out of all positive predictions made. This tells us how many of the predicted positives were actually correct.
Recall: The proportion of true positive predictions out of all actual positive examples. This measures how well the model identifies all actual positives.
F1 Score: A metric that combines precision and recall into a single score by taking their weighted average. This balances the trade-off between precision and recall, making it especially useful for imbalanced datasets.

We can calculate these metrics using the columns from the confusion matrix we created earlier.

accuracy_expr = (_.TP + _.TN) / (_.TP + _.TN + _.FP + _.FN)
precision_expr = _.TP / (_.TP + _.FP)
recall_expr = _.TP / (_.TP + _.FN)
f1_score_expr = 2 * (precision_expr * recall_expr) / (precision_expr + recall_expr)

metrics = cm.select(
    accuracy=accuracy_expr,
    precision=precision_expr,
    recall=recall_expr,
    f1_score=f1_score_expr,
)

metrics

┏━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ accuracy ┃ precision ┃ recall   ┃ f1_score ┃
┡━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ float64  │ float64   │ float64  │ float64  │
├──────────┼───────────┼──────────┼──────────┤
│ 0.583333 │  0.666667 │ 0.571429 │ 0.615385 │
└──────────┴───────────┴──────────┴──────────┘

A more efficient approach

In the illustrative example above, we used a case expression and pivoted the data to demonstrate where the values would fall in the confusion matrix before performing our metric calculations using the pivoted data. We can actually skip this step using column aggregation.

tp = (t.actual * t.prediction).sum()
fp = t.prediction.sum() - tp
fn = t.actual.sum() - tp
tn = t.actual.count() - tp - fp - fn

accuracy_expr = (t.actual == t.prediction).mean()
precision_expr = tp / t.prediction.sum()
recall_expr = tp / t.actual.sum()
f1_score_expr = 2 * tp / (t.actual.sum() + t.prediction.sum())

print(
    f"accuracy={accuracy_expr.to_pyarrow().as_py()}",
    f"precision={precision_expr.to_pyarrow().as_py()}",
    f"recall_expr={recall_expr.to_pyarrow().as_py()}",
    f"f1_score={f1_score_expr.to_pyarrow().as_py()}",
    sep="\n",
)

accuracy=0.5833333333333334
precision=0.6666666666666666
recall_expr=0.5714285714285714
f1_score=0.6153846153846154

Conclusion

By pushing the computation down to the backend, the performance is as powerful as the backend we’re connected to. This capability allows us to easily scale to different backends without modifying any code.

We hope you give this a try and let us know how it goes. If you have any questions or feedback, please reach out to us on GitHub or Zulip.

Taking a random cube for a walk and making it talk

Cody Peterson — Thu, 26 Sep 2024 00:00:00 GMT

Synthetic data with Ibis, DuckDB, Python UDFs, and Faker.

To follow along, install the required libraries:

pip install 'ibis-framework[duckdb]' faker plotly

A random cube

We’ll generate a random cube of data with Ibis (default DuckDB backend) and visualize it as a 3D line plot:

Show me the code!

From query to plot: Exploring GeoParquet Overture Maps with Ibis, DuckDB, and Lonboard

Naty Clementi and Kyle Barron — Wed, 25 Sep 2024 00:00:00 GMT

With the release of DuckDB 1.1.1, now we have support for reading GeoParquet files! With this exciting update we can query rich datasets from Overture Maps using python via Ibis with the performance of DuckDB.

But the good news doesn’t stop there, since Ibis 9.2, lonboard can plot data directly from an Ibis table, adding more simplicity and speed to your geospatial analysis.

Let’s dive into how these tools come together.

Installation

First make sure you have duckdb>=1.1.1, then install Ibis with the dependencies needed to work with geospatial data using DuckDB.

$ pip install 'duckdb>=1.1.1'
$ pip install 'ibis-framework[duckdb,geospatial]' lonboard

Motivation

Overture Maps is an open-source initiative that provides high-quality, interoperable map data by integrating contributions from leading companies and open data sources to support a wide range of applications.

Overture Maps offers a variety of datasets to query. For example, there is plenty of information about power infrastructure.

Let’s create some plots of the U.S. power infrastructure. We’ll look into power plants and power lines for the lower 48 states (excluding Hawaii and Alaska for simplicity of the bounding box).

Download data

First we import Ibis, its deferred expression object _ , and we use our default backend, DuckDB:

import ibis
from ibis import _

con = ibis.get_backend() # default duckdb backend

With Ibis and DuckDB we can be more specific about the data we want thanks to the filter push down. For example, if we want to select only a few columns and look only at the power infrastructure when can do this as follow.

# look into type infrastructure
url = (
    "s3://overturemaps-us-west-2/release/2024-07-22.0/theme=base/type=infrastructure/*"
)
t = con.read_parquet(url, table_name="infra-usa")

# filter for USA bounding box, subtype="power", and selecting only few columns
expr = t.filter(
    _.bbox.xmin > -125.0,
    _.bbox.ymin > 24.8,
    _.bbox.xmax < -65.8,
    _.bbox.ymax < 49.2,
    _.subtype == "power",
).select(["names", "geometry", "bbox", "class", "sources", "source_tags"])

Note

If you inspect expr, you can see that the filters and projections get pushed down, meaning you only download the data that you asked for.

con.to_parquet(expr, "power-infra-usa.geoparquet")

Now that we have the data lets explore it in Ibis interactive mode and make some beautiful maps.

Data exploration

To explore the data interactively we turn on the interactive mode:

ibis.options.interactive = True

usa_power_infra = con.read_parquet("power-infra-usa.geoparquet")
usa_power_infra

Let’s quickly rename the class column, since this is a reserved word and causes conflicts when using the deferred operator:

usa_power_infra = usa_power_infra.rename(infra_class="class")

We take a look at the different classes of infrastructure under the subtype power:

usa_power_infra.infra_class.value_counts().order_by(
    ibis.desc("infra_class_count")
).preview(max_rows=15)

Looks like we have plant, power_line and minor_line among others.

plants = usa_power_infra.filter(_.infra_class=="plant")
power_lines = usa_power_infra.filter(_.infra_class=="power_line")
minor_lines = usa_power_infra.filter(_.infra_class=="minor_line")

Plotting with Lonboard

Lonboard is a Python plotting library optimized for efficient visualizations of large geospatial data. It integrates well with Ibis and DuckDB, making interactive plotting scalable.

Note

You can try this in your machine, for the purpose the blog file size, we will show screenshots of the visualization

import lonboard
from lonboard.basemap import CartoBasemap # to choose color of basemap

Let’s visualize the power plants

lonboard.viz(
    plants,
    scatterplot_kwargs={"get_fill_color": "red"},
    polygon_kwargs={"get_fill_color": "red"},
    map_kwargs={
        "basemap_style": CartoBasemap.Positron,
        "view_state": {"longitude": -100, "latitude": 36, "zoom": 3},
    },
)

Power plants in the USA

If you are visualizing this in your machine, you can zoom in and see some of the geometry where the plants are located. As an example, we can plot in a small area of California:

plants_CA = plants.filter(
    _.bbox.xmin.between(-118.6, -117.9), _.bbox.ymin.between(34.5, 35.3)
).select(_.names.primary, _.geometry)

lonboard.viz(
    plants_CA,
    scatterplot_kwargs={"get_fill_color": "red"},
    polygon_kwargs={"get_fill_color": "red"},
    map_kwargs={
        "basemap_style": CartoBasemap.Positron,
    },
)

Power plants near Lancaster, CA

We can also visualize together the power_lines and the minor_lines by doing:

lonboard.viz([minor_lines, power_lines])

Minor and Power lines of USA

and that’s how you can visualize ~7 million coordinates from the comfort of your laptop.

>>> power_lines.geometry.n_points().sum()
5329836
>>> minor_lines.geometry.n_points().sum()
1430042

With Ibis and DuckDB working with geospatial data has never been easier or faster. We saw how to query a dataset from Overture Maps with the simplicity of Python and the performance of DuckDB. Last but not least, we saw how simple and quick Lonboard got us from query-to-plot. Together, these libraries make exploring and handling geospatial data a breeze.

Resources

Chat with us on Zulip:

Ibis Zulip Chat

Better PyPI stats with Python

Cody Peterson — Tue, 03 Sep 2024 00:00:00 GMT

Ibis + ClickHouse + Shiny for Python = better PyPI stats.

Overview

PyPI Stats is a great resource for Python package download statistics from PyPI. However, it only contains 180 days of data and lacks more detailed analysis we might be interested in. In this post, we’ll build a dynamic Python application for better PyPI stats using ClickHouse as our data platform, Ibis as our Python data interface, and Shiny for Python as our dashboarding tool.

What about ClickPy?

ClickPy is an existing open source and reproducible project built on the same data with ClickHouse. The primary difference is that ClickPy uses SQL and JavaScript whereas this project is in Python. We also focus on different visualizations and metrics.

Prerequisites

Install the required dependencies:

pip install 'ibis-framework[clickhouse]' plotly

Then run imports and setup:

import ibis
import plotly.express as px
import clickhouse_connect

px.defaults.template = "plotly_dark"
ibis.options.interactive = True

Connecting to ClickHouse

You can connect to the public ClickHouse playground’s PyPI database:

host = "clickpy-clickhouse.clickhouse.com"
port = 443
user = "play"
database = "pypi"

con = ibis.clickhouse.connect(
    host=host,
    port=port,
    user=user,
    database=database,
)
con.list_tables()

['countries',
 'countries_dict',
 'last_updated_dict',
 'projects',
 'pypi',
 'pypi_downloads',
 'pypi_downloads_by_version',
 'pypi_downloads_by_version_mv',
 'pypi_downloads_max_min',
 'pypi_downloads_max_min_mv',
 'pypi_downloads_mv',
 'pypi_downloads_per_day',
 'pypi_downloads_per_day_by_version',
 'pypi_downloads_per_day_by_version_by_country',
 'pypi_downloads_per_day_by_version_by_country_mv',
 'pypi_downloads_per_day_by_version_by_file_type',
 'pypi_downloads_per_day_by_version_by_file_type_mv',
 'pypi_downloads_per_day_by_version_by_installer_by_type',
 'pypi_downloads_per_day_by_version_by_installer_by_type_by_country',
 'pypi_downloads_per_day_by_version_by_installer_by_type_by_country_mv',
 'pypi_downloads_per_day_by_version_by_installer_by_type_mv',
 'pypi_downloads_per_day_by_version_by_python',
 'pypi_downloads_per_day_by_version_by_python_by_country',
 'pypi_downloads_per_day_by_version_by_python_by_country_mv',
 'pypi_downloads_per_day_by_version_by_python_mv',
 'pypi_downloads_per_day_by_version_by_system',
 'pypi_downloads_per_day_by_version_by_system_by_country',
 'pypi_downloads_per_day_by_version_by_system_by_country_mv',
 'pypi_downloads_per_day_by_version_by_system_mv',
 'pypi_downloads_per_day_by_version_mv',
 'pypi_downloads_per_day_mv',
 'pypi_downloads_per_month',
 'pypi_downloads_per_month_mv']

Top packages by downloads

Let’s start by looking at the most downloaded packages:

overall_t = con.table("pypi_downloads")

top_k = 10_000
overall_t = (
    overall_t.order_by(ibis.desc("count"))
    .limit(top_k)
    .mutate(rank=1 + ibis.row_number().over(order_by=ibis.desc("count")))
    .rename({"downloads": "count"})
    .relocate("rank")
    .order_by("rank")
)
overall_t

┏━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ rank  ┃ project         ┃ downloads   ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ int64 │ !string         │ !int64      │
├───────┼─────────────────┼─────────────┤
│     1 │ boto3           │ 25779423674 │
│     2 │ urllib3         │ 16339094961 │
│     3 │ botocore        │ 15210036424 │
│     4 │ requests        │ 13514466781 │
│     5 │ setuptools      │ 12656259406 │
│     6 │ idna            │ 11843633399 │
│     7 │ certifi         │ 11823436764 │
│     8 │ s3transfer      │ 11059293057 │
│     9 │ six             │ 10982223268 │
│    10 │ python-dateutil │ 10756832919 │
│     … │ …               │           … │
└───────┴─────────────────┴─────────────┘

Analyzing downloads for a package

Let’s choose a package to analyze:

project = "clickhouse-connect"

And see where it ranks in the top downloads:

overall_t.filter(overall_t["project"] == project)

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ rank  ┃ project            ┃ downloads ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ int64 │ !string            │ !int64    │
├───────┼────────────────────┼───────────┤
│  2303 │ clickhouse-connect │  23148161 │
└───────┴────────────────────┴───────────┘

Let’s look at downloads per day by various categories for this package:

downloads_t = con.table(
    "pypi_downloads_per_day_by_version_by_installer_by_type_by_country"
).filter(ibis._["project"] == project)
downloads_t

┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ project            ┃ version ┃ date       ┃ installer    ┃ type        ┃ country_code ┃ count  ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━┩
│ !string            │ !string │ !date      │ !string      │ !string     │ !string      │ !int64 │
├────────────────────┼─────────┼────────────┼──────────────┼─────────────┼──────────────┼────────┤
│ clickhouse-connect │ 0.2.10  │ 2024-08-29 │ bandersnatch │ bdist_wheel │ US           │     18 │
│ clickhouse-connect │ 0.2.10  │ 2024-08-29 │ bandersnatch │ sdist       │ US           │      1 │
│ clickhouse-connect │ 0.2.8   │ 2024-08-29 │ bandersnatch │ bdist_wheel │ US           │      9 │
│ clickhouse-connect │ 0.2.9   │ 2024-08-29 │ bandersnatch │ bdist_wheel │ US           │     16 │
│ clickhouse-connect │ 0.2.9   │ 2024-08-29 │ bandersnatch │ sdist       │ US           │      1 │
│ clickhouse-connect │ 0.3.0   │ 2024-08-29 │ bandersnatch │ bdist_wheel │ US           │     19 │
│ clickhouse-connect │ 0.3.0   │ 2024-08-29 │ bandersnatch │ sdist       │ US           │      1 │
│ clickhouse-connect │ 0.3.1   │ 2024-08-29 │ ~            │ bdist_wheel │ US           │     26 │
│ clickhouse-connect │ 0.3.1   │ 2024-08-29 │ ~            │ sdist       │ US           │      1 │
│ clickhouse-connect │ 0.3.1   │ 2024-08-29 │ Browser      │ bdist_wheel │ US           │      1 │
│ …                  │ …       │ …          │ …            │ …           │ …            │      … │
└────────────────────┴─────────┴────────────┴──────────────┴─────────────┴──────────────┴────────┘

We might be interested in the day-of-week seasonality of downloads:

def day_of_week_bar(t):
    t = t.mutate(day_of_week=t["date"].day_of_week.full_name())
    t = t.group_by("day_of_week").agg(downloads=ibis._["count"].sum())
    c = px.bar(
        t,
        x="day_of_week",
        y="downloads",
        category_orders={
            "day_of_week": [
                "Sunday",
                "Monday",
                "Tuesday",
                "Wednesday",
                "Thursday",
                "Friday",
                "Saturday",
            ]
        },
    )
    return c


day_of_week_bar(downloads_t)

Or the rolling 28-day downloads metric:

def rolling_downloads(t, days=28):
    t = t.mutate(
        timestamp=t["date"].cast("timestamp"),
    )
    t = t.group_by("timestamp").agg(downloads=ibis._["count"].sum())
    t = t.select(
        "timestamp",
        rolling_downloads=ibis._["downloads"]
        .sum()
        .over(
            ibis.window(
                order_by="timestamp",
                preceding=days,
                following=0,
            )
        ),
    ).order_by("timestamp")

    c = px.line(
        t,
        x="timestamp",
        y="rolling_downloads",
    )

    return c


rolling_downloads(downloads_t)

Or rolling 28-days downloads by version with a few options for how to group versions:

def rolling_downloads_by_version(t, days=28, version_style="major.minor"):
    t = t.mutate(
        timestamp=t["date"].cast("timestamp"),
    )

    match version_style:
        case "major":
            t = t.mutate(version=t["version"].split(".")[0])
        case "major.minor":
            t = t.mutate(
                version=t["version"].split(".")[0] + "." + t["version"].split(".")[1]
            )
        case _:
            pass

    t = t.group_by("timestamp", "version").agg(downloads=ibis._["count"].sum())

    t = t.select(
        "timestamp",
        "version",
        rolling_downloads=ibis._["downloads"]
        .sum()
        .over(
            ibis.window(
                order_by="timestamp",
                group_by="version",
                preceding=28,
                following=0,
            )
        ),
    ).order_by("timestamp")

    c = px.line(
        t,
        x="timestamp",
        y="rolling_downloads",
        color="version",
        category_orders={
            "version": reversed(
                sorted(
                    t.distinct(on="version")["version"].to_pyarrow().to_pylist(),
                    key=lambda x: tuple(int(y) for y in x.split(".") if y.isdigit()),
                )
            )
        },
    )
    return c


rolling_downloads_by_version(downloads_t)

Or a bar chart of downloads grouped by a category:

def group_bar(t, group_by="installer", log_y=True):
    t = t.mutate(timestamp=t["date"].cast("timestamp"))
    t = t.group_by(group_by).agg(downloads=ibis._["count"].sum())
    t = t.order_by(ibis.desc("downloads"))

    c = px.bar(
        t,
        x=group_by,
        y="downloads",
        log_y=log_y,
    )
    return c


group_bar(downloads_t)

More examples

Since we’re just writing Python, we’ve already organized code into functions for reuse. We can rerun our above analytics on a different package by changing the project variable and adjusting our table accordingly. We’ll demonstrate this with a few more packages below.

Notice you could also pass in Ibis tables from different backends, not just ClickHouse, to these functions!

package = "pyarrow"

t = con.table(
    "pypi_downloads_per_day_by_version_by_installer_by_type_by_country"
).filter(ibis._["project"] == package)

day_of_week_bar(t)

rolling_downloads(t)

rolling_downloads_by_version(t, version_style="major")

group_bar(t, group_by="installer")

package = "chdb"

t = con.table(
    "pypi_downloads_per_day_by_version_by_installer_by_type_by_country"
).filter(ibis._["project"] == package)

day_of_week_bar(t)

rolling_downloads(t)

rolling_downloads_by_version(t, version_style="major.minor")

group_bar(t, group_by="installer")

package = "ibis-framework"

t = con.table(
    "pypi_downloads_per_day_by_version_by_installer_by_type_by_country"
).filter(ibis._["project"] == package)

day_of_week_bar(t)

rolling_downloads(t)

rolling_downloads_by_version(t, version_style="major")

group_bar(t, group_by="installer")

Shiny for Python application

We can create an interactive Shiny with Python application using the code above to serve as a dashboard for better PyPI stats:

Tip

See the GitHub repository for the most up-to-date code.

Reproducing and contributing

The code is available on GitHub. Feel free to open an issue or pull request if you have any suggested improvements.

Farewell pandas, and thanks for all the fish.

Gil Forsyth — Mon, 26 Aug 2024 00:00:00 GMT

TL; DR: we are deprecating the pandas and dask backends and will be removing them in version 10.0.

There is no feature gap between the pandas backend and our default DuckDB backend, and DuckDB is much more performant. pandas DataFrames will still be available as format for getting data to and from Ibis, we just won’t support using pandas to execute queries.

Most of the rationale below applies to the Dask backend since it has so much in common with pandas. Dask is a great project and people should continue to use it outside the Ibis context.

Why `pandas`? And a bit of Ibis history

Way back in the early days of Ibis, there was only one backend: Impala. Not everyone used Impala (mindblowing, we know), and so it wasn’t too long until the Postgres backend was added (by the inimitable Phillip Cloud).

These two backends were both featureful, but there was a big problem with adoption: Want to try out Ibis? You need to install Impala or Postgres first.

Not an insurmountable problem, but a LOT more work than “just pip install ” – which prompted the question, how can a prospective Ibis user take the API for a spin without requiring a DBA or extra infrastructure beyond a laptop?

The obvious answer (at the time) was to use the only in-memory DataFrame engine around and wire up a pandas backend.

The agony and the agony

pandas was the best option at the time, and it allowed new users to try out Ibis. But, it never fit well into the model of data analysis that Ibis strives for. The pandas backend has more specialized code than any other backend, because it is so fundamentally different than all the other systems Ibis works with.

Deferred vs Eager

pandas is inherently an eager engine – every time you hit Enter you are computing an intermediate result. Ibis uses a deferred execution model, similar to what nearly all SQL backends use, that enables query planning and optimization passes.

Trying to make a pandas interface that behaves in a deferred way is hard.

One of the unfortunate effects of this mismatch is that, unlike our other backends, the pandas backend is often much slower than just using pandas directly.

And to provide this suboptimal experience, we have a few thousand lines of code that are only used in the pandas backend.

`NaN` vs `NULL`

The choice was made a long time ago to accept using NaN as the marker for missing values in pandas. This is because NumPy has a notion of NaN, but a Python None would lead to an object-dtype and poor performance.

Practicality beats purity, but this is a horrible decision to have to make. Ibis doesn’t have to make it with any other backend, because NULL indicates a missing value, and NaN is Not a Number.

Those are fundamentally different ideas and it is an ongoing headache for Ibis to try to pretend that they aren’t.

Data types

The new Arrow-backed types in pandas are a great improvement and we’ll leave it at that.

Misleading new users

People reach for what is familiar. When you try Ibis for the first time, we’re asking you to both a) try Ibis and b) pick a backend. We have defaults to try to help with this, but it can be confusing at first.

We have many reports from new users that “Ibis is slow”. What this almost always means is that they tried the pandas backend (because they know pandas) and they are having a less-than-great time.

If they tried DuckDB or Polars, instead, they would have a much easier time getting things going.

Feature parity

This is the one of the strongest reasons to drop the pandas backend – it is redundant. The DuckDB backend can seamlessly query pandas DataFrames, supports several flavors of UDF, and can read and write parquet, CSV, JSON, and other formats.

There is a reason DuckDB is our default backend: it’s easy to install, it runs locally, it’s blazing fast, and it interacts well with the Python ecosystem. Those are all the reasons we added pandas as a backend in the first place, but with the added benefit of blazing-fast results, and no type-system headaches.

Using IbisML and DuckDB for a Kaggle competition: credit risk model stability

Jiting Xu — Thu, 22 Aug 2024 00:00:00 GMT

Introduction

In this post, we’ll demonstrate how to use Ibis and IbisML end-to-end for the credit risk model stability Kaggle competition.

Load data and perform feature engineering on DuckDB backend using IbisML
Perform last-mile ML data preprocessing on DuckDB backend using IbisML
Train two models using different frameworks:
- An XGBoost model within a scikit-learn pipeline.
- A neural network with PyTorch and PyTorch Lightning.

The aim of this competition is to predict which clients are more likely to default on their loans by using both internal and external data sources.

To get started with Ibis and IbisML, please refer to the websites:

Ibis: An open-source dataframe library that works with any data system.
IbisML: A library for building scalable ML pipelines.

Prerequisites

To run this example, you’ll need to download the data from Kaggle website with a Kaggle user account and install Ibis, IbisML, and the necessary modeling library.

Download data

You need a Kaggle account to download the data. If you do not have one, feel free to register one.

Option 1: Manual download
- Log into your Kaggle account and download all data from this link, unzip the files, and save them to your local disk.
Option 2: Kaggle API
- Go to your Kaggle Account Settings.
- Under the API section, click on Create New API Token. This will download the kaggle.json file to your computer.
- Place the kaggle.json file in the correct directory, normally it is under your home directory ~/.kaggle:
```
mkdir ~/.kaggle
mv ~/Downloads/kaggle.json ~/.kaggle
```
- Install Kaggle CLI and download the data:
```
pip install kaggle
kaggle competitions download -c home-credit-credit-risk-model-stability
unzip home-credit-credit-risk-model-stability.zip
```

Install libraries

To use Ibis and IbisML with the DuckDB backend for building models, you’ll need to install the necessary packages. Depending on your preferred machine learning framework, you can choose one of the following installation commands:

For PyTorch-based models:

pip install 'ibis-framework[duckdb]' ibis-ml torch pytorch-lightning

For XGBoost and scikit-learn-based models:

pip install 'ibis-framework[duckdb]' ibis-ml xgboost[scikit-learn]

Import libraries:

import ibis
import ibis.expr.datatypes as dt
from ibis import _
import ibis_ml as ml
from pathlib import Path
from glob import glob

# enable interactive mode for ibis
ibis.options.interactive = True

Set the backend for computing:

con = ibis.duckdb.connect()
# remove the black bars from duckdb's progress bar
con.raw_sql("set enable_progress_bar = false")
# DuckDB is the default backend for Ibis
ibis.set_backend(con)

Set data path:

# change the root path to yours
ROOT = Path("/Users/claypot/Downloads/home-credit-credit-risk-model-stability")
TRAIN_DIR = ROOT / "parquet_files" / "train"
TEST_DIR = ROOT / "parquet_files" / "test"

Data loading and processing

We’ll use Ibis to read the Parquet files and perform the necessary processing for the next step.

Directory structure and tables

Since there are many data files, let’s start by examining the directory structure and tables within the train directory:

# change this to your directory
tree -L 2 ~/Downloads/home-credit-credit-risk-model-stability/parquet_files/train

Click to show data files

~/Downloads/home-credit-credit-risk-model-stability/parquet_files/train
├── train_applprev_1_0.parquet
├── train_applprev_1_1.parquet
├── train_applprev_2.parquet
├── train_base.parquet
├── train_credit_bureau_a_1_0.parquet
├── train_credit_bureau_a_1_1.parquet
├── train_credit_bureau_a_1_3.parquet
├── train_credit_bureau_a_2_0.parquet
├── train_credit_bureau_a_2_1.parquet
├── train_credit_bureau_a_2_10.parquet
├── train_credit_bureau_a_2_2.parquet
├── train_credit_bureau_a_2_3.parquet
├── train_credit_bureau_a_2_4.parquet
├── train_credit_bureau_a_2_5.parquet
├── train_credit_bureau_a_2_6.parquet
├── train_credit_bureau_a_2_7.parquet
├── train_credit_bureau_a_2_8.parquet
├── train_credit_bureau_a_2_9.parquet
├── train_credit_bureau_b_1.parquet
├── train_credit_bureau_b_2.parquet
├── train_debitcard_1.parquet
├── train_deposit_1.parquet
├── train_other_1.parquet
├── train_person_1.parquet
├── train_person_2.parquet
├── train_static_0_0.parquet
├── train_static_0_1.parquet
├── train_static_cb_0.parquet
├── train_tax_registry_a_1.parquet
├── train_tax_registry_b_1.parquet
└── train_tax_registry_c_1.parquet

The train_base.parquet file is the base table, while the others are feature tables. Let’s take a quick look at these tables.

Base table

The base table (train_base.parquet) contains the unique ID, a binary target flag and other information for the training samples. This unique ID will serve as the linking key for joining with other feature tables.

case_id - This is the unique ID for each loan. You’ll need this ID to join feature tables to the base table. There are about 1.5m unique loans.
date_decision - This refers to the date when a decision was made regarding the approval of the loan.
WEEK_NUM - This is the week number used for aggregation. In the test sample, WEEK_NUM continues sequentially from the last training value of WEEK_NUM.
MONTH - This column represents the month when the approval decision was made.
target - This is the binary target flag, determined after a certain period based on whether or not the client defaulted on the specific loan.

Here is several examples from the base table:

Show code to get the top 5 rows of base table

Querying 1TB on a laptop with Python dataframes

Cody Peterson — Mon, 08 Jul 2024 00:00:00 GMT

TPC-H benchmark at sf=1024 via DuckDB, DataFusion, and Polars on a MacBook Pro with 96GiB of RAM.

pandas requires your dataframe to fit in memory. Out-of-memory (OOM) errors are common when working on larger datasets, though the corresponding size of data on disk can be surprising. The creator of pandas and Ibis noted in “Apache Arrow and the ‘10 Things I Hate About pandas’”:

To put it simply, we weren’t thinking about analyzing 100 GB or 1 TB datasets in 2011. [In 2017], my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB dataset, you should really have about 64, preferably 128 GB of RAM if you want to avoid memory management problems. This comes as a shock to users who expect to be able to analyze datasets that are within a factor of 2 or 3 the size of their computer’s RAM.

Today with Ibis you can reliably and efficiently process a 1TB dataset on a laptop with <1/10th the RAM.

Important

This represents a 50-100X improvement in RAM requirements for Python dataframes in just 7 years thanks to composable data systems and hard work by the DuckDB team.

Exploring the data with Python dataframes

I’ve generated ~1TB (sf=1024) of TPC-H data on my MacBook Pro with 96 GiB of RAM. We’ll start exploring it with pandas, Polars, and Ibis and discuss where and why they start to struggle.

Generating the data

See the previous post for instructions on generating the data. I used bench gen-data -s 1024 -n 128, partitioning the data to avoid OOM errors while it generated.

I’d recommend instead generating a smaller scale factor and copying it as many times as needed, as generating the data at sf=1024 can take a long time.

To follow along, install the required packages:

pip install pandas 'ibis-framework[duckdb,datafusion]' polars-u64-idx plotly

Why polars-u64-idx?

We need to use polars-u64-idx instead of polars to work with >4.2 billion rows.

Imports and setup:

import os
import glob
import ibis
import pandas as pd
import polars as pl
import plotly.express as px

px.defaults.template = "plotly_dark"
ibis.options.interactive = True

Let’s check the number of rows across all tables in the TPC-H data:

Show code to get number of rows in TPC-H data

Ibis benchmarking: DuckDB, DataFusion, Polars

Cody Peterson — Mon, 24 Jun 2024 00:00:00 GMT

The best benchmark is your own workload on your own data.

Key considerations

The purpose of this post is to explore some benchmarking data with Ibis. We’ll compare three modern single-node query engines, explore the Ibis API as a great choice for each of them, and discuss the results.

The benchmark

Not an official TPC-H benchmark

This is not an official TPC-H benchmark. We ran a derivate of the TPC-H benchmark.

The TPC-H benchmark is a benchmark for databases and, increasingly, dataframes! It consists of 22 queries across 8 tables. The SQL (or dataframe) code representing the queries is designed to test the performance of a query engine on a variety of tasks including filtering, aggregation, and joins. SQL queries are defined by the TPC-H benchmark. We run the SQL queries and equivalent dataframe code via Ibis and Polars APIs.

The data for the benchmark can be generated at any scale factor, which roughly corresponds to the size of the data in memory in gigabytes. For instance, a scale factor of 10 would be about 10GB of data in memory.

The engines, the API, the code

We’ll use three modern single-node OLAP engines (DuckDB, DataFusion, Polars) with the Ibis API via two coding paradigms (dataframe and SQL). Ibis provides a consistent API across 20+ backends, including these three. We run SQL code through Ibis in addition to dataframe code to get a sense of any overhead in Ibis dataframe code.

Note

Ibis dataframe code generates SQL for the DuckDB and DataFusion backends and generates Polars API dataframe code for the Polars backend.

Honorable mention: chDB

chDB would be another great single-node OLAP engine to benchmark. We don’t because it’s not currently a backend for Ibis, though there has been work done to make it one.

If you’re interested in contributing to Ibis, a new backend like chDB could be a great project for you!

9/22 queries for Ibis with the Polars backend fail from lack of scalar subquery support. Due to this and relatively experimental SQL support in Polars, we’ve opted to run on the Polars API directly in this iteration of the benchmark. This is done with the LazyFrames API and no streaming engine (per the Polars team’s recommendation). This also allows us to compare the performance of the Polars backend through Ibis with the Polars API directly for the queries that do succeed.

Failing queries

Queries fail for one of two reasons:

The query doesn’t work in the given system
The query otherwise failed on a given run

We’ll note the cases of the first below. The second is usually due to memory pressure and will be seen at higher scale factors throughout the data.

Failing DataFusion queries

Queries 16, 21, and 22 fail for the DataFusion backend via Ibis dataframe code, and query 16 fails through SQL. Note that all TPC-H SQL queries successfully run through DataFusion directly – Ibis generates SQL that hits a bug with DataFusion that has already been fixed. We expect these queries to work in the next iteration of this benchmark coming soon.

Failing Polars queries

Queries 11, 13-17, and 20-22 fail for the Polars backend via Ibis dataframe code. These all fail due to lack of scalar subquery support in the backend. I’ve opened an issue for tracking and discussion.

Interested in contributing?

Increasing coverage of operations for a backend is a great place to start!

How queries are written

See the source code for the exact queries used in this iteration of the benchmark. Polars recently updated their TPC-H queries, so the next iteration of this benchmark would use those.

Queries were adapted from Ibis TPC-H queries and Polars TPC-H queries. The first 10 Ibis dataframe queries were translated from the Polars dataframe queries, while the rest were directly adapted from the Ibis repository. The SQL strings were adapted from the Ibis repository.

How queries are run

See the source code and methodology for more details. In short:

data is generated as a Parquet file per table
- standard DuckDB Parquet writer is used
- data is always downloaded onto a compute instance (no cloud storage reads)
decimal types are converted to floats after reading
- works around several issues
- in the next iteration of this benchmark, we’ll use the decimal type
each query is run three times per configuration (system, scale factor, instance type)
we measure the time to write the results of the query to a Parquet file
- this includes reading the Parquet file(s) and executing the query

Biases

My name is Cody and I’m a Senior Technical Product Manager at Voltron Data. I am a contributor to the Ibis project and employed to work on it – I’m biased in favor of Ibis and the composable data ecosystem.

Ibis is an independently governed open source project that is not owned by Voltron Data, though several steering committee members are employed by Voltron Data. You can read more about why Voltron Data supports Ibis, in addition to open source projects like Apache Arrow and Substrait.

Voltron Data is a Gold Supporter of the DuckDB Foundation and has a commercial relationship with DuckDB Labs with regular syncs I tend to attend. I also use MotherDuck to host our Ibis analytics dashboard data.

Results and analysis

We’ll use Ibis to analyze some of the benchmarking data.

Tip

We’ll only look at a small subset of the data in this post.

All the data is public, so you can follow along with the code and explore the data yourself. You can also see the Ibis benchmarking Streamlit app for further analysis.

Reading the data

To follow along, install the required Python packages:

pip install gcsfs 'ibis-framework[duckdb]' plotly

The data is stored in a public Google Cloud Storage (GCS) bucket:

import os
import gcsfs

BUCKET = "ibis-bench"

dir_name = os.path.join(BUCKET, "bench_logs_v2", "cache")

fs = gcsfs.GCSFileSystem()
fs.ls(dir_name)[-5:]

1: Imports
2: The public GCS bucket name
3: The directory in the bucket where the data is stored
4: Create a GCS filesystem object
5: List the last 5 files in the directory

['ibis-bench/bench_logs_v2/cache/file_id=b6236086-7fff-4569-8731-b97a635243bd.parquet',
 'ibis-bench/bench_logs_v2/cache/file_id=cbc0c7b1-e659-4adb-8c80-4077cd4d39ab.parquet',
 'ibis-bench/bench_logs_v2/cache/file_id=d91454ad-2ddd-408a-bbfd-6b159dd2132b.parquet',
 'ibis-bench/bench_logs_v2/cache/file_id=debc7203-f366-44d2-94f1-2518e6f7425f.parquet',
 'ibis-bench/bench_logs_v2/cache/file_id=e875d852-f7e7-473c-9440-92b8f2445f3a.parquet']

To start exploring the data, let’s import Ibis and Plotly, set some options, and register the GCS filesystem with the default (DuckDB) backend:

import ibis
import plotly.express as px

px.defaults.template = "plotly_dark"

ibis.options.interactive = True
ibis.options.repr.interactive.max_rows = 22
ibis.options.repr.interactive.max_length = 22
ibis.options.repr.interactive.max_columns = None

con = ibis.get_backend()
con.register_filesystem(fs)

1: Import Ibis
2: Import Plotly
3: Set the Plotly template to dark
4: Enable interactive mode for Ibis
5: Set the maximum number of rows to display in interactive mode
6: Set the maximum length of nested types to display in interactive mode
7: Set the maximum number of columns to display in interactive mode
8: Get the default (DuckDB) backend
9: Register the GCS filesystem with the default backend

Now read the data and take a look at the first few rows:

t = (
    ibis.read_parquet(f"gs://{dir_name}/file_id=*.parquet")
    .mutate(
        timestamp=ibis._["timestamp"].cast("timestamp"),
    )
    .relocate(
        "instance_type",
        "system",
        "sf",
        "query_number",
        "execution_seconds",
        "timestamp",
    )
    .cache()
)
t.head()

1: Assign the table to a variable
2: Read the Parquet files from GCS
3: Cast the timestamp column to a timestamp type
4: Reorder the columns
5: Cache the table to avoid re-reading cloud data
6: Display the first few rows

┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ instance_type ┃ system          ┃ sf    ┃ query_number ┃ execution_seconds ┃ timestamp                  ┃ session_id                           ┃ n_partitions ┃ file_type ┃ file_id                                      ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ string        │ string          │ int64 │ int64        │ float64           │ timestamp(6)               │ uuid                                 │ int64        │ string    │ string                                       │
├───────────────┼─────────────────┼───────┼──────────────┼───────────────────┼────────────────────────────┼──────────────────────────────────────┼──────────────┼───────────┼──────────────────────────────────────────────┤
│ n2-standard-4 │ polars-lazy     │   128 │           16 │          9.503613 │ 2024-06-10 08:04:31.233704 │ 6708e5d3-2b8c-4ce0-adf8-65ce94e0bff1 │            1 │ parquet   │ 0949deaf-5f8f-4c29-b2ca-934e07173223.parquet │
│ n2-standard-4 │ ibis-duckdb     │    64 │            7 │         35.826295 │ 2024-06-10 21:05:18.423375 │ 9a00385f-22b4-42df-ab3d-c63ed1a33a2e │            1 │ parquet   │ 0949deaf-5f8f-4c29-b2ca-934e07173223.parquet │
│ n2-standard-4 │ ibis-duckdb     │   128 │           16 │          7.376196 │ 2024-06-11 03:44:22.901852 │ acb56c6b-b0d5-4bbc-8791-3542b62bd193 │            1 │ parquet   │ 0949deaf-5f8f-4c29-b2ca-934e07173223.parquet │
│ n2-standard-4 │ ibis-datafusion │    16 │            7 │          8.655290 │ 2024-06-09 20:29:31.833510 │ a07fe07d-7a08-4802-b8ae-918e66e2d868 │            1 │ parquet   │ 0949deaf-5f8f-4c29-b2ca-934e07173223.parquet │
│ n2-standard-4 │ ibis-duckdb-sql │     1 │           10 │          0.447325 │ 2024-06-10 08:11:31.244609 │ d523eec6-d2de-491d-b541-348c6b5bfc65 │            1 │ parquet   │ 0949deaf-5f8f-4c29-b2ca-934e07173223.parquet │
└───────────────┴─────────────────┴───────┴──────────────┴───────────────────┴────────────────────────────┴──────────────────────────────────────┴──────────────┴───────────┴──────────────────────────────────────────────┘

We’ll also create a table with details on each instance type including the CPU type, number of cores, and memory in gigabytes:

Show code to get instance details

Ibis - Now flying on Snowflake

Phillip Cloud — Wed, 19 Jun 2024 00:00:00 GMT

Ibis allows you to push down compute operations on your data where it lives, with the performance being as powerful as the backend you’re connected to. But what happens if Ibis is running inside the backend you’re connected to?

In this post, we will discuss how we got Ibis running on a Snowflake virtual warehouse.

Why would we want to do this?

Snowflake has released several features to enable users to execute native Python code on the platform. These features include a new notebook development interface, Streamlit in Snowflake, the Native App framework, and Python within functions and stored procedures.

If users could use Ibis directly within the platform, developers could more easily switch between a local execution engine during development and efficiently deploy and operationalize that same code on Snowflake.

But this isn’t without its challenges; there were a few things we needed to figure out, and these are the questions we will answer throughout the post.

How can we get an Ibis connection to Snowflake - from within Snowflake?
How can we use third-party packages in Snowflake?
How are we going to test this to ensure it works?

Getting the Ibis connection

The release of Ibis 9.2 includes the introduction of a new method, from_connection to provide users with a convenient mechanism to take an existing Snowpark session and create an Ibis Snowflake backend instance with it.

Here’s what this looks like:

import ibis
import snowflake.snowpark as sp

session = sp.Session.builder.create()
con = ibis.snowflake.from_connection(session)

This connection uses the same session within Snowflake, so temporary objects can be accessed using Snowpark or Ibis in the same process! The contexts of stored procedures already have a session available, meaning we can use this new method and start writing Ibis expressions.

The way this works is that Ibis plucks out an attribute on the Snowpark session, which gives us the snowflake-connector-python SnowflakeConnection instance used by Snowpark.

Since Ibis uses snowflake-connector-python for all Snowflake-related connection we just reuse that existing instance.

Uploading third-party packages

Snowflake has many packages already made available out of the box through the Snowflake Anaconda channel, but unfortunately, Ibis and a few of its dependencies aren’t available. Packages containing pure Python code can be uploaded to stages for use within the platform, so we devised a clever solution to upload and reference these to get them working.

import os
import shutil
import tempfile


def add_packages(d, session):
    import parsy
    import pyarrow_hotfix
    import rich
    import sqlglot
    import ibis

    for module in (ibis, parsy, pyarrow_hotfix, rich, sqlglot):
        pkgname = module.__name__
        pkgpath = os.path.join(d, pkgname)
        shutil.copytree(os.path.dirname(module.__file__), pkgpath)
        session.add_import(pkgname, import_path=pkgname)


d = tempfile.TemporaryDirectory()
os.chdir(d.name)
add_packages(d.name, session)

We can now register a stored procedure that imports these modules and is able to reference some of the additional dependencies that are already available.

session.sproc.register(
        ibis_sproc,
        return_type=sp.types.StructType(),
        name="THE_IBIS_SPROC",
        imports=["ibis", "parsy", "pyarrow_hotfix", "sqlglot", "rich"],
        packages=[
            "snowflake-snowpark-python",
            "toolz",
            "atpublic",
            "pyarrow",
            "pandas",
            "numpy",
        ],
)

Testing!

While this is a clever solution, we must ensure it works consistently. A special unit test has been written for this exact case! The test creates a stored procedure by adding the necessary imports to the Snowpark session. Within the stored procedure, we define an Ibis expression, and we use the Ibis to_sql method to extract the generated SQL to pass to Snowpark to return a Snowpark DataFrame!

Conclusion

While it’s usually pretty easy to add new backends with Ibis, this was the first instance of supporting an additional interface to an existing backend.

We hope you take this for a spin! If you run into any challenges or want additional support, open an issue or join us on Zulip and let us know!

Unlocking data insights with Ibis and SQLMesh

Chloe He — Tue, 21 May 2024 00:00:00 GMT

Overview

Have you ever needed to learn new dialects of database languages as a data scientist or struggled with the differences between database languages? Does your company manage different production pipelines with multiple databases or engines? Have you needed to rewrite data pipelines from experimentation to deployment?

These are challenges that SQLMesh and Ibis together can solve.

SQLMesh is a next-generation data transformation and modeling framework. It aims to be easy to use, correct, and efficient and is maintained by the Tobiko Data team. It helps you scalably, reliably and safely modify your data pipelines because it understands SQL and can make intelligent updates instead of stringing scripts together. SQLMesh boasts several future-proof features such as automatic data contracts, virtual data environments and snapshots, extensive change summaries (before updates are applied!) and column-level lineage out of the box.

We walk through an example together to show you how to harness the full potential of your data analytics workflow and how SQLMesh and Ibis can work together hand-in-hand. Read the full article on SQLMesh blog!

In today’s data-driven world, the ability to efficiently analyze and derive insights from vast amounts of data is paramount. Leveraging powerful open-source tools like SQLMesh and Ibis can streamline this process, enabling you to easily manipulate and query data.

Let us know how you’re using SQLMesh and Ibis together in your use case!

Ibis 9.0: SQLGlot-ification

Ibis team — Wed, 01 May 2024 00:00:00 GMT

Overview

Ibis 9.0 wraps up “the big refactor”, completing the transition from SQLAlchemy to SQLGlot and drastically simplifying the codebase. This is a big step toward stabilized internals and allows us to more easily add new features and backends going forward. Look out for a dedicated blog post on the refactor soon.

Tip

One long-standing issue with Ibis has been chained joins, which now work as a user would expect. This was a major motivation for the refactor.

Ibis 9.0 also adds new functionality, fixes many bugs, and welcomes two new committers to the project!

Meet the Ibis team!

Members of the core Ibis team will be at PyCon 2024 in Pittsburg, PA, USA in May and SciPy 2024 in Tacoma, WA, USA in July! Stop by at one of the talks or tutorials to say hi in English and/or Español:

Note

The schedule for SciPy 2024 is not available yet, keep an eye out for Ibis sessions.

You can also listen to a recent podcast episode with Real Python or another with :probabl. where Phillip Cloud, lead maintainer of Ibis, talks about the project.

More podcasts and talks are in the works, so stay tuned!

New committers

We’re excited to welcome Nick Crews and Naty Clementi as the newest committers to the Ibis project!

Nick has been a long-time contributor to Ibis, one of the most active people on the issue tracker, and increasingly involved in the community. We’re excited to have him on board!

Naty has been contributing to Ibis for a while now, leading the effort to add geospatial support to the DuckDB backend in addition to various other contributions. We’re excited to have her on board!

9.0 contributors

Ibis 9.0 had contributions from many people, including commits from:

Phillip Cloud
Krisztián Szűcs
Gil Forsyth
Jim Crist-Harif
Cody Peterson
Nick Crews
Naty Clementi
Chloe He
Nicola Coretti
Deepyaman Datta
Péter Gyarmati
Jiting Xu
Mehmet Fatih Aktas
Tyler White
Bryce Mecum
Riezebos
Chelsea Lin
Chip Huyen
Dan Lovell
Daniel Mesejo
Kexiang Wang
Mark Needham
Ray Bell
Thierry Jean
pieter-factful
saschahofmann

Thank you to everyone who contributed to this release! And a special thanks to Krisztián Szűcs for his work on the internal representation and SQLGlot refactor work, it has drastically improved the Ibis codebase.

With this refactor, it’s never been easier to contribute to Ibis! Get on touch via GitHub or Zulip if you’re interested.

Backends

No new backends were added in this release, unless you count the April Fools’ Day Unix backend! As always, there have been backend-specific improvements and bug fixes. Some highlights are below.

Check the full changelog for more details.

If you’re new to Ibis, see how to install and the getting started tutorial.

To follow along with this blog, ensure you’re on 'ibis-framework>=9,<10'. First, we’ll setup Ibis for interactive use:

import ibis
import ibis.selectors as s

ibis.options.interactive = True
ibis.options.repr.interactive.max_rows = 3

And fetch the penguins dataset:

t = ibis.examples.penguins.fetch()
t

┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
│ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

Snowflake

You can now run Ibis inside of a Snowflake stored procedure! Look for a blog on this coming soon.

DuckDB

In this release, asof joins are also now supported with a tolerance parameter.

With DuckDB’s addition of MySQL as a database it can attach to, we’ve added a read_mysql function to Ibis inline with read_postgres and read_sqlite.

You can now cast binary data to geometry in the DuckDB backend, enabling use of regular Parquet files:

zones = ibis.examples.zones.fetch().relocate("geom")
zones.to_parquet("zones.parquet")

1: Fetch the zones example data with the geometry column.
2: Write to a Parquet file, converting the geometry column to binary.

zones = ibis.read_parquet("zones.parquet")
zones.schema()

1: Load in the zones Paruqet file.
2: Notice the geometry column is binary.

ibis.Schema {
  geom        binary
  OBJECTID    int32
  Shape_Leng  float64
  Shape_Area  float64
  zone        string
  LocationID  int32
  borough     string
  x_cent      float64
  y_cent      float64
}

zones = zones.mutate(geom=zones["geom"].cast("geometry"))
zones.schema()

1: Cast the binary geometry column to a geometry column.
2: Notice the geometry column is now a geometry type.

ibis.Schema {
  geom        geospatial:geometry
  OBJECTID    int32
  Shape_Leng  float64
  Shape_Area  float64
  zone        string
  LocationID  int32
  borough     string
  x_cent      float64
  y_cent      float64
}

PySpark

PySpark 3.5 is now supported.

Exasol

Numerous improvements have been made to the Exasol backend by Nicola including bitwise operations, time operations, and hexdigest.

Apache Flink

Scalar user-defined functions (UDFs) are now supported in the Flink backend.

RisingWave

Streaming data definition language (DDL) support has been added to the RisingWave backend.

Breaking changes

You can view the full changelog for additional breaking changes. There is one we expect to affect many users.

What does “schema” mean?

In building a standard Python dataframe interface, we must standardize naming conventions and terminology across data systems. Many systems use the words “schema”, “database”, “catalog”, and “namespace” to mean different things.

In Ibis, a “schema” is a mapping of column names to their types:

t.schema()

ibis.Schema {
  species            string
  island             string
  bill_length_mm     float64
  bill_depth_mm      float64
  flipper_length_mm  int64
  body_mass_g        int64
  sex                string
  year               int64
}

And as of Ibis 9.0, that is all “schema” means! We have standardizes on table as the container of data which has a schema. A database is a collection of tables, and a catalog is a collection of databases. Not all backends will support all levels of hierarchy.

Existing methods using the hierarchical meaning of “schema” will now raise a deprecation warning, and the arguments throughout the API have been updated to reflect the standardization.

For example, list_schemas() will give a deprecation warning:

con = ibis.connect("duckdb://")
con.list_schemas()

/var/folders/hm/f6m13d5d4xg6v606d3mfndnh0000gn/T/ipykernel_6573/2769547408.py:2: FutureWarning:

`CanListSchema.list_schemas` is deprecated as of v9.0, removed in v10.0; Use `list_databases` instead`

['information_schema', 'main', 'pg_catalog']

Use list_databases() instead:

con.list_databases()

['information_schema', 'main', 'pg_catalog']

You can also list_catalogs():

con.list_catalogs()

['memory', 'system', 'temp']

And get the current database:

con.current_database

'main'

And the current catalog:

con.current_catalog

'memory'

Other breaking changes

There were a few more minor breaking changes, see the full changelog for more details.

Functionality

A lot of new functionality has been added in Ibis 9.0.

Python 3.12

Ibis now supports Python 3.12!

Describe

Ibis now has a describe method to get a summary of a table:

t.describe()

┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ name              ┃ type    ┃ count ┃ nulls ┃ unique ┃ mode   ┃ mean        ┃ std        ┃ min     ┃ p25      ┃ p50     ┃ p75     ┃ max     ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ string            │ string  │ int64 │ int64 │ int64  │ string │ float64     │ float64    │ float64 │ float64  │ float64 │ float64 │ float64 │
├───────────────────┼─────────┼───────┼───────┼────────┼────────┼─────────────┼────────────┼─────────┼──────────┼─────────┼─────────┼─────────┤
│ species           │ string  │   344 │     0 │      3 │ Adelie │        NULL │       NULL │    NULL │     NULL │    NULL │    NULL │    NULL │
│ island            │ string  │   344 │     0 │      3 │ Biscoe │        NULL │       NULL │    NULL │     NULL │    NULL │    NULL │    NULL │
│ bill_length_mm    │ float64 │   344 │     2 │    164 │ NULL   │   43.921930 │   5.459584 │    32.1 │   39.225 │   44.45 │    48.5 │    59.6 │
│ bill_depth_mm     │ float64 │   344 │     2 │     80 │ NULL   │   17.151170 │   1.974793 │    13.1 │   15.600 │   17.30 │    18.7 │    21.5 │
│ flipper_length_mm │ int64   │   344 │     2 │     55 │ NULL   │  200.915205 │  14.061714 │   172.0 │  190.000 │  197.00 │   213.0 │   231.0 │
│ body_mass_g       │ int64   │   344 │     2 │     94 │ NULL   │ 4201.754386 │ 801.954536 │  2700.0 │ 3550.000 │ 4050.00 │  4750.0 │  6300.0 │
│ sex               │ string  │   344 │    11 │      2 │ male   │        NULL │       NULL │    NULL │     NULL │    NULL │    NULL │    NULL │
│ year              │ int64   │   344 │     0 │      3 │ NULL   │ 2008.029070 │   0.818356 │  2007.0 │ 2007.000 │ 2008.00 │  2009.0 │  2009.0 │
└───────────────────┴─────────┴───────┴───────┴────────┴────────┴─────────────┴────────────┴─────────┴──────────┴─────────┴─────────┴─────────┘

GraphViz custom attributes

Thanks to a couple community contributions, you can now pass custom attributes to the GraphViz visualization of an expression.

Tip

You can also call expr.visualize() and pass the same arguments.

from ibis.expr.visualize import to_graph

expr = (
    t.group_by("species", "island")
    .agg(count=t.count())
    .order_by(ibis._["count"].desc())
)

to_graph(
    expr,
    label_edges=True,
    node_attr={"shape": "hexagon", "color": "green", "fontname": "Roboto Mono"},
    edge_attr={"fontsize": "12", "fontname": "Comic Sans MS"},
)

Polars input and output

You can now directly output a Polars dataframe from an Ibis table (instead of going through Apache Arrow as previously possible):

t.to_polars()

shape: (344, 8)

species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex	year
str	str	f64	f64	i64	i64	str	i64
"Adelie"	"Torgersen"	39.1	18.7	181	3750	"male"	2007
"Adelie"	"Torgersen"	39.5	17.4	186	3800	"female"	2007
"Adelie"	"Torgersen"	40.3	18.0	195	3250	"female"	2007
"Adelie"	"Torgersen"	null	null	null	null	null	2007
"Adelie"	"Torgersen"	36.7	19.3	193	3450	"female"	2007
…	…	…	…	…	…	…	…
"Chinstrap"	"Dream"	55.8	19.8	207	4000	"male"	2009
"Chinstrap"	"Dream"	43.5	18.1	202	3400	"female"	2009
"Chinstrap"	"Dream"	49.6	18.2	193	3775	"male"	2009
"Chinstrap"	"Dream"	50.8	19.0	210	4100	"male"	2009
"Chinstrap"	"Dream"	50.2	18.7	198	3775	"female"	2009

And you can directly construct an Ibis memtable from a Polars dataframe:

ibis.memtable(t.to_polars())

┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
│ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

These conversions are efficient and improve usability of Ibis and Polars together.

Scalar subqueries from expressions

You can now create scalar subqueries from expressions:

t.select("species").limit(1).as_scalar()

'Adelie'

Disconnect

You can now explicitly disconnect from a backend. First, connect to a backend:

con = ibis.connect("duckdb://")
con.list_tables()

[]

Then you can disconnect. Trying to use the connection after will raise an error:

con.disconnect()
con.list_tables()

ConnectionException: Connection Error: Connection has already been closed

Today

You can now get the current date more easily:

ibis.today()

datetime.date(2024, 5, 2)

And use it in expressions:

t.mutate(ingested_at=ibis.today()).relocate("ingested_at")

┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ ingested_at ┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ date        │ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├─────────────┼─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ 2024-05-02  │ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ 2024-05-02  │ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ 2024-05-02  │ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
│ …           │ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└─────────────┴─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

UUIDs

You can now generate universally unique identifiers (UUIDs) in Ibis more easily:

ibis.uuid()

UUID('541ab072-9f28-46dd-b402-8ebf3b697fc8')

And use them in expressions:

t.mutate(id=ibis.uuid()).relocate("id")

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ id                                   ┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ uuid                                 │ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├──────────────────────────────────────┼─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ 17b11546-2573-4ed6-84b8-0c3ee6f89060 │ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ 0acdb8bf-dc1b-4cfe-bdba-8c5be4a5f84a │ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ ac8830ac-74d1-44fc-a4dc-b5b7a0343111 │ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
│ …                                    │ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└──────────────────────────────────────┴─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

TopK includes NULLs

This could technically be considered a breaking change or bug fix. Regardless, topk now includes NULLs by default:

t["sex"].topk(3)

┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ sex    ┃ CountStar(penguins) ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ string │ int64               │
├────────┼─────────────────────┤
│ male   │                 168 │
│ female │                 165 │
│ NULL   │                  11 │
└────────┴─────────────────────┘

Order and group by variadic arguments

For consistency with other methods, order_by and group_by now accept variadic arguments:

t.order_by(t["body_mass_g"].desc(), "island", t["species"])

┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├─────────┼────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Gentoo  │ Biscoe │           49.2 │          15.2 │               221 │        6300 │ male   │  2007 │
│ Gentoo  │ Biscoe │           59.6 │          17.0 │               230 │        6050 │ male   │  2007 │
│ Gentoo  │ Biscoe │           51.1 │          16.3 │               220 │        6000 │ male   │  2008 │
│ …       │ …      │              … │             … │                 … │           … │ …      │     … │
└─────────┴────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

grouped = (
    t.group_by("species", "island")
    .agg(count=t.count())
    .order_by(ibis._["count"].desc())
)
grouped

┏━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species   ┃ island ┃ count ┃
┡━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string    │ string │ int64 │
├───────────┼────────┼───────┤
│ Gentoo    │ Biscoe │   124 │
│ Chinstrap │ Dream  │    68 │
│ Adelie    │ Dream  │    56 │
│ …         │ …      │     … │
└───────────┴────────┴───────┘

You can still pass in a list or tuple if you prefer.

Unwrap JSON values

You can now unwrap JSON values into backend-native values:

data = {
    "jstring": ['"a"', '""', None, "null"],
    "jbool": ["true", "false", "null", None],
    "jint": ["1", "null", None, "2"],
    "jfloat": ["42.42", None, "null", "37.37"],
    "jmap": ['{"a": 1}', "null", None, "{}"],
    "jarray": ["[]", "null", None, '[{},"1",2]'],
}

t = ibis.memtable(data, schema=dict.fromkeys(data.keys(), "json"))
t

┏━━━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┓
┃ jstring ┃ jbool ┃ jint ┃ jfloat ┃ jmap     ┃ jarray ┃
┡━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━┩
│ json    │ json  │ json │ json   │ json     │ json   │
├─────────┼───────┼──────┼────────┼──────────┼────────┤
│ 'a'     │ True  │ 1    │ 42.42  │ {'a': 1} │ []     │
│ ''      │ False │ None │ NULL   │ None     │ None   │
│ NULL    │ None  │ NULL │ None   │ NULL     │ NULL   │
│ …       │ …     │ …    │ …      │ …        │ …      │
└─────────┴───────┴──────┴────────┴──────────┴────────┘

t.select(unwrapped=t.jstring.unwrap_as(str), original=t.jstring)

┏━━━━━━━━━━━┳━━━━━━━━━━┓
┃ unwrapped ┃ original ┃
┡━━━━━━━━━━━╇━━━━━━━━━━┩
│ string    │ json     │
├───────────┼──────────┤
│ a         │ 'a'      │
│ ~         │ ''       │
│ NULL      │ NULL     │
│ …         │ …        │
└───────────┴──────────┘

t.select(unwrapped=t.jbool.unwrap_as("bool"), original=t.jbool)

┏━━━━━━━━━━━┳━━━━━━━━━━┓
┃ unwrapped ┃ original ┃
┡━━━━━━━━━━━╇━━━━━━━━━━┩
│ boolean   │ json     │
├───────────┼──────────┤
│ True      │ True     │
│ False     │ False    │
│ NULL      │ None     │
│ …         │ …        │
└───────────┴──────────┘

Create more backends with empty URLs

For convenience, you can now create backends with an empty URL for Dask, pandas, Polars, and DataFusion:

con = ibis.connect("dask://")
con = ibis.connect("pandas://")
con = ibis.connect("polars://")
con = ibis.connect("datafusion://")

This is in addition to the existing backend-specific connection methods:

con = ibis.dask.connect()
con = ibis.pandas.connect()
con = ibis.polars.connect()
con = ibis.datafusion.connect()

Note that URL parameters are not meaningful with these backends. The main use case is more convenient connection swapping programmatically:

for backend in [
    "duckdb",
    "sqlite",
    "pyspark",
    "dask",
    "pandas",
    "polars",
    "datafusion",
]:
    con = ibis.connect(f"{backend}://")
    print(con)

Wrapping up

Ibis 9.0 is an exciting release that completes the transition to SQLGlot, adds a lot of new functionality, and welcomes a new committer to the project!

As always, try Ibis by installing and getting started.

If you run into any issues or find support is lacking for your backend, open an issue or discussion and let us know!

Varchar in a haystack

Tyler White — Fri, 12 Apr 2024 00:00:00 GMT

The scenario

You’re a data analyst, and a new ticket landed in your queue.

Subject: Urgent: Data Discovery Needed for Critical Analysis

Hi Data Team,

I hope this message finds you well. I’m reaching out with an urgent request that directly impacts the company’s most critical project. We need to locate a specific value within our database but do not know which column it’s in. Unfortunately, we don’t have documentation for this particular table. We are looking for the value “NEEDLE” in the table.

We think it is in the X database, Y schema, and Z table. We appreciate your help with this urgent matter!

Whelp, let’s give this a try.

The table

To set up this particular problem, we can use pandas to create a table with 5 columns and 100 rows. We can use the at method to update a row with the value “NEEDLE” to simulate what we need to find.

Code

Portable dataflows with Ibis and Hamilton

Thierry Jean — Tue, 02 Apr 2024 00:00:00 GMT

Introduction

This post showcases how Ibis and Hamilton enable dataflows that span execution over SQL and Python. Ibis is a portable dataframe library to write procedural data transformations in Python and be able to execute them directly on various SQL backends (DuckDB, Snowflake, Postgres, Flink, see full list). Hamilton provides a declarative way to define testable, modular, self-documenting dataflows, that encode lineage and metadata.

Let’s introduce Ibis before exploring how it pairs with Hamilton.

Standalone Ibis

Here’s an Ibis code snippet to load data from a parquet file, compute features, select columns, and filter rows, illustrating typical feature engineering operations.

Reading the code, you’ll notice that:

We use “expression chaining”, meaning there’s a series of .method() attached one after another.
The variable ibis._ is a special character referring to the current expression e.g., ibis._.pet accesses the column “pet” of the current table.
The table method .mutate(col1=, col2=, ...) assigns new columns or overwrites existing ones.

import ibis

url = "https://storage.googleapis.com/ibis-blog-data-public/hamilton-ibis/absenteeism.parquet"
feature_set = (
  ibis.read_parquet(sources=url, table_name="absenteeism")
  .rename("snake_case")
  .mutate(  # allows us to define new columns
    has_children=ibis.ifelse(ibis._.son > 0, True, False),
    has_pet=ibis.ifelse(ibis._.pet > 0, True, False),
    is_summer_brazil=ibis._.month_of_absence.isin([1, 2, 12]),
  ).select(
    "id", "has_children", "has_pet", "is_summer_brazil",
    "service_time", "seasons", "disciplinary_failure",
    "absenteeism_time_in_hours"
  )
)

Challenge 1 – Maintain and test complex data transformations

Ibis has an SQL-like syntax and supports chaining operations, allowing for powerful queries in a few lines of code. Conversely, there’s a risk of sprawling complexity as expressions are appended, making them harder to test and debug. Preventing this issue requires a lot of upfront discipline and refactoring.

Challenge 2 – Orchestrate Ibis code in production

Ibis alleviates a major pain point by enabling data transformations to work across backends. However, moving from dev to prod still requires some code changes such as changing backend connectors, swapping unsupported operators, adding some orchestration and logging execution, wanting to reuse prior code, etc. This is outside the scope of the Ibis project and is expected to be enabled by other means, which usually means bespoke constructs that turn into technical debt.

What is Hamilton?

Hamilton is a general-purpose framework to write dataflows using regular Python functions. At the core, each function defines a transformation and its parameters indicates its dependencies. Hamilton automatically connects individual functions into a directed acyclic graph (DAG) that can be executed, visualized, optimized, and reported on.

The ABC of Hamilton

How Hamilton complements Ibis

Hamilton was initially developed to structure pandas code for a large catalog of features, and has since been adopted by multiple organizations since, and expanded to cover any python object type (Polars, PySpark, ML Models, Numpy, you custom type, etc). Its syntax encourages users to chunk code into meaningful and reusable components, which facilitates documentation, unit testing, code reviews, and improves iteration speed, and the dev to production process. These benefits directly translate to organizing Ibis code.

Solution 1 – Structure your Ibis code with Hamilton

Now, we’ll refactor the above Ibis code to use Hamilton. Users have the flexibility to chunk code (i.e., what the contents of a function is), at the table or the column-level depending on the needed granularity. This modularity is particularly beneficial to Ibis because:

Well-scoped functions with type annotations and docstring are easier to understand for new Ibis users and facilitate onboarding.
Unit testing and data validation becomes easier with smaller expressions. These checks become more important when working across backends since the operation coverage varies and bugs may arise.

Table-level dataflow

Table-level operations might feel most familiar to SQL and Spark users. Also, Ibis + Hamilton is reminiscent of dbt for the Python ecosystem.

Working with tables is very efficient when your number of columns/features is limited, and you don’t need full column level lineage. As you want to reuse components, you can progressively breakdown “table-level code” in to “column-level code”.

The initial Ibis code is now 3 functions with type annotations and docstrings. We have a clear sense of the expected external outputs and we could implement schema checks between functions.

from typing import Optional
import ibis
import ibis.expr.types as ir

def raw_table(raw_data_path: str) -> ir.Table:
    """Load parquet from `raw_data_path` into a Table expression
    and format column names to snakecase
    """
    return (
        ibis.read_parquet(sources=raw_data_path, table_name="absenteism")
        .rename("snake_case")
    )

def feature_table(raw_table: ir.Table) -> ir.Table:
    """Add to `raw_table` the feature columns `has_children`
    `has_pet`, and `is_summer_brazil`
    """
    return raw_table.mutate(
        has_children=(ibis.ifelse(ibis._.son > 0, True, False)),
        has_pet=ibis.ifelse(ibis._.pet > 0, True, False),
        is_summer_brazil=ibis._.month_of_absence.isin([1, 2, 12]),
    )

def feature_set(
    feature_table: ir.Table,
    feature_selection: list[str],
    condition: Optional[ibis.common.deferred.Deferred] = None,
) -> ir.Table:
    """Select feature columns and filter rows"""
    return feature_table[feature_selection].filter(condition)

Table-level lineage with Hamilton

Column-level dataflow

Hamilton was initially built to expose and manage column-level operations, which is most common in dataframe libraries (pandas, Dask, polars).

Column-level code leads to fully-reusable feature definitions and a highly granular level of lineage. Notably, this allows one to trace sensitive data and evaluate downstream impacts of code changes. However, it is more verbose to get started with, but remember that code is read more often than written.

Now, the raw_table is loaded and the columns son, pet, and month_of_absence are extracted to engineer new features. After transformations, features are joined with raw_table to create feature_table.

import ibis
import ibis.expr.types as ir
from hamilton.function_modifiers import extract_columns
from hamilton.plugins import ibis_extensions

# extract specific columns from the table
@extract_columns("son", "pet", "month_of_absence")
def raw_table(raw_data_path: str) -> ir.Table:
    """Load the parquet found at `raw_data_path` into a Table expression
    and format columns to snakecase
    """
    return (
        ibis.read_parquet(sources=raw_data_path, table_name="absenteism")
        .rename("snake_case")
    )

# accesses a single column from `raw_table`
def has_children(son: ir.Column) -> ir.BooleanColumn:
    """True if someone has any children"""
    return ibis.ifelse(son > 0, True, False)

# narrows the return type from `ir.Column` to `ir.BooleanColumn`
def has_pet(pet: ir.Column) -> ir.BooleanColumn:
    """True if someone has any pets"""
    return ibis.ifelse(pet > 0, True, False).cast(bool)

# typing and docstring provides business context to features
def is_summer_brazil(month_of_absence: ir.Column) -> ir.BooleanColumn:
    """True if it is summer in Brazil during this month

    People in the northern hemisphere are likely to take vacations
    to warm places when it's cold locally
    """
    return month_of_absence.isin([1, 2, 12])

def feature_table(
    raw_table: ir.Table,
    has_children: ir.BooleanColumn,
    has_pet: ir.BooleanColumn,
    is_summer_brazil: ir.BooleanColumn,
) -> ir.Table:
    """Join computed features to the `raw_data` table"""
    return raw_table.mutate(
        has_children=has_children,
        has_pet=has_pet,
        is_summer_brazil=is_summer_brazil,
    )

def feature_set(
    feature_table: ir.Table,
    feature_selection: list[str],
    condition: Optional[ibis.common.deferred.Deferred] = None,
) -> ir.Table:
    """Select feature columns and filter rows"""
    return feature_table[feature_selection].filter(condition)

Column-level lineage with Hamilton

Solution 2 – Orchestrate Ibis anywhere

Hamilton is an ideal way to orchestrate Ibis code because it has a very small dependency footprint and will run anywhere Python does (script, notebook, FastAPI, Streamlit, pyodide, etc.) In fact, the Hamilton library only has four dependencies. You don’t need “framework code” to get started, just plain Python functions. When moving to production, Hamilton has all the necessary features to complement Ibis such as swapping components, configurations, and lifecycle hooks for logging, alerting, and telemetry.

A simple usage pattern of Hamilton + Ibis is to use the @config.when function modifier. In the following example, we have alternative implementations for the backend connection, which will be used for computing and storing results. When running your code, specify in your config backend="duckdb" or backend="bigquery" to swap between the two.

# ibis_dataflow.py
import ibis
import ibis.expr.types as ir
from hamilton.function_modifiers import config

# ... entire dataflow definition

@config.when(backend="duckdb")
def backend_connection__duckdb(
    connection_string: str
) -> ibis.backends.BaseBackend:
    """Connect to DuckDB backend"""
    return ibis.duckdb.connect(connection_string)

@config.when(backend="bigquery")
def backend_connection__bigquery(
    project_id: str,
    dataset_id: str,
) -> ibis.backends.BaseBackend:
    """Connect to BigQuery backend
    Install dependencies via `pip install ibis-framework[bigquery]`
    """
    return ibis.bigquery.connect(
        project_id=project_id,
        dataset_id=dataset_id,
    )

def insert_results(
    backend_connection: ibis.backends.BaseBackend,
    result_table: ir.Table,
    table_name: str
) -> None:
    """Execute expression and insert results"""
    backend_connection.insert(
        table_name=table_name,
        obj=result_table
    )

How Ibis complements Hamilton

Performance boost

Leveraging DuckDB as the default backend, Hamilton users migrating to Ibis should immediately find performance improvements both for local dev and production. In addition, the portability of Ibis has the potential to greatly reduce development time.

Atomic data transformation documentation

Hamilton can directly produce a dataflow visualization from code, helping with project documentation. Ibis pushes this one step further by providing a detailed view of the query plan and schemas. See this Ibis visualization for the column-level Hamilton dataflow defined above. It includes all renaming, type casting, and transformations steps (Please open the image in a new tab and zoom in 🔎).

Working across rows with user-defined functions (UDFs)

Hamilton and most backends are designed to work primarily on tables and columns, but sometimes you’d like to operate over a row (think of pd.DataFrame.apply()). However, pivoting tables is costly and manually iterating over rows to collect values and create a new column is quickly inconvenient. By using scalar user-defined functions (UDFs), Ibis makes it possible to execute arbitrary Python code on rows directly on the backend.

Note

Using @ibis.udf.scalar.python creates a non-vectorized function that iterates row-by-row. See the docs to use backend-specific UDFs with @ibis.udf.scalar.builtin and create vectorized scalar UDFs.

For instance, you could embed rows of a text column using an LLM API using your existing data warehouse infrastructure.

import ibis
import ibis.expr.types as ir

def documents(path: str) -> ir.Table:
    """load text documents from file"""
    return ibis.read_parquet(sources=path, table_name="documents")

# function name starts with `_` to prevent being added as a node
@ibis.udf.scalar.python
def _generate_summary(author: str, text: str, prompt_template: str) -> str:
    """UDF Function to call the OpenAI API line by line"""
    prompt = prompt_template.format(author=author, text=text)
    client = openai.OpenAI(...)
    try:
        response = client.chat.completions.create(...)
        return_value = response.choices[0].message.content
    except Exception:
        return_value = ""
    return return_value


def prompt_template() -> str:
    return """summarize the following text from {author} and add
    contextual notes based on it biography and other written work

    TEXT
    {text}
    """

def summaries(documents: ir.Table, prompt_template: str) -> ir.Table:
    """Compute the UDF against the family"""
    return documents.mutate(
        summary=_generated_summary(
            _.author,
            _.text,
            prompt_template=prompt_template
        )
    )

Ibis + Hamilton – a natural pairing

What works in dev works in prod: Ibis and Hamilton allows you to write and structure code data transformations that are portable across backends for small and big data alike. The two being lightweight libraries, installing dependencies on remote workers is fast and you’re unlikely to ever encounter dependency conflicts.
Maintainable and testable code: Modular functions facilitates writing high quality code and promotes reusability, compounding your engineering efforts. It becomes easier for new users to contribute to a dataflow and pull requests are merged faster.
Greater visibility: With Hamilton and Ibis, you have incredible visualizations directly derived from your code. This is a superpower for documentation, allowing users to make sense of a dataflow, and also reason about changes.

Scaling to infinity and beyond: the Unix backend

Phillip Cloud — Mon, 01 Apr 2024 00:00:00 GMT

The Unix backend for Ibis

We’re happy to announce a new Ibis backend built on the world’s best known web scale technology: Unix pipes.

Why?

Why not? Pipes rock and they automatically stream data between operators and scale to your hard drive.

What’s not to love?

Demo

All production ready backends ship with amazing demos.

The Unix backend is no different. Let’s see it in action.

First we’ll install the Unix backend.

pip install ibish

Like all production-ready libraries ibish depends on the latest commit of ibis-framework.

Next we’ll download some data.

!curl -LsS 'https://storage.googleapis.com/ibis-examples/penguins/20240322T125036Z-9aae2/penguins.csv.gz' | zcat > penguins.csv

import ibis
import ibish


ibis.options.interactive = True

unix = ibish.connect({"p": "penguins.csv"})

t = unix.table("p")
t

┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string    │ float64        │ float64       │ float64           │ float64     │ string │ int64 │
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Torgersen │           39.1 │          18.7 │             181.0 │      3750.0 │ male   │  2007 │
│ Adelie  │ Torgersen │           39.5 │          17.4 │             186.0 │      3800.0 │ female │  2007 │
│ Adelie  │ Torgersen │           40.3 │          18.0 │             195.0 │      3250.0 │ female │  2007 │
│ Adelie  │ Torgersen │           NULL │          NULL │              NULL │        NULL │ NULL   │  2007 │
│ Adelie  │ Torgersen │           36.7 │          19.3 │             193.0 │      3450.0 │ female │  2007 │
│ Adelie  │ Torgersen │           39.3 │          20.6 │             190.0 │      3650.0 │ male   │  2007 │
│ Adelie  │ Torgersen │           38.9 │          17.8 │             181.0 │      3625.0 │ female │  2007 │
│ Adelie  │ Torgersen │           39.2 │          19.6 │             195.0 │      4675.0 │ male   │  2007 │
│ Adelie  │ Torgersen │           34.1 │          18.1 │             193.0 │      3475.0 │ NULL   │  2007 │
│ Adelie  │ Torgersen │           42.0 │          20.2 │             190.0 │      4250.0 │ NULL   │  2007 │
│ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

Sweet, huh?

Let’s filter the data and look at only the year 2009.

expr = t.filter(t.year == 2009)
expr

┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string │ float64        │ float64       │ float64           │ float64     │ string │ int64 │
├─────────┼────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Biscoe │           35.0 │          17.9 │             192.0 │      3725.0 │ female │  2009 │
│ Adelie  │ Biscoe │           41.0 │          20.0 │             203.0 │      4725.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           37.7 │          16.0 │             183.0 │      3075.0 │ female │  2009 │
│ Adelie  │ Biscoe │           37.8 │          20.0 │             190.0 │      4250.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           37.9 │          18.6 │             193.0 │      2925.0 │ female │  2009 │
│ Adelie  │ Biscoe │           39.7 │          18.9 │             184.0 │      3550.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           38.6 │          17.2 │             199.0 │      3750.0 │ female │  2009 │
│ Adelie  │ Biscoe │           38.2 │          20.0 │             190.0 │      3900.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           38.1 │          17.0 │             181.0 │      3175.0 │ female │  2009 │
│ Adelie  │ Biscoe │           43.2 │          19.0 │             197.0 │      4775.0 │ male   │  2009 │
│ …       │ …      │              … │             … │                 … │           … │ …      │     … │
└─────────┴────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

We can sort the result of that too, and filter again.

expr = (
    expr.order_by("species", ibis.desc("bill_length_mm"))
    .filter(lambda t: t.island == "Biscoe")
)
expr

┏━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string │ float64        │ float64       │ float64           │ float64     │ string │ int64 │
├─────────┼────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Biscoe │           45.6 │          20.3 │             191.0 │      4600.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           43.2 │          19.0 │             197.0 │      4775.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           42.7 │          18.3 │             196.0 │      4075.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           42.2 │          19.5 │             197.0 │      4275.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           41.0 │          20.0 │             203.0 │      4725.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           39.7 │          17.7 │             193.0 │      3200.0 │ female │  2009 │
│ Adelie  │ Biscoe │           39.7 │          18.9 │             184.0 │      3550.0 │ male   │  2009 │
│ Adelie  │ Biscoe │           39.6 │          20.7 │             191.0 │      3900.0 │ female │  2009 │
│ Adelie  │ Biscoe │           38.6 │          17.2 │             199.0 │      3750.0 │ female │  2009 │
│ Adelie  │ Biscoe │           38.2 │          20.0 │             190.0 │      3900.0 │ male   │  2009 │
│ …       │ …      │              … │             … │                 … │           … │ …      │     … │
└─────────┴────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

There’s even support for joins and aggregations!

Let’s count the number of island, species pairs and sort descending by the count.

expr = (
    t.group_by("island", "species")
    .agg(n=lambda t: t.count())
    .order_by(ibis.desc("n"))
)
expr

┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
┃ island    ┃ species   ┃ n     ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
│ string    │ string    │ int64 │
├───────────┼───────────┼───────┤
│ Biscoe    │ Gentoo    │   124 │
│ Dream     │ Chinstrap │    68 │
│ Dream     │ Adelie    │    56 │
│ Torgersen │ Adelie    │    52 │
│ Biscoe    │ Adelie    │    44 │
└───────────┴───────────┴───────┘

For kicks, let’s compare that to the DuckDB backend to make sure we’re able to count stuff.

To be extra awesome, we’ll reuse the same expression to do the computation.

ddb = ibis.duckdb.connect()
ddb.read_csv("penguins.csv", table_name="p")
ibis.memtable(ddb.to_pyarrow(expr.unbind()))

1: The read_csv is necessary so that the expression’s table name–p–matches one inside the DuckDB database.

┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
┃ island    ┃ species   ┃ n     ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
│ string    │ string    │ int64 │
├───────────┼───────────┼───────┤
│ Biscoe    │ Gentoo    │   124 │
│ Dream     │ Chinstrap │    68 │
│ Dream     │ Adelie    │    56 │
│ Torgersen │ Adelie    │    52 │
│ Biscoe    │ Adelie    │    44 │
└───────────┴───────────┴───────┘

How does it work?

Glad you asked!

The Unix backend for Ibis was built over the course of a few hours, which is about the time it takes to make a production ready Ibis backend.

Broadly speaking, the Unix backend:

Produces a shell command for each Ibis table operation.
Produces a nominal output location for the output of that command, in the form of a named pipe opened in write mode.
Reads output from the named pipe output location of the root of the expression tree.
Calls pandas.read_csv on that output.

Why named pipes?

Shell commands only allow a single input from stdin.

However, joins accept > 1 input so we need a way to stream more than one input to a join operation.

Named pipes support the semantics of “unnamed” pipes (FIFO queue behavior) but can be used in pipelines with nodes that have more a single input since they exist as paths on the file system.

Expressions

Ibis expressions are an abstract representation of an analytics computation over tabular data.

Ibis ships a public API, whose instances we call expressions.

Expressions have an associated type–accessible via their type() method–that determines what methods are available on them.

Expressions are ignorant of their underlying implementation: their composability is determined solely by their type.

This type is determined by the expression’s underlying operation.

The two-layer model makes it easy to describe operations in terms of the data types produced by an expression, rather than as instances of a specific class in a hierarchy.

This allows Ibis maintainers to alter expression API implementations without changing those APIs making it easier to maintain and easier to keep stable than if we had a complex (but not necessarily deep!) class hierarchy.

Operations, though, are really where the nitty gritty implementation details start.

Operations

Ibis operations are lightweight classes that model the tree structure of a computation.

They have zero or more inputs, whose types and values are constrained by Ibis’s type system.

Notably operations are not part of Ibis’s public API.

When we talk about “compilation” in Ibis, we’re talking about the process of converting an operation into something that the backend knows how to execute.

In the case of this 1̵0̸0̵%̵ p̶̺̑r̴̛ͅo̵̒ͅḍ̴̌u̷͇͒c̵̠̈t̷͍̿i̶̪͐o̸̳̾n̷͓̄-r̵̡̫̞͓͆̂̏ẽ̸̪̱̽ͅā̸̤̹̘̅̓͝d̵͇̞̏̂̔̽y̴̝͎̫̬͋̇̒̅ Unix backend, each operation is compiled into a list of strings that represent the shell command to run to execute the operation.

In other backends, like DuckDB, these compilation rules produce a sqlglot object.

The compile method is also the place where the backend has a chance to invoke custom rewrite rules over operations.

Rewrites are a very useful tool for the Unix backend. For example, the join command (yep, it’s in coreutils!) that we use to execute inner joins with this backend requires that the inputs be sorted, otherwise the results won’t be correct. So, I added a rewrite rule that replaces the left and right relations in a join operation with equivalent relations sorted on the join keys.

Once you obtain the output of compile, it’s up to the backend what to do next.

Backend implementation

At this point we’ve got our shell commands and some output locations created as named pipes.

What next?

Well, we need to execute the commands and write their output to the corresponding named pipe.

You might think

I’ll just loop over the operations, open the pipe in write mode and call subprocess.Popen(cmd, stdout=named_pipe).

Not a bad thought, but the semantics of named pipes do not abide such thoughts :)

Named pipes, when opened in write mode, will block until a corresponding handle is opened in read mode.

Futures using a scoped thread pool are a decent way to handle this.

The idea is to launch every node concurrently and then read from the last node’s output. This initial read of the root node’s output pipe kicks off the cascade of other reads necessary to move data through the pipeline.

The Unix backend thus constructs a scoped ThreadPoolExecutor() using a context manager and submits a task for each operation to the executor. Importantly, opening the named pipe in write mode happens inside the task, to avoid blocking the main thread while waiting for a reader to be opened.

The final output task’s path is then passed directly to read_csv, and we’ve now got the result of our computation.

Show me the commands already!

Roger that.

expr = (
    t.filter([t.year == 2009])
    .select(
        "year", "species", "flipper_length_mm", island=lambda t: t.island.lower()
    )
    .group_by("island", "species")
    .agg(n=lambda t: t.count(), avg=lambda t: t.island.upper().length().mean())
    .order_by("n")
    .mutate(ilength=lambda t: t.island.length())
    .limit(5)
)
print(unix.explain(expr))

1: explain isn’t a public method and not likely to become one any time soon.

tail --lines +2 /home/cloud/src/ibis/docs/posts/unix-backend/penguins.csv > t0
awk -F , '{ if (($8 == 2009)) { print }}' t0 > t1
awk -F , '{ print $8 "," $1 "," $5 "," tolower($2) }' t1 > t2
awk -F , '{
  agg0[$4","$2]++
  agg1[$4","$2] += length(toupper($4))
}
END { for (key in agg0) print key "," agg0[key] "," agg1[key]/NR }' t2 > t3
sort -t , -k 3,3n t3 > t4
awk -F , '{ print $1 "," $2 "," $3 "," $4 "," length($1) }' t4 > t5
head --lines 5 t5 > t6

Conclusion

If you’ve gotten this far hopefully you’ve had a good laugh.

Let’s wrap up with some final thoughts.

Things to do

Join our Zulip!
Open a GitHub issue or discussion!

Things to avoid doing

Putting this into production

Snow IO: loading data from other DBs into Snowflake

Phillip Cloud — Wed, 06 Mar 2024 00:00:00 GMT

Recap

We’ve blogged about Snowflake IO before, in the context of getting local files into Snowflake as fast as possible.

In this post, we’ll show how to insert query results from another system into Snowflake, using Ibis.

Setup

Connect to your non-Snowflake system

We’ll connect to a postgres database running locally in a container. You should be able to swap in your own connection details as needed.

from ibis.interactive import *

pg_con = ibis.connect("postgres://postgres:postgres@localhost/postgres")

1: Import Ibis for maximum productivity in interactive analysis.

We’ll use a test dataset that contains some baseball batting statistics.

Ibis provides that example data, so we can dump that into postgres.

pg_batting = pg_con.create_table(
    "batting",
    ibis.examples.Batting.fetch().to_pandas(),
    temp=True,
)

1: Yep, I’m using pandas here!
2: Use a temporary table to avoid cluttering up the database.

Connect to Snowflake

import os

# snowflake://user:pass@account/database/schema?warehouse=my_warehouse
snow_con = ibis.connect(os.environ["SNOWFLAKE_URL"])

1: Set the SNOWFLAKE_URL environment variable to your Snowflake connection string.

Profit

Construct an Ibis expression from the postgres data

Let’s build an Ibis expression based on the batting table in our postgres database.

pg_batting

┏━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ player_id ┃ year_id ┃ stint ┃ team_id ┃ lg_id  ┃ g     ┃ ab    ┃ r     ┃ h     ┃ x2b   ┃ x3b   ┃ hr    ┃ rbi     ┃ sb      ┃ cs      ┃ bb    ┃ so      ┃ ibb     ┃ hbp     ┃ sh      ┃ sf      ┃ gidp    ┃
┡━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ string    │ int64   │ int64 │ string  │ string │ int64 │ int64 │ int64 │ int64 │ int64 │ int64 │ int64 │ float64 │ float64 │ float64 │ int64 │ float64 │ float64 │ float64 │ float64 │ float64 │ float64 │
├───────────┼─────────┼───────┼─────────┼────────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼─────────┼─────────┼─────────┼───────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ abercda01 │    1871 │     1 │ TRO     │ NA     │     1 │     4 │     0 │     0 │     0 │     0 │     0 │     0.0 │     0.0 │     0.0 │     0 │     0.0 │     nan │     nan │     nan │     nan │     0.0 │
│ addybo01  │    1871 │     1 │ RC1     │ NA     │    25 │   118 │    30 │    32 │     6 │     0 │     0 │    13.0 │     8.0 │     1.0 │     4 │     0.0 │     nan │     nan │     nan │     nan │     0.0 │
│ allisar01 │    1871 │     1 │ CL1     │ NA     │    29 │   137 │    28 │    40 │     4 │     5 │     0 │    19.0 │     3.0 │     1.0 │     2 │     5.0 │     nan │     nan │     nan │     nan │     1.0 │
│ allisdo01 │    1871 │     1 │ WS3     │ NA     │    27 │   133 │    28 │    44 │    10 │     2 │     2 │    27.0 │     1.0 │     1.0 │     0 │     2.0 │     nan │     nan │     nan │     nan │     0.0 │
│ ansonca01 │    1871 │     1 │ RC1     │ NA     │    25 │   120 │    29 │    39 │    11 │     3 │     0 │    16.0 │     6.0 │     2.0 │     2 │     1.0 │     nan │     nan │     nan │     nan │     0.0 │
│ armstbo01 │    1871 │     1 │ FW1     │ NA     │    12 │    49 │     9 │    11 │     2 │     1 │     0 │     5.0 │     0.0 │     1.0 │     0 │     1.0 │     nan │     nan │     nan │     nan │     0.0 │
│ barkeal01 │    1871 │     1 │ RC1     │ NA     │     1 │     4 │     0 │     1 │     0 │     0 │     0 │     2.0 │     0.0 │     0.0 │     1 │     0.0 │     nan │     nan │     nan │     nan │     0.0 │
│ barnero01 │    1871 │     1 │ BS1     │ NA     │    31 │   157 │    66 │    63 │    10 │     9 │     0 │    34.0 │    11.0 │     6.0 │    13 │     1.0 │     nan │     nan │     nan │     nan │     1.0 │
│ barrebi01 │    1871 │     1 │ FW1     │ NA     │     1 │     5 │     1 │     1 │     1 │     0 │     0 │     1.0 │     0.0 │     0.0 │     0 │     0.0 │     nan │     nan │     nan │     nan │     0.0 │
│ barrofr01 │    1871 │     1 │ BS1     │ NA     │    18 │    86 │    13 │    13 │     2 │     1 │     0 │    11.0 │     1.0 │     0.0 │     0 │     0.0 │     nan │     nan │     nan │     nan │     0.0 │
│ …         │       … │     … │ …       │ …      │     … │     … │     … │     … │     … │     … │     … │       … │       … │       … │     … │       … │       … │       … │       … │       … │       … │
└───────────┴─────────┴───────┴─────────┴────────┴───────┴───────┴───────┴───────┴───────┴───────┴───────┴─────────┴─────────┴─────────┴───────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

We can compute the average RBI per year per team.

pg_expr = pg_batting.group_by(("year_id", "team_id")).agg(avg_rbi=_.rbi.mean())
pg_expr

┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ year_id ┃ team_id ┃ avg_rbi   ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│ int64   │ string  │ float64   │
├─────────┼─────────┼───────────┤
│    1891 │ PIT     │ 22.782609 │
│    1895 │ BSN     │ 34.363636 │
│    1940 │ SLA     │ 22.343750 │
│    1981 │ HOU     │  9.972973 │
│    1913 │ CLE     │ 13.512821 │
│    1971 │ MON     │ 17.181818 │
│    2008 │ PIT     │ 15.000000 │
│    1895 │ WAS     │ 23.096774 │
│    2011 │ KCA     │ 16.785714 │
│    2007 │ MIL     │ 19.350000 │
│       … │ …       │         … │
└─────────┴─────────┴───────────┘

We can also rename columns to be more consistent with typical Snowflake usage.

pg_expr = pg_expr.rename("ALL_CAPS")
pg_expr

┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ YEAR_ID ┃ TEAM_ID ┃ AVG_RBI   ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│ int64   │ string  │ float64   │
├─────────┼─────────┼───────────┤
│    1891 │ PIT     │ 22.782609 │
│    1895 │ BSN     │ 34.363636 │
│    1940 │ SLA     │ 22.343750 │
│    1981 │ HOU     │  9.972973 │
│    1913 │ CLE     │ 13.512821 │
│    1971 │ MON     │ 17.181818 │
│    2008 │ PIT     │ 15.000000 │
│    1895 │ WAS     │ 23.096774 │
│    2011 │ KCA     │ 16.785714 │
│    2007 │ MIL     │ 19.350000 │
│       … │ …       │         … │
└─────────┴─────────┴───────────┘

Let’s show how many rows we have in the result.

pg_expr.count()

Insert the computed results into Snowflake

Because all Ibis backends implement the to_pyarrow() method, we can get data out of another system and into Snowflake with a few lines of code.

First we’ll create a table in Snowflake to hold the data.

Ibis helps here by providing an API to access the schema from the postgres-based expression, and automatically translates postgres types into Snowflake types.

snow_table = snow_con.create_table("pg_batting", schema=pg_expr.schema(), temp=True)

1: By default the table will be created in the database and schema of the current connection.

We’ll show that the table is empty to sanity check ourselves.

snow_table

┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ YEAR_ID ┃ TEAM_ID ┃ AVG_RBI ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ int64   │ string  │ float64 │
└─────────┴─────────┴─────────┘

Insert the expression’s result table into Snowflake.

snow_con.insert("pg_batting", pg_expr.to_pyarrow())

To sanity check what we’ve done let’s peek at the table.

snow_table

┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ YEAR_ID ┃ TEAM_ID ┃ AVG_RBI   ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│ int64   │ string  │ float64   │
├─────────┼─────────┼───────────┤
│    1891 │ PIT     │ 22.782609 │
│    1895 │ BSN     │ 34.363636 │
│    1940 │ SLA     │ 22.343750 │
│    1981 │ HOU     │  9.972973 │
│    1913 │ CLE     │ 13.512821 │
│    1971 │ MON     │ 17.181818 │
│    2008 │ PIT     │ 15.000000 │
│    1895 │ WAS     │ 23.096774 │
│    2011 │ KCA     │ 16.785714 │
│    2007 │ MIL     │ 19.350000 │
│       … │ …       │         … │
└─────────┴─────────┴───────────┘

We’ll count them too, to be extra sure.

snow_table.count()

Conclusion

In this post we show how easy it is to move data from one backend into Snowflake using Ibis.

Please try it out and get in touch on Zulip or GitHub, we’d love to hear from you!

Analysis of World of Warcraft data

Tyler White — Thu, 29 Feb 2024 00:00:00 GMT

Introduction

I grew up playing games, and with the recent re-release of World of Warcraft Classic, it seems like a perfect time to analyze some in-game data!

This dataset is the product of a Horde player’s diligent recording throughout 2008, capturing the transitional phase between the Burning Crusade and Wrath of the Lich King expansions. Notably, starting November 13, 2008, the data showcases numerous characters venturing into new territories and advancing beyond the former level cap of 70.

Analysis

We’ll determine who logged in the most, who leveled from 70 to 80 the fastest, and what activities these players engaged with based on zones. Let’s get to work.

Getting started

Ibis ships with an examples module, which includes this specific data. We’ll use DuckDB here, but this is possible with other backends, and we encourage you to experiment. DuckDB is the default Ibis backend, so it’ll be easy to use with this example.

You can execute pip install ibis-framework[duckdb,examples] to work with Ibis and the example data.

from ibis.interactive import *

wowah_data = ex.wowah_data_raw.fetch()

wowah_data

┏━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ char  ┃ level ┃ race   ┃ charclass ┃ zone                   ┃ guild ┃ timestamp           ┃
┡━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ int32 │ int32 │ string │ string    │ string                 │ int32 │ timestamp(6)        │
├───────┼───────┼────────┼───────────┼────────────────────────┼───────┼─────────────────────┤
│ 59425 │     1 │ Orc    │ Rogue     │ Orgrimmar              │   165 │ 2008-01-01 00:02:04 │
│ 65494 │     9 │ Orc    │ Hunter    │ Durotar                │    -1 │ 2008-01-01 00:02:04 │
│ 65325 │    14 │ Orc    │ Warrior   │ Ghostlands             │    -1 │ 2008-01-01 00:02:04 │
│ 65490 │    18 │ Orc    │ Hunter    │ Ghostlands             │    -1 │ 2008-01-01 00:02:04 │
│  2288 │    60 │ Orc    │ Hunter    │ Hellfire Peninsula     │    -1 │ 2008-01-01 00:02:09 │
│  2289 │    60 │ Orc    │ Hunter    │ Hellfire Peninsula     │    -1 │ 2008-01-01 00:02:09 │
│ 61239 │    68 │ Orc    │ Hunter    │ Blade's Edge Mountains │   243 │ 2008-01-01 00:02:14 │
│ 59772 │    69 │ Orc    │ Warrior   │ Shadowmoon Valley      │    35 │ 2008-01-01 00:02:14 │
│ 22937 │    69 │ Orc    │ Rogue     │ Warsong Gulch          │   243 │ 2008-01-01 00:02:14 │
│ 23062 │    69 │ Orc    │ Shaman    │ Shattrath City         │   103 │ 2008-01-01 00:02:14 │
│     … │     … │ …      │ …         │ …                      │     … │ …                   │
└───────┴───────┴────────┴───────────┴────────────────────────┴───────┴─────────────────────┘

Getting table info

Let’s learn more about these fields. Are there any nulls we should consider? We can use the info method on our Ibis expression.

wowah_data.info()

┏━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━┓
┃ name      ┃ type         ┃ nullable ┃ nulls ┃ non_nulls ┃ null_frac ┃ pos  ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━┩
│ string    │ string       │ boolean  │ int64 │ int64     │ float64   │ int8 │
├───────────┼──────────────┼──────────┼───────┼───────────┼───────────┼──────┤
│ char      │ int32        │ True     │     0 │  10826734 │       0.0 │    0 │
│ level     │ int32        │ True     │     0 │  10826734 │       0.0 │    1 │
│ race      │ string       │ True     │     0 │  10826734 │       0.0 │    2 │
│ charclass │ string       │ True     │     0 │  10826734 │       0.0 │    3 │
│ zone      │ string       │ True     │     0 │  10826734 │       0.0 │    4 │
│ guild     │ int32        │ True     │     0 │  10826734 │       0.0 │    5 │
│ timestamp │ timestamp(6) │ True     │     0 │  10826734 │       0.0 │    6 │
└───────────┴──────────────┴──────────┴───────┴───────────┴───────────┴──────┘

We can also use value_counts on specific columns if we want to learn more.

wowah_data.race.value_counts()

┏━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ race      ┃ race_count ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ string    │ int64      │
├───────────┼────────────┤
│ Undead    │    2530156 │
│ Orc       │     933056 │
│ Troll     │    1102409 │
│ Blood Elf │    3929995 │
│ Tauren    │    2331118 │
└───────────┴────────────┘

We don’t have any missing values, and the data value_counts results match what I would expect.

How about duplicates? We can check the count of unique rows against the total count.

print(wowah_data.count())
print(wowah_data.nunique())
print(wowah_data.count() == wowah_data.nunique())

10826734
10823177
False

So we have some duplicates. What could the duplicate rows be?

We can find them like this.

wowah_duplicates = wowah_data.mutate(
    row_num=ibis.row_number().over(
        ibis.window(group_by=wowah_data.columns, order_by=_.timestamp)
    )
).filter(_.row_num > 0)

wowah_duplicates

┏━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ char  ┃ level ┃ race      ┃ charclass ┃ zone               ┃ guild ┃ timestamp           ┃ row_num ┃
┡━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ int32 │ int32 │ string    │ string    │ string             │ int32 │ timestamp(6)        │ int64   │
├───────┼───────┼───────────┼───────────┼────────────────────┼───────┼─────────────────────┼─────────┤
│   341 │    70 │ Undead    │ Rogue     │ Terokkar Forest    │   204 │ 2008-06-05 01:57:37 │       1 │
│   980 │    70 │ Orc       │ Hunter    │ Isle of Quel'Danas │    79 │ 2008-10-25 05:05:03 │       1 │
│  1321 │    70 │ Tauren    │ Druid     │ Isle of Quel'Danas │     4 │ 2008-04-17 15:38:46 │       1 │
│  2866 │    70 │ Undead    │ Priest    │ Nagrand            │   103 │ 2008-10-23 21:01:32 │       1 │
│  4318 │    70 │ Undead    │ Warrior   │ Karazhan           │    19 │ 2008-07-12 18:05:06 │       1 │
│ 11316 │    70 │ Undead    │ Mage      │ Alterac Valley     │   104 │ 2008-03-22 00:30:48 │       1 │
│ 17774 │    70 │ Blood Elf │ Hunter    │ Shattrath City     │   271 │ 2008-07-12 18:06:12 │       1 │
│ 19598 │    70 │ Tauren    │ Shaman    │ The Mechanar       │   101 │ 2008-06-05 11:10:57 │       1 │
│ 21828 │    70 │ Tauren    │ Shaman    │ Durotar            │    -1 │ 2008-10-23 19:10:31 │       1 │
│ 22484 │    70 │ Tauren    │ Hunter    │ Warsong Gulch      │   315 │ 2008-07-12 18:03:23 │       1 │
│     … │     … │ …         │ …         │ …                  │     … │ …                   │       … │
└───────┴───────┴───────────┴───────────┴────────────────────┴───────┴─────────────────────┴─────────┘

I suspect this data was captured by a single player spamming “/who” in the game, most likely using an AddOn, about every ten minutes. Some players could have been captured twice, depending on how the command was being filtered.

We can go ahead and remove these duplicates.

wowah_data = wowah_data.distinct()

Which player logged in the most?

We mentioned that there was a single player likely capturing these results. Let’s find out who that is.

(
    wowah_data
    .group_by([_.char, _.race, _.charclass])
    .agg(sessions=_.count())
    .order_by(_.sessions.desc())
)

┏━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ char  ┃ race      ┃ charclass ┃ sessions ┃
┡━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ int32 │ string    │ string    │ int64    │
├───────┼───────────┼───────────┼──────────┤
│   182 │ Troll     │ Hunter    │    42770 │
│ 57741 │ Undead    │ Warlock   │    16237 │
│  1384 │ Undead    │ Warlock   │    15878 │
│ 59489 │ Blood Elf │ Priest    │    13977 │
│ 62239 │ Undead    │ Mage      │    13776 │
│ 62446 │ Blood Elf │ Mage      │    13011 │
│ 31184 │ Undead    │ Rogue     │    12019 │
│ 24126 │ Blood Elf │ Warlock   │    11791 │
│ 61105 │ Blood Elf │ Priest    │    11731 │
│ 35072 │ Blood Elf │ Paladin   │    11399 │
│     … │ …         │ …         │        … │
└───────┴───────────┴───────────┴──────────┘

That Troll Hunter that never exceeded level 1 is likely our person with 42,770 sessions.

Who leveled the fastest from 70–80?

At the end of the year, there were 884 level 80s. Who leveled the fastest?

Finding this answer will involve filtering, grouping, and aggregating to compute each character’s time taken to level from 70 to 80.

Let’s start by creating an expression to filter to only the level 80 characters, then join it to filter and identify only where they were level 70 or 80. We’re only concerned with three columns so that we will select only those.

max_level_chars = wowah_data.filter(_.level == 80).select(_.char).distinct()
wowah_data_filtered = (
    wowah_data
    .join(max_level_chars, "char", how="inner")
    .filter(_.level.isin([70, 80]))
    .select(_.char, _.level, _.timestamp)
)
wowah_data_filtered

┏━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ char  ┃ level ┃ timestamp           ┃
┡━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ int32 │ int32 │ timestamp(6)        │
├───────┼───────┼─────────────────────┤
│ 62226 │    70 │ 2008-01-09 22:00:20 │
│ 21828 │    70 │ 2008-01-09 22:00:30 │
│ 62763 │    70 │ 2008-01-09 22:01:06 │
│ 27547 │    70 │ 2008-01-09 22:01:51 │
│  5730 │    70 │ 2008-01-09 22:02:07 │
│ 34216 │    70 │ 2008-01-09 22:09:27 │
│ 40951 │    70 │ 2008-01-09 22:09:32 │
│ 45552 │    70 │ 2008-01-09 22:10:08 │
│ 19481 │    70 │ 2008-01-09 22:10:23 │
│ 19085 │    70 │ 2008-01-09 22:10:23 │
│     … │     … │ …                   │
└───────┴───────┴─────────────────────┘

Let’s use the where option to help with the aggregation.

level_calc = (
    wowah_data_filtered.group_by(["char"])
    .mutate(
        ts_70=_.timestamp.max(where=_.level == 70),
        ts_80=_.timestamp.min(where=_.level == 80),
    )
    .drop(["level", "timestamp"])
    .distinct()
    .mutate(days_from_70_to_80=(_.ts_80.delta(_.ts_70, "day")))
    .order_by(_.days_from_70_to_80)
)

The data is filtered and grouped by character, and two new columns are created to represent timestamps for levels 70 and 80. Then we drop what we no longer need, get the distinct values, and calculate the time taken to level from 70 to 80. Then we sort it!

level_calc

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ char  ┃ ts_70               ┃ ts_80               ┃ days_from_70_to_80 ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ int32 │ timestamp(6)        │ timestamp(6)        │ int64              │
├───────┼─────────────────────┼─────────────────────┼────────────────────┤
│ 68544 │ 2008-11-27 01:58:13 │ 2008-11-30 16:19:04 │                  3 │
│  1450 │ 2008-11-18 02:46:37 │ 2008-11-21 11:45:35 │                  3 │
│   399 │ 2008-11-18 00:02:40 │ 2008-11-22 20:23:57 │                  4 │
│ 86264 │ 2008-11-20 23:09:17 │ 2008-11-24 16:35:46 │                  4 │
│ 51738 │ 2008-12-03 20:10:00 │ 2008-12-07 11:26:08 │                  4 │
│  1003 │ 2008-11-18 00:29:25 │ 2008-11-22 23:47:06 │                  4 │
│ 40483 │ 2008-11-17 22:35:42 │ 2008-11-21 15:31:59 │                  4 │
│ 88331 │ 2008-12-12 01:47:57 │ 2008-12-16 10:19:23 │                  4 │
│ 86396 │ 2008-12-03 05:04:42 │ 2008-12-07 17:04:43 │                  4 │
│ 86265 │ 2008-11-20 23:28:40 │ 2008-11-24 07:47:08 │                  4 │
│     … │ …                   │ …                   │                  … │
└───────┴─────────────────────┴─────────────────────┴────────────────────┘

This isn’t perfect, as I found a case where there was a player who seemed to have quit in March and then returned for the new expansion. They hit 71 before it looks like their login at 70 was captured later. If you’re curious, take a look at char=21951 for yourself.

How did they level?

Let’s grab all the details from the previous result and join it back to get the timestamp and zone data.

leveler_zones = (
    level_calc.join(wowah_data, "char", how="inner")
    .filter(_.timestamp.between(_.ts_70, _.ts_80))
    .group_by([_.char, _.zone])
    .agg(zone_count=_.zone.count())
)
leveler_zones

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ char  ┃ zone              ┃ zone_count ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ int32 │ string            │ int64      │
├───────┼───────────────────┼────────────┤
│ 40309 │ Borean Tundra     │         70 │
│ 75845 │ Dragonblight      │         57 │
│  4175 │ Halls of Stone    │          4 │
│ 40309 │ Undercity         │         28 │
│ 78122 │ Dalaran           │          6 │
│ 75845 │ Howling Fjord     │         33 │
│ 35625 │ Borean Tundra     │         63 │
│ 67626 │ Shadowmoon Valley │         12 │
│ 54676 │ Icecrown          │         22 │
│  2877 │ Orgrimmar         │          9 │
│     … │ …                 │          … │
└───────┴───────────────────┴────────────┘

This code summarizes how often those characters appear in different zones while leveling up from level 70 to 80. It combines two sets of data based on character names, selects records within the leveling timeframe, groups data by character and zone, and counts the number of times each character was found in each zone.

There is another example table we can join to figure out the Zone information. I’m only interested in two columns, so I’ll filter this further and rename the columns.

zones = ex.wowah_zones_raw.fetch()
zones = zones.select(zone=_.Zone, zone_type=_.Type)
zones

┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ zone                 ┃ zone_type ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ string               │ string    │
├──────────────────────┼───────────┤
│ Durotar              │ Zone      │
│ The Barrens          │ Zone      │
│ Silverpine Forest    │ Zone      │
│ Stonetalon Mountains │ Zone      │
│ Thunder Bluff        │ City      │
│ Dustwallow Marsh     │ Zone      │
│ Durotar              │ City      │
│ Tirisfal Glades      │ City      │
│ Ashenvale            │ Zone      │
│ Stranglethorn Vale   │ Zone      │
│ …                    │ …         │
└──────────────────────┴───────────┘

Making use of pivot_wider and joining back to our leveler_zones expression will make this a breeze!

zones_pivot = (
    leveler_zones.join(zones, "zone")
    .group_by([_.char, _.zone_type])
    .agg(zone_type_count=_.zone.count())
    .pivot_wider(names_from="zone_type", values_from="zone_type_count")
)
zones_pivot

┏━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┓
┃ char  ┃ Zone  ┃ City  ┃ Sea   ┃ Battleground ┃ Dungeon ┃ Arena ┃
┡━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━┩
│ int32 │ int64 │ int64 │ int64 │ int64        │ int64   │ int64 │
├───────┼───────┼───────┼───────┼──────────────┼─────────┼───────┤
│ 30491 │    14 │     3 │  NULL │            1 │      17 │     3 │
│ 54357 │    12 │     2 │  NULL │            1 │      12 │     2 │
│ 24239 │    12 │     3 │  NULL │            1 │      14 │     3 │
│ 59778 │    13 │  NULL │  NULL │            2 │      11 │     1 │
│ 71918 │    17 │     3 │  NULL │            1 │      14 │     2 │
│ 73557 │    18 │     1 │     1 │            1 │      18 │     3 │
│ 87205 │    14 │     1 │  NULL │            1 │      14 │     3 │
│ 31900 │    30 │     2 │  NULL │            1 │      36 │     4 │
│   925 │    11 │  NULL │  NULL │            1 │      11 │     2 │
│ 86261 │    14 │  NULL │     1 │            1 │      14 │     2 │
│     … │     … │     … │     … │            … │       … │     … │
└───────┴───────┴───────┴───────┴──────────────┴─────────┴───────┘

If they have a high value in the “Zone” column, they were likely questing. Other players opted to venture into dungeons.

Next steps

It’s pretty easy to do complex analysis with Ibis. We churned through over 10 million rows in no time.

Get in touch with us on GitHub or Zulip, we’d love to see more analyses of this data set.

Ibis

Dynamic UDF Rewriting with Predicate Pushdowns

Introduction

ML models meet SQL

Smart UDFs with Ibis

Understanding tree pruning

Quickgrove: prunable GBDT models

Scalar PyArrow UDFs in Ibis

Making Ibis UDFs predicate-aware

1. Collecting predicates

2. Pruning model and creating a new UDF

3. Rewriting the plan

Diff

Putting it all together

Performance impact

LetSQL: simplifying UDF rewriting

Best practices & considerations

Conclusion

Resources

Does Ibis understand SQL?

Tl;dr

How does Ibis work?

Building an expression

Compiling the expression

Executing the compiled expression

So, does Ibis understand SQL?

Further reading

Querying Amazon Athena from the comfort of your Python interpreter

Why?

Installation Prerequisites

Installation

Data

Demo

How does this all work?

Classification metrics on the backend

Classification models

Model evaluation

Confusion matrix

Building a confusion matrix

Sample dataset

Metrics

A more efficient approach

Conclusion

Taking a random cube for a walk and making it talk

A random cube

From query to plot: Exploring GeoParquet Overture Maps with Ibis, DuckDB, and Lonboard

Installation

Motivation

Download data

Data exploration

Plotting with Lonboard

Resources

Better PyPI stats with Python

Overview

Prerequisites

Connecting to ClickHouse

Top packages by downloads

Analyzing downloads for a package

Shiny for Python application

Reproducing and contributing

Farewell pandas, and thanks for all the fish.

Why pandas? And a bit of Ibis history

The agony and the agony

Deferred vs Eager

NaN vs NULL

Data types

Misleading new users

Feature parity

Using IbisML and DuckDB for a Kaggle competition: credit risk model stability

Introduction

Prerequisites

Download data

Install libraries

Data loading and processing

Directory structure and tables

Base table

Querying 1TB on a laptop with Python dataframes

Exploring the data with Python dataframes

Ibis benchmarking: DuckDB, DataFusion, Polars

Key considerations

Why `pandas`? And a bit of Ibis history

`NaN` vs `NULL`