thomasjpfan.com

Six Years as a scikit-learn maintainer - Feature Retrospective

2025-05-15T09:29:00-05:00

It's hard to believe that it's been over six years since I joined the scikit-learn team as a maintainer. As of today, I have 1,374 commits and reviewed 3,179 pull requests. Behind these numbers, I am grateful for all the thoughtful discussions I have had with the community to push scikit-learn forward. Reviewing my commits, I would like to showcase some of my favorite features that I worked on:

1. Everything Trees 🌲🌲🌲

Native categorical support in Histogram-based Gradient Boosting Trees. (gh-26411, gh-18394)
Native missing value support in Random Forest & Trees. (gh-23595, gh-26391)
Cost complexity pruning In Trees. (gh-12887)

2. DataFrame interoperability 🖼️

Pandas and Polars DataFrame output with the set_output API. (gh-27315)
get_feature_names_out: Mapping input feature names to output feature names. (gh-18444)

3. Preprocessing 🕰️

TargetEncoder: Use the target to encode categorical data. (gh-25334)
Group infrequent categories in OrdinalEncoder and OneHotEncoder. (gh-25677)
KNN-based missing value imputation. (gh-12852)

4. Visualizations 📊

HTML Representation to visualize estimators in Jupyter notebooks. (gh-14180)
Plotting API for evaluating or inspecting estimators. (gh-14357)

5. Experimental GPU support 🏎️

Integrate Array API to run natively with PyTorch or CuPy arrays on a GPU. (gh-22554)

I hope you found some of these features useful or discovered some of them here 😁.

torch.export For Serializing Models and Faster Loading

2025-04-26T09:29:00-05:00

While torch.compile is great for Just in time (JIT) compilation, it adds significant startup time during prediction time. With PyTorch 2.6, torch.export can Ahead of Time (AOT) compile your PyTorch code and serialize it into a single zip file. When it works, torch.export's AOT approach has faster startup times compared to torch.compile's JIT. In this post, we learn about how to serialize and load a model using torch.export.

Using `torch.export`

First, we load a ResNet152 nn.Module model onto a GPU and data that represents typical model input:

from torchvision.models import resnet152

model = resnet152().to(torch.float32).cuda()
X = torch.randn(32, 3, 128, 128).to(torch.float32).cuda()

Next, we run export on the model and the input, while configuring a dynamic batch dimension:

from torch.export import export, Dim

batch_dim = Dim("batch", min=1, max=512)
exported_program = export(model, args=(X,), dynamic_shapes=({0: batch_dim}))

Normally, export will AOT compile the model to only work with the exact shape of the input. By setting dynamic_shape=({0: batch_dim}), we configure the compiler to allow the 0th indexed batch dimension to be a value between 1 and 512.

With the exported_program, we can run the model through a forward pass by calling .module() to extract the module:

exported_module = exported_program.module()
X_out = exported_module(X)

Finally, we can save the exported model by calling save with a file path:

from torch.export import save

save(exported_program, "resnet152.pt2")

Loading the model

With a serialized model, we can now run torch.export.load to load the model and run a forward pass with the data:

from torch.export import load

model = timed(load, "resnet152.pt2")
# Duration: 1.2s

result = timed(model.module(), X)
# Duration: 0.2s

The model takes 1.2s to load and 0.2s to use the loaded model to do a forward pass. For comparison, we can compile the same model with torch.compile's JIT:

model_opt = torch.compile(model, mode="reduce-overhead", fullgraph=True)
timed(model_opt, X)
# Duration: 21.1s

The first run takes 21.1s, because torch.compile is compiling from scratch. When we run it again with a warm cache, it takes 8.9s. Comparing the total time for initially loading the model and running a prediction, we get these total runtimes:

`torch.export`	`torch.compile`: warm cache	`torch.compile`: cold cache
1.4s	8.9s	21.1s

Using torch.export's AOT has faster startup times compared to torch.compile's JIT!

Conclusion

Although torch.export is faster compared to torch.compile, export has one big tradeoff:

torch.export requires full graph capture, which is more restrictive. torch.compile has the option to fall back to a Python interpreter for untraceable code, enabling compile to run any arbitrary Python code.

If you can capture the full graph, then torch.export has two major benefits:

You can load the serialized model from a non-Python environment such as C++. To learn more about this feature see the AOTInductor documentation for torch.export.
As shown in this post, the startup times are faster.

Overall, I find it remarkable that we have the choice between using AOT and JIT for running our PyTorch code, each with its own tradeoffs. You can learn more in torch.export's documentation.

Keep Warm with Portable torch.compile Caches

2025-04-06T09:29:00-05:00

PyTorch's compile function improves performance of your code by compiling and caching the computational graph for later use. torch.compile has a notion of a "cold compile" and a "warm compile", where "cold" is the first run and "warm" is faster by using a cache. The latest torch nightly wheel introduces a portable caching solution loadable on separate machines. In this post, we learn about this new caching solution and how it keeps your cache warm!

Cold vs Warm Compile Runtimes

In this example, we run torch.compile on a ResNet152 nn.Module without a cache and measure the cold start run:

from torchvision.models import resnet152

model = resnet152().to(torch.float32).cuda()
model_opt = torch.compile(model, mode="reduce-overhead", fullgraph=True)
X = torch.randn(32, 3, 128, 128).to(torch.float32).cuda()

timed(model_opt, X)
# Duration: 21.1s

The first run takes 21.1s, because torch.compile is compiling from scratch. When we run again, the cache is warm and the duration goes down to 8.9s which is 2x faster than a cold start!

Portable Cache

What if we wanted to populate the cache once on one machine and use that to warm up the cache for many machines? I have found success by adjusting the cache directory using TORCHINDUCTOR_CACHE_DIR, saving the entire directory, and loading the directory on the new machine. With the latest torch nightly, there is a new option: torch.compiler.save_cache_artifacts and torch.compiler.load_cache_artifacts.

With torch.compiler.save_cache_artifacts, we have a Python API for saving the cache as a binary file. In this example, we save the artifact as "artifact.bin":

artifact_path = Path("artifact.bin")

artifact_bytes, _ = torch.compiler.save_cache_artifacts()
save_artifact_path.write_bytes(artifact_bytes)

On a separate machine, we load the "artifact.bin" which warms up the cache!

artifact_bytes = artifact_path.read_bytes()
torch.compiler.load_cache_artifacts(artifact_bytes)

To learn more, refer to the documentation for save_cache_artifacts and load_cache_artifacts.

Conclusion

Now that frameworks, such as SGLang and vLLM, use torch.compile to speed up workloads, it is also important to optimize for startup times so we can scale up faster to meet demand. Keeping the torch cache warm is one dimension where we can improve startup times. The new {save,load}_cache_artifact API gives us a more portable way to keep the cache warm on multiple machines.

PyTorch Graphs Three Ways: Data-Dependent Control Flow

2025-03-16T09:29:00-05:00

Over the past few years, PyTorch went through a few iterations for turning Python code into a graph to improve performance:

TorchScript can trace or parse your code to generate a TorchScript intermediate representation that works on a subset of Python. Not in active development.
FX Graphs: torch.fx.symbolic_trace traces your code to produce a FX Graph we can mutate for optimizations.
Torch Compile: torch.compile reads the Python bytecode to generate FX Graphs while also falling back to Python for code it does not recognize.

This blog looks at how each system handles data dependent control flow with a simple example. You can run the code in this post on Google Colab.

Simple Function¶

First, we define a simple function that branches based on the input's data, i.e. if the input contains all positive values:

import torch

def func(x: torch.Tensor):
    all_pos = torch.all(x >= 0)
    if all_pos:
        return x + 10
    else:
        return x - 10

here, we pass all positive and all negative tensors to show the different code paths:

x_pos = torch.asarray([1, 2, 3])
x_neg = torch.asarray([-1, -2, -3])

func(x_neg), func(x_pos)

Out[3]:

(tensor([-11, -12, -13]), tensor([11, 12, 13]))

TorchScript¶

TorchScript allows us to trace a function, by passing in a sample input and running jit.trace:

func_jit_trace = torch.jit.trace(func, (x_pos,))

/var/folders/9l/pvs3_wlj23z_qxd4m8dv9w300000gn/T/ipykernel_31962/766430457.py:5: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if all_pos:

We see a warning because it notices the control flow and that only one of the branches will be traced. This means that the resulting function is incorrect for a negative tensor:

func_jit_trace(x_pos), func_jit_trace(x_neg)

Out[5]:

(tensor([11, 12, 13]), tensor([9, 8, 7]))

Note, TorchScript also has a jit.script which can parse the control flow logic and give the correct result:

func_jit_script = torch.jit.script(func)

func_jit_script(x_neg), func_jit_script(x_pos)

Out[7]:

(tensor([-11, -12, -13]), tensor([11, 12, 13]))

Torch FX¶

When we run torch.fx.symbolic_trace on the same function, it'll throw an error because it can not convert the function into a single FX graph because of the control flow:

try:
    torch.fx.symbolic_trace(func)
except Exception as e:
    print(e)

symbolically traced variables cannot be used as inputs to control flow

A workaround is to provide a concrete_args, so the tracing only goes down one of the code paths:

fx_func = torch.fx.symbolic_trace(func, concrete_args={"x": x_pos})

/Users/thomasfan/micromamba/envs/torch/lib/python3.12/site-packages/torch/fx/_symbolic_trace.py:906: UserWarning: Was not able to add assertion to guarantee correct input x to specialized function. It is up to the user to make sure that your inputs match the inputs you specialized the function with.
  warnings.warn(

But running this functino will give the incorrect results for x_neg:

fx_func(x_neg), fx_func(x_pos)

Out[10]:

(tensor([11, 12, 13]), tensor([11, 12, 13]))

torch.compile¶

The newer torch.compile will compile the function and gives the correct results by default:

compiled_func = torch.compile(func)

compiled_func(x_neg), compiled_func(x_pos)

Out[12]:

(tensor([-11, -12, -13]), tensor([11, 12, 13]))

Under the covers, torch.compile builds two graphs and uses a graph break to handle the control flow:

explained = torch._dynamo.explain(func)(x_pos)

print(f"""Graphs: {explained.graph_count}
Graph Breaks: {explained.graph_break_count}""")

Graphs: 2
Graph Breaks: 1

If you want the most performance, then it's best to avoid the graph breaks, using fullgraph=True:

compiled_full_bad = torch.compile(func, fullgraph=True)

But this will result in an error because the conditional requires a graph break:

try:
    compiled_full_bad(x_pos)
except Exception as e:
    print(e)

Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond to explicitly capture the control flow. For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#cond-operands

from user code:
   File "/var/folders/9l/pvs3_wlj23z_qxd4m8dv9w300000gn/T/ipykernel_31962/766430457.py", line 5, in func
    if all_pos:

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

To work around the graph break, we can use torch.cond to build the full graph:

def func_cond(x):
    return torch.cond(
        torch.all(x >= 0), lambda x: x + 10, lambda x: x - 10, (x,)
    )

compiled_full_good = torch.compile(func_cond, fullgraph=True)

compiled_full_good(x_neg), compiled_full_good(x_pos)

Out[17]:

(tensor([-11, -12, -13]), tensor([11, 12, 13]))

We see that there are zero graph breaks by using explain:

explained_cond = torch._dynamo.explain(func_cond)(x_pos)

print(f"""Graphs: {explained_cond.graph_count}
Graph Breaks: {explained_cond.graph_break_count}""")

Graphs: 1
Graph Breaks: 0

Conclusion¶

TorchScript’s usability issue stems from only supporting a subset of Python, which means we likely need to update code to make it work. FX Graph has a limited symbolic tracer producing an intermediate representation that we can mutate and passed to a compiler.

Using Python’s bytecode, torch.compile still produces FX graphs, but it “just work” with any other Python code. For code torch.compile does not recognize it will fall back to the Python interpreter. This means you can torch.compile to get some initial improvements and iterate to make your code run faster by reducing the number of graph breaks.

Python Extensions in Rust with Jupyter Notebooks

2023-12-27T09:29:00-05:00

The Rust programming language has gotten more prominent for writing compiled Python extensions. Currently, there is a bunch of boilerplate for wrapping writing up a Rust function and making it callable from Python. I enjoy exploring and prototyping code in Jupyter Notebooks, so I developed rustimport_jupyter to compile Rust code in Jupyter and have the compiled code available in Python! In this blog post, I will showcase a simple function, NumPy function, and Polar expression plugins. This blog post is runnable as a notebook on Google Colab.

Simple Rust Functions¶

rustimport_jupyter builds on top of rustimport to compile Python extensions written in Rust from Jupyter notebooks. After installing the rustimport_jupyter package from PyPI, we load the magic from within a Jupyter notebook:

%load_ext rustimport_jupyter

Next, we define a double function in Rust and prefixing the cell with the %%rustimport marker:

%%rustimport
use pyo3::prelude::*;

#[pyfunction]
fn double(x: i32) -> i32 {
    2 * x
}

The %%rustimport marker compiles the Rust code and imports the double function into the Jupyter notebook environment. This means, we can directly call it from Python!

double(34)

Out[4]:

By default, %%rustimport is compiles without Rust optimizations. We can enable these optimizations by adding the --release flag:

%%rustimport --release
use pyo3::prelude::*;

#[pyfunction]
fn triple(x: i32) -> i32 {
    3 * x
}

triple(7)

Out[6]:

NumPy in Rust¶

Rust's ecosystem contains many third party libraries that is useful for writing our custom functions. rustimport defines a custom //: comment syntax that we can use to pull in write our extensions. In this next example, we use PyO3/rust-numpy to define a NumPy function that computes a*x-y in Rust:

%%rustimport --release
//: [dependencies]
//: pyo3 = { version = "0.20", features = ["extension-module"] }
//: numpy = "0.20"

use pyo3::prelude::*;
use numpy::ndarray::{ArrayD, ArrayViewD};
use numpy::{IntoPyArray, PyArrayDyn, PyReadonlyArrayDyn};

fn axsy(a: f64, x: ArrayViewD<'_, f64>, y: ArrayViewD<'_, f64>) -> ArrayD<f64> {
    a * &x - &y
}

#[pyfunction]
#[pyo3(name = "axsy")]
fn axsy_py<'py>(
    py: Python<'py>,
    a: f64,
    x: PyReadonlyArrayDyn<'py, f64>,
    y: PyReadonlyArrayDyn<'py, f64>,
) -> &'py PyArrayDyn<f64> {
    let x = x.as_array();
    let y = y.as_array();
    let z = axsy(a, x, y);
    z.into_pyarray(py)
}

The pyo3(name = "axsy") Rust macro exports the compiled function as axsy in Python. We can now use axsy directly in Jupyter:

import numpy as np

a = 2.4
x = np.array([1.0, -3.0, 4.0], dtype=np.float64)
y = np.array([2.1, 1.0, 4.0], dtype=np.float64)

axsy(a, x, y)

Out[8]:

array([ 0.3, -8.2,  5.6])

Polars Expression Plugin¶

Recently, Polars added support for expression plugins to create user defined functions. With rustimport_jupyter, we can prototype quickly on an Polars expression directly in Jupyter! In this example, we compile a pig-laten expression as seen in Polar's user guide:

%%rustimport --module-path-variable=polars_pig_latin_module
//: [dependencies]
//: polars = { version = "*" }
//: pyo3 = { version = "*", features = ["extension-module"] }
//: pyo3-polars = { version = "0.9", features = ["derive"] }
//: serde = { version = "*", features = ["derive"] }

use pyo3::prelude::*;
use polars::prelude::*;
use pyo3_polars::derive::polars_expr;
use std::fmt::Write;

fn pig_latin_str(value: &str, output: &mut String) {
    if let Some(first_char) = value.chars().next() {
        write!(output, "{}{}ay", &value[1..], first_char).unwrap()
    }
}

#[polars_expr(output_type=Utf8)]
fn pig_latinnify(inputs: &[Series]) -> PolarsResult<Series> {
    let ca = inputs[0].utf8()?;
    let out: Utf8Chunked = ca.apply_to_buffer(pig_latin_str);
    Ok(out.into_series())
}

Note that we use --module-path-variable=polars_pig_latin_module, which saves the compiled module path as polars_pig_latin_module. With polars_pig_latin_module defined, we configure a language namespace for the Polars DataFrame:

import polars as pl

@pl.api.register_expr_namespace("language")
class Language:
    def __init__(self, expr: pl.Expr):
        self._expr = expr

    def pig_latinnify(self) -> pl.Expr:
        return self._expr.register_plugin(
            lib=polars_pig_latin_module,
            symbol="pig_latinnify",
            is_elementwise=True,
        )

With the language namepsace defined, we can now use it with Polars:

df = pl.DataFrame(
    {
        "convert": ["pig", "latin", "is", "silly"],
    }
)

out = df.with_columns(
    pig_latin=pl.col("convert").language.pig_latinnify(),
)
print(out)

shape: (4, 2)
┌─────────┬───────────┐
│ convert ┆ pig_latin │
│ ---     ┆ ---       │
│ str     ┆ str       │
╞═════════╪═══════════╡
│ pig     ┆ igpay     │
│ latin   ┆ atinlay   │
│ is      ┆ siay      │
│ silly   ┆ illysay   │
└─────────┴───────────┘

Conclusion¶

For those who like prototyping and exploring in Jupyter notebooks, rustimport_jupyter enables you to explore the Rust ecosystem while easily connecting it to your Python code. You can try out the library by installing it with: pip install rustimport_jupyter 🚀!

Quick NumPy UFuncs with Cython 3.0

2023-08-15T09:29:00-05:00

Cython 3.0 was finally released on July 17, 2023 and it comes with many features. There are many exciting features such as a Pure Python Mode and Initial support for Python's Limited API. In this blog post, we use Cython 3.0's @cython.ufunc decorator to quickly generate a NumPy Universal function (UFunc) for fused multiply-add (fma). This blog post is runnable as a notebook on Google Colab.

Fused Multiply-Add and UFuncs¶

Fused multiply-add computes (x * y) + z as if to infinite precision and rounded once to fit the return type. As of August 15, 2023, fma is not a part of NumPy, but it is part of the C99 standard. With Cython 3.0, we can quickly generate a NumPy UFunc for fma:

import numpy as np
np_include = np.get_include()

%%cython -I$np_include
cimport cython
from libc.math cimport fma

@cython.ufunc
cdef double fma_ufunc(double x, double y, double z) nogil:
    return fma(x, y, z)

With only a @cython.ufunc decorator, we get a function that operates on NumPy arrays:

import numpy as np

a = np.array([1.0, 2.0])
b = np.array([4.5, 8.0])
c = np.array([-2.0, 3.0])
fma_ufunc(a, b, c)

Out[5]:

array([ 2.5, 19. ])

Since fma_ufunc is a UFunc, it has broadcasting out of the box! In the following example, a has shape (2,) and b_column has shape (1, 2), so the resulting matrix follows NumPy's broadcasting rules to return an array of shape (2, 2).

b_column = np.array([[4.0], [-3.0]])
fma_ufunc(a, b_column, -4.5)

Out[6]:

array([[ -0.5,   3.5],
       [ -7.5, -10.5]])

Moreover, NumPy's UFunc allows us to specific an output array to place the results in with the out keyword argument:

out = np.empty((2, 2))
fma_ufunc(a, b_column, -4.5, out=out)
out

Out[7]:

array([[ -0.5,   3.5],
       [ -7.5, -10.5]])

Depending on the use case, the out keyword argument can help with reducing the peak memory usage of your program.

Supporting Floats and Doubles¶

C99 defines a fused multiple-add for single and double precision floating point numbers. We use Cython's Fused Types to extend our UFunc to single precision floats:

%%cython -I$np_include
cimport cython
from cython cimport floating
from libc.math cimport fma, fmaf

@cython.ufunc
cdef floating fma_fused_ufunc(floating x, floating y, floating z) nogil:
    if floating is double:
        return fma(x, y, z)
    else:  # floating is float in this case
        return fmaf(x, y, z)

The floating fused type consist of single and double precision types, which we use to call the corresponding version of fused multiply-add from C. When all the input arrays are float32, then the resulting array is also float32:

a = np.array([1.0, 2.0], dtype=np.float32)
b = np.array([4.5, 8.0], dtype=np.float32)
c = np.array([-2.0, 3.0], dtype=np.float32)

fma_fused_ufunc(a, b, c)

Out[11]:

array([ 2.5, 19. ], dtype=float32)

Conclusion¶

An alternative approach is Numba's @vectorize decorator, which provides a Just in Time compiled UFunc. On the other hand, Cython provides a Ahead of Time compiled solution using the @cython.ufunc decorator to create a NumPy UFunc. In either case, you can make use of many UFunc's features such as array broadcasting or specifying an output array with the out keyword argument 😊. This blog post runnable as a notebook on Google Colab.

Accessing Data from Python's DataFrame Interchange Protocol

2023-05-14T09:29:00-05:00

Python's DataFrame interchange protocol specifies a zero-copy data interchange between Python DataFrame libraries, such as Pandas, Vaex, and Polars. This blog post explores how to read data from the DataFrame Interchange Protocol and perform a simple computation using Python's ctypes module. We use Cython to access the data without the GIL and perform the same calculation. This blog post is runnable as a notebook on Google Colab.

Polars DataFrame and the Exchange Protocol¶

First, we create a small DataFrame with a single column and missing values using Polars:

import polars as pl

df = pl.DataFrame(
    {
        "first": [None, 1, 2, 3, 8, None, 1, None, 10, -2, -1],
    },
    schema={"first": pl.Int64}
)

The __dataframe__ method of a DataFrame returns an object that implements the DataFrame Interchange Protocol:

df_protocol = df.__dataframe__()

We get the column from the API specification and access the buffer that contains the data:

column = df_protocol.get_column_by_name("first")
buffer = column.get_buffers()

The buffer object is a dictionary composed of buffers representing the data and validity:

from pprint import pprint

pprint(buffer)

{'data': (PyArrowBuffer({'bufsize': 88, 'ptr': 4491764957280, 'device': 'CPU'}),
          (<DtypeKind.INT: 0>, 64, 'l', '=')),
 'offsets': None,
 'validity': (PyArrowBuffer({'bufsize': 2, 'ptr': 4491764629696, 'device': 'CPU'}),
              (<DtypeKind.BOOL: 20>, 1, 'b', '='))}

The 'data' entry consists of the underlying data's buffer and type. We compute the number of items in the buffer by dividing the buffer's size by the data type's size:

buffer_size_in_bits = buffer["data"][0].bufsize * 8
buffer_dtype_size = buffer["data"][1][1]
n_items = buffer_size_in_bits // buffer_dtype_size
print(n_items)

With the ctypes module, we access the buffer using the pointer address:

import ctypes
data = (ctypes.c_int64 * n_items).from_address(buffer["data"][0].ptr)

print(list(data))

[0, 1, 2, 3, 8, 0, 1, 0, 10, -2, -1]

Validity buffer¶

This array does not give the whole picture of the column. The original column contains null values, represented by a mask stored in the validity buffer. The interchange API tells us that the validity buffer is a bit mask:

column.describe_null

Out[9]:

(<ColumnNullType.USE_BITMASK: 3>, 0)

Looking at the validity buffer, we see that the buffer size is 2 bytes and the data type has a size of 1 bit, which is consistent with being a bit mask:

pprint(buffer["validity"])

(PyArrowBuffer({'bufsize': 2, 'ptr': 4491764629696, 'device': 'CPU'}),
 (<DtypeKind.BOOL: 20>, 1, 'b', '='))

There are no ctypes that represent one bit of data. However, we can use unsigned 8-bit integers to store the validity buffer:

n_items_validity = buffer["validity"][0].bufsize
validity = (ctypes.c_uint8 * n_items_validity).from_address(buffer["validity"][0].ptr)

The 16 bits are enough space to store the bit-mask of the original 11 bits. We use bit-wise operations to access the bit mask:

for i in range(n_items):
    val_idx = i // 8
    val_remainer = i % 8
    val = (validity[val_idx] >> val_remainer) & 1

    end = ", " if i < n_items - 1 else ""
    print(f"{val}", end=end)

0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1

Computing the nan mean¶

With the data and validity buffer, we can use Python to perform the meanwhile ignoring the null values:

def nan_mean(data, validity):
    total = 0.0
    count = 0
    for i in range(len(data)):
        val_idx = i // 8
        val_remainder = i % 8
        val = (validity[val_idx] >> val_remainder) & 1
        if val:
            total += data[i]
            count += 1
    return total / count

print(nan_mean(data, validity))

2.75

This value is consistent with the value computed using Polars from the original DataFrame, which also ignores the null values:

print(df.mean())

shape: (1, 1)
┌───────┐
│ first │
│ ---   │
│ f64   │
╞═══════╡
│ 2.75  │
└───────┘

One awesome fact about the ctype objects is that they also implement Python's Buffer Protocol. With the Buffer Protocol, we can write a Cython function with memoryviews to perform the nan-mean, while releasing the GIL:

%load_ext Cython

%%cython
cimport cython

@cython.boundscheck(False)
@cython.wraparound(False)
def nan_mean_cython(long[::1] array, unsigned char[::1] validity):
    cdef:
        Py_ssize_t idx, val_idx, val_remainder
        double output = 0.0
        Py_ssize_t count = 0

    with nogil:
        for idx in range(array.shape[0]):
            val_idx = idx // 8
            val_remainder = idx % 8
            if (validity[val_idx] >> val_remainder) & 1:
                output += array[idx]
                count += 1

    return output / count

nan_mean_cython(data, validity)

Out[17]:

2.75

Why?¶

Python's DataFrame Interchange Protocol provides a uniform API for libraries to write for! In other words, the Cython function above works not only with Polars DataFrames but also with Pandas or Vaex DataFrames. Data access does not require the GIL, so we can release the gil and use native programming languages for acceleration!

Survival Regression Analysis on Customer Churn

2018-09-12T09:29:00-05:00

In this post, we will analyze Telcon's Customer Churn Dataset and figure out what factors contribute to churn. By definition, a customer churns when they unsubscribe or leaves a service. With survival analysis, the customer churn event is analogous to "death". Armed with the survival function, we will calculate what is the optimum monthly rate to maximize a customers lifetime value. The source of this post and instructions to reproduce this analysis can be found at the thomasjpfan/ml-journal repo.

Overview of the dataset¶

The dataset consist of many featuers associated with a customer. For regular survivial analysis, we only need the tenure and Churn features. The tenure is the number of time a customer has stayed with the service. The boolean Churn feature states if the customer churned or not:

df = pd.read_csv("data/WA_Fn-UseC_-Telco-Customer-Churn.csv")
df[['tenure', 'Churn']].head()

Out[9]:

	tenure	Churn
0	1	No
1	34	No
2	2	Yes
3	45	No
4	2	Yes

For customers that did not churn yet, they may churn in the future. Since this is data from the future, it is not recorded in our dataset. Datasets exhibiting this behavior are called right-censored. Luckily the Cox's model is able to handle right-censored data.

Loading data and fitting the model¶

We use the lifelines project to train a Cox’s Proportional Hazard model. This model is able to do regression on the other featuers in the dataset.

from lifelines import CoxPHFitter

events = convert_cat(df)
cph = CoxPHFitter()
_ = cph.fit(events, duration_col='tenure', event_col='Churn')

With the fitted survivial regression model, we take a look at how each feature affects the survivial function:

The standardized cofficients gives a sense of the impact of each feature. The closer the cofficient is to zero, the less effect it has on the survivial function. The survivial function defines the probability the churn event has not occured yet at a given month, $t$: $$ S(t) = P(T > t) $$ For example, when $t=0$, the probabilty $P(T > 0) = 1$, because on an infinite time scale, a customer will always churn. The boolean automatic_payment feature denotes if a customer has automatic payments enabled. We plot the survivial function with or without automatic payments:

The green and red curve represents the survivial function when automatic payment is on or off respectively. The result is expected, the green curve is always above the red curve, i.e. enabling automatic payments increaese the probability of survivial. The other boolean features also help with customer churn:

The survivial function for various contract lengths shows the expected result, i.e. longer contracts prevents customers from leaving:

Deep dive into monthly rates¶

In this section, we will calculate how much to charge a customer to maximize lifetime value. First, we visualize the monthly rate distrubution:

Next, we plot the survivial function for different monthly rates:

Again the result is expected, the higher the monthly rate, the lower the survivial function. With these survivial functions, we can calculate the average number of months a customer will stay for different monthly rate. Multiplying the average number of months with the monthly rate, gives the lifetime value of a customer at each price point:

In this case, the maximum expected lifetime value is 7139 USD, using a monthly rate of 179 USD.

Whats next?¶

Survival analysis is a powerful way to look at customer churn data. We calculated the impact of each feature on the survivial curve. Moreover, we used the survival curve to calculate the expected lifetime value of a customer for various monthly rates. The next step is to do the same analysis in a bayesian point of view, which adds a measure of uncertainty into the model, enhancing our understanding of the underlying processes.

Nuclei Image Segmentation Tutorial

2018-07-31T09:29:00-05:00

In this tutorial, we will implement a UNet to solve Kaggle's 2018 Data Science Bowl Competition. The challenge asks participants to find the location of nuclei from images of cells. The source of this tutorial and instructions to reproduce this analysis can be found at the thomasjpfan/ml-journal repo.

Exploring the Data¶

We can now define the datasets training and validiation datasets:

samples_dirs = list(d for d in Path('data/cells/').iterdir() if d.is_dir())

train_dirs, valid_dirs = train_test_split(
    samples_dirs, test_size=0.2, random_state=42)

train_cell_ds = CellsDataset(train_dirs)
valid_cell_ds = CellsDataset(valid_dirs)

Overall the cell images come in different sizes, and fall in three different categories:

Most of the data is of Type 2. Training a single model to be able to find the nuclei for all types may not be the best option, but we will give it a try! For reference here are the corresponding masks for the above three types:

In order to train a neutral net, each image we feed in must be the same size. For our dataset, we break our images up into 256x256 patches. The UNet architecture typically has a hard time dealing with objects on the edge of an image. In order to deal with this issue, we pad our images by 16 using reflection. The image augmentation is handled by PatchedDataset. Its implementation can be found in dataset.py.

train_ds = PatchedDataset(
    train_cell_ds, patch_size=(256, 256), padding=16, random_flips=True)
val_ds = PatchedDataset(
    valid_cell_ds, patch_size=(256, 256), padding=16, random_flips=False)

Defining the Module¶

Now we define the UNet module with the pretrained VGG16_bn as a feature encoder. The details of this module can be found in model.py:

module = UNet(pretrained=True)

Freezer¶

The features generated by VGG16_bn are prefixed with conv. These weights will be frozen, which restricts training to only our decoder layers.

from skorch.callbacks import Freezer
freezer = Freezer('conv*')

Learning Rate Scheduler¶

We use a Cyclic Learning Rate scheduler to train our neutral network.

from skorch.callbacks import LRScheduler
from skorch.callbacks.lr_scheduler import CyclicLR

cyclicLR = LRScheduler(policy=CyclicLR, 
                       base_lr=0.002, 
                       max_lr=0.2,
                       step_size_up=540,
                       step_size_down=540)

Why is step_size_up 540?

Since we are using a batch size of 32, each epoch will have about 54 (len(train_ds)//32) training iterations. We are also setting max_epochs to 20, which gives a total of 1080 (max_epochs*54) training iterations. We construct our Cyclic Learning Rate policy to peak at the 10th epoch by setting step_size_up to 540. This can be shown with a plot of the learning rate:

_, ax = plt.subplots(figsize=(10, 5))
ax.set_title('Cyclic Learning Rate Scheduler')
ax.set_xlabel('Training iteration')
ax.set_ylabel('Learning Rate')
ax.plot(cyclicLR.simulate(1080, 0.002));

Checkpoint¶

A checkpoint is used to save the model weights with the best loss:

from skorch.callbacks import Checkpoint

checkpoint = Checkpoint(dirname='unet')

Custom Loss Module¶

Since we have padded our images and mask, the loss function will need to ignore the padding when calculating the binary log loss. We define a BCEWithLogitsLossPadding to filter out the padding:

class BCEWithLogitsLossPadding(nn.Module):
    def __init__(self, padding=16):
        super().__init__()
        self.padding = padding

    def forward(self, input, target):
        input = input.squeeze_(
            dim=1)[:, self.padding:-self.padding, self.padding:-self.padding]
        target = target.squeeze_(
            dim=1)[:, self.padding:-self.padding, self.padding:-self.padding]
        return binary_cross_entropy_with_logits(input, target)

Training Skorch NeutralNet¶

Now we can define the skorch NeutralNet to train out UNet!

from skorch.net import NeuralNet
from skorch.helper import predefined_split

net = NeuralNet(
    module,
    criterion=BCEWithLogitsLossPadding,
    criterion__padding=16,
    batch_size=32,
    max_epochs=20,
    optimizer__momentum=0.9,
    iterator_train__shuffle=True,
    iterator_train__num_workers=4,
    iterator_valid__shuffle=False,
    iterator_valid__num_workers=4,
    train_split=predefined_split(val_ds),
    callbacks=[('freezer', freezer),
               ('cycleLR', cyclicLR), 
               ('checkpoint', checkpoint)],
    device='cuda' 
)

Let's highlight some parametesr in our NeutralNet:

criterion__padding=16 - Passes the padding to our BCEWithLogitsLossPadding initializer.
train_split=predefined_split(val_ds) - Sets the val_ds to be the validation set during training.
callbacks=[(..., Checkpoint(f_params='best_params.pt'))] - Saves the best parameters to best_params.pt.

Next we train our UNet with the training dataset:

net.fit(train_ds);

  epoch    train_loss    valid_loss    cp      dur
-------  ------------  ------------  ----  -------
      1        0.4901        0.4193     +  53.9509
      2        0.3803        0.3331     +  46.7676
      3        0.2797        0.2307     +  46.9844
      4        0.1653        0.1053     +  46.9767
      5        0.1076        0.1025     +  46.9547
      6        0.0825        0.0780     +  47.0113
      7        0.0765        0.0747     +  47.1332
      8        0.0732        0.0641     +  47.0073
      9        0.0632        0.0548     +  47.0701
     10        0.0574        0.0537     +  46.9553
     11        0.0565        0.0537        47.1040
     12        0.0544        0.0536     +  47.1731
     13        0.0543        0.0513     +  47.2048
     14        0.0523        0.0513        47.1222
     15        0.0520        0.0503     +  47.3969
     16        0.0515        0.0512        47.1741
     17        0.0514        0.0503     +  46.9930
     18        0.0522        0.0501     +  47.0438
     19        0.0517        0.0501        47.3764
     20        0.0515        0.0519        47.2810

Before we evaluate our model, we load the checkpoint with the best weights into the net object:

net.load_params(checkpoint=checkpoint)

Evaluating our model¶

Now that we trained our model, lets see how we did with the three types presented at the beginning of this tutorial. Since our UNet module, is designed to output logits, we must convert these values to probabilities:

val_masks = net.predict(val_ds).squeeze(1)
val_prob_masks = 1/(1 + np.exp(-val_masks))

We plot the predicted mask with its corresponding true mask and original image:

Our UNet is able to predict the location of the nuclei for all three types of cell images!

Whats next?¶

In this tutorial, we used skorch to train a UNet to predict the location of nuclei in an image. There are still areas that can be improved with our solution:

Since there are three types of images in our dataset, we can improve our results by having three different UNet models for each of the three types.
We can use traditional image processing to fill in the holes that our UNet produced.
Our loss function can include a loss analogous to the compeititons metric of intersection over union.

Rodents Of NYC

2017-08-28T09:29:00-05:00

On July 2017, NYC has announced a $32 million plan towards reducing rodent populations. The plan will fully take into affect at the end of 2017. We will analyze how NYC's current process to control rodent population is successful by using 311 complaint data and rodent inspection data.

Rodent Inspections/Baiting¶

All rodent inspection data are available at NYC Open Data. We will be working with data from Jan 2013 to July 2017:

Out[5]:

	INSPECTION_TYPE	JOB_ID	JOB_PROGRESS	ZIP_CODE	LATITUDE	LONGITUDE	INSPECTION_DATE	RESULT
929989	INITIAL	PO1000000	1	10457	40.850778	-73.887226	2015-07-03 01:18:47	Passed Inspection
929990	INITIAL	PO1000003	1	11419	40.691459	-73.820706	2015-07-08 02:55:49	Passed Inspection
929991	INITIAL	PO1000004	1	11354	40.764792	-73.828963	2015-07-10 04:15:13	Active Rat Signs
288082	COMPLIANCE	PO1000004	2	11354	40.764792	-73.828963	2015-10-01 11:00:36	Passed Inspection
929992	INITIAL	PO1000018	1	11368	40.747279	-73.862460	2015-07-21 10:30:12	Problem Conditions

Each site is given a JOB_ID, where JOB_PROGRESS increases with every revisit to the location. There are a total of four inspection types:

Initial Inspection (INITIAL) - Inspection responding to 311 call
Compliance Inspection (COMPLIANCE) - After failing initial inspection, a follow up will be conducted.
Baiting - Application of rodenticide or monitoring
Clean Up - Removal of garbage and clutter

Taking a look at the number of inspections done since 2015, there is a noticeable uptick in inspections during the first half of 2017:

As of this blog post, we only have data for the first 7 months in 2017, lets compare the change in inspection count for the first 7 months year over year:

In the first 7 months of 2017, there was 27,000 more more initial inspections compared to 2016. Over the same period, there has been 4,000 more compliance inspections. Similarly, there is a increase activity in baiting during 2017. With clean ups per week in the teens and baiting per week in the thousands, NYC overwhelming prefers baiting than physical clean ups:

In the past two years, there has been an increase in 7,000 baiting events relative to the previous year:

Is it working?¶

From the rodent inspection data, we can see that there is increase activity in trying to get rid of rodents. But how can we tell if it is working? The inspectors do return to a baited location to see if the location is cleared of rats, but that could just mean the rats just moved to another location. One solution is to observe how many rodent related 311 calls are coming in since January 2013. 311 data can be downloaded from NYC Open Data. Here is a sample of the data we will be working with:

WARNING:root:Requests made without an app_token will be subject to strict throttling limits.

Out[10]:

	incident_zip	created_date	unique_key	borough
0	10462	2016-05-16 12:00:00	33371211	BRONX
1	10306	2016-05-17 12:00:00	33375487	STATEN ISLAND
2	10463	2016-05-17 12:00:00	33379343	BRONX
3	10462	2016-05-17 12:00:00	33379676	BRONX
4	10463	2016-05-18 12:00:00	33384535	BRONX

Most of the columns were filtered since we are focusing on how the count changes through time. The number of rodent related calls is periodic and is slowly increasing since 2011. Unsurprisingly, there are less rodent complaints during the winter months:

The number of rodent calls have been increasing about 2,300 year over year for the past three years:

Conclusion¶

Although there is an increase in baiting and inspection activity, there is still an increase of rodent related 311 calls in the same time period. We did manage to decreases the change in rodent calls for 2016, but it came back up to 2,600 in 2017. Mayor Blasio's $34 million plan will fully go into affect at the end of 2017. After the plan comes into affect, we will come back to see if it was able to control the rodent infestation.

Hassle Free UNets

2017-08-05T09:29:00-05:00

Constructing a UNet requires you to keep track of every signal size that flow through the UNet. This can lead to size mismatches when constructing the neutral network. To remedy this issue, I created a small PyTorch UNet module that calculates the sizes for you. You can even customize the building blocks used to construct the UNet. The UNet module can be found on github

Introduction

The UNet architecture, introduced in this paper, has the following structure:

The primary use for a UNet is to perform segmentation. In the above case, the UNet is used to detect cancerous regions in the input image. There are four blocks to constructing a UNet: split, center, merge, and final blocks. My implementation of UNet has the following initialization method:

class UNet(nn.Module):

    def __init__(self, *,
                 input_shape,
                 num_classes,
                 layers=4,
                 features_root=16,
                 split_block=SplitBlock,
                 merge_block=MergeBlock,
                 center_block=CenterBlock,
                 final_block=FinalBlock,
                 double_center_features=True):
                 ...

You can alter any of the four blocks used to generate the UNet. The layers parameter determines how tall the UNet is. For example, the image above has four layers. Each split layer upscales the signal to 2**layer*features_root where layer is the zero-indexed layer number.

I will use the short hand, (features, size), in my diagrams to denote the shape features x size x size of my signals. For example, (112, 32), means the signal has the shape 112x32x32.

Split Block

The split block has two outputs and one input:

The shapes of the outputs are calculated for you, all you have to do is provide your custom implementation of the split block. For reference, here is the default split_block implementation:

class SplitBlock(nn.Module):

    def __init__(self, in_shape, up_shape, hor_shape, layer):
        ...

    def forward(self, x):
        ...
        return self.max_pool(hor), hor

The extra layer index is passed in, just in case you want to adjust the block for different layers. To create your custom implementation, just copy the SplitBlock implementation and change the bodies of __init__ and forward. Make sure that the forward call returns two values, the first being the up signal and second being the hor signal. The parameters {}_shape uses the convention (features, size). You can use these parameters to perform assertions on the signal shapes during initialization.

Center Block

The center block has one input and one output:

This block does not change the size of the signal, only the number of features change. Thus the default center_block implementation just needs the feature number:

class CenterBlock(nn.Module):

    def __init__(self, in_feats, out_feats):
        ...

    def forward(self, x):
        return self.layers(x)

If double_center_features from UNet initialization is True, out_feats is two times in_feats, otherwise they are the same.

Merge Block

The merge block has two inputs and one output:

The default implementation of this block is:

class MergeBlock(nn.Module):
    def __init__(self, in_shape, out_shape, hor_shape, layer):
        ...

    def forward(self, x, hor):
        return out

The in_shape and out_shape parameters depends on the double_center_features. If the features were doubled at the center block, the features going down the merge block will also be doubled. forward takes in two inputs and outputs one signal.

Final Block

The final block has one input and one output:

The default final block just has a single convolution layer:

class FinalBlock(nn.Module):

    def __init__(self, in_feats, out_feats):
        self.layer = nn.Conv2d(in_feats, out_feats, kernel_size=1)

    def forward(self, x):
        return self.layer(x)

The final block outputs a signal of size (num_classes, input_size), which which was passed into UNet initialization. Each pixel is given a logit value for each class. This can transform to a probability by using a sigmoid layer.

Conclusion

This UNet module allowed me to quickly experiment with different blocks and hyper-parameters for any segmentation problem. Hopefully you it can help you too! :D

Bayesian Coin Flips

2015-09-12T09:29:00-05:00

In this blog post, we will look at the coin flip problem in a bayesian point of view. Most of this information is already widely available through the web, but I want to write it up anyways, so I can go into more involved bayesian concepts in future posts.

Likelihood¶

Lets say we flip a coin, and get $h$ heads and $t$ tails, the probability follows a binomial distribution:

$$ P(D|\theta) = {_{h+t}}C_{h}\theta^{h}(1-\theta)^{t} $$

where $D$ is the event of getting $h$ heads and $t$ tails, $\theta$ is the probability of heads, and $1-\theta$ is the probability of tails. Let say we want to flip the conditional probability using Bayes' theorem:

$$ P(\theta|D) = \dfrac{P(D|\theta)P(\theta)}{P(D)} $$

Why do we want to write the conditional probability this way?

The conditional probability, $P(\theta|D)$, treats the probability of heads, $\theta$, as a random variable. It is the probability of $\theta$, given that we observed the event $D$. To make speaking of these probabilies easier they are given names:

$P(\theta)$: the prior
$P(\theta|D)$: the posterior
$P(D|\theta)$: the likelihood

For example, lets say we flipped some coins and observed 3 heads and 5 tails, ($D$ is the event of 3 heads and 5 tails), the posterior allows us to obtain the probabilities of $P(\theta=0.1|D)$ or $P(\theta=0.7|D)$, etc. The posterior givens us probabilities for all possible values of $\theta$ (the probability of heads).

Prior¶

Next, lets look at the prior, $P(\theta)$, this is the probability of $\theta$ before any coin flips. In other words, this is the measure of the belief before we perform the experiment. For the coin flipping example, we normally come across coins that have $\theta=0.5$, so our prior should center around 0.5. For now, lets pick a beta distribution with $\alpha=2$ and $\beta=2$ as our prior:

$$ P(\theta) = \dfrac{1}{B(\alpha,\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1} $$

where $B(\alpha, \beta)$ is the Beta Function. This prior is centered at 0.5 and is lower for all other values. Let's graph the beta prior distribution:

Show Code

import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns

# configure style
mpl.rc('text', usetex=True)
mpl.rc('font', size=26)
sns.set_style("darkgrid")
sns.set_context("talk", rc={"figure.figsize": (12, 8)}, font_scale=1.5)
current_palette = sns.color_palette()

def plot_prior(alpha, beta, ax=None):
    x = np.linspace(0, 1, 1000)
    y = scipy.stats.beta.pdf(x, alpha, beta)

    if not ax:
        fig, ax = plt.subplots()
    ax.plot(x, y)
    ax.set_xlabel(r"$\theta$", fontsize=20)
    ax.set_ylabel(r"$P(\theta)$", fontsize=20)
    ax.set_title("Prior: BetaPDF({},{})".format(alpha,beta));
    
plot_prior(alpha=2, beta=2)

The maximum of our prior is centered at 0.5 and is lower for other values. This means that we normally see coins which are fair, but do not rule out that there is a chance that the coin could be unfair.

P(D)¶

The last thing we need to get the posterior is the denominator of bayes theorem, $P(D)$, which is the probability of the event happening. In general, this is calculated by integrating over all the possible values of $\theta$:

$$ P(D) = \int_0^1 P(D|\theta)P(\theta)d\theta $$

Normally this integral would not be possible to do analytically, but since our prior is a beta distribution and our likelihood is a binomial distribution, this integral would be worked out to be:

$$ P(D) = {_{h+t}}C_{h}\dfrac{B(h+\alpha, t+\beta)}{B(\alpha, \beta)} $$

For other priors, the integral would not be able to be computed, and other techniques are used to get the posterior, which I will get into in a future blog post.

Putting it together¶

Putting $P(D)$, the prior, and likelihood together into Bayes' theorem to get the posterior: $$ P(\theta|D) = \dfrac{1}{B(h+\alpha, t+\beta)}\theta^{h+\alpha-1}(1-\theta)^{t+\alpha-1} $$

Example¶

Recall, for our example, $\alpha=2$ and $\beta=2$, thus posterior becomes:

$$ P(\theta|D) = \dfrac{1}{B(h+2, t+2)}\theta^{h+1}(1-\theta)^{t+1} $$

Let's say create function in python to plot the posterior:

def plot_posterior(heads, tails, alpha, beta, ax=None):
    x = np.linspace(0, 1, 1000)
    y = scipy.stats.beta.pdf(x, heads+alpha, tails+beta)
        
    if not ax:
        fig, ax = plt.subplots()
    ax.plot(x, y)
    ax.set_xlabel(r"$\theta$", fontsize=20)
    ax.set_ylabel(r"$P(\theta|D)$", fontsize=20)
    ax.set_title("Posterior after {} heads, {} tails, \
                 Prior: BetaPDF({},{})".format(heads, tails, alpha, beta));

Lets say we flipped the coin 17 times and observed 5 heads and 12 tails, our posterior becomes:

plot_posterior(heads=5, tails=12, alpha=2, beta=2)

With 5 heads and 12 tails, our belief of the possible values of $\theta$ shifts to the left, suggesting that $\theta$ is more likely to be lower than $0.5$. Now lets say we flipped 75 times and observed 50 heads and 25 tails:

plot_posterior(heads=50, tails=25, alpha=2, beta=2)

With that many heads, the posterior shifts to the right, implying that $\theta$ is higher. Notice that the distrubution for 75 flips is narrower than the 17 times. With 75 flips, we have a clearer picture of what the value of $\theta$ should be.

Different Priors¶

What would happen when we choose other priors?

We'll explore how to handle non-beta priors in a future blog post. Right now, lets look at what happens when we choose different beta priors. Lets say we come from a world where coins are not 50-50, but are biased toward a bigger $\theta$:

plot_prior(alpha=20, beta=4)

Let's see what happens to the posterior when we flip a coin and get:

4 heads 5 tails
20 heads 20 tails
50 heads 49 tails
75 heads 74 tails
400 heads 399 tails

fig, axes = plt.subplots(5)
flips = [(4, 5), (20, 20), (50, 49), (75, 74), (400, 399)]
for i, flip in enumerate(flips):
    plot_posterior(heads=flip[0], tails=flip[1], alpha=20, beta=4, ax=axes[i])
    axes[i].set_yticks([])
fig.subplots_adjust(hspace=0.4)
for ax in axes:
    ax.set_yticks([])
    ax.set_xticks([])
    ax.set_xlabel("")
axes[4].set_xticks([0.5]);

When we only flip 9 coins in the first case, our belief does not change much and is still skewed. But as we flip more coins and get an 50-50 distrubution of heads and tails, our belief changes to a distrubution around $\theta=0.5$, i.e. a fair coin. Since our prior was so skewed, even with 400 heads and 399 tails, the peak of the posterior distrubution is still not 0.5.

Closing¶

In the next post, we will look at what happens when we have a non-beta prior, and use priors that are more strange. Specifically, we will look at situations where $P(D)$ can not be solve analytically, and must switch to other methods to obtain the posterior distrubution.

Statistical Power of Coin Flips

2015-08-29T09:29:00-05:00

There are a few topics that I wish were taught in an introduction to statistics undergraduate course. One of those topics is Bayesian Statistics, the other is Statistical Power. In this post, I go through the analysis of flipping coins, and how to calculate statistical power for determining if a coin is biased or fair.

When we flip a coin $n$ times with probability $p$ of getting heads, the probability distribution is binomial. We can apply hypothesis testing where the null hypothesis states: the coin is fair ( $p=0.5$ ), and the alternative hypothesis states: the coin is not fair, ( $p\neq0.5$ ). Additionally, we fix $\alpha=0.05$, meaning when the null hypothesis is true, we will reject it 5% of the time (Type I error). We can find the confidence interval:

from scipy.stats import binom

alpha = 0.05
n = 100 # number of flips
p = 0.5 # fair
binom.interval(1 - alpha, 100, 0.5)

Out[4]:

(40.0, 60.0)

For 100 coin flips, if we get a number of heads between 40 and 60, we "fail to reject the null hypothesis", otherwise we "reject the null hypothesis." With $\alpha=0.05$, we would incorrectly "reject the hypothesis" 5% of the time.

We may further ask, what is the probability that we "reject the null hypothesis" when $p\neq0.5$. Lets say that the coin has bias with $p=0.55$, the probability of not getting 40 to 60 heads is:

# cdf = cumulative distrubution function
1 - (binom.cdf(60, 100, 0.55) - binom.cdf(39, 100, 0.55))

Out[5]:

0.13519227295357183

If $p=0.55$, then there is a 13.5% chance that we will "reject the null hypothesis". In other words, 13.5% of the time we would be able to distinguish between a fair coin from a biased one. This value is commonly known as statistical power. Lets write a function to calculate the statistical power for different values of $p$ and flips, $n$:

def coin_power(n, p, alpha=0.05):
    """Calculates statistical power
    Arguments:
    n -- number of flips
    p -- actual probability for heads
    """
    # confidence interval for p = 0.5 for n flips
    lower, upper = binom.interval(1 - alpha, n, 0.5)
    return 1 - (binom.cdf(upper, n, p) - binom.cdf(lower - 1, n, p))

For 100 flips, we plot the statistical power as a function of the actual heads probability, $p$:

import matplotlib.pyplot as plt

fig, ax = plt.subplots(1, 1)
ax.set_title("Power vs Actual Heads Probability for n = 100")

def plot_power(n, ax, **kwargs):
    """Plots power vs actual heads
    Arguments:
    n -- number of flips
    ax -- matplotlib axes
    kwargs -- keyword arguments for plotting
    """
    p_values = np.linspace(0,1,1000)
    ax.plot(p_values, coin_power(n, p_values), **kwargs)
    ax.set_xticks(np.arange(0,1.1, 0.1))
    ax.set_yticks(np.arange(0,1.1, 0.1))
    ax.set_ybound((0, 1.05))
    ax.set_ylabel("Power")
    ax.set_xlabel("Actual Heads Probability")
    
plot_power(100, ax, color='r')

For 100 flips, if the actual heads probability is 0.4, then the power is 0.5, which means we would not be able to tell the different between a bias coin and fair coin 50% of the time. One way to increase the power is to increase the number of flips, n:

fig, ax = plt.subplots(1, 1)

def plot_powers(ax):
    ax.set_title("Power vs Actual Heads Probability")
    plot_power(100, ax, color='r', label="flips = 100")
    plot_power(250, ax, color='b', label="flips = 250")
    plot_power(1000, ax, color='g', label="flips = 1000")
    plot_power(3000, ax, color='y', label="flips = 3000")
    ax.legend(bbox_to_anchor=(0.75, 0.6), loc=2, prop={'size':16})
    
plot_powers(ax)

As we increase the number of flips, the power is greater for broader values of $p$. Lets focus on the region of power greater than 0.9:

fig, ax = plt.subplots(1, 1)
plot_powers(ax)
ax.annotate('Power > 0.9', xy=(0.4758, 0.95), size=16)
ax.fill_between([0, 1.0], [0.9, 0.9], [1.0, 1.0], alpha=0.2)
ax.set_ybound((0.85, 1.01))
ax.set_xbound((0.3, 0.7))
ax.set_xticks(np.arange(0.30,0.70,0.04));

Recall that power is the probability that we can distinguish between a bias coin and a fair coin. By inspecting the above plot, we can make the following approximations:

If we flip 100 coins, we get at least 0.9 power for a bias coin where $p<0.34$ or $p>0.66$.
If we flip 250 coins, we get at least 0.9 power for a bias coin where $p<0.40$ or $p>0.60$.
If we flip 1000 coins, we get at least 0.9 power for a bias coin where $p<0.45$ or $p>0.55$.
If we flip 3000 coins, we get at least 0.9 power for a bias coin where $p<0.47$ or $p>0.53$.

With 100 flips, we can only distinguish a very bias coin where $p<0.34$ or $p>0.66$ from a fair coin 90% of the time. With 10 times more flips (1000), we can distinguish a less bias coin where $p<0.45$ or $p>0.55$ from a fair coin 90% of the time.

It is important that experiements have a large enough sample size so that there is enough statistical power to detect differences. If the sample size is too small, we can only detect a difference when there is a massive difference from the norm.

PCA Before Regression

2015-08-15T19:29:00-05:00

Sometimes it can be quite harmful to remove features in a data set. This entry gives an example of when principle component analysis can drastically change the result of a simple linear regression. I am also trying out importing Jupyter notebooks into this post, so you will see In[] and Out[] tags, which signify the input and output of the notebook.

Suppose there are two variables, A and B, that have a positive correlation of 0.6. There is also a dependent variable Y, determined by Y = A - B, which is unknown. The goal is to obtain the linear relationship, Y = A - B, from data points (a, b, y). To make this more concrete, let's generate some data points with python:

import numpy as np

rho = 0.6 # positive correlation
size = 5000 # number of values

np.random.seed(42) # meaning of life
A = np.random.randn(size)
B = rho * A + np.sqrt(1 - rho**2) * np.random.randn(size)
Y = A - B

We can easily check that the correlation is 0.6:

print(np.corrcoef(A, B))

[[ 1.          0.59371757]
 [ 0.59371757  1.        ]]

To visualize the relationship between A and B:

import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6,6))
plt.scatter(A, B, marker='*', c='b')
plt.xlabel('A')
plt.ylabel('B');

Princple Component Analysis (PCA) looks for the directions in A-B space where the data points are more spread out. One can do this in Python like so:

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
pca.fit(np.c_[A,B])

Out[5]:

PCA(copy=True, n_components=2, whiten=False)

Ploting the two components found by PCA:

pca_comp = pca.explained_variance_.reshape((2,1)) * pca.components_
pca_cord = np.r_[A.mean(),B.mean()] + pca_comp * 2
fig, ax = plt.subplots(figsize=(8,8))
ax.plot([pca_cord[0][0], -pca_cord[0][0]], [pca_cord[0][1], -pca_cord[0][1]], 'b-', lw=3)
ax.plot([pca_cord[1][0], -pca_cord[1][0]], [pca_cord[1][1], -pca_cord[1][1]], 'r-', lw=3)
ax.scatter(A, B, marker='*', c='b')
ax.set_xbound((-4,4))
ax.set_ybound((-4,4))
ax.set_xlabel('A')
ax.set_ylabel('B');

There is a red and blue direction, and the blue direction explains most of the spread (variance) of the data. How much of the variance is explained by the blue direction? We can compare them with a graph:

fig, ax = plt.subplots(figsize=(10,5))
width = 0.8
ind = np.arange(len(pca.explained_variance_ratio_)) + (1-width)/2
ax.bar(ind, pca.explained_variance_ratio_, width=width, color=['b', 'r'])
ax.set_ybound((0,1))
ax.set_ylabel('Ratio of Variance Explained')
ax.set_xlabel('PCA Components')
ax.set_xticks(ind + width/2)
ax.set_xticklabels(('First', 'Second'));

The first/blue component of PCA explains about 0.8 of the variance, while the second one explains 0.2. The components of the two directions are:

pca.components_

Out[8]:

array([[-0.70227741, -0.71190339],
       [-0.71190339,  0.70227741]])

To make the make future formulas shorter, we will estimate 0.7022 as 0.70 amd 0.7119 as 0.71. The calculations in python will keep all significant figures. The transformation from A-B space into these components would be:

$$\begin{aligned} M = 0.70A+0.71B-(0.70\mu_A + 0.71\mu_B) \\ N = 0.70A-0.71B-(0.70\mu_A - 0.71\mu_B) \end{aligned}$$

where $\mu_A$ is the mean of A, $\mu_B$ is the mean of B, M and N are the values in the first and second princple component space.

Most textbooks/courses would suggest that we can just ignore the second component and use the first princple component. In other words, we will ignore N and keep M. So lets only keep the first component in PCA:

pca_1 = PCA(n_components=1) # keeping only one component
M = pca_1.fit_transform(np.c_[A,B])

Then we perform simple linear regression on this variable:

from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(M, Y)
 
# r^2 value
lr.score(M, Y)

Out[10]:

0.00018173249112451995

The $r^2$ value of this regression is extremely small. The transformation from variables A and B to M, consist of the sum of A and B, with very close coefficients, in other words our regression model is trying to fit a line to:

$$Y = \alpha + \beta*M \approx \alpha' + \beta'(A+B)$$

Recall for this example, Y was generated with the formula:

$$ Y = A-B$$

The regression model would never be able to fit and get the correct coefficients because $M\approx A+B$. If we go back and just perform linear regression on the original variables:

lr_orig = LinearRegression()
lr_orig.fit(np.c_[A,B], Y)

# r^2 value
print(lr_orig.coef_)
print(lr_orig.intercept_)
print(lr_orig.score(np.c_[A,B], Y))

[ 1. -1.]
0.0
1.0

We get the values that are consistent with our contrived model for Y.

One should be careful when performing feature engineering, you may accidentally lose some crucial information.

High n-Dimensional Spheres Are Only Skin Deep

2014-08-15T09:29:00-05:00

For the past few months, I've been implementing machine learning algorithms and one of the small details about higher dimensional spheres caught me by surprise.

Back in Real Analysis, every student goes through solving for the volume of a n-Dimensional sphere and gets:

$$ V_n(r) = \dfrac{\pi^{n/2}}{\Gamma\left(\dfrac{n}{2}+1\right)} r^d $$

where $r$ is the radius, $n$ is the dimension, and $\Gamma$ is the Gamma Function. Even without knowing the above equation, it is natural to assume that $V_n(r)$ is proportional to $r^n$ or $V_n(r)=Ar^n$, where $A$ is a constant.

To make this example more concrete, lets say we want to find the fraction of the volume of a sphere of radius, $R$, that lies between $r=0.95R$ and $r=R$. In other words, we want to find the fractional volume of the blue region:

Using basic algebra the fractional volume is:

$$ f = \dfrac{V_D(R)-V_D(0.95R)}{V_D(R)} = 1 - 0.95^N, $$

where $f$ is fractional volume. As the dimension, N, increases the fractional volume tends to one, as shown in this table:

In other words, in higher dimensions, almost all the volume of the sphere is in the shell that has a thickness equal to 5% of the radius. Mind-blowing?

Hold on Tight!

2014-07-04T09:29:00-05:00

There are quite a few places where one can go indoor rock climbing in NYC. To keep the climber safe, the climber wears a harness that is attached to to a rope. The rope is then wrapped around a pipe at least one time and then at least 160 degrees more depending on where your buddy is holding on to the other end. The amount of force needed to hold a climber up is surprisingly low because of how friction helps prevents the rope from slipping off the pipe.

Roughly, the rope that wraps around the pipe looks something like this:

The force that is used to hold the climber up is called a belaying force. To calculate the relationship between the force of the climber and the belaying force, we go back to undergraduate physics and draw a free body diagram of a small piece of the rope:

where $T$ is tension at one end, $T+dT$ is the tension at the other end, $N$ is the normal force, and $f$ is the friction force that is resisting slipping. In equilibrium, we sum the forces in the vertical:

$$ \sum F_y = dN - T\sin\left(\dfrac{d\theta}{2}\right) - (T+dT)\sin\left(\dfrac{d\theta}{2}\right) = 0 $$

and horizontal directions:

$$ \sum F_x = T\cos\left(\dfrac{d\theta}{2}\right) - (T+dT)\cos\left(\dfrac{d\theta}{2}\right) + df = 0 $$

Since $d\theta$ is small, the trigonometric functions can be simplified as follows:

$$ \sin\left(\dfrac{d\theta}{2}\right) = \dfrac{d\theta}{2} \textrm{ and } \cos\left(\dfrac{d\theta}{2}\right) = 1 $$

With these simplifications the vertical equation becomes:

$$ \dfrac{dN}{d\theta} = T $$

Recalling that $f=\mu N$ where $\mu$ is the static coefficient of friction, the horizontal equation becomes:

$$ \dfrac{dT}{d\theta} = \dfrac{df}{d\theta} = \mu\dfrac{dN}{d\theta} $$

Putting the above two equations together, we get a simple differential equation:

$$ \dfrac{dT}{d\theta} = \mu T $$

with the solution:

$$ T = T_0e^{\mu\theta} $$

where $T_0$ is the tension of belaying force and $T$ is force from the climber. If you want to calculate the force required to hold a climber up, we just solve for $T_0$:

$$ T_0 = Te^{-\mu\theta} $$

If the rope is wrapped around once and an addition 160 degrees, then theta is 160+360=520 degrees or 9.08 radians. Letting the coefficient of friction between a nylon rope and a steel pipe be 0.20 and weight of the climber be 120.0 lbs, then the force needed to hold the climber up is:

$$ T_0 = (120.0)e^{-(0.20)(9.08)} = 19.5\textrm{ lbs} $$

In other words, we only need to provide 16.3% of the weight of the climber to hold him/her up. Hoping your buddy uses two hands, thats only 10 lbs of force per hand. Kind of crazy huh?

Side Note

When ships dock, sailors use rope wrapped around a post to keep the ship from floating away. The rope is wrapped around the post several times like so:

In this picture, the rope wrapped around the post about five times. Assuming coefficient of friction is $0.20$, the knot tied to the dock only needs 0.2% of tugging force from to ship to keep it docked.

Maximum of free damped oscillators

2014-05-02T09:29:00-05:00

While reading a blog entry on free damped oscillators by Dr. Drang, I was caught off guard by the fact that the maximum of the damped oscillation solution is different from the envelope that is tangent to it. Looking back, this was obviously since the exponential is a monotonic decreasing function which would change the position of the local maximum of the cosine function. What I took for granted was the fact that the distance between the maximum is the same as the period of the cosine function.

The solution of the free damped oscillator is

$$ x(t) = Ae^{-\eta t}\cos(\omega t - \phi), $$

where $\eta$ is a damping constant, $A$ is the amplitude, $\omega$ is the angular frequency, and $\phi$ is a phase shift that depends on initial conditions. With $\eta=0.5$, $\omega=2\pi$, and $\phi=0$, this function looks like this:

The red function is $\pm e^{-\eta t}$, which is the envelope for $x(t)$ (the middle function). If you zoom into the peak at $t=1$, the difference between the peak of $x(t)$ and the envelope is apparent:

What is the actual difference between them? Well that just takes some simple calculus:

$$ \dfrac{dx}{dt} = -\eta\omega A e^{-\eta t}\cos(\omega t-\phi) - \omega A e^{-\eta t}\sin(\omega t -\phi) $$

Then taking the derivative and setting it equal to zero:

$$ -\eta\omega A e^{-\eta t}\cos(\omega t-\phi) - \omega A e^{-\eta t}\sin(\omega t -\phi) = 0 $$

and solving for $t$:

$$ t_{max} = -\dfrac{1}{\omega}\arctan(\eta) + \dfrac{\phi+2\pi n}{\omega} $$

where $n$ is a natural number. The tagent value can be obtained by setting $x(t)$ equal to the envelope:

$$ Ae^{-\eta t} = Ae^{-\eta t}\cos(\omega t - \phi) $$

and then solving for $t$:

$$ t_{tang} = \dfrac{\phi + 2 \pi n}{\omega} $$

where $n$ is a natural number. These are the maximum values for a normal cosine function.

Summary

The difference between $t_{max}$ and $t_{tang}$ is:

$$ t_{tang}-t_{max} = \dfrac{1}{\omega}\arctan(\eta) $$

This difference is the same for every peak, thus distance between the maximum of $x(t)$ is the same as the period of the cosine function. Honestly, this post is not that revolutionary, but I just wanted to test out how math renders on my site.

Code for Plots

For those who are interested, I used matplotlib to generated the graphs above:

import matplotlib.pyplot as plt
from numpy import exp, cos, pi, linspace
from scipy.optimize import minimize

eta = 0.5
omega = 2 * pi


def damped_os(t):
    return exp(-eta * t) * cos(omega * t)

t = linspace(0, 4, num=2000)
x = damped_os(t)
x_u = exp(-eta * t)
x_d = -exp(-eta * t)

fig, axes = plt.subplots()
axes.plot(t, x, 'black')
axes.plot(t, x_u, 'red')
axes.plot(t, x_d, 'red')

axes.set_xlabel('t')
axes.set_ylabel('x/A')

# plt.savefig('{filename}/images/201405_Maximum_of_damped_free_vibrations/freedampedfull.png')

# Zooming
axes.set_xlim(0.9, 1.1)
axes.set_ylim(0.55, 0.65)


def neg_damped_os(t):
    return -damped_os(t)

result = minimize(neg_damped_os, 1)

# maximum arrow
x_max, y_max = result.x, -result.fun
dx, dy = 0.02, 0.03
plt.arrow(x_max - dx, y_max - dy, dx, dy,
          width=0.0001, length_includes_head=True)
plt.annotate("max", xy=(x_max - dx - 0.01, y_max - dy - 0.005))

x_tang, y_tang = 1.0, damped_os(1.0)
dx, dy = 0.03, 0.01
plt.arrow(x_tang + dx, y_tang + dy, -dx, -dy,
          width=0.0001, length_includes_head=True)
plt.annotate("tangent", xy=(x_tang + dx + 0.005, y_tang + dy))

plt.savefig('images/201405_Maximum_of_damped_free_vibrations/freedampedzoom.png')

My Current Cpp Makefile

2014-01-04T14:51:00-05:00

When I took my first programming course at Polytech, I used an integrated development environment (IDE) called Visual Studio. It magically complied all my files together and formed an executable. Now a days, I've moved on from using IDEs and switch to using a text editor, Sublime Text. As the C++ code I wrote started to get more complicated and span multiple files, I needed a way to systematically complied the files together. Thus, I created a Makefile to build and organize my C++ files.

Makefile Logic

CXX = g++
CXXFLAGS=-g -O2 -Wall -Wconversion

The first two lines of my Makefile defines the complier and its flags. I especially like the -Wconversion flag, since I prefer code that have explicit type conversions. This flag issues a warning when a type conversion is implicit.

SOURCES=$(wildcard src/**/*.cpp src/*.cpp)
OBJECTS=$(patsubst src/%.cpp,bin/%.o,$(SOURCES))

The wildcard function uses a pattern to find the sources for compilation. In this case, all the filenames in src folder and its subfolders are stored into SOURCES. The patsubst function changes the extension of SOURCES to .o extension and defines a destination folder, bin, for the complied objects.

EXECUT=build/test.exe
MAIN=src/main.cpp

This defines the location of the main() function and the output executable. I have gotten into the habit of creating an executable for quick testing and having a single file, main.cpp, defining the main function of my program.

LIB_SOURCES=$(filter-out $(MAIN), $(SOURCES))
LIB_OBJ=$(patsubst src/%.cpp,bin/%.o,$(LIB_SOURCES))
LIB_TARGET=build/libYOURLIBRARY.a
SO_TARGET=$(patsubst %.a,%.so,$(LIB_TARGET))

If you wanted to build libraries, these variables give the sources and destination for the libraries. filter-out removes the main.cpp from the library compilation. There are two types of libraries, static and dynamic. The former uses a .a extension while the latter uses a .so extension.

Makefile Usage

With all the variables defined, the Makefile is used to do various actions. My most used action is the to build an executable from all the source files:

exec: build $(OBJECTS)
    $(CXX) $(CXXFLAGS) -o $(EXECUT) $(OBJECTS)
    @./$(EXECUT)

This code is called by running make exec in the same directory as the Makefile. The components build and $(OBJECTS) on the same line of exec must run or compile before the body can be executed. The body of exec compiles the executable and then runs it. On the last line, @ is used to issue a command on the shell.

build:
    @mkdir -p build
    @mkdir -p bin

The first component of exec, build, simply makes two directories, build and bin. The second component, $(OBJECTS), is syntactic sugar to reference the variable, OBJECTS. The make utility knows to take these files and compile them when it is a component to an action. In this case, it is a component to exec. If OBJECTS were not redefined to be in the new folder, bin, the make utility would compile the .cpp files into the same location as the source files, doubling the number of files in the src folder. Since the OBJECTS were given a new home in bin, the following is needed:

bin/%.o: src/%.cpp
    $(CXX) -c $(CXXFLAGS) -o $@ $<

This tells the make utility to compile all the .cpp files into the bin folder. There is some more syntactic sugar here: $@ refers to the action name, bin/%.o and $< refers to component, src/%.cpp.

clean:
    rm -rf build bin $(OBJECTS)

The clean action removes all the output files created by the Makefile. This gives you a clean slate, with only the source code. You can run this action with make clean.

The rest of my Makefile defines actions for building static and dynamic libraries. I put the exec action first, which makes it the default action. In other words, I can just run make to run make exec. It is quite exhilarating using Makefile for compilation rather than an IDE.

The Makefile in its Entirety

CXX = g++
CXXFLAGS=-g -O2 -Wall -Wconversion

SOURCES=$(wildcard src/**/*.cpp src/*.cpp)
OBJECTS=$(patsubst src/%.cpp,bin/%.o,$(SOURCES))

EXECUT=build/test.exe
MAIN=src/main.cpp

LIB_SOURCES=$(filter-out $(MAIN), $(SOURCES))
LIB_OBJ=$(patsubst src/%.cpp,bin/%.o,$(LIB_SOURCES))
LIB_TARGET=build/libYOURLIBRARY.a
SO_TARGET=$(patsubst %.a,%.so,$(LIB_TARGET))

exec: build $(OBJECTS)
    $(CXX) $(CXXFLAGS) -o $(EXECUT) $(OBJECTS)
    @./$(EXECUT)

all: build $(OBJECTS) lib dyn exec

libexec: build $(OBJECTS) lib
    $(CXX) $(CXXFLAGS) -o $(EXECUT) $(MAIN) $(LIB_TARGET)

test:
    gdb $(EXECUT)

bin/%.o: src/%.cpp
    $(CXX) -c $(CXXFLAGS) -o $@ $<

run:
    @./$(EXECUT)

lib: build $(LIB_OBJ)
    ar rcs $(LIB_TARGET) $(LIB_OBJ)
    ranlib $(LIB_TARGET)

dyn: build $(LIB_OBJ)
    $(CXX) -shared -o $(SO_TARGET) $(LIB_OBJ)

build:
    @mkdir -p build
    @mkdir -p bin

clean:
    rm -rf build bin $(OBJECTS)

thomasjpfan.com

Six Years as a scikit-learn maintainer - Feature Retrospective

1. Everything Trees 🌲🌲🌲

2. DataFrame interoperability 🖼️

3. Preprocessing 🕰️

4. Visualizations 📊

5. Experimental GPU support 🏎️

torch.export For Serializing Models and Faster Loading

Using torch.export

Loading the model

Conclusion

Keep Warm with Portable torch.compile Caches

Cold vs Warm Compile Runtimes

Portable Cache

Conclusion

PyTorch Graphs Three Ways: Data-Dependent Control Flow

Simple Function¶

TorchScript¶

Torch FX¶

torch.compile¶

Conclusion¶

Python Extensions in Rust with Jupyter Notebooks

Simple Rust Functions¶

NumPy in Rust¶

Polars Expression Plugin¶

Conclusion¶

Quick NumPy UFuncs with Cython 3.0

Fused Multiply-Add and UFuncs¶

Supporting Floats and Doubles¶

Conclusion¶

Accessing Data from Python's DataFrame Interchange Protocol

Polars DataFrame and the Exchange Protocol¶

Validity buffer¶

Computing the nan mean¶

Why?¶

Survival Regression Analysis on Customer Churn

Overview of the dataset¶

Loading data and fitting the model¶

Deep dive into monthly rates¶

Whats next?¶

Nuclei Image Segmentation Tutorial

Exploring the Data¶

Defining the Module¶

Freezer¶

Learning Rate Scheduler¶

Checkpoint¶

Custom Loss Module¶

Training Skorch NeutralNet¶

Evaluating our model¶

Whats next?¶

Rodents Of NYC

Rodent Inspections/Baiting¶

Is it working?¶

Conclusion¶

Hassle Free UNets

Introduction

Split Block

Center Block

Merge Block

Final Block

Conclusion

Bayesian Coin Flips

Likelihood¶

Prior¶

P(D)¶

Putting it together¶

Example¶

Different Priors¶

Closing¶

Statistical Power of Coin Flips

PCA Before Regression

High n-Dimensional Spheres Are Only Skin Deep

Hold on Tight!

Side Note

Maximum of free damped oscillators

Summary

Code for Plots

My Current Cpp Makefile

Makefile Logic

Makefile Usage

Using `torch.export`