TJ Palanca

3 ways to connect to a homelab

Thu, 30 Jan 2025 11:12:36 +0800

I bought an M1 Mac Mini (configured with additional RAM), and figured that it would be a great homelab for occasional use and tinkering. It would also open me up to enabling a primarily local instead of cloud-based smart home using Home Assistant.

My homelab has been great at allowing me to remove subscriptions for seldom-used cloud services. (Image by ChatGPT 4o)

I also wanted it to be the main place where compute happens for hobby projects, whether I am using my iPad, Chromebook, or any other device. ¹

One of the key decisions was how to enable secure connections from the outside when I need to do compute from my iPad while outside my home. I added a way to access the desktop remotely and development tools accessible via a browser, such as VS Code and RStudio.

Requirements for connection methods

I needed the solution to fulfil the following requirements:

allows access from anywhere in the world
adds a layer of authentication via consumer-grade SSO such as Google or GitHub on top of the web applications
trusted and reliable; the tool has a track record of delivering secure access over a decently long amount of time.

as well as are some negative requirements:

does not expose any ports to the open internet: there must always be a trusted proxy service that not just I maintain
no open-source or hand rolled authentication mechanisms that could degrade security if I don’t keep up with vulnerabilities
does not require a static IP; my internet provider charges extra

In my search, I’ve found a few solutions that fit the bill.

VS Code Tunnels

VS Code has a great feature where you could connect to an instance of VS Code running on the homelab via vscode.dev. The experience is virtually identical to a local instance of VS Code — extensions, settings, and other customisations all work quite well.

Although the established SSH tunnel between your remote machine and homelab is end-to-end encrypted, your traffic is routed through an Azure service². You’ll need to trust that Microsoft has put in place security measures to prevent that from being misused³.

The main limitation of this approach is that you are confined to using VS Code — although that may not really be a dealbreaker as everything you need to develop is already inside the tool.

Tailscale

Tailscale is a “mesh VPN” that allows you to connect all kinds of devices to each other by simply using the same shared credentials. It works by having a “coordination server” to allow your devices to discover each other and connect directly to each other. Your actual data flow is handled peer-to-peer, and Tailscale itself isn’t privy to this traffic.

One limitation of Tailscale is that it cannot usually work with another VPN, and this is true on my iPad as there is a system limitation of 1 only VPN connected at a time. As far as I can tell, you can’t have both IP anonymisation and access to local homelab networks.

Cloudflare Tunnels

Cloudflare is a lot of things, but one product that they offer is Cloudflare Tunnels. You run the tunnel on your homelab, and then it connects to the cloudflare network without creating a publicly accessible IP address. You can then access these services using a specially created domain, or if you already use Cloudflare as a DNS, domains or subdomains that you won. In addition to that, you can layer on Cloudflare Zero Trust to add authentication on top of your applications running behind the tunnel.

When you create a Cloudflare Tunnel, you connect your host to CloudFlare’s edge network. This means that you’ll inherently be trusting that they will protect access to your network and not use it for any nefarious purposes themselves.

Which to pick?

You’ll have to consider your needs, but I think I’ve boiled down the distinction to the following.

If you require both IP anonymisation and access to HomeLab at the same time, or want to expose some homelab services to the internet with or without authentication, then Cloudflare Tunnels is a good choice.
If you aren’t comfortable trusting a third-party service with your traffic, then Tailscale’s mesh VPN will keep your traffic between your devices only.
If you are only interested in coding via VS Code from a Chromebook or iPad, then VS Code Tunnels is the easiest way to go.

I ended up going with Cloudflare Tunnel for the following reasons:

I was already trusting Cloudflare as my DNS provider and proxy, so I felt that using Cloudflare Tunnels had plenty of good ecosystem effects with minimal marginal risk.
Most of my computing time is spent on iOS devices, so I’d need a solution that offered IP anonymisation and a connection to my homelab, so I could not use Tailscale.
Based on an unscientific test of connecting from different places, it offered the best performance.

I really only use an iPad in conjunction with my Mac Mini server, when I used to also have a Macbook Air and a Chromebook in the past. I find that the Mac Mini on a desktop enforces discipline when doing “real work”. ↩︎
VS Code Tunnels - How are tunnels secured ↩︎
VS Code Dev Tunnels Security ↩︎

dbt Singapore meetup: Using dbt for accounting and investment reporting

Fri, 13 Sep 2024 19:57:47 +0800

I’ve always believed that there is still a lot of value in tackling bread and butter reporting using the modern stack of data tools. One of the most valuable and important reporting workflows in any business isn’t in the marketing, product, or engineering teams - it’s in the finance team.

Reporting on the financial performance and health of companies and investments is usually done by large teams exerting a large amount of elbow grease. By leveraging dbt in our firm’s accounting and investment reporting, we were able to deliver both accounting and investment reporting on a daily basis and in an automated fashion.

The presentation was intended to give an overview of what dbt users who are tackling similar challenges could face and what worked best for us in overcoming those challenges.

We also wrote about this talk on the Titanium Birch blog.

Exploring the Polars Dataframe Library

Fri, 06 Sep 2024 13:14:22 +0800

Polars is a relatively new data frame library that I don’t yet have experience, but I do want to use it for storing data related to this library, so I’m recording my experiments here.

import polars as pl

Quick Start

The syntax looks very much like {dplyr}, following simple verbs that can be chained together via method chaining. You can use the backslash for line continuation but I prefer using parentheses.

df = pl.read_csv("https://j.mp/iriscsv")
(
    df.filter(pl.col("sepal_length") > 5)
    .groupby("species")
    .agg(pl.col("sepal_length").sum())
)

Polars Expressions

An expression is essentially a definition of a data transformation on a polars data frame. By being able to optimize the query on that expression and parallelizing, Polars is able to achieve performance.

# Select the column foo, sort it ascending, and get first 2 elements
pl.col("foo").sort().head(2)

# You can pipe expressions together
df.select(
    [pl.col("foo").sort().head(2), pl.col("bar").filter(pl.col("foo") == 1).sum()]
)

import polars as pl
import numpy as np

np.random.seed(12)

df = pl.DataFrame(
    {
        "nrs": [1, 2, 3, None, 5],
        "names": ["foo", "ham", "spam", "egg", None],
        "random": list(np.random.rand(5)),
        "groups": ["A", "A", "B", "C", "B"],
    }
)
df

# Take the names vector, count how many, unique, and name a column
df.select(
    [
        pl.col("names").n_unique().alias("unique_names_1"),
        pl.col("names").unique().count().alias("unique_names_2"),
    ]
)

# Summing, minimum, maximum, std dev, and variance
df.select(
    [
        pl.sum("random").alias("sum"),
        pl.min("random").alias("min"),
        pl.max("random").alias("max"),
        pl.col("random").max().alias("other_max"),
        pl.std("random").alias("std dev"),
        pl.var("random").alias("variance"),
    ]
)

# Get all names where the name contains am at the end
df.select(
    [
        pl.col("names").filter(pl.col("names").str.contains(r"am$")).count(),
    ]
)

# When the random field is >0.5, then use 0 else use the random, multiple by nrs
df.select(
    [
        pl.when(pl.col("random") > 0.5).then(0).otherwise(pl.col("random"))
        * pl.sum("nrs"),
    ]
)

# Window functions in SQL, get the sum of random over groups, and get the list of random over names
df.select(
    [
        pl.col("*"),  # select all
        pl.col("random").sum().over("groups").alias("sum[random]/groups"),
        pl.col("random").list().over("names").alias("random/name"),
    ]
)

Expression Contexts

Expression contexts define how the expression is evaluated.

Select Context

Does operations over columns
Must produce Series of same length or length 1 (broadcasted)
You can add pl.all() to make it more like {dplyr}’s mutate(), otherwise

it behaves like transmute().

df.select(
    [
        pl.all(),
        pl.sum("nrs").alias("nrs2"),
        pl.col("names").sort().alias("names2"),
        pl.col("names").first().alias("first name"),
        (pl.mean("nrs") * 10).alias("10xnrs"),
    ]
)

Add Columns

df.with_columns(
    [
        pl.sum("nrs").alias("nrs_sum"),
        pl.col("random").count().alias("count"),
    ]
)

Behaves like dplyr::mutate().

GroupBy Context

df.groupby("groups").agg(
    [
        pl.sum("nrs"),  # sum nrs by groups
        pl.col("random").count().alias("count"),  # count group members
        # sum random where name != null
        pl.col("random").filter(pl.col("names").is_not_null()).sum().suffix("_sum"),
        pl.col("names").reverse().alias(("reversed names")),
    ]
)

Expressions work on groups
You may return any length
You use agg() after a a groupby()

Thoughts

I do think this API is much more user-friendly, tractable, and less error-prone than the pandas API. I would like, as much as possible, for my workflow sot use polars whenever I start something new, and avoid having to use pandas unless there is some package compatibility quirk that needs to be resolved.

#8: Writing is thinking, weak passport problems, and economic growth

Tue, 20 Aug 2024 19:52:00 +0800

No amount of organising beats writing when it comes to synthesising new information

I consume a lot of media. On a regular day, my feed reader pushes the latest gadget reviews from The Verge, updates on Southeast Asia from Dari Mulut Ke Mulut, or my Kindle asks me to finish reading The Montessori Toddler, along with all the other books in my ever-growing reading list.

I tell myself that this isn’t mindless consumption. I imagine myself quoting a book I read 2 years ago, or relating some tidbit in a newsletter to a problem I’m facing at work, or applying some sage parenting advice while playing with my daughter.

This wasn’t really happening, though. There was something missing in the process — I wasn’t taking the time to coalesce my thoughts from amorphous human brain file format to concrete written prose. Worse, the more time I spent without writing anything, the more I tried to keep things in my head, taking up cycles and leaving me scatterbrained.

I set myself a goal of 300 words per day in Ulysses, a beautiful writing app that tracks this for me. Did it work? I’m not sure yet, but I definitely feel better!

Book: Everybody Writes by Ann Handley

I finished reading Everybody Writes by Ann Handley, and assembled some book notes. It’s a great book on writing from a marketing perspective. Even if you don’t write or market for a living, I find that the advice applies well to blogs, newsletters, and speeches.

Some of my favourite takeaways:

Words are your ambassador, and can make or break you.
If you want to get better at writing, just write! Edit and refine later.
High-quality content should have utility, inspiration, and empathy.

I’d highly encourage anyone who does any sort of business writing to give it a look.

Renewing my US Visa

I renewed my US visa recently as a Philippine passport holder in Singapore. Here’s my detailed write-up of the experience. For those who don’t know, the US visa is also useful for obtaining an electronic travel authorisation to Canada.

The process was not well documented, but once you know what to do it’s a very straightforward process.

Reading

Here’s a brief rundown of the most notable articles from my recent reading history:

The importance of looking outside of “the data” — Randy Au breaks down Nike’s recent crash due to an over-reliance on the data they currently collect, not realising that it’s the data that they don’t collect (churned and non-customers) that they needed to focus on.

Primary Residence Value As A Percentage Of Net Worth Guide — As a percentage of your long-term net worth, Financial Samurai says your primary residence should be no more than 30%.

Global economic growth has always fallen short of expectations — I’m currently reading The Imagination Machine, and one of the most striking visuals at the start of the book illustrates this point:

The authors make the case that this is because growth opportunities now require a lot more “imagination” than before, where process development and technological innovation were enough.

I actually produced a newsletter post beyond the original “I am restarting my newsletter” post. 🎉 I’d appreciate your feedback, just hit reply.

#7: Digital gardening didn't work for me, getting serious about writing, and my new role

Wed, 24 Jul 2024 22:25:36 +0800

Welcome back to what is essentially an annual newsletter 🥲. In this issue, find out why digital gardening didn’t work for me, how I want to own my content, and my new gig combining accounting and data engineering.

Obsidian’s graph view was very fun to look at, but I quickly realised that it wasn’t going to really help with my writing.

A reflection on digital gardening

In my previous newsletter, I was singing the praises of digital gardens, and how they were supposed to lower the barrier for entry for writing.

Well, I’m pleased to report that it was successful in lowering the barrier to entry, I’ve added many notes to my garden, from new Julia packages to the economy of Somalia. However, I still failed to produce any substantial writing 😕.

The whole “idea emergence” thing, where if I link notes together and then somehow use the network to generate new ideas, didn’t really work for me.

Whenever I tried to synthesise my thoughts, the result looked more like a review of related literature than anything original or “emergent”. I felt compelled to stick to the confines of the linked material, rather than consciously think through the topic and form my thoughts.

Another reason I decided to abandon full-blown digital gardening is that I realised that the end product wasn’t really enjoyable to read. My garden was overgrown, and it became too much work to “tend” to it. You could start on a very polished note and then end up with the bits and ends which are of no value to anyone but me.

Publish on your site, syndicate elsewhere (POSSE)

Yet another iteration of this website, hopefully it’ll stick! 🤞

If you’re getting this newsletter in your inbox and the sender is unfamiliar, it’s because I’ve moved over to a service called Micro.Blog. It started as a kickstarter in 2017 and has grown since then. The same privacy policy applies — your email is only used to send you this newsletter and nothing else.

If there’s one thing that Twitter’s chaotic few years has pushed me to do, is to make sure all the content I create isn’t confined to one place. Micro.blog’s founder Manton Reece has written a book that lays out his vision for people to blog on their domain names and syndicate that out with feeds.

I resonate with that vision.

I post all my writing, including microposts (or tweets, or toots, or whatever you call them), on tjpalanca.com first (publish your own site). Afterward, a combination of micro.blog’s syndication tools, Zapier, and Buffer, syndicate the content out to Facebook, Twitter, Mastodon, Medium, and any other network, protocol, or site now and in the future.

If any of these networks decides it no longer wants to exist, my content is still safely, and originally, on my website.

Working at Titanium Birch: Standardising disparate data feeds for investment reporting and accounting

I started a new role as CFO / Software Engineer at Titanium Birch, a single family office started by the co-founder of ExpressVPN.

I’ve written a blog post on the company blog about what we have been working on recently — turning data feeds from disparate banks and brokers into useful and actionable reports and financial statements.

It’s quite interesting to apply a data engineering lens to a very well established problem such as bookkeeping — the underlying principles remain the same: standardise your source feeds into objects, and then obey that interface when creating business objects — this allows standardisation code and analysis code to evolve independently.

That’s the first installment — hoping I can keep up my newsletter habit, at least for the next trip around the sun. See you on the next one!

Review: dBrand Skin for iPhone and iPad

Sun, 07 Jul 2024 22:13:11 +0800

dbrand Matte Black skin and the Moft snap case

Since we had a baby, we added AppleCare+ ¹to all our Apple devices, as the risk is now much higher that our devices meet an untimely demise.

A side effect of the warranty is that it made me brave enough to go caseless. However, I quickly noticed after a few days that a bare iPhone is too slippery and cold to the touch. I needed something that didn’t add any additional heft while making the texture a lot more comfortable.

That’s where the dbrand skin comes in.

TABLE OF CONTENTS

Selecting a pattern and ordering

The dbrand checkout experience makes it easy to pick and compare styles, but the reflectivity of the skin isn’t as obvious.

I ordered from Singapore, with free delivery on orders above USD 60. I ordered 2 skins for my iPhone 15 Plus and 1 skin for iPad, assuming that the iPhone skins would wear earlier.

For those who wanted the leather skin like I did, the iPhone leather case didn’t cover the sides, and the iPad leather case was noted to interfere with the Magic Keyboard connection.

The website was pretty clear and easy to navigate. While they do a good job of representing the skin in the pictures, one thing that is difficult to judge is how it reflects the light. In a previous order, I had gotten the “Swarm” pattern as I loved the hexagons, but it ended up being so bedazzled and shiny that it didn’t look premium at all. I swapped it for a Matte Black version instead.

The package arrived within a week with no issues; it was well protected and all the skins were not folded or bent.

Applying the skin

DBrand has some well-made how-to videos on applying the skin. They’re pretty clear and all you need is some concentration and a hair dryer. Allot at least an hour to apply the skin properly, with nothing rushing you, as it is very possible to botch the application.²

What helped was to make sure that everything is completely, perfectly aligned as soon as the skin touches the device. Removing and re-applying the skin is no issue as long as it hasn’t been a while.

First impressions

iPhone 15 Plus with Matte Black dbrand skin

It felt a little silly to make this purchase, when I could have bought a basic case for much less. The feeling disappeared when I put the skin on and walked around with the phone for a full day. The phone was so much easier to handle and felt warm to the touch. Using the phone on the subway or walking around was much more enjoyable (and safe).

iPad Pro M4 in Matte Black and Magic Keyboard in Matrix

The iPad skin was more basic — it was just a simple skin over the back (not the sides). It largely served the purpose of adding grippiness while protecting the back from scratches. I also chose the one without the logo cutout as it was usually in landscape, but the logo was not.

After a week of use

Inevitably, my 2-year-old picked up and dropped the phone, and there is now a slight scratch on the skin and the phone itself. The skin definitely doesn’t protect as well as a case, but as far as micro-scratches from keys or anything your device might share a pocket with, it’s been great.

Slight bump after falling on a hard floor from a 2-meter height

I did notice a small tear in the corner where the skin wasn’t completely applied. I’m pretty sure this will become an issue in the future. By then, I’ll apply my second skin, but this is something to note — any slight imperfection in the application will probably result in the skin peeling off in some way.

Conclusion

There are legitimate reasons to get this setup:

you find any iPhone case bulky, especially with Plus or Pro models,
you already have AppleCare+, so aren’t worried about catastrophic falls, and
you don’t mind having minor bumps and scratches that aren’t covered.

If this sounds like you, I’d recommend you give a dbrand skin a try. If you’re just looking for the cheapest way to protect your case or don’t have AppleCare+, even the most basic case will be more protection at a fraction of the cost and hassle.

AppleCare+ is an additional purchase that extends the warranty to 2 years and includes accidental damage. ↩︎
I’ve heard that dbrand may sometimes send you a replacement, but that isn’t an established policy that I’ve read anywhere. ↩︎

Moving Shares from Shareworks Solium to Interactive Brokers

Tue, 04 Jul 2023 00:00:00 +0800

Stock and Crypto Market Values. Photo by Maxim Hopman on Unsplash

I was lucky enough to be part of two companies that went public via an IPO: Uber and Grab, both ridesharing companies.

A common platform for managing and distributing these employee shares is Shareworks/Solium (now part of Morgan Stanley at work). While it’s great for seeing your stock compensation as you earn it, it’s not really a platform that you could use once those shares are granted to you and tradeable in the public market.

I use Interactive Brokers¹ for managing my non-fund equity investments. It’s a great platform with very low fees. I wanted to transfer my employee shares to IBKR so I can manage them and liquidate them with low fees.²

This TIL is just my personal experience, your mileage may vary. Please try with a small test transfer before sending a larger amount of shares.

First, initiate the transfer in IBKR by going to Transfer & Pay > Transfer Positions.

Afterward, select United States as your region (as both Solium/Shareworks and IBKR are located in the US).

Select Free of Payment Transfer of US Securities as your preferred method. This is the best method as it does not involve any fees (hence the name).

Enter Solium Capital as your broker, your Soliium Employee Number as the account number, and other relevant information. You also need to put in which and how many shares you would like to transfer.

Sign your name in transfer verification.

Click on View Form to get the information that you need to give Solium. Note the DTC Number 0534 which is critical for making sure the shares move to the right broker.

Now, log into solium to initiate the transfer from their end. In Shareworks/Solium:

View your portfolio, then select Sell or Transfer. Input the Brokerage information that you have received before. The critical parts are the DTC Number and the Interactive Brokers Account Number.
Go through confirmation steps to request Solium to transfer shares to IBKR.

It takes 3-5 business days for the transfer to proceed. Since FOP notifications expire after 5 business days, make sure to take these steps in one sitting rather than doing them in parts.

Writing/Learning/Moving+Shares+from+Shareworks+Solium+to+Interactive+Brokers

This is a referral link! If you don’t want to use it, just go to the regular website. ↩︎
Incidentally, it’s also a great place to do currency conversion. The rates are near spot with very minimal fixed commissions. You shouldn’t use the broker as a currency exchange (they will take action against your account for that), but if you are looking to change your currency exposure over a long period of time then it’s great. ↩︎

Digital gardens, Parenting, and Coffee

Thu, 25 May 2023 00:00:00 +0800

Newborn Sleep Patterns

Welcome to my digital garden

I’ve long lamented the inactivity of my website - but not for lack of trying. In the old website, I had 10 drafts sitting in the inbox queue, but could never bring myself to hit publish because they weren’t exactly as polished as the other posts from almost a decade ago.

Also, the whole publishing process as quite laborious - I didn’t have an easy way to just go from my internal Notion ideas to the Quarto website that I set up last year - I had to be at my laptop, completely focused, to publish anything, which is a rare occurrence now that On Newborn Parenting|we are of child.

I think I’ve found a good solution, by creating a digital garden:

This lowers the barrier to entry and publishing of ideas, I can keep them knocking around in the digital garden and make sense of all the information before I publish them into the writing section of my website, and
I’ve integrated all my thinking into one tool (my Building a Second Brain|second brain for both thinking and publishing in Obsidian and its Obsidian Publish service. If you go to my site now, you’ll see that it’s changed to that form. I’ll write more at length about this setup in the future.

So, that’s inspired a fresh new attempt to revive my regular newsletter habit.

Privacy-friendly Google Analytics?

I wrote about Nerfing Google Analytics to be more privacy-friendly. As I revamped the website, I thought I’d turn to free tools, as opposed to my Plausible Analytics self-hosted setup.

But it’s no longer as simple as slapping Google Analytics on the website and calling it a day due to changes in privacy regulation. This post writes down how I’ve managed to lock down all the unnecessary ad tracking and data sharing agreements to keep my Privacy Policy|privacy policy as simple as “I collect anonymized analytics on pageviews so I can make the site better, and nothing else.”

Newborn parenting

I wrote a short piece On Newborn Parenting and the various adjustments we’ve had to make as a result of now having a beautiful daughter in our home.

Reading highlights

10 Thoughts From the Fourth Trimester by Tim Urban · Highlights · Original

This was a hilarious point of view of Parenting from Tim Urban of Wait But Why fame. My favorite points were:

The labor experience for couples and how they are wonderfully mundane compared to what we see in the movies.
How newborns are basically fetuses outside the womb and they aren’t cute at all until they’re actually babies at 3 months.
How there is no regulation at all around parenting despite being completely in control of a child’s life

A Brief History & Ethos of the Digital Garden by Maggie Appleton · Highlights · Original

As I got deeper and deeper into building a second brain, I’ve turned my focus to my website and how it’s gotten so stale. A digital garden is all about lowering the barriers to publishing my moving away from “articles” and “essays” where everything must be perfect, to publishing field notes or other unfinished work for continual refinement in public. Maggie Appleton describes this style of publishing excellently and with amazing pictures.

As Filipinos Crave More for Local Java, Philippine Coffee Farms Brew a Plan by Isabel Martinez · Highlights · Original

Philippine Agriculture has always been about rice - not even increasing yield or reducing distribution costs - just about import tariffs and trade controls. It’s great to see that despite great odds the coffee industry in the Philippines has started to see green shoots, thanks to increasing domestic (read: robust and less recession-sensitive) demand for artisanal, high value coffee, and an increase in coffee productvity.

newsletter/2023-05-25

On Newborn Parenting

Tue, 23 May 2023 00:00:00 +0800

Photo by Kelly Sikkema (Unsplash)

We recently added a beautiful baby girl to our family, and it’s been a wild ride. Since tech workers are a fairly new phenomenon, I suspect that a majority of tech workers are now currently in the same life stage. As this happens, I suspect that tech companies will change how work is done, now that more and more of the workforce leaves their 20s and moves into their 30s.

Life adjustments

Simplify your life. The financial cost of having a child is well-understood and requires no elaboration, but having a child does have a huge time cost (well spent!). In that I’ve found that eliminating some distractions, like not following consumer tech so much, eliminating excess subscriptions, has allowed me to focus on work, family life, and one other hobby without sacrificing quality.

Maintaining friendships

Do the work to maintain friendships. It’s very common for new parents to slink away into the world and cease to develop friendships. I didn’t want to do that, so I deliberately put in the work to schedule and follow through on meeting and spending time with friends (whether or not they had kids of their own!). I also think that’s why it’s important to still have hobbies, so that you can talk about something that isn’t your kids.

Reading about parenting

It’s more important than ever to load up on science-backed ways to effectively parent your child. There’s an abundance of (sometimes unsolicited) parenting advice out there, so make sure you are only selecting a few trusted sources: people, books, websites. Otherwise, everything will conflict and be very confusing.

Tips and Tricks

Invest in a good set of wireless earbuds (I prefer Airpods). There is an non-trivial opportunity to listen to your favorite podcasts while carrying the baby.
Learn to not panic when the baby is crying - it’s just the baby’s way of communicating that something is not right. It’s very easy to get overwhelmed.

content/thoughts/2023-02-25-newborn-parenting/index.html
thoughts/on-newborn-parenting
content/thoughts/2023-02-25-newborn-parenting

Nerfing Google Analytics to be more privacy-friendly

Sat, 20 May 2023 01:00:00 +0800

Photo by Lianhao Qu on Unsplash

I am not a lawyer and this is not legal advice. I’m only sharing how I navigated privacy legislation and attempted compliance on a personal website.

I recently revamped my personal website to connect more closely to my “second brain” which is in Obsidian. It’s now published via Obsidian Publish. I’m pretty happy with it so far!

As I polished it up, I realized that I wanted to understand how readers read, interacted, and understood, especially as they go through the digital garden part of the site, hence my exploration of website tracking tools.

Privacy regulations have changed

As a data professional, I already knew that the privacy landscape has changed dramatically, including GDPR in the European Union, CCPR in the US State of California, and a whole host of privacy regulations across ASEAN. Even though my site as small, I wanted to make it as transparent and privacy-friendly.

Exploring website tracking solutions

I used to use (and may eventually move over to¹) a self-hosted version of Plausible Analytics for privacy-friendly Google Analytics. However, it is still tracking that I need to disclose on my privacy policy, and I’m not sure about whether self-hosting analytics solutions is really that responsible compared to relying on services who have teams looking after the security of their data.

So, I’ve decided to take a look at the oft-maligned Google Analytics to see if I can use the new Google Analytics 4 in a more privacy-friendly way, while providing me with free, simple, easy-to-use, and reliable website analytics.

Nerfing Google Analytics

I’ve done quite a bit of research into the various configuration options, and I’ve settled on the following adjustments (read: nerfs), to make Google Analytics just do website analytics without all the other stuff.

By default, here are some assurances:

IP addresses are not logged or stored as long as you use the new GA4 tags.

Here are some of the adjustments that need to be made to the analytics itself:

Configured extra features in the initial config call, such as:
- disabling Google Signals which allows Google to infer interests and other demographics for ad targeting,
- disabling ad personalization signals which allows the events from the tag to be used to personalize ads served to that user.
- enabling restricted data processing which disallows using it for various re-marketing purposes
- disabling URL passthrough which is a way to track across sessions without using cookies.

// Load Google Analytics with optional features disabled
gtag('config', ga_measurement_id, {
	'allow_google_signals': false,
	'allow_ad_personalization_signals': false,
	'restricted_data_processing': true,
	'url_passthrough': false
});

Enabled consent mode for google analytics, with ad_storage always set to false since I had no use for the analytics, and analytics_storage being opt-in for European Union + UK users (due to GDPR) and opt-out for everyone else.

// Consent Setup
// Default: Opt-Out of Analytics
gtag('consent', 'default', {  
	'ad_storage': 'denied',  
	'analytics_storage': 'granted'
});
// European Union and the United Kingdom: Opt-In to Analytics
gtag('consent', 'default', {
	'ad_storage': 'denied',  
	'analytics_storage': 'denied',
	'region': ['BE', 'BG', 'CZ', 'DK', 'DE', 'EE', 'IE', 'GR', 'GB', 'ES', 'FR', 'HR', 'IT', 'CY', 'LV', 'LT', 'LU', 'HU', 'MT', 'NL', 'AT', 'PL', 'PT', 'RO', 'SI', 'SK', 'FI', 'SE']
});

Added a tracking consent banner that asks users whether to accept or reject tracking, that then modifies the default consent settings based on whether users accepted or rejected the tracking by setting:
- the following for when consent is granted (with the consent lasting for 365 days to make it finite), and

gtag('consent', 'update', {'analytics_storage': 'granted'});
Cookies.set(consent_cookie, 'accept', { expires: 365 });

the following for when consent is not granted. Adding the window variable is important because even when denied, GA4 still sends “cookieless pings” for modelling which for me is a gray area but still a form of tracking I don’t need.

window[`ga-disable-${ga_measurement_id}`] = true;
gtag('consent', 'update', {'analytics_storage': 'denied'});

Some options that need to be configured on the Google Analytics Property:

Disabled granular location data which collects city-level data (not the actual geolocation) and some user agent and device model information. These are not directly sensitive but may be used for fingerprinting, so I’ve decided to do away with it since I have no real use for this information.
Turned off all data sharing with Google. Any data sharing needs to be disclosed to users and I had no real use for sharing this information with Google since I did not use digital ads nor serve them on my site. I still linked it to Search Console as that only shared limited data for SEO optimization which I need.
Set a retention period. I use 14 months because I want to see seasonality in the data, and I disclosed that in my privacy policy.

I added a section to my privacy policy

I’ve written down, in as clear language as possible, how I do this in my Privacy Policy. In it, I attempt to explain:

what tools I used to process data,
what data those tools collect,
what uses I have for that data,
how long I retain that data, and
what uses I prohibit for the data.

This is the section I have for Google Analytics:

Google Analytics Privacy Policy

I use Google Analytics to understand how users use the website, navigate through the content, and subscribe to the newsletter. This allows me to improve the website’s content, understand pain points in navigating the site, and make a more compelling newsletter.

I do not collect or link any personally identifiable information, including the email collected for the newsletter to this traffic. My use case requires only aggregated and anonymized data and not any specific user’s activity.

All of the following have been done to ensure that we protect privacy as much as possible:

Google Signals measurement is disabled

Google Ad Personalization is disabled

Restricted Data Processing is enabled

URL Passthrough is disabled

Data Sharing is disabled except for sharing anonymized statistics with Google Search Console, to improve the discoverability of my website.

Consent mode is activated in the website. For users in the European Union, consent is opt-in, while consent is opt-out for the rest of the world.

Data is retained for a period of 14 months, to ensure that I can understand the trends in seasonality and be able to provide good content at the right time.

I believe it’s clear and transparent enough that anyone coming into the website should be able to understand collection and use. I think it’s definitely better than those auto-generated ones full of legalese.

Writing/Learning/Nerfing+Google+Analytics+to+be+more+privacy-friendly

If I ever move back to Plausible, I’m just going to pay for their hosted version. ↩︎

How the 2008 Financial Crisis Still Affects Us

Tue, 08 Nov 2022 11:00:00 +0800

I had recently stumbled upon this video that stitches together the great recession of 2008 to the crisis we are experiencing now. It’s a great video, and I recommend watching it in its entirety.

One thing I found super fun to build was a map of the root causes and effects stemming from 20 years ago all the way to today, and I wanted to share it here in case it helps anyone make sense of the mess.

flowchart TD
    classDef root fill:#f66;
    classDef effect fill:lightblue;
    classDef speculation fill:lightgreen,stroke:dash;
    PEP --> HPK
	GSA[[1999, Repeal of Glass-Steagall Act]]:::root --> BRI[Banks could start taking risky\ninvestments on their own balance sheet] 
    GSA --> IBI
	S11[[2001, Sep 11 Attacks and\nEnron Scandals]]:::root --> LCC[Low consumer confidence] --> FLR[Fed lowering rates\nto 1% from 7%] --> PEP
	PEP[Period of economic prosperity] --> HON[Everyone who could\nown a home already did]
	HPK[Home prices kept going up] --> SPM
	HON --> SPM[Mortgage lenders started\nissuing adjustable rate\nsub-prime mortgages]
	SPM --> ORM[Mortgages needed to offload\nrisky mortgages]
    FLR --> FLY[Funds looking for\nhigh yield investments]
    FLY --> CDO[CDOs, amalgamation of\n debt divided into tranches, were created] 
    ORM --> CDO
    CRP[Credit rating profits\nhad perverse incentives] --> CRA
    IBI[Investment banks\nhad perverse incentives,\n no penalty for creating risk] --> CDO
    IBI --> FOR[Banks would\nforum shop for credit rating agencies] --> CRP
    CRA[Credit Rating Agencies rated as AAA] --> CDO
    FGE --> AVO[Avoided calling CDSes insurance,\nas they would not be regulated] --> CDS
    FGE[Geniuses were hired to create\nincreasingly complex products] 
    CDS[Mortgage credit default swap or CDS,\ninsurance for failed mortgages,\nwere created]
    CDO --> CDI[CDOs exploded in\nadoption] 
    CDI --> BCDS[Banks continued betting and hedging on\nmortgages and CDOs using CDS] 
    CDS --> BCDS 
    BRI --> BLA[Banks could take on even more risk] --> BCDS
    SEC[[2004, SEC relaxed liquidity requirements\nof 5 biggest banks]]:::root --> BLA
    BCDS --> BLI[Bank Liabilities\nexploded]
    ERE[[2004, economy recovered]]:::root --> FRR[Fed raises rates] 
    FRR --> SPU[Sub-prime mortgages\nstarted to default] 
    SPM --> SPU
    SPU --> SPL[Sub-prime lenders\nstarted to go bankrupt]
    BLI --> BBS[[2008, Bank balance sheets\nwere in trouble]]:::root
    SPL --> CDD[No one wanted to buy\nthe CDOs anymore] --> VM[Value of real estate investments\nplummeted] --> BBS
    BBS --> BSD[Bear Stearns goes bankrupt] --> JPM[JPM bought Bear Stearns at a 98% discount] --> BSF
    BBS --> LBD[Lehman Brothers goes bankrupt due to\n 30x leverage on real estate, hidden by\n accounting gimmicks] --> BSF
    BBS --> GSM[Goldman Sachs and Morgan Stanley bet against\nthe instruments they were still selling to save themselves] --> BSF
    BBS --> ML[Merill-Lynch merges with\nBank of America to save themselves] --> BSF
    BSF[Banking system was frozen,\n no one would lend anything] --> FL[Firms like GE can't get even short-term credit,\n wages were not paid] 
    BBS --> BOS[[2008, bailouts started,\nUSD2.5T]]:::root --> AIG[Federal Reserve buys AIG,\n because it was deemed too big to fail]
    FL --> BOS --> DTI[Distrust in institutions] --> BTC[[Bitcoin was created]]:::effect
    DTI --> OWS[[Occupy Wall Street started]]:::effect
    BOS --> LTB[Loans to large banks to\nconsolidate and buy up smaller banks]
    BSF --> ICE[Iceland government is in financial trouble\ndue to bank bailouts] --> EUR
    BSF --> GRC[Greece is in financial trouble] --> EUR
    BSF --> UK[UK had to bailout its banking system] --> EUR
    EUR[[Eurozone Debt Crisis\nGreece, Ireland, Portugal, Spain Italy]]:::effect
    BSF --> UN[[6 million unemployed in the US]]:::effect --> DFA
    DTI --> DFA[[Dodd-Frank Act]]:::effect --> DS[Did not seem to work] --> BCSO
    BCSO[Bespoke Collateralized Synthetic Obligations\non Corporate Debt] --> CS[Systemic corporate debt effect?\n Hopefully not]:::speculation
    UN --> REAL[[Real GDP per capita has never recovered since 2008\nA whole generation was poorer than their parents]]:::effect
    EUR --> REAL --> CBR[Central banks kept lowering interest rates]
    CBR --> INT[Cheap interest rates became the norm]
    REAL --> PROD[Productivity of labor has not increased since 2008] 
    COVID[[2020, COVID-19 Pandemic]]:::root --> MP[Further monetary stimulus\nvia asset purchases] 
    MP --> SM[Stock market has been going up] --> INF
    MP --> NFT[Investments in risky assets such as NFTs] --> INF
    COVID --> PRD[Supply chain disruptions] 
    MP --> INF[[Inflation]]:::effect
    PRD --> INF
    INF --> CBR2[Central banks raise interest rates]
    CBR2 --> DMD[Demand for everything reduced]
    DMD --> REC[[2022, Another Recession!]]:::speculation
    INT --> REC 
    PROD --> REC
    INT --> VRM[Lots of variable rate mortgages\nsnapping to market rates soon\nin Australia, UK, EU] --> REC
    CBR2 --> BNK
    INT --> BNK[Credit Suisse is in trouble\nwith their debt] --> REC

Here's an image version.

Here’s the original video embedded in case you want to watch the whole video:

content/thoughts/2022-11-08-financial-crisis/2022-11-08-financial-crisis.html
thoughts/financial-crisis-still-affects-us

Losing the Wings

Sun, 06 Nov 2022 15:00:00 +0800

Portents of economic malaise, from the bird to the elephant, and the ticking time bomb of the Philippines rural banking system.

Self-hosting Mastodon

Update: I’ve since moved to Micro.Blog because I didn’t really see myself constantly browsing mastodon feeds and using it in the same way I used Twitter. I’m very happy to have cut out a huge time sink in my life.

Behold, Mastodon!

I got deep into Mastodon today - mainly driven by the unsustainability of the platform, but also because the idea of a federated social network is quite interesting. The idea of owning my own piece of the network while still retaining connectivity appeals to me, who once tried to move off of Google’s ecosystem and go completely self-hosted.

My Mastodon profile is here, and you can follow me from any public or self-hosted server; Mastodon’s protocols will take care of the communication between our nodes.

The terraform code to deploy onto the main kubernetes environment is in the TJCloud Repo. It requires the Mastodon Docker image, which is available on Docker Hub, a Postgres database, and a redis instance. There’s an option to add ElasticSearch for full-text search capabilities but I didn’t bother.

The main gotchas were ensuring that you had valid redirects for /.well-known/host-meta and /.well-known/webfinger at the root of the domain if you weren’t serving Mastodon at the top level (I served mine out of mastodon.tjpalanca.com). I used nifty Cloudflare redirect rules (in beta) to make that happen.

The user experience is better than I expected, although there are still quite a few rough edges. Some accounts I could not follow presumably due to some server incompatibility, and there’s some sluggishness that is expected because of all the nodes that have to communicate. It is a usable experience though, so the crucial thing is just getting the right communities onto Mastodon so that it can be a viable alternative to Twitter.

Hard economic times

Ever since the pandemic shut down every aspect of production globally for a good few months, I’ve been wondering where the economic devastation would hit. Now it’s clear that those times are upon us, I’ve not worked in a [[recession]], and find myself wondering what I should do differently - do I prioritize job security, or building up savings, or do I just keep doing what I’m doing and hope for the best?

What I’ve been reading

November 2022 FOMC Press Conference - Inflation is high, and the Fed continues to raise rates - FOMC meetings are super exciting again. The idea of steering whole economies with just a single policy instrument is so fascinating and brings out the old econ nerd in me.
The Philippines’ rural banks are facing an analog-to-digital existential crisis - The Ken (Paywalled) - The Philippine rural banking system is in a period of change due to: (a) increased capital requirements, (b) e-wallets coming for their deposit base, and (c) digitization with staff that are content to stay in less urban areas of the country. Send me a message if you’d like a gift link to the full article.
How dbt fails - Benn Stancil - Benn narrates the evolution of dbt and its company and does a “pre-mortem” on the company:
- dbt was initially sold without any preconditions - it’s just SQL, but with the new semantic layer stuff introduced in 2022, it is no longer that simple, product development didn’t keep up with demand, and it made dbt opaque.
- dbt is still a very good product, because it makes thousands of people capable of doing things they otherwise couldn’t, but now it’s an incumbent and it’s going to have to continue to fight to stay relevant.
- honestly, this was a little depressing, but the fact that we can’t imagine dbt Labs crashing and burning is remarkable.

content/news/2022-11-06.html
newsletter/2022-11-06

Stack Upgrades

Mon, 17 Oct 2022 15:00:00 +0800

1Password can sign git commits for you

You can use the beta version of 1Password 8 to sign git commits! You can generate a private key with it and then protect that key with your Touch ID. Pretty nifty way to ensure your commits aren’t being impersonated.

Transitioning to Terraform Cloud

I’ve always been loathe to manage cloud infrastructure directly, especially for my personal stack. I’ve historically used Cloud66 to manage a Kubernetes cluster and related applications, but apart from the cost, I’ve necessarily reached some limitations when accessing more Kubernetes features, or with the domain specific YAML configurations that C66 provides.

I’ve since then moved to Infrastructure-as-Code (Terraform Cloud) for my personal stack here. It has Kubernetes on Linode, Cloudflare, Ingresses, OAuth2 Gateways with Keycloak, PGAdmin, a development stack, and my Plausible Analytics tracking - basically everything I need to be fully productive on the cloud, without ever needing to develop locally.

From the notes: FastAPI

I’ve never seriously taken a look at FastAPI since I’ve not had a use case for it, but I finally found the time to do a thorough read through of the documentation. The notes are here: FastAPI.

It’s pretty nifty, and my next task is to look at leveraging Hotwire to build server-render applications with minimal JavaScript, and extend that to an idea I had about making Business Intelligence more impactful.

What I’ve been reading

Ukraine’s Path to Victory - Foreign Policy is a thorough run-down of how Ukraine can wrest victory from Russia’s hands. It’s a great article but I’m not sure I can really validate all the claims that it makes.
Why The Ethereum Merge is a Big Deal - Every is a great explainer on the changes with the Ethereum merge. It’s quite exciting to see cryptocurrencies come up with some really grown-up governance process and utilize it to create sweeping changes, particularly to address energy use.
Towards a Principled Bayesian Workflow - Michael Betancourt is one of the gems I’ve re-read due to a personal project I’m tackling. Hands down still one of the best pieces of workflow writing in years, and one that really got me into using Bayesian statistics for my data science work for a while.

content/news/2022-10-17.html
newsletter/2022-10-17

Notebook-driven development, ASEAN, and Operational Excellence

Sun, 14 Aug 2022 16:00:00 +0800

This newsletter post is about new Python tools like {nbdev}, and what I’ve encountered trawling through the web.

From the field notes

I dove into {nbdev} from fast.ai, which is a way of developing python software purely from notebooks, which sounds terrifying at first, but becomes a little more sane upon looking in detail. See this article for their announcement post for nbdev v2 and integration with Quarto. I ultimately ended up deciding not to adopt this approach in the end, because:

It lacks good editor support - I would love for an extension to help with these #| comments similar to what Quarto has done with its VS Code extension,
it’s tied to {setuptools} and doesn’t integrate with pyproject.toml and {poetry} - I love poetry’s dependency management and can’t abandon it for nbdev, and
there’s just a lot of boilerplate which is just a personal preference.

What I’ve been reading

Even Superman cannot solve the Myanmar problem - One of my favorite newsletters focusing on ASEAN writes about the difficulties ASEAN has in exerting leverage on the Myanmar junta to create a peaceful resolution to the current coup. It highlights how without any serious attempt at more economic and political integration, the bloc will not be able to withstand both internal and external pressure.
People-first data stacks - It’s a good reminder to ensure that good product management practices are adopted whenever we attempt migrating to new “Modern Data Stack” tech. Adoption is not just a function of fit-for-purpose products but also a process that encourages proper data tool adoption.
HBR - Why Do We Undervalue Competent Management? - This is an old one from 2017, but I think the findings are durable:
- It’s hard to achieve operational excellence even with the right structures in place, due to the high variance in management quality across and within countries.
- Imporving management practice is just as costly as capital investment (buildings and equipment)
- Good management correlates with performance ¹
- Ultimately $\text{Results} = \text{Strategy} \times \text{Execution}$ , and it’s important to instill in managers that “being strategic” (always a point of constructive feedback given to first-time managers) does not substitute for operational excellence.

newsletter/2022-08-14
content/news/2022-08-14.html

One skeptical thought though is that because this is based on a survey, the causality might be reversed - good performance is associated with increased investment in management practices due to lower pressure. ↩︎

Restoring modification timestamps in cloned git repos

Sun, 07 Aug 2022 09:00:00 +0800

I have a GitHub Actions Workflow that clones the repo, renders the content, then pushes it up to the gh-pages branch to be published. However, the ordering of the field notes was dependent on the file modification timestamps. I imagine workflows that use Make will also be affected by this issue.

I needed a solution that restored the modification timestamps after the repo is cloned. I chanced upon git-tools which contained a utility called git-restore-mtime that served this purpose perfectly.

To achieve this, I added a step to install and execute after cloning. Additionally, the whole history needs to be cloned by setting fetch-depth to 0 for the actions/checkout action.¹

- name: Check out repository
  uses: actions/checkout@v2
  with: 
    fetch-depth: 0

- name: Restore modification timestamps 
  run: sudo apt-get install -y git-restore-mtime && git restore-mtime

Note that this requires git and a modern python version. GitHub Actions does come with Python pre-installed, so this works well.

content/notes/til/git_restore_mtime.html
learning/git-restore-mtime

I presume that the modification times are taken from the git log, but I may be mistaken. ↩︎

Experimentation, Self-Service Analytics, and Forgiveness

Sun, 07 Aug 2022 02:00:00 +0800

What I’ve been reading

Down with Experimentation Maximalism is from the Analytics Engineering Roundup, and offers a great framework for thinking about how to use experimentation and product development:
- before product-market fit (PMF), you should be developing bold hypothesis and attempting to get at the root of issues, which can just rely on descriptive statistics.
- after PMF,¹ you should more likely rely on a structured experimentation program to eke out the maximum returns from your current product’s monetization and quality.

Four frameworks for self-service analytics is another great newsletter post from the Analytics Engineering Roundup.
- Frameworks always exist, even if they are just implicitly used.
- All have assumptions, implications, and limitations, and the wisdom is knowing how to translate those into decisions, and when to use a different framework.
- In self-serve analytics, there are a few frameworks that are commonly used:
  - to save analyst time for higher-level decisions -> usually ends up in a situation where the lowest-value items have the best documentation and support
  - to break down data silos -> ends up in a situation where most time is spent on band-aid solutions that don’t address fundamental infrastructure issues
  - to provide a single source of truth for highest value issues -> author clearlypresents this as a more preferred option => ends up in a situation where the most time is spent on the highest leverage things
  - to increase data-driven decision making => too vague of a framework for anyone to most concretely facilitate in while performing their daily roles.
- I really like the iteration he’s come up with, breaking it down in 2 dimensions (positive vs normative, and within different decision types):
Quantum Resistant Crypto Hacked is a reminder that when using blockchain technologies, trust shifts from human actors to the encryption algorithms and the rules of governance encoded in that blockchain. Risk management, instead of being rendered unnecessary, shifts from those human systems to the technologies employed.

Today I learned

Restoring modification timestamps in cloned git repos - This is useful in CI/CD workflows that rely on modification timestamps.

Peaceable disposition, indeed

One of my favorite examples of sarcasm as early as the 1800s:

Mine is a most peaceable disposition. My wishes are: a humble cottage with a thatched roof, but a good bed, good food, the freshest milk and butter, flowers before my window, and a few fine trees before my door; and if God wants to make my happiness complete, he will grant me the joy of seeing some six or seven of my enemies hanging from those trees. Before death I shall, moved in my heart, forgive them all the wrong they did me in their lifetime. One must, it is true, forgive one’s enemies– but not before they have been hanged. Heinrich Heine

I did add a note here, though, that even post-PMF companies should revert back to the first approach when venturing into new markets or product lines. ↩︎

New Python Libraries, Quarto, and Viruses

Sun, 31 Jul 2022 16:00:00 +0800

In this newsletter issue, I talk about my revised Python stack, getting COVID, and Quarto.

Python Tooling

I’ve spent quite a bit of time dabbling in Python recently, so I decided to do another sweep of the ecosystem to see how things have improved. These are the new gems I’ve adopted into my set.

poetry is a python dependency manager that reminds me of npm, and it just works. Dependency management via conda or plain virtualenvs has always been such a pain, and this really resolves one of Python’s key pain points for me. It even extends to publishing modules so that we don’t have to use setuptools.
A big reason why I used R when I started learning data science was because I really disliked pandas' API. polars is an alternative data frame written in Rust that comes with a Python interface. It not only boasts superior performance through parallel processing, it also was an opportunity to design a really good interface that is closer to {dplyr}.
pydantic is a data validation library that allows you to use functional programming concepts in Python really easily. It reminds me of using types in Julia. Additionally it forms the basis for many other popular Python libraries like fastapi and sqlmodel.

Other libraries as I discover them are going to be in Python Libraries.

Quarto

It seems that RStudio has also decided to give Python a closer look now. They’ve even rebranded to Posit to reflect the fact that they would not just be working on the R ecosystem moving forward. I am personally very excited about this change.

One of the first projects they came up with is Quarto, a scientific publishing tool written in Typescript but supporting R, Python, Julia, and Javascript (of the Observable Notebook variety). This allows all scientific languages to be used in one site.

I was so excited that I spend my period of virus isolation mindlessly migrating my blog to this new format, so now I can author any piece of scientific publishing. If you notice that the blog looks slightly different, that’s because it’s now reborn as a Quarto Blog.

Testing Positive

I tested positive for COVID this week and am writing this while in isolation. I’ve ruminated on how annoying it is to spend 2 years ducking and dodging the virus only to get it now when we have a newborn, but I’ve convinced myself I’m underestimating what could have happened when my wife was still pregnant, or before we got vaccinated.

I’m almost out of it, and have learned a great deal about ART and PCR false positive and false negative rates.

content/news/2022-07-31.html
newsletter/2022-07-31

Decision Reversibility

Wed, 01 Dec 2021 15:00:00 +0800

You can always make fast decisions if decisions are reversible. Don’t get caught up in analysis paralysis when you could have committed and reverted in the same amount of time.

Ensure that analysis effort leads to better or faster decisions

One thing I always encourage data people to do is to determine exactly how analysis effort will lead to a better or faster decision:

One trick can be as simple as plotting out (with your stakeholder) what action would be taken as a result of the analysis.
Another is thinking about decision reversibility, especially in startups where your analysis bandwidth is necessarily limited and precious.

Highly reversible decisions don’t require a lot of pre-decision analysis

The amount of time spent doing pre-decision data analysis on a decision should be inversely proportional to the reversibility of that decision. It’s more beneficial to do an analysis for launching a new market (very hard to “unlaunch” a market) than changing copy of a website (just throw some ideas and see what sticks using some experimentation).

I find that I save people a lot of time on highly reversible decisions by just telling them to launch and iterate instead of spending tons of time digging into event logs and user behavior, and as a bonus, I get to spend my analysis bandwidth on less reversible, and therefore more consequential, decisions.

thoughts/decision-reversibility
content/thoughts/2021-12-01-decision-reversibility/decision-reversibility.html

Analysis quality, software engineering, and proprietary systems

Tue, 30 Nov 2021 02:00:00 +0800

Analysis quality

One of the great ironies of the analytics industry is its utter inability to measure itself. Despite “defining key metrics” for ambiguous business processes being a key responsibility in nearly every analytics role, we enthusiastically reject doing it for ourselves. - Benn Stancil

There’s been a lot of ink spilled on what constitutes a good analysis, but I’ve found there’s a common thread coalescing around the idea that in a world without counterfactuals and rife with complex (emergent) systems fast decisions are good decisions.

Making good business decisions is about making many falsifiable business decisions. Most of our business ideas are bad, and there is little we can do to improve the quality of our business decisions, so the next best course of action is to make many business decisions and ensure that we can evaluate them objectively. Eric Colson lays it out extremely well in his article. Article.
Analysis quality is all about speed to action.** Benn Stancil writes about how a very simplified, non-nuanced metric such as speed to action is a good way to measure analysis quality. I think it’s an excellent take; it forces our analysts to stop hiding behind unnecessary nuance and details and express real empathy for the decisions we hope to influence. Article.

Properietary systems

Bank Python is diverging from open source python, i.e. the atrophy of skill in proprietary software environments. Cal Peterson writes about the different monolithic systems that exist in investment banks and how they are extremely different from everything outside of it. It’s an interesting look at how technology practices harden for specific industries. Article

Software engineering

People talk about innovation a whole lot, but what they are usually looking for is cheap wins and novelty. If you truly innovate, and change the way that people have to do things, expect mostly negative feedback. If you believe in what you’re doing, and know it will really improve things, then brace yourself for a long battle. - Justin Etheridge

20 Things learned over 20 years as a software engineer. Engineering is ultimately a creative process - what that means is that amount of calories spent banging out code ona keyboard does not equal to more productivity (it may even mean less). Maturity in software engineering involves guiding the creative process in a more principled way before starting work. Article

Finding a micro-blogging platform

This information is no longer up to date as clearly I am no longer using this solution for micro-blogging.

I’ve been trying to find a good blogging platform lately. I’d like to be able to: (a) write easily on an iPad and a Mac (maybe even on iPhone), (b) create fast feedback loops for content, and (c) be able to expand more advanced non-text elements, such as images and code blocks.

I want this separate from my main blog because I want to write more updates on my life, random thoughts and interests, and keep the main blog free for more longform analysis-type articles (because after all the distill format looks super weird for informal thinking).

I tried apps like Ulysses or iA Writer, but find that they are way too focused on the writing part and only publish to separate managed platforms like Wordpress or Ghost. Code blocks, tables, and images prove to be really difficult.

I also likewise looked at things in the R world like {blogdown} but found that the feedback loops are too slow, and working with them on an iPad would be quite clunky.

I settled on what was starting me in the face anyway, which is Notion exposed in a subdomain via Cloudflare Workers (the overall solution is called Fruition). I can easily edit this anywhere (except maybe offline where I’ll use a standard markdown editor), then import afterward, it’s WYSIWYG so instant feedback loops, and can still accommodate some advanced code blocks and embeds.

newsletter/2021-11-30

The Map is not the Territory

Sat, 18 Sep 2021 15:00:00 +0800

The map is not the territory; seek ground truth whenever possible to accelerate learning.

uberHOP is a little example from my experience. The product was a point-to-point (a.k.a UV express) service Uber launched in Manila, along with Seattle and Toronto.

The way it worked was simple: you would make a request to take a specific route during peak hours, and we would batch you in with up to 6 people to take a high occupancy vehicle along the route.

uberHOP needed high occupancy to become profitable

The pricing was at a 70% discount to uberX (the traditional ride product), and drivers were guaranteed earnings, so there was a minimum average occupancy needed to hit profitability. To get to that high occupancy, we needed to ensure that the routes selected were of high quality.

Initial approach: Clustering!

My first instinct as a data person was [[clustering]]. We needed to find pairs of longitude and latitude that had enough pickup and dropoff density in them to have a decent chance of becoming profitable.

The launch routes were selected using this method, but we had limited success, even after a novelty period, cancellation rates remained high. I tried different algorithms, distance metrics, using various map features, dispatch radiuses, all for very incremental gains.

We used clustering to find initial approaches, but the results were not as expected.

Seeking ground truth

What did help was to actually seek ground truth, and the solution was embarrassingly obvious.

When we physically went to the most successful route’s pickup, the two key factors were: (a) high density residential buildings (as opposed to commercial), and (b) a driveway so drivers weren’t a moving target.

SM Light Residences was a great pickup that embodied all the factors that were required for a good pickup

We were able to turn the product profitable in a few weeks! This was easy to do because I was physically located in the market. However, this is a perennial challenge for distributed teams, so it’s even more important to consciously seek ground truth in those situations.

Here’s an abridged version in Twitter thread form:

The map is not the territory; seek ground truth whenever possible to accelerate learning. 🧵

Here's a little example from my experience: uberHOP was a point-to-point (a.k.a UV express) service we launched in Manila. We needed high occupancy so route selection was critical. pic.twitter.com/dfGdDlfE1j
— TJ Palanca (@tjpalanca) September 17, 2021

aliases:
  - content/thoughts/2021-09-18-the-map-is-not-the-territory/the-map-is-not-the-territory.html
  - posts/2021-09-18-the-map-is-not-the-territory
  - thoughts/the-map-is-not-the-territory

Data analytics in emerging markets

Thu, 16 Sep 2021 00:00:00 +0800

I was invited by the Civil Service Commission of the Philippines to provide a short talk on data analytics in emerging markets.

I wanted to provide an organization with a rough blueprint to building a sustainable data practice in an emerging market context, including to:

set a Data Strategy
building Decision Systems
achieve Data Adoption

The slides are here:

talks/data-analytics-in-emerging-markets

Harnessing the Power of Data

Tue, 24 Nov 2020 01:00:00 +0800

Prepared and presented this talk along with the data team at First Circle at Philippine Startup Week 2020. Focused on the practical aspects of actually creating a data function within a startup, from Data Strategy, to Data Engineering, and Data Analytics. We intentionally skipped Data Science.

Here are the slides for reference:

A data team's product is decisions

Sun, 11 Oct 2020 00:00:00 +0800

Photo by Franki Chamaki (Unsplash)

Making your data team’s main product decisions, as opposed to reports, models, or engineered systems, is a great way of communicating the value of the team internally and externally.

Data teams don’t come built-in

Most data teams don’t come “built-in” to most companies that don’t start with data as the primary product.

There will usually be a team in charge of sales and distribution (Sales, Marketing, Growth, Operations), a team in charge of product purchasing or development (Product, Engineering), and general administration (Finance, Leadership, HR).

By the time companies begin to even consider the concept of a data team is when they have achieved some non-trivial amount of scale (read: post Product-Market fit).

That doesn’t mean that data teams are of no utility - it just means that people who find themselves being data person #1 in a startup or large company usually encounter the issue of either finding the areas for which they or their team can be responsible or for proving relevance or value add to the business.

One thing I have found very useful is centering your data team’s product around decisions.

A data team’s product is decisions

Regardless of where the type of data work is on the spectrum between data engineering, analytics, business intelligence, or modelling, the product of a data team is not the above, but decisions.

At First Circle, our team’s goal is to ensure that all decisions, whether strategic or operational, augmented or automated with data.

Types of decisions

There are two types of decisions that are usually what a data team should target: (a) bespoke, luxury strategic decisions, and (b) mass-produced, repeatable, and fast operational decisions.

Strategic Decisions

Strategic decisions are luxury, bespoke, and artisanal decisions; decisions that are made infrequently but have a large impact. There is a large amount of uncertainty and it’s unlikely there is going to be a frequent feedback loop. For augmenting these decisions one would benefit from a business mindset, communication skills, and an ability to extract information from limited data (heavily custom linear models or Bayesian techniques).

Examples of strategic decisions are product or market expansions, budget and headcount planning, competitive analysis, mergers, and acquisitions.

Operational Decisions

Operational decisions are mass-produced, repeatable decisions that are made hundreds or thousands of times a day. They may not be that impactful individually, but the sheer volume makes optimizing these kinds of decisions important as well. The decision systems behind operational decisions are usually algorithmic in order to be cost-beneficial and would need machine learning, data science, and strong data engineering to productionize a system.

Examples of operational decisions are any per-transaction decisions, such as a bank’s personal loan grant algorithm, a ridesharing company’s driver dispatch algorithm, or an e-commerce company’s recommendation algorithm.

Individuals who come into data science usually only think of the second type of a decision, but it’s critical that data teams address both if they are the only team dealing with data, as running a business successfully requires both types of decisions to be data-driven.

Considering both types in a data team’s roadmap

When designing a data team’s roadmap, I’ve found that it’s important to be conscious of the difference between the two types of “products”, and to ensure that the mix is appropriate for the business.

Too much spent on operational decisions may mean a long, drawn-out slog to first value delivered, as a lot of data collection and experimentation has to take place.

Too much spent on strategic decisions may mean that your company doesn’t actually realize true technology value from data, you will then operate much like a traditional company that is just a little smarter with their information.

Worse and more common is mixing up the two; treating strategic decisions as operational or vice versa. I’ve seen many situations where people are using random forest variable important for a one-time decision (“insighting”), or where a human’s role in a process is only pushing buttons and not actually applying any judgment.

What I suggest is laying these out in a board and highlighting the high priority items in both buckets. This exercise should be done in conjunction with your team’s planning cycle.

In a future post, I’ll write about how to structure a team that addresses these issues in a holistic way, and avoids the common pitfalls associated with “pin factory” model that many new data teams adopt.

posts/2020-10-11-data-team-value
articles/data-team-value

Dealing with model uncertainty in data products

Thu, 06 Aug 2020 00:00:00 +0800

Uncertainty can stall the development of data products, particularly in areas where there are domain experts that don’t necessarily understand the end goal.

As software moves closer and closer to traditional industry (think ridesharing, FinTech, AgriTech, etc.), more and more data products are being built to support or automate decisions that would normally be made by human experts. As this happens, there typically is a valid doubt as to whether a data product can automate a solution to a high enough standard that it becomes worth the investment.

When unmanaged, this uncertainty can stall the production of otherwise good data products. Either the data product will be developed in a way that materially harms the business because the data scientists overestimated the value of their models, or human experts will reject the product entirely and never realise any value from their data.

I’m trying to write down some tips in dealing with design uncertainty in data products: (a) build in parameterization or agility into your data products, and (b) evolve your data products in an agile way. These concepts are typically not new to product management, but as data scientists find themselves in situations where they have to be their own product manager, I think it is useful to phrase it through this lens.

Building in parameterization where we don’t have data

Should we cap credit limits at 10 million PHP or 15 million PHP? Sometimes, there is no data-driven way to answer this question because we don’t have examples of how users would react to this policy. The domain expert may also be uncomfortable about making a strong decision on what that number should be, but in a typical software development context, there is a demand to spec that user story down to the exact parameter.

In these cases, the best course of action is to invest time in parameterizing these decisions. We can set them at the value that makes sense for now but builds into the product functionality to quickly switch this parameter once we have new information (and to replace this with some machine-learned decision once we have enough examples).

This way, product development can move forward without having to make potentially costly decisions that are hard to reverse down the line.

Put another way: you can either build perfect prediction such that you know exactly what will happen as you roll out your data product, or you can build in split-second reaction times such that you’ll never really need the forecast.

Evolving data products

Not everything has to be a neural network at the start; in fact, if you are starting a project with statistical modelling or machine learning top of mind, you’ll probably face a lot of issues.

A way to deal with model performance uncertainty is to take smaller steps in your automation journey. You can see here three distinct levels where data is taking an increasingly important role in making that decision. While progression from “Intuitive” to “Data-Driven” to “AI-Driven” is clear and probably no one will find issues with trying to progress up that ladder, it’s typically the green boxes (the level of AI involvement) where projects tend to trip up.

What one can do here is progress up the green boxes by really just starting with typical software - rules-based logic with fact-based inputs. Once we get comfortable and continue to collect data/reduce uncertainty around how to make the right decisions, you can progress up the chain and add more modeling elements to it.

Two factors influence how fast you can progress across the green axis:

Cost of failure - this is inherent to the decision you are trying to automate. The cost of failure of a poorly underwritten loan is going to be much lower than serving up the wrong recommendation on an e-commerce site.
Data completeness - this is where data engineering fundamentals are absolutely critical. A problem like image processing is data complete; whatever info is needed to 100% say whether something is a dog or a cat is already in the image. Whereas, when you are trying to lend someone money, you can hardly observe every aspect of how that person manages his/her finances or runs his/her business. If you can automate data collection in a cheap and automated way, then you are much more likely to progress to more advanced forms of decision automation.

content/thoughts/2020-08-06-dealing-with-model-uncertainty-in-data-products/dealing-with-model-uncertainty-in-data-products.html
posts/2020-08-06-dealing-with-model-uncertainty-in-data-products
thoughts/dealing-with-model-uncertainty-in-data-products

Data Science for Operational Impact

Sat, 23 Jun 2018 01:00:00 +0800

Data science for operational impact is quite an interesting field of study that has more intersections with social sciences and requires more organizational savvy.

Much of the data science discourse at the time was geared towards what I call Product Data Science, where the goal was to build highly scalable machine learning systems that solve a general problem (think Uber’s surge algorithm). However, I think an equally interesting area of data science is what I’d call Operational Data Science, where there is significantly more iteration, working with domain experts (local marketers, country managers), to solve problems with “human-in-the-loop”.

Thinking about data science this way has enabled me to overcome many adoption barriers in my past work.

I presented this at SGInnovate, a data science incubator bootcamp in Singapore, while I was still at Uber.