Mikhail Popov

License

Sat, 23 Sep 2023 00:00:00 +0000

My blog posts and tutorials are released under a Creative Commons Attribution-ShareAlike 4.0 International License.

Privacy Policy

Sat, 23 Sep 2023 00:00:00 +0000

Who I am

My website address is: https://mpopov.com/.

This website is administered by Mikhail Popov ( mikhail@mpopov.com).

What data I collect

I do not use any cookies and I does not send tracking data to any third-party services.

Wikipedia Preview for R Markdown documents

Sat, 09 Apr 2022 00:00:00 +0000

Wikipedia Preview (developed by Wikimedia’s Inuka team) is so cool:

When readers navigate in and out of a webpage through interacting with several hyperlinks, they can easily lose context of what they were reading in the first place. Content sites would like their readers to read and engage with their content and understand it without having to get contextual information elsewhere. Wikipedia Preview can solve this problem for content providers by allowing readers to have concise and visual contextual information from Wikipedia within a content provider’s mobile properties - website or webapp.

Even faster matrix math in R on macOS with M1

Sun, 10 Oct 2021 00:00:00 +0000

Instructions for switching R to use Apple’s math library optimized for Apple Silicon and some benchmarks comparing the performance.

Making Of: Session Tick visualization

Sat, 13 Mar 2021 00:00:00 +0000

In this post I will walk through my R code for a data visualization I created for the session length dataset project at the Wikimedia Foundation.

Animation of optimization in torch

Sun, 28 Feb 2021 00:00:00 +0000

In this post I will show you how to use the {gganimate} R package to make an animated GIF illustrating Adam optimization of a function using {torch}:

library(torch)
library(gganimate)
library(tidyverse)

We will use torch::optim_adam() to find the value of x that minimizes the following function:

f <- function(x) (6 * x - 2) ^ 2 * sin(12 * x - 4)

The function looks as follows:

The adam_iters dataset will contain an iter column (for the iteration/step identifier) and an x column (the value of x after each iteration):

Pivoting posteriors

Mon, 07 Sep 2020 00:00:00 +0000

In Stan, when a parameter is declared as an array, the samples/draws data frame will have columns that use the [i] notation to denote the i-th element of the array. For example, suppose we had a model with two parameters – \(\lambda_1\) and a \(\lambda_2\). Instead of declaring them individually – e.g. lambda1 and lambda2, respectively – we may declare them as a single lambda array of size 2:

parameters {
  real lambda[2];
}

When we sample from that model, we will end up with samples for lambda[1] and lambda[2]. We want to extract the i from [i] and the name of the parameter into separate columns, yielding a tidy dataset.

Git Forensics

Tue, 25 Aug 2020 00:00:00 +0000

Earlier today I was helping a coworker with a question about data related to block messages on mobile, like this:

Lol.

— Katherine Maher, October 20, 2018

I did not anticipate my investigation to become what I might best describe as “git forensics”. First, let’s introduce our dramatis personae:

MobileFrontend: When you browse the mobile (“m.” subdomain) version of Wikipedia in a browser, what you see rendered is largely due to the MobileFrontend extension for MediaWiki (the software that powers Wikipedia, Wikimedia Commons, and Wikimedia projects).
Git: Git is a free and open source system for version control.
Phabricator: Phabricator is an open source toolkit for code review, repository hosting, bug tracking, and project management. If you’re familiar with Jira it’s kind of like that.

The story begins with the Phabricator task T260218 to “Add logging for measuring impact of our work on improving the mobile block messages.” The task mentions that this has been done in the past and links to a task from August 2018: T201719. What happened to that tracking? Is that data still available?

Using R to help my wife manage Sims screenshots

Sun, 02 Aug 2020 00:00:00 +0000

I grew up with The Sims and remember spending what is probably hundreds of hours of my childhood with that game, so it was a special feeling to share that with my wife and introduce her to The Sims 3 a few years ago.

Coming from Animal Crossing: Happy Home Designer, the expanded toolset for interior design (AND addition of architecture tools) hooked her, but she wasn't that into the non-building part of playing The Sims. It actually wasn't until last year when we got The Sims 4 (TS4) at a discount that she got into the full Sims experience, because TS4 is waaaaaaay better in every way (from building houses, to playing with your sims).

Replacing the knitr engine for Stan

Thu, 30 Jul 2020 00:00:00 +0000

2020-08-03 UPDATE: Good news! A version of this engine is now included in versions 0.1.1 and later of {CmdStanR}. Use cmdstanr::register_knitr_engine() at the top of the R Markdown document to register it as the engine for stan chunks. See the vignette R Markdown CmdStan Engine for examples. Shoutout to the maintainers Jonah Gabry & Rok Češnovar for a super positive code review experience with the pull request for this.

I originally dabbled with custom {knitr} engine creation last month, when I made {dotnet} which enables R Markdown users to write chunks with C# and F# programs in them.

Introducing 'dotnet' knitr engine for C# & F# chunks in R Markdown

Wed, 10 Jun 2020 00:00:00 +0000

I had a thought “wouldn’t it be cool to do a blog post about Bayesian inference with Infer.NET?” and then a follow-up thought “wouldn’t it be even cooler to have the probabilistic programs as R Markdown chunks that would be actually built/compiled and then run/executed just like Python and Julia chunks would be?”

And that’s how I ended up spending an evening learning how to make custom language engines for {knitr} and making one for C# and F# languages.

Strings in R 4.x vs 3.x (and earlier)

Fri, 22 May 2020 10:25:00 +0000

Among the several user-facing changes listed in R 4.0.0’s release notes was this point:

There is a new syntax for specifying raw character constants similar to the one used in C++: r"(...)" with ... any character sequence not containing the sequence )". This makes it easier to write strings that contain backslashes or both single and double quotes. For more details see ?Quotes.

To get a better sense of this (wonderful) feature addition, I thought it’d be useful to see some before/after examples.

Ordinary Differential Equations with Stan in R

Fri, 22 May 2020 00:00:00 +0000

A tutorial on fitting a Bayesian ODE model with Stan, R, and {cmdstanr} package.

Approximating probabilities

Sat, 10 Aug 2019 00:00:00 +0000

A tutorial on using R and Monte Carlo simulation as a substitute for analytical solutions to “what is the probability of?” problems.

Faster matrix math in R on macOS

Tue, 04 Jun 2019 00:00:00 +0000

Update (October 2021): macOS 10.14 “Big Sur” and later do not ship with Accelerate BLAS dynamic libraries in the filesystem, so this trick only works up to macOS 10.13 “High Sierra”

If you want faster matrix operations in R on your Mac, you can use Apple’s BLAS (Basic Linear Algebra Subprograms) library from their Accelerate framework instead of the library which comes with the R binary that you get from CRAN. (Unless you built R from source yourself.) CRAN recommends against this, saying:

Modeling a logic puzzle as a constraint satisfaction problem with OMPR

Sat, 04 May 2019 00:00:00 +0000

A tutorial on treating a logic puzzle as constraint satisfaction problem and coding it as an integer program to solve using the R package {ompr}.

Bayesian Optimization in R

Tue, 16 Apr 2019 00:00:00 +0000

A tutorial on using Bayesian optimization to find the minimum of a function with only a few evaluations of the functions, using different approaches to identify the best next value to evaluate the function at.

My recipe for the best breakfast potatoes (and terrific bacon)

Wed, 15 Aug 2018 00:00:00 +0000

Everyone I treat with these bomb-ass potatoes always tells me how amazing they are and it’s a bit of an elaborate process to describe, so I decided to write it up here. There are actually two recipes in this post and one is (kind of) a prerequisite for the other, but if you’re vegetarian/vegan or don’t eat pork for religious (or other) reasons, feel free to skip to the second stage.

Data Analyst vs Data Scientist: Industry Perspectives

Thu, 24 May 2018 00:00:00 +0000

Both “Data Analyst” (DA) and “Data Scientist” (DS) are titles that vary greatly between industries and even amongst individual organizations within industries. As the roles behind titles change over time, it is natural for some teams to ask themselves the following questions: should we have distinct roles or just stick to one? How would we differentiate the roles in a way that fulfills our organization’s needs and is generally consistent with similar organizations? Do we want to consider a DS to be equivalent to a Sr. DA, the only difference being the title? Answering these questions not only establishes clear responsibilities and expectations, but enables hiring managers and recruiters to communicate clearly with potential applicants in the future (in job postings, for example).

Resources for learning to visualize data with R/ggplot2

Wed, 21 Mar 2018 00:00:00 +0000

I’m currently learning visualisation with R/ggplot2 and was wondering whether you could share tips/links/videos/books/resources that helped you in your journey :-)

Sure! Here ya go:

Tips

The only tip I’ll give is that you should strive to make every chart look exactly how you want it to look and say exactly what you want it to say. You will learn in the process of doing. When it’s time to visualize the data and you have an idea for a very specific look and story, don’t give it up or compromise on your vision just because you don’t know how to do it. Trust me, there is so much documentation out there and so many posts on Stack Overflow that you will be able to figure it out. (But also it’s totally fine to get 90-95% of the way there and call it done if that last 5-10% sprint is driving you bonkers.)

The journey so far…

Thu, 21 Sep 2017 00:00:00 +0000

I recently received an email which said, “I’m interested in learning more about you and your journey to where you are today,” so I thought I’d describe how I went from studying visual arts to analyzing data at Wikimedia Foundation (WMF).

Growing up I excelled in visual arts and mathematics at school, and they continued to be my strongest subjects. My parents and I immigrated to US from Russia when I was 10, and I spent the first few years focused on learning English – which was especially difficult because I was the only Russian-speaking person at my school. I was okay at English when I entered 6th grade, having learned a lot of it from The Simpsons of all things. That was also the year I joined band and started learning trombone, but that wouldn’t last.

Advice for graduates applying for data science jobs

Wed, 16 Aug 2017 00:00:00 +0000

2019-08-01 update

Things were a little different when I wrote this in 2017. These days I constantly see new/junior data scientists get rejected because they don’t have the experience. Even those who have an impressive portfolio of projects to show off that they have the technical know-how get thumbs down. I firmly believe this is a failure of employers, not the new generation of recently graduated data scientists entering the field.

Installing GPU version of TensorFlow™ for use in R on Windows

Sun, 11 Jun 2017 00:00:00 +0000

Intro

The other night I got TensorFlow™ (TF) and Keras-based text classifier in R to successfully run on my gaming PC that has Windows 10 and an NVIDIA GeForce GTX 980 graphics card, so I figured I’d write up a full walkthrough, since I had to make minor detours and the official instructions assume – in my opinion – a certain level of knowledge that might make the process inaccessible to some folks.

Probabilistic programming languages for statistical inference

Tue, 10 Jan 2017 00:00:00 +0000

Introduction

This post was inspired by a question about JAGS vs BUGS vs Stan:

right, that’s what got me confused! so they.. do the same thing?

— Andrew MacDonald, January 10, 2017

Explaining the differences would be too much for Twitter, so I’m just gonna give a quick explanation here.

Mostly-free resources for learning data science

Tue, 22 Dec 2015 00:00:00 +0000

In the past year or two I’ve had several friends approach me about learning statistics because their employer/organization was moving toward a more data-driven approach to decision making. (This brought me a lot of joy.) I firmly believe you don’t actually need a fancy degree and tens of thousands of dollars in tuition debt to be able to engage with data, glean insights, and make inferences from it. And now, thanks to many wonderful statisticians on the Internet, there is now a plethora of freely accessible resources that enable curious minds to learn the art and science of statistics.

Guide to Shiny apps with Shiny Server on Amazon EC2

Sun, 01 Jun 2014 00:00:00 +0000

Preface: posting this for archive purposes only. This was the first of its kind and has been succeeded by better guides.

Introduction

I am writing this guide because this guide did not exist when I decided to put my 2010 US Census Shiny App on Amazon’s servers. Surely I can’t be the only one who’s never had any experience with EC2 (or SSH or vi, for that matter).

Quartile-Frame Scatterplot with ggplot2

Sun, 01 Jun 2014 00:00:00 +0000

Inspired by The Visual Display of Quantitative Information by Edward R. Tufte

The goal is to make the axes tell a better story about the data. This is done by turning the axes into quartile plots (cleaner boxplots).

Usage Example:

Only x and y are required, everything else is optional.

qsplot(
  x = mtcars$wt, y = mtcars$mpg,
  main = "Vehicle Weight-Gas Mileage Relationship",
  xlab = "Vehicle Weight", ylab = "Miles per Gallon",
  font.family = "Gill Sans" # alternatively: "Times New Roman"
)

The R code can be found on GitHub.

Miscellaneous

Wed, 01 Jan 2014 00:00:00 +0000

Talks

No one GLM should have all that power: Power analysis of multilevel/hierarchical generalized models
A Workflow For Bayesian Modeling and Reporting in R: Talk on Bayesian modeling, model comparison, and presentation of results for Pittsburgh useR meetup, December 2018
A Workflow For Bayesian Modelling and Reporting in R: Talk on Bayesian modeling, model comparison, and presentation of results with R for Friendly Tech Space folks
Open knowledge in R with Wikimedia APIs: Presented at Cascadia R Conference on 2 June 2017

Guest Appearances, Interviews, and Blogs

Wikimedia Blog published my post explaining how my team hires data scientists
NSF-Census Research Network featured me in their NCRN Newsletter Volume 3, Issue 2’s section on the program’s alumni
Master’s in Data Science interviewed me for their Life of a Statistician feature
One More Turn: 5 Questions ( Watch on YouTube)

Projects

Wed, 01 Jan 2014 00:00:00 +0000

R Packages

API wrappers

waxer: R wrapper for the Wikimedia Analytics Query Service (AQS). This particular wrapper is for the /metrics endpoint of the REST API which provides data and metrics around traffic, users, and content on Wikimedia sites. Get started…
WikidataQueryServiceR: Interface to Wikidata Query Service API for querying Wikidata using SPARQL and getting back data.frames in R. Available on CRAN.

R Markdown

wikipediapreview: Enables easy integration of Wikipedia Preview popup context cards in R Markdown documents – compatible with distill, pkgdown, and blogdown – refer to the blog post for more info
wmfpar: Template based on memor, for use by the Wikimedia Foundation’s Product Analytics team for PDF reports written in R Markdown

RStudio add-ins

tinydensR: An RStudio add-in for playing with distribution parameters and visualizing the resulting probability density and mass functions.

Machine learning

dpmclust: Implements the DP-means algorithm introduced by Kulis and Jordan in their article Revisiting k-means: New Algorithms via Bayesian Nonparametrics. Instead of specifying how many clusters to partition the data into, like one would with k-means, user specifies a penalty parameter λ which controls if/when new clusters are created during iterations.
maltese: Little utility R package for transforming time series data into a format that’s more machine learning-friendly – previous p observations become features.
MultiLabel Prediction Using Gibbs Sampling: Users can employ an external package (e.g. ‘randomForest’, ‘C50’), or supply their own. New observations are classified using a Gibbs sampler since each unobserved label is conditioned on the others. The package includes methods for evaluating the predictions for accuracy and aggregating across iterations and models to produce binary or probabilistic classifications. Available on CRAN.

Python packages

gsc-utils: Utilities for accessing and downloading the statistics on a site’s presence in Google’s search results via Search Console API.

Games and apps

taskviewr: Shiny application for browsing R packages listed on CRAN’s Task Views. It includes their URLs and licensing details, which can be very helpful if you are looking for, say, a machine learning package that is MIT-licensed.

My other Shiny applications include freelancr (for figuring out freelancing hourly rates) and the Discovery Dashboards, which I maintain as a Data Scientist on the Product Analytics team at the Wikimedia Foundation.