Cruise Location Heatmaps

Sat, 05 Aug 2017 00:00:00 +0000

Continuing my fascination with cruise ships, I decided to try and build heatmaps from the cruise ship location data I gathered in a previous post.

After looking into a few heatmap options, I settled on using heatmap.py because of its rich feature set and simple interface. I avoided using the Open Street Maps integration due to its licensing issues and instead used an equirectangular world map from Wikipedia.

Generating Maps

I originally tried generating heatmaps on my Macbook, which worked great for inputs under around 50,000 rows. Beyond that, heatmap.py begins consuming so much memory it is impossible to use. There are options you can use with heatmap.py (--range and --decay=0) that reduce its memory footprint, but I didn’t like the resulting images when using a decay of 0 so I had to upgrade my hardware.

Enter Google Cloud Platform’s Compute Engine. I spun up a 4 vCPU instance with 20GB of memory, which tore through a 2.1 million point map in about 30 minutes. The instance costs about $0.15 an hour, so this is pretty cost-effective for static images.

Static Maps

The first map I built was of all major cruise liners for all time. It contains the above-mentioned 2.1 million distinct coordinates.

python heatmap.py \
--ignore_csv_header \
-I base.jpg \
-P equirectangular \
-W 2058 -H 1036 \
-B 0.66 -e="-89,-180,89,180" \
-o out.png coords_map_major_cruise_lines.csv

Next, I wanted to compare the routes different cruise lines use. I built datasets with three cruise lines’ ships in them and mapped them:

Royal Caribbean International

Carnival Cruises

Disney Cruises

As expected, all three cruise lines hit the same major routes: the Caribbean, Mexico, and Alaska. Carnival seems to focus heavily on the Caribbean while Royal Caribbean has more diverse itineraries in the Mediterranean, Asia and South America. Disney only operates four ships, so their routes are understandably limited.

Animated Maps

heatmap.py has an animate option that I found very intriguing. To test it out, I built a dataset of one ship’s coordinates in ascending time order to show where the ship travels over time. Since I can’t start small, I used the Norwegian Pearl for this experiment, the ship with the most coordinates in the dataset.

The animate option does pretty much what you’d expect: it generates incremental PNG frames for each point in the input dataset and combines them all using ffmpeg. Since the Pearl’s coordinate dataset contains 42,000 points, it generated 42,000 heatmaps. The process of building the below animated heatmap took over 72 hours on the Compute Engine instance I was using.

It’s interesting to see the ship bounce back and forth between Alaska and the Caribbean via the Panama Canal. It makes sense that ships running the Alaskan route don’t stay in the North East Pacific during the offseason, and instead head to warmer waters where cruises run year-round.

Cruise Location Dataset

Tue, 04 Jul 2017 00:00:00 +0000

I just got back from my honeymoon, my first cruise to Alaska on the Explorer of the Seas. Aside from having a fantastic time with my wife, I gained a new fascination with the breadth and scale of cruise travel. Our ship housed upwards of 2,000 passengers and was often one of two or three ships docked in a port. Multiply that by the 100+ ships that are currently running cruises, you can have 200,000 people on the ocean at a time visiting dozens of ports across the globe.

When we got back, I began looking into cruise tracking sites and found one site that offers historical ship location data for 200+ ships going back to 2003. Using that data, I hoped to compile a dataset of ships, cities and locations for analysis and publishing.

Extract

The scripts used to extract and clean the raw data are available on Github. They are quick hacks, but they got the job done.

Transformation

Once the raw data was loaded into BigQuery, I began transforming it. First step, I want to join the location stamps to port cities and find the closest port city to that location. I tried using a Cartesian join on the whole dataset:

SELECT callsign, created_at, loc.lat as lat, loc.lon as lon, wind_from, wind_knots,
       wind_gust, barometer, air_temp, dew_point, city.lat as city_lat, city.lon as city_lon,
  	   city.id as city_id
FROM `cruise_scraper.locations_clean` loc, 
     `cruise_scraper.port_cities` city

Since the raw dataset is rather large, about 4.7 million location stamps, BigQuery couldn’t quite handle it. I limited the query to a few months worth of data and that worked, so I ran the same query 47 times and finally ended up with 2.5 billion rows of Cartesian joined goodness.

The next step is to eliminate the rows where the location stamp is paired with any city other than the one closest to it:

SELECT callsign, created_at, lat, lon, city_id as nearest_port_city, distance_to_port
FROM (
  SELECT
    callsign, created_at, lat, lon, wind_from, wind_knots, wind_gust, barometer, air_temp,
    dew_point, city_lat as city_lat, city_lon as city_lon, city_id as city_id,
    cartesianDistance(lat, lon, city_lat, city_lon) as distance_to_port,
    row_number() over(partition by callsign, created_at order by cartesianDistance(lat, lon, city_lat, city_lon)) as rnk
  FROM `cruise_scraper.locations_city_join_*`
  GROUP BY 1,2,3,4,5,6,7,8,9,10,11,12,13,14
)
WHERE rnk=1

The cartesianDistance function in the above query is a UDF. BigQuery accepts Javascript UDF’s, this one computes the distance in meters between two (lat,lon) pairs:

CREATE TEMPORARY FUNCTION cartesianDistance(lat1 FLOAT64, lon1 FLOAT64, lat2 FLOAT64, lon2 FLOAT64)
RETURNS FLOAT64
LANGUAGE js AS """
  var R = 6378.137; // Radius of earth in KM
  var dLat = lat2 * Math.PI / 180 - lat1 * Math.PI / 180;
  var dLon = lon2 * Math.PI / 180 - lon1 * Math.PI / 180;
  var a = Math.sin(dLat/2) * Math.sin(dLat/2) +
  Math.cos(lat1 * Math.PI / 180) * Math.cos(lat2 * Math.PI / 180) *
  Math.sin(dLon/2) * Math.sin(dLon/2);
  var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
  var d = R * c;
  return d * 1000; // meters
""";

Using the nearest city location stamp data, I aggregated down to daily granularity:

SELECT 
  trim(ships_v.name) as ship_name,
  trim(loc.callsign) as callsign,
  trim(ships_v.major_cruise_line) as major_cruise_line,
  date, 
  cartesianDistance(x1, y1, x2, y2) as meters_traveled,
  x1 as lat_start,
  y1 as lon_start,
  x2 as lat_end,
  y2 as lon_end,
  case when stopped_minutes >= 180 then TRUE else FALSE end as in_port, -- 3 hours spent stationary
  stopped_minutes,
  num_periods,
  port_city as port_city_id,
  cities.city as port_city_name,
  cities.country_code as port_city_country
FROM (
  SELECT 
    callsign,
    date(created_at) as date,
    max(case when day_start=1 then lat else null end) as x1,
    max(case when day_start=1 then lon else null end) as y1,
    max(case when day_end=1 then lat else null end) as x2,
    max(case when day_end=1 then lon else null end) as y2,
    sum(case when abs(lat-last_lat) < 0.1 and abs(lon-last_lon) < 0.1 then timestamp_diff(created_at, case when last_created_at is null then timestamp(date(created_at)) else last_created_at end, MINUTE) else 0 end) as stopped_minutes,
    sum(timestamp_diff(created_at, last_created_at, MINUTE)) as total_minutes,
    max(case 
             when last_lat is null and last_lon is null then nearest_port_city
             when abs(lat-last_lat) < 0.1 and abs(lon-last_lon) < 0.1 then nearest_port_city 
             else null end) as port_city,
    count(*) as num_periods
  FROM (
    SELECT 
      callsign,
      created_at,
      lat,
      lon,
      nearest_port_city,
      distance_to_port,
      lag(lat) over(partition by callsign, date(created_at) order by created_at) as last_lat,
      lag(lon) over(partition by callsign, date(created_at) order by created_at) as last_lon,
      lag(created_at) over(partition by callsign, date(created_at) order by created_at) as last_created_at,
      row_number() over (partition by callsign, date(created_at) order by created_at) as day_start,
      row_number() over (partition by callsign, date(created_at) order by created_at desc) as day_end
    FROM cruise_scraper.locations_with_city
    -- WHERE created_at > timestamp('2017-06-01')
    GROUP BY 1,2,3,4,5,6
  )
  GROUP BY 1,2
) loc
LEFT JOIN `cruise_scraper.ships_v` ships_v ON LOWER(TRIM(loc.callsign))=LOWER(TRIM(ships_v.callsign))
LEFT JOIN `cruise_scraper.port_cities` cities ON port_city = cities.id

Analysis

Which cruise line is operating the most ships:

SELECT major_cruise_line, COUNT(*) 
FROM cruise_scraper.ships_v
GROUP BY 1
ORDER BY 2 DESC

major_cruise_line	count
Other	149
Royal Caribbean International	25
Carnival Cruise Line	24
Norwegian Cruise Line	16
Holland America Line	15
Costa Crociere	15
Princess Cruises	14
Celebrity Cruises	11
Silversea Cruises	4
Disney Cruises	3

How many day/ship combinations are missing data?

SELECT 
  COUNT(*)
FROM(
  SELECT 
    ship_name, 
    date,
    lag(date) over(partition by callsign order by date) as last_date
  FROM `home-data-warehouse.cruise_scraper.daily_ship_location`
) 
WHERE date_diff(date, last_date, DAY) > 1

83,106

What are the most popular port cities?

SELECT
  port_city_name as city,
  port_city_country as country,
  COUNT(*) as count
FROM cruise_scraper.daily_ship_location
WHERE major_cruise_line <> 'Other' and in_port = True
GROUP BY 1,2
ORDER BY 3 DESC

city	country	count
Fort Lauderdale (Port Everglades)	US	3705
Vancouver	CA	3236
Cozumel	MX	3074
San Diego	US	2762
St. Maarten	SX	2046

Data Issues

The raw data comes from an unmonitored data pipeline, there may be data missing. Additionally, it looks like ships do not report location details when they are outside of service areas. This leads to many days where there’s only one reported location.

Since my definition of an in-port day depends on a time difference between stationary locations, these days cannot be shown as in-port. This may be leading to under-reported port days.

March Madness 2017

Tue, 04 Apr 2017 00:00:00 +0000

For this year’s March Madness tournament, I built an algorithmic bracket picker based on Dungeons and Dragons dice-roll battles. The bracket it generated, that I submitted for my company’s bracket competition, did surprisingly well: it correctly picked UNC to win the tournament and placed first overall in our bracket competition.

How accurate was it?

To examine the generated bracket’s accuracy, I loaded its predicted games and the tournament’s actual games into SQL tables, actual_games and predicted_games:

CREATE TABLE actual_games (
    round INT,
    division VARCHAR(7),
    game_number INT,
    team_a VARCHAR(255),
    team_b VARCHAR(255),
    winner VARCHAR(255),
    PRIMARY KEY(round, division, game_number)
);

To see how many games whose winners were accurately predicted by D&D, compare the winners:

SELECT picked_correctly, COUNT(*)
FROM (
    SELECT 
    	actual.round, 
    	actual.division, 
    	actual.game_number, 
    	actual.winner=predicted.winner as picked_correctly
    FROM actual_games as actual 
    INNER JOIN predicted_games as predicted
        ON actual.round = predicted.round AND
            actual.division = predicted.division AND
            actual.game_number = predicted.game_number
) games GROUP BY 1;

Games Predicted Accurately

As the tournament goes on, previous mistakes make it increasingly harder to make correct picks. I broke the accuracy down by round number to see how it performed as the picks got harder.

SELECT round, COUNT(*), SUM(picked_correctly), SUM(picked_correctly)/COUNT(*) as percentage_correct FROM (
    SELECT 
    	actual.round, 
    	actual.winner=predicted.winner as picked_correctly
    FROM actual_games as actual
    INNER JOIN predicted_games as predicted
        ON actual.round = predicted.round AND
           actual.division = predicted.division AND
           actual.game_number = predicted.game_number
) x 
GROUP BY 1 
ORDER BY 1 DESC;

Games Predicted Accurately by Round

At least 50% accuracy throughout the tournament, not bad. The 100% in the championship round is misleading since there’s obviously only one game in that round.

The randomness/upset factor

It would be easy to just rank teams on seed number and come up with a fairly accurate bracket. The real magic of the tournament is in the Cinderella stories and huge unexpected upsets. The D&D algorithm accounts for these by introducing random factors like dragon attacks, which immediately end the battle and award a random team the victory. I wanted to see how well my random upset picks performed:

SELECT picked_correctly, COUNT(*)
FROM (
    SELECT 
    	actual_v.round, 
    	actual_v.division, 
    	actual_v.game_number, 
    	actual_v.winner=predicted_v.winner as picked_correctly 
    FROM actual_v 
    INNER JOIN predicted_v
        ON actual_v.round = predicted_v.round AND
            actual_v.division = predicted_v.division AND
            actual_v.game_number = predicted_v.game_number
    WHERE (
        (predicted_v.team_a_seed < predicted_v.team_b_seed and predicted_v.winner = predicted_v.team_b) OR
        (predicted_v.team_b_seed < predicted_v.team_a_seed and predicted_v.winner = predicted_v.team_a)
)) x 
GROUP BY 1;

Upsets Predicted Accurately

The bracket went 3-11 predicting upsets where a higher seeded team beat a lower seed. Not a great showing, but those games are really hard to predict using just the data the algorithm uses.

Next Year

For next year’s bracket, I will try to incorporate more metrics into the model to make battle logic smarter. This year’s iteration was only based on seed value and points scored and allowed. I’ll also take a few dragons out to cut down on false upsets.

Podcasts Dataset

Sat, 18 Mar 2017 00:00:00 +0000

This March several podcast publishers are participating in “Try pod,” a campaign to encourage people to give podcasts a try. Unlike most people using the #trypod hashtag, I’m not going to use the event as an excuse to hawk my own podcast (I don’t have one.) Instead, I am publishing a large dataset of podcasts and episodes on data.world in the hopes that it will inspire some cool analytics or tools.

In this post, I’ll demonstrate some of the ETL logic used to create the dataset and run some queries on the dataset using Google BigQuery.

ETL

The first step in creating this dataset was to create a large list of podcast feeds to scrape. Since the de facto podcast provider, iTunes, doesn’t expose any public API I turned to remnants of the early days of podcasts, feed directories.

The build_lists directory contains scripts that scrape some of the largest podcast directories I found that were still operating, including All Podcasts, Podcastpedia, and Godcast1000. Fearing those directories are out-of-date, I also wrote a script to scrape the top podcasts in each category from iTunes.

Shows extract

Once the list of feeds was compiled, I wanted to extract and transform show objects. The transform script iterates over each tag element in the feed and creates a show object with it.

Since the list of feeds I extracted basically spanned the entire history of podcasting, there were a lot of feeds that didn’t follow today’s best practices or even use valid XML. Trying to accomodate those broken feeds led to a pretty messy transform step with lots of cases like the following:

# handle different date formats
elif tag == 'lastbuilddate':
	try:
		if re.search('[+-][0-9]+$', child.text.strip()):
			dt = datetime.datetime.strptime(child.text.strip()[0:-5].strip(), '%a, %d %b %Y %H:%M:%S')
		else:
			dt = datetime.datetime.strptime(child.text.strip().strip(), '%a, %d %b %Y %H:%M:%S')
	except (ValueError,AttributeError):
		dt = datetime.datetime.now()
				
obj['last_build_date'] = dt.strftime('%Y-%m-%d %H:%M:%S')

Episodes extract

Using the unique table of transformed shows, I then extracted episodes from the feeds. As with the shows transform step, the logic is full of try/catches looking for missing or malformed elements.

# some feeds use seconds, some use [HH:]MM:SS
elif tag == 'duration': 
	if child.text and ':' in child.text:
		lengths = child.text.split(':')[::-1]
		duration = 0

		for i in range(0, len(lengths)):
			try:
				duration += max((i*60), 1) * int(float(lengths[i]))
			except (ValueError, TypeError):
				pass
	else:
		try:
			duration = int(child.text)
		except (ValueError, TypeError):
			duration = None

	obj['length'] = duration

Analysis

To run analytics on the dataset, I loaded the CSV files into Google BigQuery. One of the advantages of using BigQuery is that it makes aggregations on a single column, even over millions of rows, absurdly fast. Running some of the same aggregations I ran below took 10 times as long on my local MySQL database.

Note: the episodes_flat table referenced below is a materialized view joining shows and episodes. Joined logical views in BQ tend to be more expensive to query than the cost of just unloading the results into a physical table since they do full-table scans to perform the join.

Completeness

The first question I want to answer with this dataset is how much of the podcast universe did I manage to scrape and ingest? According to a 2015 Myndset article, there are somewhere between 100,000 and 200,000 podcasts. My extract found and ingested 32,832 shows, so I found 15-30% of that total number.

As a check against that number, I wanted to see how many of the shows I subscribe to are included in the dataset. I used Exofudge’s Pocketcasts API lib to get a list of the 94 shows I subscribe to on Pocketcasts and compared it to the shows dataset:

SELECT in_dataset, COUNT(*)
FROM (
  SELECT 
    subs.title, case when shows.id is null then false else true end as in_dataset
  FROM podcasts.pocketcasts_subs subs
    LEFT JOIN podcasts.shows shows ON subs.title = shows.title
  GROUP BY 1,2
)
GROUP BY 1;

in_dataset	count
true	62
false	32

So the dataset contains 66% of the shows I subscribe to. That number is likely skewed higher by the fact that I listen to mainly top-100 shows on iTunes, which one extract was specifically intended to cover. In any case, I’m pleased with the coverage I achieved.

Basic aggregations

How many episodes are marked explicit?

SELECT 
  episodes_explicit as explicit,
  COUNT(*) as episode_count
FROM podcasts.episodes_flat
GROUP BY 1 ORDER BY 2;

explicit	count
true	127,784
false	1,135,940

What’s the average time between episodes?

SELECT 
  avg((episodes_pub_date - prev_pub_date)/(24000000)) as diff_in_hours,
  avg((episodes_pub_date - prev_pub_date)/(24000000 * 60 * 60)) as diff_in_days
FROM (
  SELECT 
    episodes_id,
    show_id,
    episodes_pub_date,
    LAG(episodes_pub_date, 1) OVER (PARTITION BY show_id ORDER BY episodes_pub_date) as prev_pub_date
  FROM podcasts.episodes_flat
);

diff_in_hours	diff_in_days
38,647.74	10.73

On average, how many episodes are in a feed?

SELECT AVG(episode_count) as avg, MIN(episode_count) as min, MAX(episode_count) as max
FROM (
  SELECT show_id, COUNT(*) as episode_count
  FROM podcasts.episodes_flat
  GROUP BY 1
);

avg	min	max
82.954181436261	1	24,350

The feed with 24,350 episodes is the TSN.ca podcast, which has published an average of 13 episodes a day since 2011. They keep their entire podcast history through 2011 in their feed, which is a whopping 31 MB.

It’s not a data post without some charts

What are the most popular audio encodings?

SELECT episodes_audio_mime_type, COUNT(*)
FROM podcasts.episodes_flat
GROUP BY 1 HAVING COUNT(*) > 1
ORDER BY 2 DESC;

Episode category over time

SELECT DATE(episodes_pub_date), categories.category, count(*) 
FROM (
  SELECT category, COUNT(*) 
  FROM podcasts.shows 
  WHERE category is not null
  GROUP BY 1
  ORDER BY 2 DESC
  LIMIT 15) categories
INNER JOIN podcasts.episodes_flat ep ON categories.category=ep.show_category
WHERE episodes_pub_date > timestamp('2006-11-27') and episodes_pub_date < timestamp('2016-11-27')
GROUP BY 1, 2
ORDER BY 1 DESC, 2;

Further work

There is a lot of unstructured text data in this dataset, using a tool like Elastic Search to mine insights from the episode descriptions could yield some interesting information. This would also be a great dataset to join with popularity data, but as far as I know there is no good source for this available.

This dataset could also provide a good starting point for a show recommendation engine based on NLP processing of descriptions and tag analysis.

Making something cool with this data? Let me know!

Hacker News Salary Survey

Tue, 07 Feb 2017 00:00:00 +0000

A while back, there was a thread on Hacker News asking people to submit their employer and salary information. Being in the start-up scene and following Hacker News pretty closely, I was instantly very interested in the data this survey was going to produce. I started working on cleaning it up that night, and published the code to Github. In this post I’ll be presenting the ETL code I wrote and analyzing the results.

Extract and Transform

I decided to load the data into a postgres database to make joins and categorization easier. Extraction was just a matter of loading the result CSV into a DataFrame and unloading it to SQL.

db = sqlalchemy.create_engine('postgresql://localhost:5432/hackernews')

dataframe = pandas.read_csv(sheet_url, header=0, 
		names=['ts', 'employer', 'location', 'job_title', 
			'employer_experience_years', 'total_experience_years', 
			'annual_base_pay', 'signing_bonus', 'annual_bonus', 
			'stock_value_bonus', 'gender', 'comments'])

dataframe.to_sql('salary_staging', db, index_label='id')
db.dispose()

A similar process is used to load dimension tables with external data into the database for joins.

The next step is to cleanse the messy data generated by free-text fields. Each step listed below contains sample queries, the full cleanse and transform process is available in transform.py

Dimension Tables

Three more dimension tables, locations, employers and job titles, are created from the survey data.

INSERT INTO locations
(
	location_name
)
SELECT DISTINCT
	lower(trim(location))
FROM salary_staging
WHERE 
	lower(trim(location)) NOT IN(SELECT location_name FROM locations) 
		and location IS NOT NULL
;

Decorate locations

Additional data, like state and coordinates, are added to the locations table from external sources.

UPDATE locations SET
	location_state = abbreviation,
	location_country = 'US'
FROM
	us_states
WHERE (locations.location_name LIKE '%' || LOWER(us_states.name) || '%'
		OR locations.location_name LIKE '%, ' || LOWER(us_states.abbreviation) || '%'
		OR locations.location_name LIKE '% ' || LOWER(us_states.abbreviation))
	AND locations.location_state IS NULL AND locations.location_country IS NULL
;

UPDATE locations
	SET location_latitude="Lat",
		location_longitude="Long"
FROM zip_codes
WHERE location_city = lower("City") and location_state="State"

Categorize job titles

Add a job category and rank to job titles.

UPDATE job_titles
	SET job_title_category='Web'
WHERE (job_title LIKE 'full_stack%'
			OR job_title LIKE '%fullstack%'
			OR job_title LIKE '%front_end%'
			OR job_title LIKE '%frontend%'
			OR job_title LIKE '%web %'
		) AND job_title_category IS NULL
;

UPDATE job_titles
	SET job_title_rank = 'Junior'
WHERE (job_title LIKE 'junior%'
		OR job_title LIKE 'jr_%')
	  AND job_title_rank IS NULL

Finally, insert the salaries into a normalized table.

INSERT INTO salaries
(
	salary_id,
	employer_key,
	location_key,
	job_title_key,
	submitted_at,
	total_experience_years,
	employer_experience_years,
	annual_base_pay,
	signing_bonus,
	annual_bonus,
	stock_value_bonus,
	gender,
	comments
)
SELECT
	stg.id as salary_id,
	em.employer_key as employer_key,
	lo.location_key as location_key,
	jt.job_title_key as job_title_key,
	stg.ts::TIMESTAMP as submitted_at,
	stg.total_experience_years::DECIMAL(6,2),
	stg.employer_experience_years::DECIMAL(6,2),
	stg.annual_base_pay::DECIMAL(12,2),
	stg.signing_bonus::DECIMAL(12,2),
	stg.annual_bonus::DECIMAL(12,2),
	stg.stock_value_bonus,
	stg.gender::gender_type,
	stg.comments
FROM salary_staging stg
	LEFT JOIN employers em ON lower(trim(stg.employer))=em.employer_name
	LEFT JOIN locations lo ON lower(trim(stg.location))=lo.location_name
	LEFT JOIN job_titles jt ON lower(trim(stg.job_title))=jt.job_title
WHERE stg.id NOT IN (SELECT salary_id FROM salaries);
;

Analysis

I’ll be examining the results of some of the queries in basic_analysis.sql.

First, let’s see if we can identify careers with the highest starting salaries. I’ll group by job category and calculate the median salary per year working and median salary per year at current employer. A much higher salary per year at current job than the median per year working indicates a higher median starting salary.

Not surprisingly, management appears to have a great median starting salary. Apart from that, data and operations jobs also have better than average starting salaries.

Next, let’s see how salaries trend as employees gain experience. I’d expect every employment category to trend upward as time goes on, but maybe there are fields with larger increases.

As expected, all of the employment categories trend upward, some more than others. Software has the highest unexperienced salary and ends up in the upper part of the 8-10+ range, while Web development seems to stay pretty stagnant in the high five figures. One concerning thing as a data engineer is the dive Data jobs take after 8 years experience. I believe (hope) this is just a lack of good data about experienced data employees rather than an actual trend. I’ll let you know in a few years ;)

Lastly, I want to see where the Hacker News community works. I’ll group by city and count the results.

Somewhat surprisingly, the overwhelmingly largest result is New York City. I knew NYC had a large start-up community, but I definitely wasn’t expecting it to outrank the Bay Area. Looking at the other high-rated results, several Silicon Valley cities show up. Just out of curiosity, I want to see if the number of jobs within a few miles of San Francisco is larger than NYC.

-- distance from SF: 37.762240, -122.446640
SELECT COUNT(*)
FROM salaries_v 
WHERE location_city IS NOT NULL 
	AND ABS(37.762240 - location_latitude) < 1.5
	AND ABS(-122.446640 - location_longitude) < 1.5;

That makes more sense. There were 80 responses within approximately 20 miles of the SF city center. The Silicon Valley area is comprised of many smaller cities rather than one large city like New York, so it makes sense for SV jobs to have diverse city names.

Conclusion

This post took the survey results at face-value and attempted to draw some conclusions. Overall, I think the data presented here is accurate and valuable, but there are quite a few outlier responses that might have skewed some results. In a future post, I plan to match the data in this survey with larger datasets and some external data sources to verify its integrity and expand on the insights here.

Searching for the Perfect Salad Bar

Mon, 16 Jan 2017 00:00:00 +0000

My fiancée and I are always on the lookout for restaurants with a good salad. We love going out to eat, but hate the calories and fat that typically come along with it. My goal in this post is to find new restaurants with good salads or salad bars in my neighborhood.

Since ranking nearby restaurants immediately brings Yelp to mind and they have a REST API, Yelp is the natural choice for a data source here. To run the extract code below, you’ll need to register an app with the Yelp API. I’ll be using python and pandas DataFrames to manipulate the data.

Extraction

Yelp’s API uses straight-forward oauth2 authentication which is easy to implement with the requests library:

def yelp_login(client_id, client_secret):
    params = {
        'client_id': client_id,
        'client_secret': client_secret,
        'grant_type': 'client_credentials'
    }
    
    resp = requests.post('https://api.yelp.com/oauth2/token', params)
    
    if resp.status_code == 200:
        obj = resp.json()
        if 'access_token' in obj:
            return obj['access_token']

    return None

The two API endpoints I’ll be using use the same pagination technique, so a generic request method isolates the pagination logic and keeps code simpler:

def yelp_request(endpoint, access_token, params, object_name):
    headers = {
        'Authorization': 'Bearer '+access_token
    }
    
    params['limit'] = 50
    params['offset'] = 0
    keep_extracting = True
    
    while keep_extracting:
        resp = requests.get(yelp_api_base + endpoint, headers=headers, params=params)

        if resp.status_code == 200:
            json = resp.json()
            
            if len(json[object_name]) < params['limit']:
                keep_extracting = False
            
            for b in json[object_name]:
                yield b
                time.sleep(2)
            
            params['offset'] += 50
        else:
            print(resp.json())
            keep_extracting = False

The yelp_request method makes business and review extraction straight-forward: I just need to pass the endpoint path and query parameters for each call to the function.

def yelp_search_businesses(access_token, search_term, location, radius=40000, categories=None):
    params = {
        'term': search_term,
        'location': location,
        'radius': radius
    }
    
    if categories:
        params['categories'] = categories
    
    return yelp_request('businesses/search', access_token, params, 'businesses')
    

def yelp_reviews(access_token, business_id):
    return yelp_request('businesses/{id}/reviews'.format(id=business_id), access_token, {}, 'reviews')

Finally, putting it all together to extract businesses matching ‘salad bar’ within the search radius from my ZIP code and adding available reviews:

yelp_api_base = 'https://api.yelp.com/v3/'
access_token = yelp_login(app_id, app_secret)

places = []

for b in yelp_search_businesses(access_token, 'salad bar', '91307'):
	b['reviews'] = []
	for r in yelp_reviews(access_token, b['id']):
		b['reviews'].append(r)

	places.append(json.loads(b))

df = pandas.DataFrame.from_records(places)
print(len(df))

1000

Hmm. Yelp seems to have cut me off after 1,000 records, despite showing more than 8,000 total records in my search. That shouldn’t be a problem for this dataset: since the Yelp API is sorting by adjusted rating, anything below the 1,000^th rank isn’t likely to be very appetizing anyway.

Additionally, the reviews endpoint only returns three results. I’m not sure which three reviews are returned, so they may not be a representative sample of the business’s overall reviews.

Analysis

Cities by Restaurant Rating

First, I’ll group by city to rank the cities in my area by average restaurant rating. The mask call removes results more than 10 kilometers from my home.

cities = (df.mask(lambda x: x['distance'] > 10000)[['city', 'rating']]
            .groupby('city')
            .mean()
            .sort_values('rating', ascending=False))

cities.to_csv('data/the-perfect-salad-bar/cities_by_ranking.csv')

Astute readers may notice that the city with the highest overall rating, Wodland Hills, does not exist. Disregarding that typo, we now know where the highest rated restaurants likely are.

One interesting note: West Hills, the city I live in, came up dead last in rating. This confirms my previous belief that there are few good salad places in West Hills.

Reviews Containing “Salad”

Next, I want to find restaurants where the reviews actually mention salads. It’s usually a good sign when reviewers like the salads enough to mention them in reviews.

salads = (df[['name', 'reviews', 'rating', 'city', 'distance', 'price']]
            .where(df['reviews'].str.contains('salad'))
            .dropna()
            .sort_values('rating', ascending=False))

salads_by_occurrance = (salads
                        .groupby(['name'])['reviews']
                        .apply(lambda x: x[x.str.contains('salad')].count())
                        .sort_values(0, ascending=False))

salads_by_occurrance.to_csv('data/the-perfect-salad-bar/salad_reviews.csv')

The above chart plots restaurants by distance from my location and star rating. The heavily populated 4-star spot about 30 kilometers from West Hills is the trendy Westside LA neighborhood, home to plenty of excellent healthy restaurants.

Final List

Finally, I’ll merge and sort the two DataFrames above to come up with a ranked list of restaurants to try in nearby cities.

cities = pd.DataFrame.from_csv('data/cities_by_ranking.csv', index_col=None)

merged = salads.merge(cities, on='city')

(merged.sort_values(['rating_x', 'price', 'rating_y'], ascending=[False, True, True])[['name', 'city', 'rating_x', 'price']]
       .head(10)
       .to_csv('data/sorted_salad_restaurants.csv', index=False))

Wrap up

The above list satisfies most of the criteria I defined going in to this project: they’re relatively close-by, theoretically have good salads, and I’ve only eaten at one of them.

Overall, the Yelp API is very easy to use and is very fast. Unfortunately, since Yelp is not in the business of giving their data away, its results are perhaps too limited to make this a comprehensive analysis. Some areas to come back and reconsider are:

Investigate and fix rate limiting issue that cut off the extract at 1,000 rows.
Try the reviews API endpoint multiple times to see if it returns random results to build a larger review set.
Incorporate Yelp check-ins to rank similarity to previous places I’ve eaten.

About Everyday Data

Fri, 30 Dec 2016 00:00:00 +0000

I’m Brandon, a Data Engineer at Loot Crate. This blog is my attempt at examining the data we generate (and largely ignore) in our daily lives. I plan to use the projects posted here as a chance to learn new things about data extraction, cleansing, visualizations, and statistics.

Posts on Everyday Data will all follow the same basic format:

Description

What is the data source being examined? How common is it? Does it apply to anyone but me? (Hopefully, yes!) Explain the question being explored and any initial hypotheses.

Extraction

How is the data obtained? This section will typically include code snippets and sample data.

Analysis

Statistics and visualizations go here. Do the visualizations reveal anything unexpected? Explain the methodology behind the visualizations used.

Wrap up

Examine the original question and hypothesis. Was the hypothesis accurate? If not, why not?

Cruise Location Heatmaps

Generating Maps

Static Maps

Royal Caribbean International

Carnival Cruises

Disney Cruises

Animated Maps

Cruise Location Dataset

Extract

Transformation

Analysis

Data Issues

March Madness 2017

How accurate was it?

Games Predicted Accurately

Games Predicted Accurately by Round

The randomness/upset factor

Upsets Predicted Accurately

Next Year

Podcasts Dataset

ETL

Shows extract

Episodes extract

Analysis

Completeness

Basic aggregations

It’s not a data post without some charts

Further work

Hacker News Salary Survey

Extract and Transform

Dimension Tables

Decorate locations

Categorize job titles

Analysis

Conclusion

Searching for the Perfect Salad Bar

Extraction

Analysis

Cities by Restaurant Rating

Reviews Containing “Salad”

Final List

Wrap up

About Everyday Data

Description

Extraction

Analysis

Wrap up

Further reading