I’m liking Leaflet. It’s got all the great flexibility I love in Polymaps, quickly compositing various layers of raster and vector data into one fancy map. But unlike Polymaps it runs well on iOS devices. And it’s got an active community; Polymaps feels like a dead end.

A few things are missing:

There’s no way to specify the z-ordering of layers, it’s all just in whatever order they are. The developers have said they’re working on this. (Z ordering basically doesn’t exist in SVG, it’s all document order. That’s dumb.)

Scaling transitions aren’t beautiful. Leaflet animates the raster layer but hides my vectors and then has them blink back into place. Bah.

Screenshot of work in progress. Lots going on here… Purple is steps. Orange squiggles are my walks. Faded = older walks. Redder = longer walks. There’s some interaction, too. Click on a track to highlight it and get a popup showing length and age.

A blog post I wrote on a foreign laptop ended up with byte values 0x92 for apostrophes and 0x93 and 0x94 for quotes. This is an old Windows “smart quotes” and makes sense in Windows code page 1252. It is not valid ISO-Latin-1; that encoding doesn’t have typographical quote marks in it.

Easy enough to fix once the encoding is identified: “iconv -f windows-1252 -t utf-8”

I should make a website that has one web page per code point. So you could go to a page named “0x92.html” and see what all it translates to in the world’s various codecs. There’s a bunch of one-page-per-Unicode-character sites out there, but I don’t know of a resource that lets you easily guess a mystery encoding by a few examples.

I wanted to download 1300+ images from a Dropbox gallery. Unfortunately Dropbox’s UI is so dynamic with Javascript there’s no simple list of URLs. So instead of trying to reverse engineer their code I opened up the Javascript console after the page had loaded to extract the dynamically generated HTML links.

 

l = document.querySelectorAll("a.filename-link")
a = []
for (i = 0; i < l.length; i++) { a.push(l[i].href); }
for (i = 0; i < a.length; i++) { console.log(a[i]); }

Pretty straightforward, but a lot more tinkering during the process. Those URLs lead to the image page; to get to the actual JPG you have to append “?dl=1” to the URL.

I keep playing with this dataset of all the steps of San Francisco from OpenStreetMap. I made a GeoJSON file with all the steps of San Francisco. It’s a snap to add to a Web map with Leaflet. A few thoughts:

GeoJSON is a good format for geodata when you’re working cross-language. PostGIS can emit it, Python and Javascript can easily parse it, Leaflet can easily draw it. Half the time I just get a string from PostGIS and never even parse it in Python, just pass it along.

The Python geojson library doesn’t really help very much. If you’re using geopy already to work with geodata, then maybe it’s helpful. But if you’re just building up arbitrary JSON blobs with some geodata it’s not needed.

The world could use a GeoJSON validator. Extra credit if it rendered the GeoJSON blob on a map, wouldn’t be too hard. I should set this up as a web service.

JSON float precision continues to contribute to file bloat; Python’s default 15 digits precision translates to subatomic location positioning. Fortunately in Python 2.7 it’s pretty easy to round off numbers, you just have to do it.

The GeoJSON format I settled on was a FeatureCollection. Each feature has the name, length (in metres), and OSM ID of one set of steps. Then two geometries: the centroid of the steps and the path of the steps. Example:

{   "type": "Feature",
    "properties": {
        "id": 8916413,
        "length": 146.58,
        "name": "Harry Street"
    },
    "geometry": {
        "type": "GeometryCollection",
            "geometries": [
            {   "type": "Point",
                "coordinates": [ -122.431219, 37.740099 ],
            },
            {   "type": "LineString",
                "coordinates": [
                    [ -122.430741, 37.740424 ],
                    [ -122.43075,  37.740351 ],
                    [ -122.431759, 37.739803 ]]
            }
]}}

Here’s the SQL query for all the steps I’m using. There’s more Python code than I’d like to collect all these rows and emit the GeoJSON, but I’m not pasting it all here.

select
    planet_osm_line.name,
    ST_Length(planet_osm_line.way) as meters,
    ST_AsGeoJSON(ST_Transform(ST_Centroid(planet_osm_line.way),4326), 6),
    ST_AsGeoJSON(ST_Transform(planet_osm_line.way, 4326), 6),
    planet_osm_line.osm_id
from planet_osm_line
    inner join planet_osm_polygon
    on ST_Intersects(planet_osm_line.way, planet_osm_polygon.way)
where
    planet_osm_polygon.osm_id = '-111968'
    and planet_osm_line.highway = 'steps'
order by meters desc;

I’m exploring Leaflet as an alternative to Polymaps, mostly because Leaflet has some momentum and is faster to render on iOS. here’s a quick demo of drawing a raster map with a GeoJSON overlay.

var map = new L.Map('map');
var rasterLayer = new L.TileLayer('http://{s}.tile.stamen.com/terrain/{z}/{x}/{y}.jpg', {
    attribution: 'Stamen Terrain',
    maxZoom: 18
});
var view = new L.LatLng(37.75, -122.43); // geographical point (longitude and latitude)
var trackLayer = new L.GeoJSON(track);
map.setView(view, 14);
map.addLayer(rasterLayer);
map.addLayer(trackLayer);

“track” was preloaded as a JSONP script. ogr2ogr barfs on converting my GPX files from Runkeeper directly to GeoJSON, so I exported them out of PostGIS instead with

select ST_AsGeoJSON(wkb_geometry)
from tracks
where name like '%4/21/12%'

Leaflet uses SVG for the GeoJSON layer. Pretty simple structure, svg > g > path, I think one per GPS track.

With just a bit more work it was easy to load 65 walking tracks via Ajax (165k data total) and plot them all in a browser. Performance is pretty good on iOS; the main thing I don’t like is the way the GeoJSON layers disappear during a resize.

I hacked up a KML of all my geotagged tweets. Here’s a map view.

My whole work environment is UTF-8. Except Python. Python’s “print” will encode in UTF-8 if its printing directly to a tty. But as soon as you pipe that tty to something Python reverts to ASCII. And it’s a real PITA to overcome that.

Here’s one ugly kludge to make Python always treat stdout as UTF-8. It causes problems with some tools, but is sufficient for hacking around:

import sys, codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)

I moved a road. OpenStreetMap had one path for an obscure little road in Nevada City, CA called Woodwardia Place. I was curious about the exact path so I drove down it twice with a GPS logger, then used JOSM to upload a change. It’s cool how fast the upload took effect; the raster tiles for the map were generated anew within 30 seconds of my upload. JOSM is a bit confusing, and I never could figure out how to upload the GPX track for OSM’s records.

Here’s the edit; old road is in grey, new road is in yellow moved slightly north.

Then I split the road. There’s a segment that’s a bridge, so I made a change that split the road into two ways and then tagged the short bridge segment with bridge=yes.

Finally I went to a different part of the map and removed a road segment. I’d driven blind down that road expecting it to go through, but the map was wrong. Never again! Removing a road segment is a bit tricky; I had to select the node on the western (non-existent) T intersection, “unglue it”, delete the incorrect line segments, then “merge nodes” to reglue the now-T intersection that does exist on the map.

I have a bunch of GPX tracks generated by iSmoothRun, a GPS tracker for the iPhone that tries to correct for location error by using the accelerometer as a pedometer. The tracks it produces are pretty lumpy too, so I’m trying to use ST_Simplify afterwards to clean them up.

ST_Simplify takes as a second parameter the “tolerance”, how much to smooth. This parameter is not well documented but it seems to be measured in the units of the geometry you are simplifying. For my walking tracks 0.00005 seems to be about right. If that were degrees, that’d correspond to about 15 – 20′ in San Francisco.

Here’s a quicky chart of the average percentage difference between the original track length and the simplified track length, for different tolerances.

0.00001   0.1%
0.00003   0.7%
0.00005   1.8%
0.00007   2.8%
0.0001    4.4%
0.001    19.9%

0.00005 seems to be a reasonable compromise. It produces tracks with about 1/3 the number of points as the original. Below is a picture of the original track and the simplified drawn on top of it, to show the geometry change.

I wish I understood SRIDs more intuitively, I feel like I’m bashing about. Half my data is WGS84 and half is Google Mercator, and then below I’m using SRID 2163 to do the distance calculation. That’s a bit of cargo cult coding, I think it’s NAD83 which is close enough to WGS84 to get away with.

-- Create a simplified view of an ogr2ogr imported track
create or replace view simple_tracks as
select
ogc_fid,
ST_Simplify(wkb_geometry,0.0001) as wkb_geometry,
name
from tracks;

-- Display the difference between original and simplified
select
  miles-smiles as difference,
  round((miles-smiles)/miles*100) as pctlength,
  round(spoints*100/points) as pctpoints,
  * from
(select
tracks.name,
ST_Length(ST_Transform(simple_tracks.wkb_geometry, 2163)) /1609 as smiles,
ST_Length(ST_Transform(tracks.wkb_geometry, 2163)) /1609 as miles,
ST_NPoints(simple_tracks.wkb_geometry) as spoints,
ST_NPoints(tracks.wkb_geometry) as points
from simple_tracks inner join tracks
on simple_tracks.name = tracks.name order by name) as m;

-- Calculate just the average difference
select avg((miles-smiles)/miles*100) from
(select
  tracks.name,
  ST_Length(ST_Transform(simple_tracks.wkb_geometry, 2163)) /1609 as smiles,
  ST_Length(ST_Transform(tracks.wkb_geometry, 2163)) /1609 as miles
from simple_tracks inner join tracks
on simple_tracks.name = tracks.name order by name) as m
where miles > 1;

I imported 60 different GPX tracks from RunKeeper into my database, using this hack. I’ve been dorking around with the data using TileMill as a sort of GIS app; it’s nicer than QGIS! But it’s silly for me to try to draw my own complete map from OSM data, as fun as that is, so I’ll probably switch to some GeoJSON overlay on top of a slippy map with nice terrain tiles next. Anyway, here’s a map of just the walks and then parks in SF.

I like the way the transparency + the GPS noise naturally shows streets I walk on more frequently.