Polymaps has the ability to easily pull in raster tiles from various sources to make Slippy maps. I have code that lets me use tiles made by Google, Cloudmade, Bing, GDAL’s tile script, even Minecraft. I have no idea how it works, though. The underlying thing I don’t understand is Polymaps’ internal coordinate representation. Here’s an example, for a map tile centered on San Francisco.

• Polymaps coordinate:
  column: 327, row: 791, zoom: 11
  key: “11/327/791”
  x: -186.5, y: -273.5
• Google terrain: (327,791), zoom 11
  http://mt0.google.com/vt/lyrs=t@126,r@145&hl=en&x=327&y=791&z=11
• TMS, the Tile Map Service: (327,1256), zoom 11
  http://example.com/tiles/11/1256/327.jpg
• QuadTree (Bing?): 02301020333
  http://ecn.t3.tiles.virtualearth.net/tiles/r02301020333?g=637&mkt=en-us&lbl=l1&stl=h&shading=hill&n=z

This tells me Polymaps is natively using Google’s coordinate system. TMS is similar but numbered differently, here’s my cargo cult code to work with it. Mike tells me 1 << c.zoom is the number of tiles at the given zoom level, which explains why it’s used as a modulus everywhere.

    // Set X to c.column, but wrapped around in the range [0, numTiles)
    var x = c.column % (1 << c.zoom);
    if (x < 0) {
      x = x + (1 << c.zoom)
    }
    // Set Y to numTiles - c.row, to account for TMS doing lat backwards
    var y = (1 << c.zoom) - c.row - 1;
    var url = "http://example.com/tiles/" + c.zoom + "/" + y + "/" + x + ".jpg";

More info: interactive coordinate exploration tool, Google docs, OSM / TMS docs.

Below is the result of my map experiments: the track of a flight with Ken flying practice approaches at Salinas overlayed on top of two approach charts and the Google Terrain base layer. It’s a bit of a jumble.

• Transparent brown line: flight track
• Black scratchy lines: approach chart
• Brown terrain contours: approach chart
• Green terrain: Google maps

What I’m really trying to highlight is the juxtaposition of the GPS track to the black approach path on the procedure plate. The other stuff is context, or noise. It’s less confusing if you only draw one approach chart, not two. Making parts of the approach plate transparent is a big help too; it’d be nice to eliminate the terrain, etc entirely. Maybe just let the black lines come through.

I updated Quantum GIS to 1.6.0. It claims some improvements for georeferencing. I also learned better how it works.

Transformation Types are documented in the Quantum GIS manual. The docs suggest Polynomial 3 or Helmert are the best choices. Polynomial 3 is a nonstarter since it requires at least 10 GCPs and Helmert produced an oddly rotated image that was not correct.  Linear, Polynomial 1, Projective, and Thin Plate Spline all produced very similar results. Linear isn’t even a proper transform, really, but it seems close enough for my purposes. I’m shooting in the dark here, not knowing how the source NACO plates are projected, but at this scale and for my purposes any errors are not too big a deal.

The Target SRS you set in the Georeferencer is just an annotation, it doesn’t magically cause some warping. I’d tried setting it to EPSG:900913 for the Google (scale: 1 = 1m or something) but was typing latitude and longitudes in the tool (scale: 1 = 1 degree). Works better if I set the Target SRS to EPSG:4326 for WGS 84, produce a GeoTIFF, then later warp the image to Google mercator.

Raster / Warp will let you do the warping to Google. All it’s doing is calling gdalwarp, something like gdalwarp -s_srs EPSG:4326 -t_srs EPSG:900913 -r cubicspline -of GTiff input.tif output.tif. I wonder if this is really even warping anything or it’s just rescaling?

Bottom line:

1. Draw 4 or so georeferencing control points on the chart. Type latitude and longitude.
2. Set Transformation Settings to Polynomial 1, Cubic Spline, LZW, EPSG:4326.
3. Warp the image to Google mercator

QGIS has a bunch of Raster tools where it calls GDAL. GDAL invocations were failing. I think I tracked it down with help from the IRC channel: it wasn’t finding the GDAL binary path. Setting it in the Settings makes stuff work. Details at http://trac.osgeo.org/qgis/ticket/3476

I’m taking a crack at quick and dirty geo-referencing of NACO approach plates for pilots. They come as PDF images, ungeoreferenced, but they contain points with precise latitude and longitude you can align by hand. The result won’t be accurate, certainly not for navigation, but maybe it’s good enough. Here’s some notes on what I’ve tried.

1. Download a PDF plate from NACO.
2. Convert it to PNG via pdftoppm -r 300 | pnmtopng -comp 9. The result file will be 1613×2476, about 700k.
3. Fire up Quantum GIS (I’m using 1.5.0). Launch Plugins/Georeferencer/Georeferencer.
4. File /Open Raster to load your PNG.
5. Zoom in and add some georeference points via Edit / Add Point. I picked a few intermediate GPS fixes on a GPS approach and looked up their coordinates via fltplan.com. Don’t forget that X is longitude, and is negative. How many points? 2 is minimum, I entered 4. It all depends on what projection the NACO plates uses, and I don’t know what that is.
6. File / Save GCP Points, to save your hand edited work.
7. Settings / Transformation Settings. Lots of options I don’t understand. Target SRS is EPSG:4326 for WGS84. LZW compression gets the filesize from 12 megs to 2 megs.
8. Look at the table of points below the view of your map. The last 3 columns (dX, dY, residual) are some sort of measurement of the error / warping requirements.
9. File / Start georeferencing to generate the georeferenced TIFF output. Takes a few seconds.
10. gdalinfo on the resulting .tif file to see whether the result looks reasonable.
11. Close the georeferencer plugin.
12. Start a new Quantum GIS project. Settings / Project Properties to set the CRS to EPSG:4326.
13. Layer / Add Raster Layer to load the georeferenced TIFF. It should show up in the window, with coordinates of lat/long and a scale in the vicinity of 1:300,000.
14. Add more layers to juxtapose other data on top of the georeferenced plate. State boundaries are helpful for getting oriented, I also overlayed a GPS flight track.
15. The NACO plate background is RGB 236, 236, 236. You can make that transparent with the Transparency properties of the plate layer, adding a band for just that one RGB value with 100% transparent.

One thing I haven’t figured out how to do in Quantum GIS is draw my plates on top of some reasonable map, like Google terrain maps. I can load Google maps via the OpenLayers plugin but the coordinate system those maps use is something screwy with X/Y in the hundreds of thousands. I’m not even sure Quantum GIS will warp raster layers on the fly, nor have I figured out what target SRS I should be emitting to be in the same format.

My real goal is to end up loading this stuff in Javascript, with Polymaps.

I got a Spot for Christmas. In addition to emergency use they have a tracking option where it records your position via satellite every 10 minutes, live. The website is awful and they don’t keep your data very long, but the data is there. And they have an API of sorts. Related search: [findmespot completexml]. Some notes:

• You can get an XML view of a public shared page by going to a URL like http://share.findmespot.com/messageService/guestlinkservlet?glId=NONCE&completeXml=true. The NONCE is the same blob that’s in the glId tag in the public shared page.
• Data comes in a custom XML format. It’s pretty simple, each message is an XML blob with a latitude, longitude, and timestamp (in some random timezone!). Also some other junk.
• The data is cached and only updated every 15 minutes. I’ve been told there’s more latency than that.
• The shared page only shows data for the last 7 days.
• Spot keeps your data for only 30 days. There’s also a manual download option.
• Spot has a fancy site for hosting trip diaries at http://www.spotadventures.com. It includes KML and GPX downloads, photos, notes, etc. It looks like a pretty complex and featureful product with very few users.

I haven’t found any sort of hacker community for Spot. There is at least one third party service, though: Spotwalla.

Sort of related: Spot’s new system includes a “Type & Send” feature where you can send arbitrary 41 character messages via the satellite network. They charge $0.10/message in bulk.

Back in 1993 I ran a Usenet server. Usenet’s mostly dead now but there’s a lively trade on  Usenet still in pirated media, particularly pornography. The challenge is posting a 1 gigabyte .AVI file in a medium that’s limited to small text files. Here’s some of the tech in use these days:

• nzb, a metadata file for downloading a large file spread across multiple posts. It serves a function analagous to .torrent files in BitTorrent; doesn’t contain any content, but describes content. There are Usenet search engines that return NZB files, you can then take one to download the actual release off your favourite NNTP server.
  An NZB file is an XML document that describes a set of files which, when combined, make the release. Each file is described by a Usenet group and a list of segments. Each segment is a single Usenet message, named by Message ID. (Note: you can’t create the NZB file until after all the message parts have been uploaded).
• rar, a compression format. It’s like zip, but tighter. Rar also does multipart well.
• par2, an error correcting code format. Sort of the opposite of compression; transmit redundant data so a large archive can be rebuilt even if a few little pieces are missing.
• uuencode, the historical encoding of binary data for Usenet. It’s a base 64 encoding with some very primitive file wrapper stuff. Still in use.
• yEnc, the modern encoding of binary data for Usenet. It’s a full 8 bit encoding with escaping for a few characters (NULL, LF, CR, =). It also has a simple file wrapper and the ability to split a file amongst multiple messages. yEnc seems like MIME-lite, not sure what problem it solves that MIME can’t do.
• sfv files are checksums on other files, used to verify your parts.
• nfo files are text files (well, ansi graphics), typically a README from the release group.

I took a close look at a single NZB for a 300MB AVI file. Here’s what it contains:

• 15 files: an SFV, a PAR2, 6 .RAR parts, and 7 PAR2 pieces.
• Each RAR part is 50 megs and consists of 201 segments of length 249600.
• The encoding makes each RAR file 70 megs on the server. Last chunk is smaller, total RAR text size is 410MB.
• The PAR2 chunks total another 40MB.
• It’s roughly 450MB to post a 300MB AVI to Usenet, or 50% overhead.

Here’s some Windows software I found useful:

• GrabIt, a free NNTP client that’s good at downloading large binaries. The client is free and will download any NZB you give it, but they really want you to pay $25/year to use their NZB search engine.
• QuickPar takes a PAR fileset and assembles the single file out of its components.
• YEnc Power Post seems to be what people posting to Usenet use, but I can’t find a canonical linkf or it I trust.

What I’m missing is a reliable, free NZB search engine. There’s a variety of for-pay options of varying levels of sleaziness. binsearch.info is promising.

I want short names to refer to a specific latitude/longitude. Mostly so I can type them easily into an aviation GPS. For my purposes I need accuracy of about 1 mile, so 1 minute or .01 degree. I only need coordinates valid in the continental US, bounded roughly by 20, -125 to 50, -65. Some options, with the encoding for San Francisco (roughly at 37.75, -122.34).

• 377512243. Lat/long base 10. 9 characters per location, and dead-ass easy to understand.
• 9q8yvv. Geohash, a hierarchical base 32 encoding. 6 characters per location. See here for visualization.
• KO46D. “Grid Fixes” from FltPlan. 5 characters per location. Can’t find documentation but it seems to have a resolution of 10 minutes latitude, 1 degree longitude. Not really accurate enough.
• CM87SR. 6 characters per location. Grid Squares, aka Maidenhead Locator System, a ham radio thing, here’s a map view. Accurate to 2.5′ latitude and 5′ longitude, or roughly 4 miles, so not good enough for my purposes. There is an 8 character extended locator option.
• ?????. Perfect rectangular coding, base 32. 5 characters per location. Need (50-20)*60 = 1800 = 11 bits of latitude, and (125-65)*60 = 3600 = 12 bits of longitude. 23 bits can be encoded in 5 characters in a base 32 representation. A base 52 representation would get it down to 4 characters.

I think if I were actually building this I’d probably go for Geohash. It has a nice property that you can specify arbitrary precision, then lop characters off the right to make a shorter code that’s less specific.

It’s been a long time since I’ve written C++ code. Or C code, for that matter. Like before STL, before g++ really did templates right, before namespaces. I sort of miss the drama of C hacking, but I sure don’t miss all the stupid stuff about C.

A friend of mine wrote a brand new Minecraft map renderer, in C++. It’s really fast. But compiling it was such a mess. g++ 4.4 apparently moved some include files around, so I had to manually include stdint.h in a few places. Why? So you can get an honest-to-goodness 32 bit wide int reliably, since of course C still assumes int is some random width that happens to be the default in that version of the compiler on that kernel. Even with stdint.h included the compiler still couldn’t pick up a macro from inside it. I never could figure out why, I finally just copypastaed the macro into the code. Awesome.

Then there’s linking. What a bizarre thing to have to do with code. The nice thing about linking is you can also link -static, and get a binary that really will run pretty much on any compatible system. Java can do the equivalent reasonably well, but Python is a total nightmare for distributing portable code.

At least I haven’t seen any segmentation faults in the running code yet. Nor buffer overruns. Nor typecast problems. Ah, C, your edges are so sharp!

Let’s say I make some great SRTM altitude tiles, where can I host them that I don’t go broke? Map tiles are funny, they’re just static files so you don’t need a big server, just cheap bandwidth. I did a quick look around at some options:

• Amazon S3 / CDN: $150/mo = 1TB, no server
• Google AppEngine: $120/mo = 1TB. Limitations may make it impossible.
• oneandone.net: $150/mo = 3TB with a server. SkyVector is here.
• godaddy: $135/mo = 1.5TB with a server. Used by PDFPlates? NACOmatic was serving 20TB / month for awhile, not sure how that worked out
• Voxel: $100/mo = 1TB. FlightAware is here.
• one guy I know uses GoDaddy, hostgator, justhost, and linode to push about 2TB / month. HostGator supposedly cut him off without notification at some point.
• I heard ReadyHosting shut down a 20 TB / month project
• random “unmetered” hosting: the challenge is to find the one that’s actually unmetered and doesn’t have hidden throttles or termination policies.

I don’t know much about hosting. Static map tiles are a weird kind of thing to host.

Update: chunkhost offers $66/mo = 3.2TB with a decent Xen slice.