I’m new to Javascript hackery, my little aviation weather hack had me doing a lot of new stuff with Ajax and dynamic HTML. So I asked my friend Mike for a code review. He’s the smartest Javascript hacker I know, and we both worked at Google where rigorous code reviews were a thing.

Here’s some complicated things I learned from him:

1. There’s ways to create private namespaces for libraries and classes. Douglas Crockford’s “Private Members in JavaScript” is a good introduction, some examples from Mike are kmeans.js and crimespotting.js. The key thing is creating a closure to scope the private stuff, then assigning the public functions to a module object. There’s potential per-instance overhead to doing it this way.
2. Reassigning the CSS class of a DOM element at runtime is totally OK. In jQuery, it’s like doing foo.attr("class", "someStyle"). It’s a handy way to make the document suddenly look different, but it made me nervous.
3. Editing the CSS style of a DOM element is also OK. In jQuery, it’s doing foo.css("font-weight", "bold"). Presumably it’s using element.style.setProperty() to do the manipulation.
4. jQuery.ajax() gives different types types depending on what it thinks the response is. XML gives back a DOM object, HTML gives back a string, JSON gives back a Javascript blob. My code is getting HTML and then uses the magic jQuery(htmlString) capability to parse the HTML into a DOM and then query it. (BTW, turning strings into DOM is not easy in Javascript.)

Also some simple things:

1. Javascript has Array.join() just like Python. Handy to avoid lots of String concatenation.
2. Not clear that testing if (foo === 3) is worth typing the extra =. There’s circumstances where it saves you, but in practice may not be a big deal.
3. Array.push() can take multiple arguments, to avoid repeated calls
4. return "" + x is an easy way to ensure the return type is always a string
5. Be sure to use “var” for all local variables. This is rookie Javascript, but I still forget.

I learned a lot. Code reviews are a great way to teach and learn, it’d be nice to have an online community that offered them along the lines of StackOverflow or github. Both for basic education, but also helping clean up important open source libraries. Key tech piece: a really good browser based code annotation tool.

Now as a favour in kind I’m going to try porting my code from using jQuery to using his D3 library. D3 is more of a visualization library than a jQuery replacement, but it happens to have the selectors and Ajax support I need.

Inspired by my playing with Chrome Web Apps, I sat down and read up on two of the ways HTML 5 allows Javascript to use local storage.

Offline Web Applications let you build webapps that work even when the user has no network connection. The key thing is you create a “cache manifest” file which contains a list of the files that should be cached locally. If the user is offline (or if the site is down?) then the browser will use the cached version. There’s complicated extra stuff for fallbacks and wildcards that allow for webapps to opportunistically cache just the stuff the user saw for some other reason. And there’s DOM events to let the webapp know what’s going on with caching and network status.

Local Storage is a variety of technologies for a webapp to read and write data on the local hard drive. The main new one is the localStorage object, a persistent Array. Basically a key:value store with String being the only datatype. Apps get 5 megabytes of local store. There’s also some work being done on SQL and object databases, but nothing finished yet. Miraculously, local storage seems to work in every browser, even IE 8 and above.

Over on my main blog I just posted What’s a Chrome Web App? Long story short, it’s basically either a bookmark or a Chrome extension.

As part of my research I turned my aviation weather experiment into a Chrome Application. It took about 15 minutes, and most of that was resizing the icon. Here’s the mainfest.json file that makes it go:

{ "name": "AvWx",
  "description": "Experimental aviation weather page",
  "version": "1",
  "app": { "launch": {
      "web_url": "http://www.somebits.com/avwx/static.html",
      "container": "panel",
      "width": 720,
      "height": 640
  } },
  "icons": { "128": "avwx-128.png" } }

The most interesting part of that is the “container: panel”. That tells Chrome to launch it without any navigation UI, so it looks like a standalone application. I could also be asking for extended privileges, like accessing geolocation or delivering notifications.

I bought a Chumby. An “Insignia”, to be specific, from Best Buy for sale $50. Totally on a whim, but I bought two with the idea the other is a gift. What’s a Chumby? It’s an Internet-enabled alarm clock. Really it’s a tiny little hacker-friendly embedded system (Linux?) with WiFi, a 320×240 touchscreen, and a small form factor. It’s a toy.

Pretty bad user experience out of the box. Here’s the horror so far on first use. My goal was to see what time it is.

• Plug it in, first thing you do is calibrate the touchscreen. Despite my dragging the little circle into the ring (visually), it doesn’t seem to register until the third try.
• Second thing you do is set it up on your WiFi. It doesn’t see any networks. I type my SSID manually. It’s still lost. I type my WEP key in manually on the terrible touchscreen, unsure if it is even the kind of key it’s looking for. Press connect, 60 seconds later a timeout and no joy. Set up a second wifi network with no password. Chumby can’t find it either, even manually. Go online, read support, try rebooting the device. It finds the network! It’s online! Including a charmingly nerdy screen full of info about MAC addresses, Tx packet counts, etc. Lol.
• Eager to move along my one page quick start manual, I find I’m not being asked to set the time. Which makes sense; it’s online afterall, but then why does the manual say it should?
• Asked to register my device online. Have to create yet another @#*$#$ website account, with another username that’s unique, with another password, with another email verification step, with another “click here to not get spam” choice. I’m tired of Internet accounts. Fortunately, the device pairing goes smoothly, including a nice little AJAX notification in the browser.
• Start the device. Maybe I can see a clock now? No, I get some headline scroll of TUAW feeds, including the word “RSS” prominently featured in the Chumby UI.
• Stuck, not sure how to get to the clock, I randomly hit buttons. Suddenly presented with the option to upgrade the Chumby. OK, sure, and now for 5 minutes I’m being admonished “Do not disconnect the power at any time.”
• Chumby finally reboots. I’m treated to a scrolling slideshow of apps I don’t want. What time is it?!
• Find the button on top of the device (not the touchscreen), which lets me choose apps. Scroll through “featured apps” to 10 of 16, which is a clock. Find it’s 15 minutes after I started, then the app disappears.
• Sometime later my Chumby starts scrolling apps again, including one to a page with ads selling me more crap from the manufacturer. Charming.

I think I’ll be setting up the other Chumby at home before presenting it as a gift. I like to troll my Apple-loving friends with how pretentious, closed, and expensive Apple devices are. But I’ll give Apple this, they have an excellent first-time user experience. OTOH their equivalent device, the iPod Touch, is 5x as expensive.

Snark aside, it’s not a bad device for $50. The main UI is “scroll through a bunch of Flash apps”. That doesn’t work for me, so I’ve set it to run a single app, the Flickr + clock app. It’s also able to play Internet radio streams in the background. That’s pretty slick.

About once a month someone links to my blog and I get all excited to see how many clicks they drove. Despite 10+ years of Javascript tracking systems like Google Analytics, I still don’t have a proper tool to just quickly show me “what’s driving traffic today?”. I usually cobble together some hideous thing that starts awk { print $7, $11 } | grep -v ... and then get annoyed. So today I wrote a proper Python script:

#!/usr/bin/python

import itertools, re

skipRE = re.compile(r'(js|png|gif|jpg|css|rss091|atom|ico)$')
def skip(url, referer):
    return skipRE.search(url) is not None or \
        referer == '-' or \
        referer.startswith('http://www.somebits.com')

for l in itertools.chain(file("/var/log/apache2/access.log.1"), file("/var/log/apache2/access.log")):
    d = l.split()
    url = d[6]
    referer = d[10].strip('"')
    if not skip(url, referer):
        print url, referer

The output is then piped through a Python script I wrote years ago which counts up the most common lines.

One remaining mystery; the large majority of my pageviews have a referer of -, or empty. I know folks aren’t pasting URLs in by hand, where does this come from? Do redirectors somehow clear Referer?

No hacking; that time got spent playing Minecraft instead. It’s an amazing indie game and while I’m mostly done with it, today my friends had a one-day PvP event on the server that was sort of interesting. It doesn’t make for a very good PvP game, honestly.

In a bit of a yak shaving exercise, I’m looking for a really simple blog post writer that’s more convenient than using wordpress.com’s web editor. Windows Live Writer works with wordpress.com, how good is its HTML?

$(data).find('option[value^="ruc"],option[value^=eta]').each(
    function(i, e) {
        var timeString = $(e).text();
        var filename = $(e).attr("value");
        r[timeString] = filename;
    });

There’s no support for <pre> tags, but I found a plugin that does it. (Update: not suitable for wordpress.com, since it requires Javascript. Use wordpress.com’s builtin tag instead) The ribbon UI is pretty bad and there’s a lot of junk between me and my thoughts. But it’s not terrible. Here’s a picture:

Yesterday’s hack was finishing up AvWx, a quick Javascript thing I wrote to show me forecast data useful for flight planning. It’s an iPhone friendly web page that shows forecast maps for the next 3 days at 1800Z. 3 images for general weather (from Wunderground) and 3 images for winds (from ADDS). This is the quick and dirty data I look at when planning a multiday cross-country flight.

Like many of my recent hacks most of the work is done client side, in Javascript. Unfortunately I had to have a server-side CGI to work around the stupid same origin policy. There’s a couple of new-to-me Javascript tricks in there.

One is I’m using jQuery and AJAX to fetch a form off of ADDS and parse it to extract the filenames for the wind maps. The image naming was complicated enough it was better just to use the HTML. I’d done AJAX stuff before but always with XML or JSON data, never HTML. It’s not so bad:

    $(data).find('option[value^="ruc"],option[value^=eta]').each(
        function(i, e) {
            var timeString = $(e).text();
            var filename = $(e).attr("value");
            r[timeString] = filename;
        });

The other trick I did was scaling and cropping the wind images client-side. ADDS returns a 680×680 image but I want to scale it to 640 wide, then slice about 450 pixels out of the middle of it to display. I did this with the “CSS background cropping” technique; basically you create an empty div and set its background image property to your image, then use CSS to position, scale, and crop. Total hack, but it looks good in Chrome and the iPhone.

.etacrop {
      display: block;  width: 680px;  height: 452px;
      background-size: 640px 640px;
      background-repeat: no-repeat;
      background-position: 0px -86px;
   }
   target.css("background-image",
      "url(http://aviationweather.gov/adds/data/winds/" + winds[i] + ")");

There was more work involved, but that’s the nut of it. I ended up writing a pretty comprehensive QUnit test suite for the parsing and filename munging. Felt like overkill, but already has paid off in letting me test and improve the code.

Today’s hacking project is getting up to speed with modern GIS. Specifically, PostGIS (Postgres with spatial features) as a database, QGis as a graphical frontend for viewing graphics data, and GeoDjango for building webapps.

QGis is awesome: you can quickly add layers, including layers that come from PostGIS. Loaded layers are on the tray to the left, you can right click them to set display options, enable a query to filter what’s displayed, etc. There’s also an edit mode for editing geometry.

PostGIS is encouraging. Some notes:

• Tutorial: Getting to know PostGIS has you setting up the database in part 1, then inserting useful shapefile data in part 2. Some of the details are out of date.
• Tutorial: Getting Started With PostGIS: An almost Idiot’s Guide is more Windows oriented, but looked useful.
• Manual: Create a spatially-enabled database tells you the extra incantations required to add GIS capability to a Postgres database. It’s a little funky, you have to enable PL/pgSQL language and then load some predefined functions.
• Manual: reference to all functions and types.

After finishing all the setup I loaded the cities and counties file referenced in that first tutorial (at USGS) It was easy to load them up into QGis for display. “Show me county outlines, and the names and locations of all cities with more than 100,000 residents”.

Playing in PostGIS, here’s some interesting queries:

select county, ST_Area(wkb_geometry) as area from us_counties
where state = 'CA'  order by area desc limit 5;
Biggest county. Units of Area are.. degrees?
                       county                       |       area
----------------------------------------------------+------------------
 San Bernardino County                              | 5.13534296766738
 Inyo County                                        | 2.66716021408502
 Kern County                                        | 2.09407653001836
 Riverside County                                   | 1.84319135357509
 Siskiyou County                                    | 1.77534424474288

select county, ST_AsText(ST_Simplify(wkb_geometry, 0.1))
  from us_counties where state = 'CA'  limit 1;
(returns a simplified polygon of the county outline)

\d us_counties
shows all the columns. The interesting one is
  wkb_geometry | geometry
Geometry is a polymorphic type, in the case of this shapefile it's a Polygon. The "WKB"
name is just a descriptive, but implies the import source was in "well known binary" format. 

select name, ST_X(wkb_geometry), ST_Y(wkb_geometry)
  from us_cities where name like 'San %' and state = 'CA';
 San Anselmo                                      | -122.561653137207 | 37.9746437072754
 San Rafael                                       | -122.531089782715 | 37.9735336303711
 San Quentin                                      | -122.487464904785 | 37.9412689208984

select name, state,
  floor(ST_Distance_Sphere(wkb_geometry, ST_GeomFromText('POINT(-122.4522 37.7555)'))) as distance_from_SF
  from us_cities where state='CA' order by distance_from_sf   limit 5;

                       name                       | state | distance_from_sf
--------------------------------------------------+-------+------------------
 San Francisco                                    | CA    |             3602
 Daly City                                        | CA    |             5596
 Broadmoor                                        | CA    |             8207
 Colma                                            | CA    |             8767
 Brisbane                                         | CA    |             9495


select county from us_counties where
  ST_Contains(wkb_geometry, ST_GeomFromText('POINT(-122.4522 37.7555)'));

shows which county contains an arbitrary lat/lon

select county, ST_Area(wkb_geometry) as area from us_counties where state = ‘CA’  order by area desc limit 5;
county                       |       area
—————————————————-+——————
San Bernardino County                              | 5.13534296766738
Inyo County                                        | 2.66716021408502
Kern County                                        | 2.09407653001836
Riverside County                                   | 1.84319135357509
Siskiyou County                                    | 1.77534424474288

select county, ST_Area(wkb_geometry) as area from us_counties where state = ‘CA’  order by area desc limit 5;
county                       |       area
—————————————————-+——————
San Bernardino County                              | 5.13534296766738
Inyo County                                        | 2.66716021408502
Kern County                                        | 2.09407653001836
Riverside County                                   | 1.84319135357509
Siskiyou County                                    | 1.77534424474288

I’ve been working with Adam Fast to build a Python library to parse FADDS data. FADDS data is a US aviation database from the FAA, it contains things like locations of airports, nav fixes, etc. The primary branch is at https://github.com/adamfast/faddsdata, my fork is at https://github.com/NelsonMinar/faddsdata. So far we’re parsing two of the files (APT.txt and NATFIX.txt), adding new files is pretty easy.