Josh Werts

Monkey-patching the ESRI JSAPI to mock services

Fri, 23 Mar 2018 13:41:03 -0400

Lately I’ve been working on a project where I’m unable to access the client’s ArcGIS Server services directly and don’t have the data/environment to publish locally. Working directly on the server is slow and I wanted a way to mock in a few services just to get started on some UI elements. There are some online services that help w/ mocking HTTP calls, but I didn’t really want to put the data out there in the cloud….

So how about some monkey-patching?

Concept

Here’s the basic concept - since nothing is really closed in JavaScript, we can intercept global functions, perform some action and then allow the function to continue as usual. Here’s a simple example - let’s say we want to log the url of all XHR requests to the console:

// place this anonymous closure somewhere in your code before any service calls.
// could be in a script tag in index.html for instance.
(function (open) {

  // REPLACE the open function with a new function
  XMLHttpRequest.prototype.open = function (method, url, async, user, password) {

    // log out the url
    console.log('XHR request to: ', url);

    // after we do our work, call the ORIGINAL open function so we continue normally.
    open.apply(this, arguments);
  };

  // pass the ORIGINAL function in as a parameter.
})(XMLHttpRequest.prototype.open);

Example output in Chrome console:

FeatureLayer (XHR) Example

Some of the JSAPI uses XHR to pass queries. For instance, this is an example of intercepting and mocking out service calls from a JSAPI 4.6 FeatureLayer. In this case, we’re just checking the URL on each call and changing it to our JSON (which is simply previous ArcGIS Server calls saved out into .json files). There are 3 steps here based on how ESRI has implemented the FeatureLayer:

When the FeatureLayer is instantiated, a call is made to the endpoint w/ f=json to retrieve the service’s schema.
At 4.x ESRI performs a returnCountOnly call to get the total count of features.
Then, the API tries to iterate through the objectIds with actual query calls to get all of the features, but we’re just returning a single response. There’s a major gotcha here: Make sure "exceededTransferLimit": false is set to false in the json or the API will just keep iterating indefinitely trying to get all of the features.

(function (open) {
  XMLHttpRequest.prototype.open = function (method, url, async, user, password) {
    if (url === 'https://server/arcgis/rest/services/myservice/MapServer/5?f=json') {
      console.log('open', method, url, async, user, password);
      url = 'https://localhost:8080/app/mocks/ags/my_ags_service_info.json';
    }

    if (url.indexOf('https://server/arcgis/rest/services/myservice/MapServer/5/query') > -1) {
      if (url.indexOf('returnCountOnly') > -1) {
        console.log('open', method, url, async, user, password);
        url = 'https://localhost:8080/app/mocks/ags/my_ags_service_5_values_count.json';
      } else {
        console.log('open', method, url, async, user, password);
        url = 'https://localhost:8080/app/mocks/ags/my_ags_service_5_values.json';
      }
    }
    open.apply(this, arguments);
  };
})(XMLHttpRequest.prototype.open);

QueryTask (jsonp) Example

Other parts of the API use jsonp instead of XHR to handle requests so we can’t intercept via XMLHttpRequest. Here’s a slightly different approach to handle a QueryTask.execute call by patching the class’s execute function itself.

// NOTE: requires here are with a Webpack setup.
import QueryTask = require('esri/tasks/QueryTask');
import Point = require('esri/geometry/Point');
import FeatureSet = require('esri/tasks/support/FeatureSet');
import esriRequest = require('esri/request');

(function (execute) {
  QueryTask.prototype.execute = function (query, requestOptions?) {
    if (this.url.indexOf('https://server/arcgis/rest/services/myservice/MapServer/5?f=json') > -1) {

      // create a new Promise and do our own call to the local mocked json response.
      return new Promise((resolve, reject) => {
        esriRequest('https://localhost:8080/app/mocks/ags/my_features.json').then(response => {
          let featureSet = new FeatureSet(response.data);

          // In some cases geometries weren't correctly created and needed to fix.
          for (let f of featureSet.features) {
            f.geometry = new Point({
              x: f.geometry.x,
              y: f.geometry.y,
              spatialReference: {
                wkid: 4326
              }
            });
          }
          // resolve the promise with our mocked FeatureSet
          resolve(featureSet);
        }, error => reject(error));
      });
    }
  }
  return execute.apply(this, arguments);
})(QueryTask.prototype.execute);

There are probably better ways to accomplish this and I definitely wouldn’t suggest doing anything like this for a production app, but it can still be useful in some cases.

Group by query in a file geodatabase

Tue, 16 Jan 2018 13:51:36 -0500

File geodatabase feature classes and tables lack some of the more advanced ability to query that a true relational database supports. Sometimes these queries can be simulated with arcpy cursors; one good example is a SQL group by query.

Typical SQL:

SELECT field, count(*) from table group by field order by count(*) desc;

In arcpy:

from collections import Counter

def group_by_count(table_or_fc, fields):
    """ Returns dictionary containing count of unique items """
    counter = Counter()
    with arcpy.da.SearchCursor(table_or_fc, fields) as curs:
        for row in curs:
            # no need to store as a tuple if only 1 field, just store the value
            if len(row) == 1:
                row = row[0]
            counter[row] += 1
    return counter


def group_by_count_formatted(table_or_fc, fields):
    """ prints out counts of unique values """
    counter = group_by_count(table_or_fc, fields)
    # sort yields highest count records first (order by count(*) desc)
    for key, count in sorted(counter.items(), reverse=True, key=lambda item: item[1]):
        print("{}: {:,}".format(str(key), count))

Example usage in ArcMap Python console (single field):

>>> group_by_count('junctions', 'ImpedanceType')
Counter({u'SmallStreet': 455145, u'LargeStreet': 28714, u'Stream': 9375, u'RailRoad': 1742})

>>> group_by_count_formatted('junctions', 'ImpedanceType')
SmallStreet: 455,145
LargeStreet: 28,714
Stream: 9,375
RailRoad: 1,742

Example usage in ArcMap Python console (multiple fields):

>>> group_by_count('junctions', ['ImpedanceType', 'InfrastructureType'])
Counter({('SmallStreet', 'New'): 318834, ('LargeStreet', 'New'): 28710, ('Stream', 'New'): 18379, ('Stream', 'Aerial'): 5806, ('RailRoad', 'New'): 4043, ('Stream', 'Underground'): 3227, ('RailRoad', 'Aerial'): 1035, ('RailRoad', 'Underground'): 668})

>>> group_by_count_formatted('junctions', ['ImpedanceType', 'InfrastructureType'])
('SmallStreet', 'New'): 318,834
('LargeStreet', 'New'): 28,710
('Stream', 'New'): 18,379
('Stream', 'Aerial'): 5,806
('RailRoad', 'New'): 4,043
('Stream', 'Underground'): 3,227
('RailRoad', 'Aerial'): 1,035
('RailRoad', 'Underground'): 668

These functions aren’t going to be as efficient as a SQL query, but they can be quite useful sometimes for ad-hoc data exploration - especially in the Arcmap/Pro console.

ESRI Javascript API 4 with Angular 2 - Transition to Webpack

Sun, 08 Jan 2017 14:21:52 -0500

Since my first post on Angular 2 and the ESRI API, I’ve transitioned from SystemJS to Webpack which I’m enjoying thoroughly.

Sample App w/ Webpack setup:
Demo
Repo

IMO, Webpack has a few major advantages over SystemJS:

Development build step is much faster and lite-server does not bog down with memory leaks (not sure why that happened with the SystemJS setup but lite-server got slower and slower until restarted).
Webpack takes care of bundling code for you (everything expect the ESRI requires).
- This setup also bundles each components’ html and css in the main bundle.
Feels like less of a hack to use with esri - simply require in each module and that’s it. Setup is simply specifying ESRI as an “external”. Nothing really special there and it just works.
No need for a grunt/gulp step to minimize, etc. Webpack is pretty comprehensive.

This app is just a map with a custom coordinate display component (bottom left):

I’ve used a similar webpack configuration to this in a couple production apps over the last several months. Unit testing w/ this setup and Typescript is finally starting to feel natural (instead of a large very verbose burden w/ es5). I’ve found it fairly straightfoward to test domain classes and models w/ business logic where I feel you get the most bang for your buck from testing. What I haven’t quite figured out yet is how to test components (and when it’s actually worth the effort). This project contains a working test for the coordinate component… It works, but I’m not yet convinced it’s the best way to go about it. Please comment if you have experience here!

Coordinate Component test:

import { TestBed, ComponentFixture } from '@angular/core/testing';
import { DebugElement } from '@angular/core';

import { CoordinateComponent } from './coordinate.component';

let fixture: ComponentFixture<CoordinateComponent>;
let component: CoordinateComponent;
let de: DebugElement;
let el: HTMLElement;
let $mapDiv;
let mapDivEl: Element;
beforeEach(() => {
  // refine the test module by declaring the test component
  TestBed.configureTestingModule({
    declarations: [CoordinateComponent]
  });

  // create component and test fixture
  fixture = TestBed.createComponent(CoordinateComponent);
  el = fixture.nativeElement;

  // get test component from the fixture
  component = fixture.componentInstance;

  // add a div to the body
  $mapDiv = $('<div>', { id: 'mapDiv' }).appendTo('body');
  mapDivEl = $mapDiv[0];

  // mock required MapView props/functions
  component.mapView = <any>{
    container: mapDivEl,
    toMap: (point) => {
      return {
        longitude: 5,
        latitude: 10
      };
    },
    zoom: 7,
    scale: 500
  };

  // runs ngOnInit
  fixture.detectChanges();
});

describe('Coordinate', () => {
  it('should update lat/long on mousemove', () => {
    mapDivEl.dispatchEvent(new Event('mousemove', {
      bubbles: true,
      cancelable: true
    }));
    // pageX, pageY
    fixture.detectChanges();
    expect(el.querySelector('#longitude').innerHTML).toBe('5.000000');
    expect(el.querySelector('#latitude').innerHTML).toBe('10.000000');
    expect(el.querySelector('#zoom').innerHTML).toBe('7');
    expect(el.querySelector('#scale').innerHTML).toBe('500');
  });
});

FeatureClass from JSON

Fri, 14 Oct 2016 10:07:44 -0400

I keep forgetting how to do this after several months pass by and it’s certainly not obvious…

ESRI provides a tool for converting JSON to FeatureClass but it requires reading the JSON from a .json file: JSON to Features. That extra step of writing out and reading from a file just seems completely unnecessary.

Here’s the alternative lesser known and not obvious way to do it:

valid_featureset_json = \
"""{
    "displayFieldName": "",
    "fieldAliases": {
        "OBJECTID": "OBJECTID",
        "Description": "Description"
    },
    "geometryType": "esriGeometryPoint",
    "spatialReference": {
        "wkid": 4326,
        "latestWkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID"
        },
        {
            "name": "Description",
            "type": "esriFieldTypeString",
            "alias": "Description",
            "length": 50
        }
    ],
    "features": [
        {
            "attributes": {
                "OBJECTID": 1,
                "Description": "This is a test feature."
            },
            "geometry": {
                "x": -123.80816799999991,
                "y": 39.40451500000006
            }
        },
        {
            "attributes": {
                "OBJECTID": 2,
                "Description": "This is aanother test feature."
            },
            "geometry": {
                "x": -123.80816839299996,
                "y": 39.404514814000095
            }
        }
    ]
}"""

# convert to RecordSet object (should be able to use this most places FeatureSet would be called for)
record_set = arcpy.AsShape(valid_featureset_json, True)

# If you need a FeatureClass, just copy into one
output_fc = r'in_memory\\output_fc'
arcpy.management.CopyFeatures(record_set, output_fc)

And that’s it! No need to write the json to file.

Fun with Python Sets (and arcpy)

Thu, 21 Jul 2016 17:38:05 -0400

I often forget about the Python set, so this is a fun reminder to myself to keep it in mind. Here’s a neat example of using a set for deleting fields.

This function deletes all of the fields except the ones you specify:

def delete_all_fields_except(fc, keep_fields):
    """ deletes all fields except those specfied as keep_fields """ 
    # create set of all possible fields but exclude OBJECTID, SHAPE, etc.
    all_fields = set(f.name for f in arcpy.ListFields(fc) if not f.required)

    # get a list of fields in the featureclass but not in the list of fields to keep
    # using set.difference()
    
    delete_fields = list(all_fields.difference(keep_fields))
    
    # delete out all the fields
    arcpy.management.DeleteField(fc, delete_fields)

Usage:

# Assume our feature class has the following fields:
print([f.name for f in arcpy.ListFields(fc)])
# ['OBJECTID', 'Shape', 'field1', 'field2', 'field3', 'field4', 'field5']

keep_fields = ['field1', 'field3']
delete_all_fields_except(fc, keep_fields)

print([f.name for f in arcpy.ListFields(fc)])
# ['OBJECTID', 'Shape', 'field1', 'field3']

This is pretty simple example and you could obviously accomplish this without using set, but set is more elegant and more efficient (probably doesn’t matter much here). Perhaps I’ll update this post with more advanced usage now that this is here to remind me to think about sets!

ESRI Javascript API 4 with Angular 2 and Typescript

Tue, 17 May 2016 20:23:08 -0400

Update 1/8/2017: I’ve transitioned to Webpack since this post: link

This could really be a post about many things.

First, I’m a believer in client-side MV*. Second, I’m trying to incorporate better testing into my front-end JS code. Angular seemed like a natural fit given these 2 primary objectives. I’ve done a fair amount of work now with Angular 1 and the ESRI JSAPI 3.x library. There have been some solid success here, but Dojo require throughout the app and especially in unit tests have been a major headache. You could perhaps mock all dependencies from ESRI in the unit tests or try to keep ESRI in it’s own untested sandbox, but both of these strategies seems better in theory than practice.

Aside: ESRI does not make any of this easy. Sure, you can get it to work… kind of…. with a lot of headaches along the way. It all boils down to Dojo. Don’t get me wrong, the ESRI API does some amazing things and some of the new 4.0 features look incredibly promising (.watch() is brilliant), but there’s really no solid reason why we’re still forced to use Dojo.

Anyway, getting to what this post is really about….

The App

Live Demo

From a high level view, the app simply adds points to the map and shows a list of those points with their geometry and “index” attribute. Nothing too special - just there to test some concepts. As an added bonus, the app is served through lite-server and constantly updates as you code. Similarly, the unit tests run continuously.

Building upon experimental work by Rene Rubalcava and Tom Wayson, I was able to get an Angular 2 app properly loading dojo dependencies both in the browser and in Karma tests.

The app consists of a “model” (Angular service) to hold the points (or domain object of your choosing) - the thought being that business logic in this model would be testable without being concerned about the objects’ relation to the map. Ideally, this model wouldn’t contain ESRI dependencies, but with the JSAPI 4’s watch capabilities, it seems to make sense to use an esri/core/Collection as the underlying data structure in the model. Also ideally, we’d just have the collection in the model, but it seems the only way to achieve the map automagically updating is if we use the points from a GraphicsLayer (passing in our Collection as the graphics property to the GraphicsLayer constructor worked in 4.0beta3, but not in 4.0 final for some reason).

PointsModel (points.model.ts)

So here’s our PointsModel (in Typescript). We’ve simply wrapped a few Collection methods and then we can add some additional business logic like getIndexSum(). index is just a made up attribute for sake of testing.

import { Injectable } from '@angular/core';

import Graphic from 'esri/Graphic';
import GraphicsLayer from 'esri/layers/GraphicsLayer';
import Collection from 'esri/core/Collection';

@Injectable()
export class PointsModel {
  private points: Collection = new Collection();
  pointsLayer: GraphicsLayer;
  constructor() {
    this.pointsLayer = new GraphicsLayer();
    this.points = this.pointsLayer.graphics;
  }
  addPoint(pointGraphic: Graphic) {
    this.points.add(pointGraphic);
  }
  addPoints(pointsGraphics: Graphic[]) {
    this.points.addMany(pointsGraphics);
  }
  getPointGraphics() {
    return this.points;
  }
  clear() {
    this.points.removeAll();
  }
  getIndexSum() {
    let sum = 0;
    if (this.points !== null) {
      this.points.forEach(p => sum += p.attributes.index);
    }
    return sum;
  }
}

AttributeComponent (attribute.component.html)

When we add a point to the model, it not only shows up in the map, but also in the attribute list which is wired up through databinding:

<div>
  <h2>Points!</h2>
  <p>Index Sum: {{pointsModel.getIndexSum()}}
  <ul>
    <li *ngFor="let point of points.toArray()">
      <span>{{point.attributes.index}} ({{point.geometry.x | number:'.5-5'}},{{point.geometry.y | number:'.5-5'}})</span>
    </li>
  </ul>
</div>

MapService (map.service.ts)

Our map binding to our PointsModel (which is just a matter of adding pointsModel.pointsLayer (our GraphicsLayer) to the map):

import { Injectable } from '@angular/core';

import Map from 'esri/Map';
import GraphicsLayer from 'esri/layers/GraphicsLayer';

import { PointsModel } from './points.model';

@Injectable()
export class MapService {
  map: Map;
  pointGraphicsLayer: GraphicsLayer;
  constructor(pointsModel: PointsModel) {
    this.map = new Map({
      basemap: 'topo'
    });
    this.map.add(pointsModel.pointsLayer);
  }
}

PointsModel Tests (points.model.spec.ts)

import { PointsModel } from './points.model';

import Graphic from 'esri/Graphic';
import Point from 'esri/geometry/Point';

describe('PointsModel tests', () => {
  let mockPointGraphic = new Graphic({
    attributes: {
      index: 1
    },
    geometry: new Point({
      x: 1,
      y: 2,
      spatialReference: {
        wkid: 4326
      }
    })
  });

  let pointsModel;
  beforeEach(() => {
    pointsModel = new PointsModel();
  });

  it('should contstruct it', () => {
    expect(pointsModel).toBeDefined();
    expect(pointsModel.getPointGraphics()).toBeDefined();
  });

  describe('adding and removing points', () => {
    it('should add a point to collection', () => {
       pointsModel.addPoint(mockPointGraphic);
       pointsModel.addPoint(mockPointGraphic);
       expect(pointsModel.getPointGraphics().length).toEqual(2);
    });

    it('should add points to collection', () => {
      pointsModel.addPoints([mockPointGraphic, mockPointGraphic]);
      expect(pointsModel.getPointGraphics().length).toEqual(2);
    });

    it('should clear points', () => {
      pointsModel.addPoint(mockPointGraphic);
      pointsModel.addPoint(mockPointGraphic);
      pointsModel.clear();
      expect(pointsModel.getPointGraphics().length).toEqual(0);
    });
  });

  describe('calculations', () => {
    it('should calculate the sum of the index attributes', () => {
      pointsModel.addPoints([mockPointGraphic, mockPointGraphic]);
      let sum = pointsModel.getIndexSum();
      expect(sum).toEqual(2);
    });
  });

});

The nice thing here is that the tests didn’t balk at using the esri Collection that must be imported into the PointsModel (this has been an incredibly difficult thing to do w/ Angular 1 / Dojo).

Setup

Setup for this is non-trivial and is based heavily on esri/esri-system-js along with custom configuration in the Karma configuration and loading of dependencies locally using the esri bower jsapi repo. In short, the esri-system-js loader loads ALL esri dependencies at the start of the application so they’re available through import statements in Typescript files. This was already figured out for the browser portion by Tom and Rene as referenced previously, so check out their repo’s for more information there.

Getting it wired up for testing was a little more difficult but here are the key highlights. Note that I’ve broken out some different configs for browser vs tests.

dojoConfigTest.js

Fairly standard setup for loading from a local bower setup.

(function(window) {
  // set up your dojoConfig
  window.dojoConfig = {
    baseUrl: 'app/node_modules/',
    deps: ['app/main'],
    packages: [
      'app',
      'dijit',
      'dojo',
      'dojox',
      'dstore',
      'dgrid',
      'esri', {
        name: 'moment',
        location: 'moment',
        main: 'moment'
      }
    ]
  };
});

karma.conf.js

These are the changes that were added to the karma.conf.js configuration included in the angular2 quick start repo:

  ...

  files: [
    ... angular files, etc.
    // ********* esri load ***********
    // must be able to serve these files for dojo require
    // NOTE: karma gives a cryptic error when 
    // files can't be found  (msg || "").replace is not a function
    { pattern: 'bower_components/dojo/**/*.*', included: false, watched: false },
    { pattern: 'bower_components/dojox/**/*.*', included: false, watched: false },
    { pattern: 'bower_components/dstore/**/*.*', included: false, watched: false },     
    { pattern: 'bower_components/dgrid/**/*.*', included: false, watched: false },
    
    { pattern: 'bower_components/dijit/**/*.*', included: false, watched: false },
    { pattern: 'bower_components/esri/**/*.*', included: false, watched: false },    
    { pattern: 'bower_components/moment/**/*.js', included: false, watched: false },   

    // load dojoConfig so dojo knows where to "require" modules from
    'dojoConfigTest.js',
    
    // we need the actual dojo startup file for "requrire" to be defined
    'bower_components/dojo/dojo.js',
    
    // load in esri's systemJs util
    'node_modules/esri-system-js/dist/esriSystem.js',
    
    // load in our array of esri dependencies
    'esriLoadConfig.js',
    
    // bootstrap in the modules using esri-system-js
    'esriSystemLoadTest.js', 

    ... more angular files
]

esriLoadConfig.js

Contains ALL esri modules required by the application:

(function(window) {
  window.esriLoadConfig = {
      modules: [
      'esri/Map',
      'esri/views/MapView',
      'esri/core/Collection',
      'esri/layers/GraphicsLayer',
      'esri/Graphic',
      'esri/geometry/Point',
      'esri/geometry/SpatialReference',
      'esri/symbols/SimpleMarkerSymbol',
      'esri/Color'
    ]
  };
}(window))

esriSystemLoadTest.js

Called by Karma to pre-load the esri modules before the tests run.

// load esri modules needed by this application
// into a System.js module called esri
start = performance.now();
esriSystem.register(esriLoadConfig.modules, function () {
  end = performance.now();
  time = end - start;
  console.log('Loaded esri modules', time / 1000.0);
});

esriSystemLoadBrowser.js

Used in the browser code (called from index.html) to load in the esri modules AND bootstrap the application once those modules are available.

// load esri modules needed by this application
// into a System.js module called esri
console.log("Loading esri modules: ", esriLoadConfig.modules);
start = performance.now();
esriSystem.register(esriLoadConfig.modules, function () {
  // then bootstrap application
  end = performance.now();
  time = end - start;
  console.log('Loaded esri modules', time / 1000.0);
  System.import('app/main').then(function () {
    console.log('app/main imported');
  }, function (error) {
    console.log("System import error:", error);
  });
});

Conclusion

~~This is highly experimental and there’s a lot of moving parts, but it’s nice to know this is possible.~~ This is getting closer to production quality. I’ve left out a lot but the full repo’s here:

https://github.com/jwerts/jsapi4-angular2

There will likely be updates to the repo as I continue to explore this concept (and learn Angular 2… and learn Typescript).

It could really benefit from a final build process. Note that I’ve used CDN for most dependencies in index.html for the gh-pages demo to avoid loading local bower and node dependencies to Github.

Edit 2016-06-16: Updated Angular to 2.0.0.RC2
Edit 2016-06-24: Updated Angular to 2.0.0.RC3
Edit 2016-07-05: Updated Angular to 2.0.0.RC4
Edit 2016-07-07: Updated to use esri-system-js 1.0 beta which now preserves esri module names and works correctly with Typescript arcgis-js-api typings.
Edit 2016-09-22: Updated Angular to 2.0.0 final

arcpy: Efficient Geometry Creation

Thu, 04 Feb 2016 15:10:09 -0500

Sometimes every second counts… and even if it doesn’t, it’s still interesting to see the quirks of a familiar library.

It turns out that object creation can be somewhat expensive (especially when you’re talking about Python –> ArcObjects –> COM). With arcpy (and underlying ArcObjects), there are some objects which can be reused to gain some efficiency.

An interesting example is simply creating a polyline from a pair of points.

Example: Creating a 2 point polyline

Simplest form:

In its simplest form, you may write:

def create_line_simple(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    start_point = arcpy.Point(*point1_x_and_y)
    end_point = arcpy.Point(*point2_x_and_y)
    array = arcpy.Array([start_point, end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline

# Usage:
polyline = create_line_simple((-81.60, 36.20), (-81.70, 36.30), arcpy.SpatialReference(4326))

There’s nothing wrong with this code. In fact, if you’re only creating a few polylines, stop here. It’s readable and gets the job done.

A little more efficient:

However, if you’re creating thousands of polylines, some time can be saved by reusing arcpy.Point objects.

# modules scoped private variables
_start_point = arcpy.Point()
_end_point = arcpy.Point()

def create_line_reuse_points(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    array = arcpy.Array([_start_point, _end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline

In this case, we’re creating 2 module scoped points only once and then setting the X and Y properties on those points. The arcpy.Polyline constructor reads X and Y from those points, but it doesn’t maintain a reference to the points. Setting properties on the existing objects is a bit more efficient than creating new objects every time and since references aren’t maintained to those objects, we’re safe from a memory perspective.

Even more efficient:

Why not go ahead and reuse the arcpy.Array as well? Once again, arcpy.Polyline() only reads data from the array and doesn’t maintain a reference. Make sure to removeAll() from array to clean up.

# modules scoped private variables
_start_point = arcpy.Point()
_end_point = arcpy.Point()
_array = arcpy.Array()

def create_line_reuse_points_array(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    _array.add(_start_point)
    _array.add(_end_point)
    polyline = arcpy.Polyline(_array, spatial_ref)
    _array.removeAll()
    return polyline

How much more efficient is this approach?

Here are the results (in seconds) for creating 100,000 polylines with each function (Python 3.4.1 w/ ArcGIS Pro on Core i7-4712HQ):

Create line simple:
0:00:21.071529

Create line reuse points:
0:00:17.813275

Create line reuse points and array:
0:00:16.277035

Is it a huge difference? Not really. But if you have a process that creates a large amount of geometries, it’s worth considering reusing a few objects.

Here’s the full test script to produce the above results:

import arcpy
from datetime import datetime as dt


def time_me(n):
    """ decorator to print total time to run function n number of times """
    def time_me_decorator(f):
        def wrapper(*args):
            start = dt.now()
            for _ in range(n):
                f(*args)
            print(dt.now() - start)
        return wrapper
    return time_me_decorator


REPETITIONS = 100000

######## Simple Case

@time_me(REPETITIONS)
def create_line_simple(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    start_point = arcpy.Point(*point1_x_and_y)
    end_point = arcpy.Point(*point2_x_and_y)
    array = arcpy.Array([start_point, end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline


######## Reuses the point objects

# modules scoped private functions
_start_point = arcpy.Point()
_end_point = arcpy.Point()

@time_me(REPETITIONS)
def create_line_reuse_points(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    array = arcpy.Array([_start_point, _end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline


######## Reuses the point and array objects

# modules scoped private functions
_start_point = arcpy.Point()
_end_point = arcpy.Point()
_array = arcpy.Array()

@time_me(REPETITIONS)
def create_line_reuse_points_array(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    _array.add(_start_point)
    _array.add(_end_point)
    polyline = arcpy.Polyline(_array, spatial_ref)
    _array.removeAll()
    return polyline


# Run our tests
if __name__ == "__main__":
    WGS_84 = arcpy.SpatialReference(4326)
    POINT1 = (-81.674525, 36.216630)
    POINT2 = (-81.675351, 36.213886)

    print("Create line simple:")
    create_line_simple(POINT1, POINT2, WGS_84)
    print("")

    print("Create line reuse points:")
    create_line_reuse_points(POINT1, POINT2, WGS_84)
    print("")

    print("Create line reuse points and array:")
    create_line_reuse_points_array(POINT1, POINT2, WGS_84)

Octopress to Hexo to Hugo

Thu, 10 Dec 2015 18:17:49 -0500

This blog post is really just to say “Wow! Hugo is freakin’ awesome!”

Octopress

Octopress definitely got the job done and introduced me to static blog generators, but I had no real desire to learn Ruby and lately with Windows 10, I’ve booted up to Linux less and less (yes, Windows 10 really is pretty good - if only it had a bash terminal). Installing Ruby on Windows just wasn’t something I wanted to bother with. Also, if I wanted to change the underlying templates, it just seemed more complicated than it really needed to be and I couldn’t find the time or motivation to really bother changing the default template much.

Hexo

Soooooooo… the next logical step seemed to be Hexo. NodeJS - easy enough on Windows and I’ve done a little node development here and there so here we go. Somehow this just seemed a bit kludgy too. It’s a decent enough project, but just didn’t quite feel right.

Hugo

I’ve been looking for excuses to learn Go, so on a whim, googled static blog generators and Go. I’m yet to really learn any Go from using Hugo (other than perusing the code), but still it’s nice to know it’s there if I need to dig through source to figure out what’s up or want to dive deeper and contribute to the project.

At any rate, Hugo just feels right. Not only was I able to migrate my existing blog fairly quickly, but it was not only easy, but also fun to start hacking on an existing theme.

Reasons I love Hugo:

Installation is just a matter of adding a .exe to your PATH and your good to go. Can’t argue with that.
Good documentation - was fairly easy to get started (one hint though: you need to specify a theme or it will seem like it’s failing silently)
Go templates seem mostly straightforward and powerful
SPEED - this wasn’t really a criteria for me since this blog is pretty small but with Hugo’s speed (100ms to generate this blog) along with its built-in live reload, development is easy and fun.

This blog is currently running off a moderately modified Steam theme. Development is fun enough that hopefully it will continue to evolve (in other words, I don’t feel like I’m fighting the framework with Hugo).

I also found these resources especially helpful for migration:

Feature Service applyEdits on Multiple Layers

Tue, 10 Nov 2015 13:09:37 -0500

If you’ve worked with any of the esri web API’s over the last several years, chances are you’ve used a Feature Service to edit an SDE feature class at some point. Using the web API’s, the obvious way to do this would be to construct a FeatureLayer and call its applyEdits method. In reality, this is just a wrapper around the ArcGIS Server REST API’s applyEdits endpoint on an individual service layer.

Example:

https://[host]/arcgis/rest/services/[folder]/[service]/[layerid]/FeatureServer/applyEdits

But what happens if you want to applyEdits to multiple layers within the Feature Service instead of just one layer? You’ve got a few standard API options after constructing multiple feature layers:

Call applyEdits on each FeatureLayer with dojo/promise/all:

// construct FeatureLayers
flayer1 = new FeatureLayer("https://MyServer/arcgis/rest/services/MyServiceFolder/MyService/0");
flayer2 = new FeatureLayer("https://MyServer/arcgis/rest/services/MyServiceFolder/MyService/1");

// call applyEdits
var deferred1 = flayer.applyEdits(adds1, deletes1, updates1);
var deferred2 = flayer.applyEdits(adds2, deletes2, updates2);

// use dojo/All to respond to success or failure when both operations finish
all([deferred1, deferred2]).then(function(results) {
  // Do something with results.
  // Note that you'll still need to check each edit in each layer for "success"
}, function(error) {
  // uh oh, one of them failed.  
});

Chain promises for each FeatureLayer:

// construct FeatureLayers
flayer1 = new FeatureLayer("https://MyServer/arcgis/rest/services/MyServiceFolder/MyService/0");
flayer2 = new FeatureLayer("https://MyServer/arcgis/rest/services/MyServiceFolder/MyService/1");

// call applyEdits
flayer.applyEdits(adds1, deletes1, updates1).then(function(result1) {
  // do something with result
  // Note that you'll still need to check each edit for "success"
  flayer2.applyEdits(adds2, deletes2, updates2).then(function(result2) {
    // do something with result
    // Note that you'll still need to check each edit for "success"
  }, function(error) {
    // flayer 2 error
  });
}, function(error) {
  // flayer 1 error
});

Either of these options would work, but what if the 2 layer edits depended on each other?

Usually when I’m editing multiple layers at once, they have some sort of relation to each other. This may or may not be defined in the database, but at any rate - if one edit fails, I’d really like the other edit to also fail so we don’t end up with an inconsistency in the database. Additionally, wouldn’t it be nice to only send one request to the server, especially when there’s many layers?

One Request to Rule Them All

It turns out that the ArcGIS Server REST API also has an applyEdits endpoint on the Feature Service itself - it’s just not wrapped in the web API’s.

This makes it possible to send multiple edits in a single request AND make sure nothing gets stored on any layers if any one of the layer edits fails.

There’s nothing in the ESRI web API’s that directly hit this endpoint, so I give to you: FeatureService.js.

bower install featureservice-esri-jsapi

Usage:

var package_path = window.location.pathname.substring(0, window.location.pathname.lastIndexOf('/'));
var dojoConfig = {
  async: true,
  packages: [
    {
      name: 'feature-service',
      location: package_path + '/js/lib/bower_components/featureservice-esri-jsapi/dist',
      main: 'FeatureService.min'
    }
  ]
};

require(
  [
    'feature-service'
  ], function(FeatureService) {
    'use strict';

    var service = new FeatureService("https://[server]/arcgis/rest/MyService");
    var edits = [
      {
        id: 2,  // id of layer in service
        adds: [
          addGraphic
        ],
        updates: [
          updateGraphic
        ],
        deletes: [
          1, 2  // just objectids of features to delete.
        ]
      },
      {
        id: 5,  // id of layer in service
        adds: [
          someGraphic,
          anotherGraphic,
          yetAnotherGraphic
        ]
      }
    ];

    var deferred = service.applyEdits(edits);
    deferred.then(function(result) {
      // result is an array
      // each array value contains an object with layer id and objectids of successful edits:
      /*
      [
        {
          id: int id of feature service layer
          adds: [oid, oid, oid, ...],
          updates: [oid, oid oid, ...],
          deletes: [oid, oid, oid, ...]
        },
        ...
      ]
      */
      // layer 2 results
      console.log(result[0].id);
      console.log(result[0].adds);
      console.log(result[0].updates);
      console.log(result[0].deletes);

      // layer 5 results
      console.log(result[1].id);
      console.log(result[1].adds);
      console.log(result[1].updates);
      console.log(result[1].deletes);

    }, function(error) {
      /* error callback can be called for 2 reasons
        1. Regular service faults (server 500 error, etc)
        2. One or more of the edits failed (but server still responded with 200)

        In case 1, the error is a standard esri error object with code and message.
        In case 2, the error is an object with code (200) and message plus an additional
        errors property which contains an array of error objects.
        Error object contains code, description, and id (layer id).
      */
      // case 2
      if (error.code === 200) {
        for (var i; i<error.errors.length; i++) {
          var err = error.errors[i];
          console.log(err.id);
          console.log(err.code);
          console.log(err.description);
        }
      }
    });
});

Currently only the applyEdits endpoint is implemented, but this is really the most useful. There are a couple of personal design decisions here:

The service always sends rollbackOnFailure as true. This seems to be the most likely use case.
If any single edit fails, the fault handler is triggered. This is different from the jsapi’s FeatureLayer, where even if there’s a failure, it will still trigger the success handler and you have to check each result for failures. This class does it for you.

If you find this useful, please leave a comment or feel free to fork!
https://github.com/jwerts/featureservice-esri-jsapi.git

As a side note, this could have been a separate post on “Unit testing esri Javascript API services with Karma/Mocha/Chai/Sinon”. Perhaps in the future…

arcpy.Project in_memory Featureclass

Thu, 10 Sep 2015 00:00:00 +0000

It’s inevitable that you eventually run into this error when scripting with arcpy (arcpy.Project_management): https://help.arcgis.com/en%20/arcgisdesktop/10.0/help/index.html#//00vp0000000m000944.htm

The standard project tool does not support in_memory workspaces.

Here’s the workaround - we just create a new featureclass using the source featureclass as a template and then exploit the spatial_reference parameter of arcpy.da.SearchCursor to project on the fly while inserting into the new featureclass.

Function:

from os.path import split
import arcpy

# create destination feature class using the source as a template to establish schema
# and set destination spatial reference
def project(source_fc, out_projected_fc, spatial_reference):
  """ projects source_fc to out_projected_fc using cursors (supports in_memory workspace) """
  path, name = split(out_projected_fc)
  arcpy.management.CreateFeatureclass(path, name,
                                      arcpy.Describe(source_fc).shapeType,
                                      template=source_fc,
                                      spatial_reference=spatial_reference)

  # specify copy of all fields from source to destination
  fields = ["Shape@"] + [f.name for f in arcpy.ListFields(source_fc) if not f.required]

  # project source geometries on the fly while inserting to destination featureclass
  with arcpy.da.SearchCursor(source_fc, fields, spatial_reference=spatial_reference) as source_curs, \
       arcpy.da.InsertCursor(out_projected_fc, fields) as ins_curs:
      for row in source_curs:
        ins_curs.insertRow(row)

Usage:

# assume we've already created this somewhere
source_fc = r"in_memory/source_fc"

# destination featureclass to be created
out_projected_fc = r"in_memory/projected_source_fc"

# destination projection
web_mercator = arcpy.SpatialReference(102100)

project(source_fc, out_projected_fc, web_mercator)

Edit 2016-02-04: Changed to function per suggestion by Andy Garfield

Debugging Server Object Extensions (SOEs) in Style

Sun, 12 Apr 2015 00:00:00 +0000

Server Object Extensions (SOEs) can be incredibly powerful, but they can also be a pain to debug.

Typical workflow:

Rebuild solution
Open ArcGIS Manager in browser -> Site -> Extensions -> Update Extension by selecting .soe file.
Restart all of the services that depended on the SOE.
Attach Visual Studio to process.

Steps 2 & 3 are the really annoying ones, but they can be automated!

It turns out this is really a post about REST and Python.

When you update an SOE through the ArcGIS Manager app (https://localhost/arcgis/manager), the web application is actually making calls to ArcGIS Server REST admin endpoints.

Automate with Python

I’ll cut to the chase.

See the full script here:
https://github.com/jwerts/soe-updater

Usage (simply run from command line after rebuilding SOE solution):

C:\Projects\_General\_Code\soe-updater>python update_soe.py

Retrieving token...
Retrieved: oUK04q-J8ORWDUrSWGPfq8zAU29u3q5_YZ79ZvcFZx8kFneOMb5Z2Y2Yf19
Uploading SOE...
Uploaded: ibd792bae-a986-4861-8ac3-c16d42f4d610
Updating SOE...
Updated!
Starting services...
Starting service_folder/service.MapServer
Started!

Your SOE will be updated and services restarted. You can re-attach to the service processes or if you’re using System.Diagnostics.Debugger.Launch(); in your Init() function, you’ll automatically be greeted with the dialog to attach your process once the script finishes and the services restart.

Now a little explanation of what’s going on…

REST API Calls

Generate token

Because we’re going to automate this, we need a preliminary step. We’ll generate a token that can be passed to subsequent steps.

https://localhost/arcgis/rest/tokens

Upload the SOE

The .soe file that gets generated after building in Visual Studio is uploaded to ArcGIS Server and returns an itemID.

https://localhost/arcgis/admin/uploads/upload

Update the SOE

Another call references the uploaded .soe through the itemID and updates the extension.

https://localhost/arcgis/admin/services/types/extensions/update

Restart services

After the SOE is updated, all services that use it are stopped and must be restarted. This can also be accomplished through REST calls:

https://localhost/arcgis/admin/services/[service_folder]/[service_name]/start

These calls are all pretty straightford to automate in a Python script using the requests library (https://docs.python-requests.org/en/latest/).

arcpy.da Cursors and Readability Part 2

Sun, 21 Dec 2014 00:00:00 +0000

arcpy.da cursors provide the simplest data structure by default (tuples). Python generators provide a pretty neat way of customizing these cursors to increase readability.

SearchCursor:

Return dict instead of tuple:

def rows_as_dicts(cursor):
  """ returns rows as dicts (does not maintain field order) """
  colnames = cursor.fields
  for row in cursor:
    yield dict(zip(colnames, row))

Usage (note the cursor is wrapped with the generator):

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
  for row in rows_as_dicts(curs):
    total_cost_by_name[row["name"]] = row["unit_cost"] * row["quantity"]

If using with an update cursor, you’ll want to use an OrderedDict so the field/value order is maintained. You’ll also need to the use dict.values() to pass an ordered tuple back into Cursor.updateRow.

UpdateCursor:

Return collections.OrderedDict instead of tuple:

import collections
def rows_as_ordered_dicts(cursor):
  """ returns rows as collections.OrderedDict """
  colnames = cursor.fields
  for row in cursor:
    yield collections.OrderedDict(zip(colnames, row))

Usage (updating):

with arcpy.da.UpdateCursor(costs_table, ["unit_cost", "quantity", "total_cost"]) as curs:
  for row in rows_as_ordered_dicts(curs):
    row["total_cost"] = row["unit_cost"] * row["quantity"]

    # call .values() to convert back to tuple
    # if we didn't use OrderedDict, the values would be in random order.
    curs.updateRow( row.values() )

arcpy.da Cursors and Readability Part 1

Thu, 07 Aug 2014 00:00:00 +0000

The arcpy.da cursors (ie: arcpy.da.SearchCursor) are far and away better than the regular cursors (ie: arcpy.SearchCursor). They’re more pythonic, incredibly flexible, faster, and allow context management (with). However, their straight-out-of-the-box-use may be a little hard to read and maintain.

Fun with Tuples

arcpy.da cursors return tuples (instead of objects that feel too much like writing ArcObjects).

Good:

It’s perfectly valid to use tuple indexes to access the fields:

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
    for row in curs:
        total_cost_by_name[row[0]] = row[1] * row[2]

Better:

This works fine, but the intent of our calculation is not really clear without some mental mapping. How about this?

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
    for row in curs:
        name = row[0]
        unit_cost = row[1]
        quantity = row[2]

        total_cost_by_name[name] = unit_cost * quantity

The intent in the calculation is much more clear here, but we’ve gained a lot of verbosity that is perhaps unnecessary.

Best:

Of course, you could set the variables all on one line name, unit_cost, quantity = row[0], row[1], row[2], but even better would be to take advantage of python’s tuple unpacking: name, unit_cost, quantity = row. Taking that one step further, why not unpack every tuple as it’s iterated over. Now we have a process that is both concise and highly readable:

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
    for name, unit_cost, quantity in curs:
        total_cost_by_name[name] = unit_cost * quantity

Now what if we want to update a total cost field in the featureclass using an UpdateCursor instead?

with arcpy.da.UpdateCursor(costs_table, ["unit_cost", "quantity", "total_cost"]) as curs:
    for unit_cost, quantity, total_cost in curs:
        total_cost = unit_cost * quantity
        curs.updateRow( (unit_cost, quantity, total_cost) )

Python: Recursion with Lists

Sat, 28 Jun 2014 00:00:00 +0000

I rarely find use cases for recursion in my every day work, but every once in a while, there’s a problem that comes up where I instantly think: “Recursion!” (typically followed by an equal mix of excitement and dread).

I often try to get away from GIS related programming when I’m not at work. With limited free time, Python’s the perfect language to focus on fundamentals without getting bogged down with other concerns.

Recursion is challenging enough already but I find it even more difficult to wrap my head around when the end result is intended to be a list. I love nested functions (and closure) in Python (as long as it doesn’t get out of hand) and think it works well in this case to provide the expected simple interface to a function despite the fact that recursion is being used internally.

Most of the code I write for work is proprietary and can’t be posted here, so here’s a classic problem I’ve been playing around with simply for learning purposes.

Code:

def prime_factors(x):
    """ returns all prime factors of x
    ex:  prime_factors(21) = [3,7]
    ex:  prime_factors(24) = [2,2,2,3]
    """

    def get_first_prime(y):
        """ returns first prime of y """
        # 0 and 1 are not prime, start with 2
        i = 2

        while i <= y:
            if y % i == 0:
                return i
            i += 1

    def get_primes(y, primes):
        """ Recursively appends primes of y
            ex:  prime_factors(24) = [2,2,2,3]

            24
           /  \
          2    12
              /  \
             2    6
                 / \
                2   3

        Appends end nodes of tree as it recurses down.
        """
        first_prime = get_first_prime(y)
        primes.append(first_prime)

        if y == first_prime:
            return

        else:
            # Number is not prime and can be divided further.
            # Divide by the first prime (left node)
            # to get the next test value (right node)
            get_primes(y / first_prime, primes)


    primes = []
    if x >= 2:
        get_primes(x, primes)

    return primes

Usage:

>>> prime_factors(24)
[2, 2, 2, 3]
>>> prime_factors(13595)
[5, 7, 13, 29]
>>> prime_factors(23049820394)
[2, 89, 139, 181, 5147]

The list primes is passed by reference into the recursive function and simply appended to throughout the operation to keep track of the values. There is no actual return value from the function.

This could be difficult to follow if the end user used: get_primes(y, primes) -> void directly but since it’s wrapped in a simpler interface: prime_factors(x) -> primes, the end result is a returned list as expected.

arcpy: Reorder Fields

Thu, 17 Apr 2014 00:00:00 +0000

Here’s another handy function for re-ordering fields in a feature class. Like the rename_fields function previously posted, it recreates the existing field mappings and modifies as necessary. The output is a new feature class with fields in the order specified.

The only caveat is that required fields always get pushed to the front (so the featureclass will start with OBJECTID, Shape, rest of fields…). After required fields are out of the way, the rest of the fields are added in the order specified in the field_order list. If any fields are missing, they are added to the end of the feature class.

import arcpy

def reorder_fields(table, out_table, field_order, add_missing=True):
    """
    Reorders fields in input featureclass/table
    :table:         input table (fc, table, layer, etc)
    :out_table:     output table (fc, table, layer, etc)
    :field_order:   order of fields (objectid, shape not necessary)
    :add_missing:   add missing fields to end if True (leave out if False)
    -> path to output table
    """
    existing_fields = arcpy.ListFields(table)
    existing_field_names = [field.name for field in existing_fields]

    existing_mapping = arcpy.FieldMappings()
    existing_mapping.addTable(table)

    new_mapping = arcpy.FieldMappings()

    def add_mapping(field_name):
        mapping_index = existing_mapping.findFieldMapIndex(field_name)

        # required fields (OBJECTID, etc) will not be in existing mappings
        # they are added automatically
        if mapping_index != -1:
            field_map = existing_mapping.fieldMappings[mapping_index]
            new_mapping.addFieldMap(field_map)

    # add user fields from field_order
    for field_name in field_order:
        if field_name not in existing_field_names:
            raise Exception("Field: {0} not in {1}".format(field_name, table))

        add_mapping(field_name)

    # add missing fields at end
    if add_missing:
        missing_fields = [f for f in existing_field_names if f not in field_order]
        for field_name in missing_fields:
            add_mapping(field_name)

    # use merge with single input just to use new field_mappings
    arcpy.Merge_management(table, out_table, new_mapping)
    return out_table

Usage:

new_field_order = ["field2", "field3", "field1"]
reorder_fields(in_fc, out_fc, new_field_order)

I’ve tested this in 10.1.1+ - it may work in 10.0 as well and should work with any license level.