Updating docs for Terrain v2 by iandees · Pull Request #151 · tilezen/joerd

iandees · 2017-07-26T20:54:07Z

Add new section for Footprints downloads / map preview under Data Sources page
Parity with vector tile docs
- Add HTTP Status Codes: https://github.com/tilezen/vector-datasource/blob/master/docs/http-status-codes.md
  - Update Mapzen docs generator config to list this new file
- Add in explanation about z, x, y coords from: https://github.com/tilezen/vector-datasource/blob/master/docs/use-service.md#specify-z-x-and-y-tile-coordinates to fix [docs] Add info about tile coordinates #109.
Update attribution.md
Please also make reference to new sources in this "what-is-sourced-at-what-zooms" table
We also need to update this date note: https://github.com/tilezen/joerd/blob/master/docs/data-sources.md#data-updates
{256,512,260,516} 512 and buffered sizes are also now supported for normal and terrarium formats when access from Mapzen's hosted service (but not direct from S3 bucket). https://github.com/tilezen/joerd/blob/master/docs/formats.md#types-of-terrain-tiles
Update general Mapzen docs about Terrain Tiles and what software is used (no longer Joerd, though we use Joerd to track the work, there's an issue tracking splitting that up):
https://mapzen.com/documentation/overview/rate-limits/#mapzen-terrain-tiles

iandees · 2017-07-26T20:54:33Z

I started off with listing the new sources based on the stuff listed as completed in https://github.com/tilezen/joerd/projects/1.

nvkelso · 2017-07-26T21:37:11Z


 The underlying data sources are a mix of:

+- [CDEM](http://geogratis.gc.ca/api/en/nrcan-rncan/ess-sst/c40acfba-c722-4be1-862e-146b80be738e.html) (Canadian Digital Elevation Model) in Canada, with variable spatial resolution (from 20-400 meters) depending on the latitude.


Please make this list alphabetical.

Sounds good! It seemed to be ordered by resolution, so I was going with that. Alpha it is!

nvkelso · 2017-07-26T23:58:27Z


+These source images are composited to form tiles that make up the Mapzen Terrain Tiles service. To determine exactly which images contributed to Mapzen Terrain Tiles in a particular area, you can download the footprints database and use it with a GIS program like [QGIS](http://www.qgis.org/) to inspect which of these sources were used.
+
+To further assist in determining which sources contributed to an individual tile, each tile in the service has an `X-Imagery-Sources` header listing the sources that contributed to the tile. Each entry in the comma-separated list follows the pattern `source/filename.tif`.


To cut down on support requests, please provide an example with say srtm and and ned and how to x-walk between the tables elsewhere in this page and the footprint database to determine things like source resolution versus tile pixel size.

@nvkelso I added an example below this paragraph, but it's not clear to me if I touched on what you were getting at.

iandees · 2017-07-27T17:54:43Z

@nvkelso want me to track down license and attribution info and add to the attribution doc, too?

nvkelso · 2017-07-27T19:41:29Z

want me to track down license and attribution info and add to the attribution doc, too?

@iandees Yes, please.

nvkelso · 2017-07-27T19:42:38Z

+These source images are composited to form tiles that make up the Mapzen Terrain Tiles service. To determine exactly which images contributed to Mapzen Terrain Tiles in a particular area, you can download the footprints database and use it with a GIS program like [QGIS](http://www.qgis.org/) to inspect which of these sources were used.
+
+* [PostgreSQL Dump](https://example.com) (8.4MB, gzipped)
+* [GeoJSON](https://example.com) (5.3MB, gzipped)


List GeoJSON first (before PostgreSQL Dump) since it can open directly in QGIS listed in prior paragraph.

nvkelso · 2017-07-27T19:44:36Z

+* [GeoJSON](https://example.com) (5.3MB, gzipped)
+
+### Source headers
+


Consider linking to explanation of HTTP headers, like https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

nvkelso · 2017-07-27T21:41:22Z

+
+[Released](https://data.linz.govt.nz/layer/1768-nz-8m-digital-elevation-model-2012/) by [LINZ](https://data.linz.govt.nz/), licensed under [Creative Commons Attribution 3.0 New Zealand](https://data.linz.govt.nz/license/attribution-3-0-new-zealand/).
+
+Attribution statement: Copyright 2011 Crown copyright (c) Land Information New Zealand and the New Zealand Government. All rights reserved


These should follow the format, wrapped to ~80 char: Attribution statement header on one line, then quoted markdown text for the actual statement.

Attribution statement:

Copyright 2011 Crown copyright (c) Land Information New Zealand and the New Zealand Government. All rights reserved

None of the other statements follow this wrapped format. Do you want me to update those, too?

More specifically, the 80-char wrapping isn't there.

nvkelso · 2017-07-27T21:42:13Z

 - [3DEP](http://nationalmap.gov/elevation.html) (formerly NED) in the United States, 10 meters outside of Alaska, 3 meter in select land and territorial water areas
- [SRTM](https://lta.cr.usgs.gov/SRTM) globally except high latitudes, 30 meters in land areas
- [GMTED](http://topotools.cr.usgs.gov/gmted_viewer/) globally, coarser resolutions at 7.5", 15", and 30" in land areas
+- [ArcticDEM](http://nga.maps.arcgis.com/apps/MapSeries/index.html?appid=cf2fba21df7540fb981f8836f2a97e25) strips of 5 meter mosaics across the majority of the land north of 60° latitude


Mention affected countries and callout Alaska here.

Nice graphic for posterity from here:

nvkelso · 2017-07-27T21:43:24Z

+- [data.gov.uk](http://environment.data.gov.uk/ds/survey/index.jsp#/survey), 2 meters over most of the United Kingdom
+- [data.gv.at](https://www.data.gv.at/katalog/dataset/b5de6975-417b-4320-afdb-eb2a9e2a1dbf), 10 meters over Austria
 - [ETOPO1](https://www.ngdc.noaa.gov/mgg/global/global.html) to fill in ocean bathymetry at 1'
+- [EUDEM](https://www.eea.europa.eu/data-and-maps/data/eu-dem#tab-original-data) in most of Europe at 30 meter resolution


Besides saying generally "Europe", please also list all affected countries. I know it's a long list, but someone will be searching for their country name.

I noticed that the UK was listed as being covered in the EUDEM metadata even though that wasn't in our footprints database, so I re-ran the UK-coverage tile and added it back 👍 .

nvkelso · 2017-07-27T21:46:26Z

Please also make reference to new sources in this "what-is-sourced-at-what-zooms" table:
https://github.com/tilezen/joerd/blob/master/docs/data-sources.md#what-is-sourced-at-what-zooms

Doesn't need to be full list, but at least point people over to the full sources list.

(This area is also where a live slippy map could be embedded.) deferred

nvkelso · 2017-07-27T21:47:02Z

We also need to update this date note:
https://github.com/tilezen/joerd/blob/master/docs/data-sources.md#data-updates

Terrain tiles were built during 2016Q2 and 2016Q3 based on available sources at that time. Regular updates are not planned.

nvkelso · 2017-07-27T21:49:05Z

{256,512,260,516} 512 and buffered sizes are also now supported for normal and terrarium formats when access from Mapzen's hosted service (but not direct from S3 bucket).
https://github.com/tilezen/joerd/blob/master/docs/formats.md#types-of-terrain-tiles

nvkelso · 2017-07-28T22:21:25Z

+### data.gv.at Digitales Geländemodell (DGM) Österreich
+
+[Released](https://www.data.gv.at/katalog/dataset/b5de6975-417b-4320-afdb-eb2a9e2a1dbf) by [data.gv.at](https://www.data.gv.at/), licensed under [Creative Commons Namensnennung 3.0 Österreich](https://creativecommons.org/licenses/by/3.0/at/deed.de)
+


data.gv.at needs attribution statement.

I couldn't find one on their website… I'll look a bit more.

I don't see a specific statement they require as attribution. The (English translation of the) license says:

You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

All the other "attribution statements" in this document are verbatim from the license or metadata, so it's odd to generate one ourselves and include it.

Our job is to make sense of the non-sense ;) I recommend:

Attribution statement: © offene Daten Österreichs – Digitales Geländemodell (DGM) Österreich.

Fixed in a02e0ed.

nvkelso · 2017-07-28T22:23:28Z

+
+These source images are composited to form tiles that make up the Mapzen Terrain Tiles service. To determine exactly which images contributed to Mapzen Terrain Tiles in a particular area, you can download the footprints database and use it with a GIS program like [QGIS](http://www.qgis.org/) to inspect which of these sources were used.
+
+* [GeoJSON](https://example.com) (5.3MB, gzipped)


These need real URLs.

nvkelso · 2017-11-07T23:01:51Z

-More details can be found on the ETOPO1 [about](https://www.ngdc.noaa.gov/mgg/global/global.html) page and [data sources](https://www.ngdc.noaa.gov/mgg/global/etopo1sources.html) page, and [description](https://www.ngdc.noaa.gov/docucomp/page?xml=NOAA/NESDIS/NGDC/MGG/DEM/iso/xml/316.xml&view=getDataView&header=none) page and [disclaimer](https://www.ngdc.noaa.gov/ngdcinfo/privacy.html#disclaimer) page.
+### data.gov.uk LIDAR Composite Digital Terrain Model
+
+[Released](https://data.gov.uk/dataset/lidar-composite-dtm-2m1) by [data.gov.uk](https://data.gov.uk/), licensed under [Open Government License version 3](http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/)


licensed under the United Kingdom [Open...

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:02:33Z

+
+### data.gv.at Digitales Geländemodell (DGM) Österreich
+
+[Released](https://www.data.gv.at/katalog/dataset/b5de6975-417b-4320-afdb-eb2a9e2a1dbf) by [data.gv.at](https://www.data.gv.at/), licensed under [Creative Commons Namensnennung 3.0 Österreich](https://creativecommons.org/licenses/by/3.0/at/deed.de)


by Austria's [data.gv...

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:02:54Z

+
+### data.kartverket.no Digital terrengmodell
+
+[Released](http://data.kartverket.no/download/content/digital-terrengmodell-10-m-utm-33) by Kartverket, [licensed](http://www.kartverket.no/data/lisens/) under [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/deed.no).


by Norway's Kartverket

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:03:33Z

+
+### National Institute of Statistics and Geography (INEGI)
+
+[Released]() under [INEGI's "free use of information" license](http://en.www.inegi.org.mx/inegi/terminos.html).


Released is missing a link. Either add one or remove the link markdown.

Released under Mexico's

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:06:04Z

@@ -20,9 +20,7 @@ Attribution needs to "appear in a place that is reasonable to the medium or mean

 3DEP (formerly NED) are several public domain datasets released by USGS:


Since we have multiple 3DEPs we should list those out a bit here? I'm thinking of the topobathy that is in California, for instance.

Or that could be added to the data sources page. Yeah, probably better there since it's also next to the footprints DB download.

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:07:40Z


 - [3DEP](http://nationalmap.gov/elevation.html) (formerly NED) in the United States, 10 meters outside of Alaska, 3 meter in select land and territorial water areas
- [SRTM](https://lta.cr.usgs.gov/SRTM) globally except high latitudes, 30 meters in land areas
+- [ArcticDEM](http://nga.maps.arcgis.com/apps/MapSeries/index.html?appid=cf2fba21df7540fb981f8836f2a97e25) strips of 5 meter mosaics across all of the land north of 60° latitude, including Alaska, Canada, Greenland, Iceland, Norway, and Russia


Sweden is also covered by ArcticDEM?

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:10:31Z

 ## Data updates

-Terrain tiles were built during 2016Q2 and 2016Q3 based on available sources at that time. Regular updates are not planned.
+Terrain tiles version 1 was built during 2016Q2 and 2016Q3 based on available sources at that time. Version 1.1 was built during 2017Q3. Regular updates are not planned.


Version 1.1 was built during 2017Q3 and released 2017Q4.

Terrain tiles version 1 was built during 2016Q2 and released in 2016Q3.

Fixed in 455a9c6.

nvkelso · 2017-11-07T23:32:14Z

@@ -34,6 +34,23 @@ Here's a sample tile in Normal format:
 http://tile.mapzen.com/mapzen/terrain/v1/normal/11/330/790.png?api_key=your-mapzen-api-key
 ```


We should add a note in the Requesting tiles section that all releases in the 1.x series are available in the v1/ file path, minor update versioning isn't available via the Mapzen API... something something.

I wonder if we should not mention 1.1 at all and instead say that we updated it rather than confuse with a minor version number?

"data pack 2017". You're probably right.

nvkelso · 2017-11-13T20:53:55Z

Oh, we still need to update in the attribution.md:

Required attribution:

Required attribution when using Mapzen's hosted service:

And I think vector tiles has something like this to cover things that don't require attribution but we should thank anyhow (like ArcticDEM):

Optional attribution when using Mapzen's hosted service:

nvkelso · 2017-11-13T20:55:56Z

In data-sources.md:

SRTM globally except high latitudes, 30 meters in land areas

We need to indicate this is 30 meters but 90 meter nominal quality. to also explain why the map below says so much of the world is 90 meters when it's the 30 meter SRTM data.

iandees · 2017-11-13T20:57:57Z

we still need to update in the attribution.md

What should be updated? Is this supposed to include all the attribution statements from the CC-BY sources? What is different about this listing than the "when using Mapzen hosted service" other than including Mapzen?

iandees · 2017-11-13T21:00:51Z

We need to indicate this is 30 meters but 90 meter nominal quality

Fixed in b6d1081.

nvkelso · 2017-11-14T04:38:22Z

Regarding #151 (comment), the reference docs are:

https://github.com/tilezen/vector-datasource/blob/master/docs/attribution.md

***Required attribution:***

> © OpenStreetMap contributors, Who's On First

***Required attribution for Mapzen's hosted service:***

> Mapzen, © OpenStreetMap contributors, Who's On First

***Optional full attribution including all data projects:***

> Mapzen, © OpenStreetMap contributors, Who's On First, Natural Earth, and openstreetmapdata.com

… gif of same

nvkelso · 2017-11-14T18:58:26Z

Added * to 90m legend item in footprints preview image in 992c877. Ties in with Ian's earlier edit in b6d1081.

nvkelso

This is ready to merge. We can take care of the final 2 tasks once it's in master in the docs generator PR.

Adding doc entries for new v2 sources

e743220

nvkelso reviewed Jul 26, 2017

View reviewed changes

iandees added 2 commits July 26, 2017 18:00

Alphabetize the sources list

838b793

Add some text discussing the footprints and imagery-sources header

c580aae

nvkelso reviewed Jul 26, 2017

View reviewed changes

Add details about using source headers and footprints dump

903bcb8

nvkelso reviewed Jul 27, 2017

View reviewed changes

List geojson first

75eb39f

nvkelso reviewed Jul 27, 2017

View reviewed changes

iandees added 4 commits July 27, 2017 15:48

Link to a Wikipedia article about HTTP response headers

abbc7ba

Add information about new sources to attribution doc

b8e7f37

Add http status codes doc

1b88ce7

Add a little documentation about tile coordinates

d2ab596

nvkelso reviewed Jul 27, 2017

View reviewed changes

iandees added 6 commits July 28, 2017 08:06

Move attribution statements into code blocks

82238d0

Add country list to ArcticDEM

b4cd288

List the countries covered by EUDEM

273bc1b

Add INEGI/Mexico attribution

b692e64

Add more details to data sources per zoom

498d5b1

Update the date statement

4c1ae9a

nvkelso reviewed Jul 28, 2017

View reviewed changes

iandees added 4 commits November 6, 2017 16:58

Point to the unversioned footprints dumps

7ddd875

Use the phrase 'NED Topobathy' instead of '3DEP' in California

414d347

Remove the v2/ prefix from S3 URLs

b7eb8c5

Add sentence about S3 versioning on the prod tiles bucket

5c159d3

nvkelso reviewed Nov 7, 2017

View reviewed changes

iandees added 3 commits November 7, 2017 18:16

Improvements from review comments

455a9c6

Add attribution and row in data sources for Australia

e1e90e7

Use a more recent, better-looking footprints screenshot.

83453d3

Mention 90-meter 'quality' of SRTM

b6d1081

nvkelso added 3 commits November 14, 2017 10:56

ignore macOS trash

2ee27eb

add * to 90m legend item

992c877

add assets to recreate footprints preview image (QGIS, PSD), animated…

0b7f57a

… gif of same

nvkelso added 2 commits November 14, 2017 11:16

add newer sources to attribution statements

c8fcd92

formatting

d8f587b

nvkelso approved these changes Nov 14, 2017

View reviewed changes

nvkelso merged commit 121b592 into master Nov 14, 2017

nvkelso deleted the iandees/v2-docs branch November 14, 2017 19:26


		The underlying data sources are a mix of:

		- [CDEM](http://geogratis.gc.ca/api/en/nrcan-rncan/ess-sst/c40acfba-c722-4be1-862e-146b80be738e.html) (Canadian Digital Elevation Model) in Canada, with variable spatial resolution (from 20-400 meters) depending on the latitude.


		These source images are composited to form tiles that make up the Mapzen Terrain Tiles service. To determine exactly which images contributed to Mapzen Terrain Tiles in a particular area, you can download the footprints database and use it with a GIS program like [QGIS](http://www.qgis.org/) to inspect which of these sources were used.

		To further assist in determining which sources contributed to an individual tile, each tile in the service has an `X-Imagery-Sources` header listing the sources that contributed to the tile. Each entry in the comma-separated list follows the pattern `source/filename.tif`.

		* [GeoJSON](https://example.com) (5.3MB, gzipped)

		### Source headers


		[Released](https://data.linz.govt.nz/layer/1768-nz-8m-digital-elevation-model-2012/) by [LINZ](https://data.linz.govt.nz/), licensed under [Creative Commons Attribution 3.0 New Zealand](https://data.linz.govt.nz/license/attribution-3-0-new-zealand/).

		Attribution statement: Copyright 2011 Crown copyright (c) Land Information New Zealand and the New Zealand Government. All rights reserved

		### data.gv.at Digitales Geländemodell (DGM) Österreich

		[Released](https://www.data.gv.at/katalog/dataset/b5de6975-417b-4320-afdb-eb2a9e2a1dbf) by [data.gv.at](https://www.data.gv.at/), licensed under [Creative Commons Namensnennung 3.0 Österreich](https://creativecommons.org/licenses/by/3.0/at/deed.de)


		### data.kartverket.no Digital terrengmodell

		[Released](http://data.kartverket.no/download/content/digital-terrengmodell-10-m-utm-33) by Kartverket, [licensed](http://www.kartverket.no/data/lisens/) under [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/deed.no).


		### National Institute of Statistics and Geography (INEGI)

		[Released]() under [INEGI's "free use of information" license](http://en.www.inegi.org.mx/inegi/terminos.html).

		@@ -20,9 +20,7 @@ Attribution needs to "appear in a place that is reasonable to the medium or mean

		3DEP (formerly NED) are several public domain datasets released by USGS:

		@@ -34,6 +34,23 @@ Here's a sample tile in Normal format:
		http://tile.mapzen.com/mapzen/terrain/v1/normal/11/330/790.png?api_key=your-mapzen-api-key
		```

Conversation

iandees commented Jul 26, 2017 • edited by nvkelso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iandees commented Jul 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iandees commented Jul 27, 2017

Uh oh!

nvkelso commented Jul 27, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvkelso commented Jul 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvkelso commented Jul 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvkelso commented Jul 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvkelso Jul 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvkelso Jul 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

iandees commented Jul 26, 2017 •

edited by nvkelso

Loading

nvkelso commented Jul 27, 2017 •

edited

Loading

nvkelso commented Jul 27, 2017 •

edited

Loading

nvkelso commented Jul 27, 2017 •

edited

Loading

nvkelso Jul 28, 2017 •

edited

Loading

nvkelso Jul 28, 2017 •

edited

Loading