[sample data] update web log geo.src field to match country code of geo.coordinates by nreese · Pull Request #110885 · elastic/kibana

nreese · 2021-09-01T20:03:45Z

Web logs sample data has inconsistencies between geo.src, geo.dest and geo. coordinates where in most cases geo. coordinates is not contained in either country denoted by geo.src or geo.dest. This inconsistency makes understanding the web logs samples impossible and diminishes the usefulness of web logs dashboard and maps.

When web logs were originally generated, geo.coordinates were assigned based on a random pick in US airport locations. This assignment did not verify that the src or dest country were US.

This PR fixes the problem by assigning geo.src for each row to "US". That way, the source country and the geo.coordinates are always consistent. Below is the script used to update the data set.

'use strict'

const fs = require('fs');
const readline = require('readline');

async function load () {
  let failures = 0;
  const logs = [];
  const readStream = fs.createReadStream('./logs.json');

  const rl = readline.createInterface({
    input: readStream,
    crlfDelay: Infinity
  });

  for await (const line of rl) {
    try {
      const log = JSON.parse(line);
      // {"srcdest":"IN:US","src":"IN","dest":"US","coordinates":{"lat":39.41042861,"lon":-88.8454325}}
      log.geo.src = 'US';
      log.geo.srcdest = 'US:' + log.geo.dest;
      logs.push(log);
    } catch (err) {
      failures ++;
      console.log(`Unable to parse line, error: ${err.message}`);
    }
  }

  const writeStream = fs.createWriteStream('./updated_logs.json');
  logs.forEach(log => {
    writeStream.write(JSON.stringify(log, null, ''));
  });
  writeStream.end();
  
  console.log(`success: ${logs.length}, failures: ${failures}`);
}

load().catch(console.log);

Why not update geo.coordinates for original geo.src country? Well, Maps has a tutorial where geo.coordinates are reverse geo coded with United States Census Bureau Combined Statistical Area (CSA). Changing geo.coordinates would have broken this tutorial. Also, changing geo.coordinates to match original geo.src country would have been harder to code.

There were a few visualizations that used geo.src. Since geo.src is now always "US", these charts had to be updated to provide more meaningful visualizations. For example, the old sanky diagram kind of sucks when geo.src is always the same value

Updated charts:

…eo.coordinates

elasticmachine · 2021-09-01T20:03:47Z

Pinging @elastic/kibana-gis (Team:Geo)

timroes

Tested sample data dashboard and it seems consistant and visualizations making all sense. thanks for taking care of it. LGTM

nreese · 2021-09-07T14:17:30Z

@elasticmachine merge upstream

thomasneirynck

❤️

kibanamachine · 2021-09-07T16:31:15Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #150426 succeeded fdf119b
💔 Build #150390 failed 98352ab

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…eo.coordinates (elastic#110885) * [sample data] update web log geo.src field to match country code of geo.coordinates * fix functional tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-09-07T16:53:33Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…eo.coordinates (#110885) (#111434) * [sample data] update web log geo.src field to match country code of geo.coordinates * fix functional tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nathan Reese <reese.nathan@gmail.com>

…-link-to-kibana-app * 'master' of github.com:elastic/kibana: (61 commits) [Logs UI] Fix alert previews for thresholds of `0` (elastic#111150) [Archive Migration][Partial] discover apps-discover (elastic#110437) [APM] Set start date of APM ML job to -4 weeks (elastic#111375) [ML] APM Latency Correlations: Code consolidation. (elastic#110790) [Discover] Fix indices permission for multiline test (elastic#111284) [Detection Rules] Add 7.15 rules (elastic#111464) [Security Solution][Endpoint][Host Isolation] Hide isolate host option in alert details rather than disabling (elastic#111064) React version of angular license view (elastic#111317) [APM] Fix link in readme (elastic#111362) [Security Solution] add agent field to generator (elastic#111428) [Dashboard] Retain Tags on Quicksave (elastic#111015) Reorder App Search ingestion methods (elastic#111361) Port performance docs to new docs system. (elastic#111063) [Security Solution][RAC] Fixes updatedAt loading bug (elastic#111010) [sample data] update web log geo.src field to match country code of geo.coordinates (elastic#110885) [Security solution] [Endpoint] Fix bad artifact migration (elastic#111294) Fix copy typo. (elastic#111203) [build] Remove empty optimize directory (elastic#111393) [Maps] fix term join not updating when editing right field (elastic#111030) [Fleet] Set default settings in component template instead of the index template (elastic#111197) ... # Conflicts: # x-pack/plugins/reporting/public/management/__snapshots__/report_listing.test.tsx.snap # x-pack/plugins/reporting/public/management/report_listing.test.tsx

…eo.coordinates (elastic#110885) * [sample data] update web log geo.src field to match country code of geo.coordinates * fix functional tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

[sample data] update web log geo.src field to match country code of g…

98352ab

…eo.coordinates

nreese added Feature:Add Data Add Data and sample data feature on Home Team:Geo Former Team Label for Geo Team. Now use Team:Presentation v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.16.0 labels Sep 1, 2021

nreese requested review from thomasneirynck and timroes September 1, 2021 20:03

nreese requested review from a team as code owners September 1, 2021 20:03

pgayvallet approved these changes Sep 1, 2021

View reviewed changes

fix functional tests

fdf119b

timroes approved these changes Sep 2, 2021

View reviewed changes

Merge branch 'master' into issue_40401

e0a53ce

thomasneirynck approved these changes Sep 7, 2021

View reviewed changes

nreese added the auto-backport Deprecated - use backport:version if exact versions are needed label Sep 7, 2021

nreese merged commit daf860c into elastic:master Sep 7, 2021

kibanamachine mentioned this pull request Sep 7, 2021

[7.x] [sample data] update web log geo.src field to match country code of geo.coordinates (#110885) #111434

Merged

nreese mentioned this pull request Mar 29, 2022

Fix upgrade maps smoke tests #128696

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sample data] update web log geo.src field to match country code of geo.coordinates#110885

[sample data] update web log geo.src field to match country code of geo.coordinates#110885
nreese merged 3 commits intoelastic:masterfrom
nreese:issue_40401

nreese commented Sep 1, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Sep 1, 2021

Uh oh!

timroes left a comment

Uh oh!

nreese commented Sep 7, 2021

Uh oh!

thomasneirynck left a comment

Uh oh!

kibanamachine commented Sep 7, 2021

Uh oh!

kibanamachine commented Sep 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

nreese commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Sep 1, 2021

Uh oh!

timroes left a comment

Choose a reason for hiding this comment

Uh oh!

nreese commented Sep 7, 2021

Uh oh!

thomasneirynck left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Sep 7, 2021

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

kibanamachine commented Sep 7, 2021

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nreese commented Sep 1, 2021 •

edited

Loading