Skip to content

First draft of the juttle tutorial with github data#15

Closed
dmehra wants to merge 2 commits intomasterfrom
juttle-tutorial
Closed

First draft of the juttle tutorial with github data#15
dmehra wants to merge 2 commits intomasterfrom
juttle-tutorial

Conversation

@dmehra
Copy link
Contributor

@dmehra dmehra commented Dec 21, 2015

@demmer this is still raw, want high level review. Missing:

  • join section that enriches the data with metadata
  • at some point, switch to use elastic for the data with big data set (ingest with read file | write elastic ?)
  • haven't validated all the juttles that run in outrigger

Ignore exact wording, not ready for review of that level yet. Mainly want review of "does this tutorial cover the right bits of juttle". Most of it came from the old tutorial, with some mods.

@dmehra dmehra mentioned this pull request Dec 27, 2015
6 tasks
@dmehra
Copy link
Contributor Author

dmehra commented Dec 28, 2015

How about review by people who're here this week: @rlgomes @go-oleg @davidvgalbraith

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmehra I can add this feature to the file adapter but would want to wait for this other PR to get in before fiddling in that are of the code: #11

@go-oleg
Copy link
Contributor

go-oleg commented Dec 28, 2015

Overall I think this does a great job of introducing juttle.

Some thoughts

  • Where will a reader typically be coming from when reading this tutorial? I think having a basic understanding of the anatomy of a juttle program shown here is prereq for going through this tutorial.
  • The to-be-added join example will also serve to demonstrate that a flowgraph can start with two sources. Perhaps it would be good to have an example in here that shows a flowgraph can end in two sinks as well (all the current examples branch and merge in the middle).
  • Weave more links to "advanced" features of juttle (e.g. features like functions, subs, and user defined reducers which allow for more modular programming).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be handy to define small data sets here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could use help with the definition: would it be "data set that can fit in memory of a Node.js process"? what's the cap - 1GB? and how much of that can be used up by the data?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmehra
Copy link
Contributor Author

dmehra commented Dec 30, 2015

Addressed the review comments (thanks guys) except these two:

  • add example that shows a flowgraph can end in two sinks - will think about where to fit it
  • Weave more links to "advanced" features of juttle (e.g. features like functions, subs, and user defined reducers which allow for more modular programming). - unclear on this one:

@go-oleg did you mean to just add them as "juttle has these features but tutorial doesn't cover them", or to actually include them in the tutorial? If the latter, i'm not coming up with useful places to put a function here, much less a reducer. There are places where I can utilize a sub.

Sections on join and ES are new and need review, @rlgomes and @davidvgalbraith please.

@henridf your review would be most welcome, as you fathered the first version of this tutorial (a lot of it stayed true to the original).

still todo:

  • add screenshots of output. I first thought of including them as links, so the user would click if interested, or proceed to do the steps themselves without "cheating". However the dense text / code is starting to seem threatening, perhaps inlining the output will make the doc an easier read. Thoughts on that?
  • figure out what to do with the ending, where we peter out on input controls, since the only supported ones are still just text and number. None of that juttle-powered goodness works today in outrigger.
  • put the data under outrigger/demos/github-tutorial

@go-oleg
Copy link
Contributor

go-oleg commented Dec 30, 2015

I was hoping throughout the tutorial, but you're right in that its going to be complicated. Maybe some links at the end of the tutorial along the lines of "go here next":

  • list of built in procs
  • subs and functions
  • custom reducers

@henridf
Copy link
Contributor

henridf commented Jan 5, 2016

A thought for further out - once there is a Google BigQuery adaptor, it would be compelling to use that to pull data rather than use a static file.

@dmehra
Copy link
Contributor Author

dmehra commented Jan 13, 2016

TODO:

  • change ES data to use single-index schema instead of daily
  • fix up the ending of tutorial with inputs as soon as they work in outrigger

@dmajda
Copy link
Contributor

dmajda commented Jan 20, 2016

As the tutorial work moved to Outrigger, I suppose this PR won’t ever get merged ⇒ closing.

@dmehra If I am mistaken, please reopen.

@dmajda dmajda closed this Jan 20, 2016
@dmajda dmajda deleted the juttle-tutorial branch January 20, 2016 08:26
@dmehra
Copy link
Contributor Author

dmehra commented Jan 20, 2016

Will reopen because we still want the main tutorial doc merged into juttle for publishing as part of the docs site, I think, so all local docs links would work (and a big part of this doesn't use outrigger).

mattnibs pushed a commit that referenced this pull request Jan 20, 2016
@dmehra dmehra restored the juttle-tutorial branch January 27, 2016 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants