Generate reference links for painless API by nik9000 · Pull Request #3 · nik9000/elasticsearch

nik9000 · 2016-11-29T21:21:04Z

Adds "Appending B. Painless API Reference", a reference of all classes and methods available from painless. Removes links to java packages because they contain methods that we don't expose and don't contain methods that we do expose (the ones in Augmentation). Instead this generates a list of every class and every exposed method using the same type information available to the interpreter/compiler/whatever-we-call-it. From there you can jump to the relevant docs.

Right now you build all the asciidoc files by running

gradle generatePainlessApi

You can also run it in an IDE safely if you pass the path to the directory in which to generate the docs as the first parameter. It'll blow away the entire directory an recreate it from scratch so be careful.

And then you can build the docs by running something like:

../docs/build_docs.pl --out ../built_docs/ --doc docs/reference/index.asciidoc --open

That is, if you have checked out https://github.com/elastic/docs in ../docs. Wait a minute or two and your browser will pop open in with all of Elasticsearch's reference documentation. If you go to http://localhost:8000/painless-api-reference.html you can see this list. Or you can get there by following the links to Modules and Scripting and painless and then clicking the link in the paragraphs below titled Appendix B. Painless API Reference.

I like having these in asciidoc because we can deep link to them from the rest of the guide with constructs like <<painless-api-reference-Object-hashCode-0>> and <<painless-api-reference->> and we get link checking. Then the only brittle link maintenance bit is the link generation for javadoc. Which sucks. But I think it is important that we reference the methods directly so they are easy to find.

WIP for a bunch of reasons:

I'm not 100% set on the linking strategy. It is certainly easier than trying to extract that javadoc of the method I'm linking to. And it should be safe from a licensing perspective. But it is much less convenient.
It isn't clear if we should generate the documentation and check it in or if we should generate it when we're generating the docs to publish. Either way works and we can do one for now and switch if we like. The process of generating the documentation is pretty much instant if you include it in another gradle execution.
We don't publish Javadoc so the Elasticsearch docs links to a place that republishes. And the versions are all wrong. And Painless's javadoc isn't available but we need to publish at least some of it to make this linking strategy work.
It may be worth moving this out of src/test/java and into src/tool/java or something. I don't know how to do that off hand so I'm not going to do it unless someone tells me it is important.
I don't really like the way the (java 9) links look. I think they are important to have. Or it is important to have something like them.

clintongormley · 2016-11-30T10:45:40Z

Hi @nik9000 - i tried building your painless_links branch but I get the following when I do gradle generatePainlessApi:

:modules:lang-painless:compileJava
Full recompilation is required because 'PainlessLexer.java' was changed. Analysis took 3.541 secs.
/Users/clinton/workspace/elasticsearch/modules/lang-painless/src/main/java/org/elasticsearch/painless/Definition.java:507: error: reference not found
    /** Collection of all simple types. Used by {@link PainlessDocGenerator} to generate an API reference. */
                                                       ^
1 error

nik9000 · 2016-12-01T13:16:03Z

@clintongormley weird. I pushed a patch that I'm sure works locally.

clintongormley · 2016-12-01T14:46:16Z

Tried again and now this:

Full recompilation is required because 'elasticsearch-6.0.0-alpha1-SNAPSHOT.jar' was changed. Analysis took 3.63 secs.
/Users/clinton/workspace/elasticsearch/modules/lang-painless/src/main/java/org/elasticsearch/painless/Definition.java:507: error: reference not found
    /** Collection of all simple types. Used by {@link PainlessDocGenerator} to generate an API reference. */
                                                       ^
1 error
:modules:lang-painless:compileJava FAILED

nik9000 · 2016-12-02T15:39:01Z

Sorry! I was able to reproduce by cleaning. Without clean I wasn't getting the failure.... I pushed a fix for it.

jdconrad · 2016-12-02T19:09:31Z

@nik9000 I think the feature is great and definitely needed. I'm not overly concerned about the code at this point because I know it's still a WIP. Will be super happy to have this in once you're happy with the code.

Field collapsing is supported from version 5.3

This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.

Instead of using Gradle-version specific compilation options, use distinct source sets. This also allows compilation of buildSrc/build-tools under IDEs that don't understand the version-specific compilation options. Relates to elastic#22669

…2717) As a follow up to elastic#22649, this changes the resent tests for parsing parts of search responses to randomly set the humanReadable() flag of the XContentBuilder that is used to render the responses. This should help to test that we can parse back thoses classes if the user specifies `?human=true` in the request url.

Docker cgroups are mounted in the wrong place (i.e., inconsistently with /proc/self/cgroup). This commit adds an undocumented hack for working around, for now. Relates elastic#22757

…ers (elastic#22703) move "es." internal headers to separate metadata set in ElasticsearchException and stop returning them as response headers Closes elastic#17593 * [TEST] remove ESExceptionTests, move its methods to ElasticsearchExceptionTests or ExceptionSerializationTests

This adds the necessary `AuthCache` needed to support preemptive authorization. By adding every host to the cache, the automatically added `RequestAuthCache` interceptor will add credentials on the first pass rather than waiting to do it after _each_ anonymous request is rejected (thus always sending everything twice when basic auth is required).

Previously the behavior where the `OpType` byte was serialized was only in master, but it was recently backported to 5.x, so the serialization version checks need to be updated as well.

5.3.0 is the first version that contains the split from headers to metadata, updated the check to reflect that. It was previously after to be able to commit to master first, and only after that backport the change. Otherwise master tests would have failed until the change was backported.

These files should have been removed in an earlier commit. This commit also simplifies usage of ProgressLoggerWrapper by using the Groovy delegation instead of using explicit delegation.

Adds "Appending B. Painless API Reference", a reference of all classes and methods available from Painless. Removes links to java packages because they contain methods that we don't expose and don't contain methods that we do expose (the ones in Augmentation). Instead this generates a list of every class and every exposed method using the same type information available to the interpreter/compiler/whatever-we-call-it. From there you can jump to the relevant docs. Right now you build all the asciidoc files by running ``` gradle generatePainlessApi ``` These files are expected to be committed because we build the docs without running `gradle`. Also changes the output of `Debug.explain` so that it is easy to search for the class in the generated reference documentation. You can also run it in an IDE safely if you pass the path to the directory in which to generate the docs as the first parameter. It'll blow away the entire directory an recreate it from scratch so be careful. And then you can build the docs by running something like: ``` ../docs/build_docs.pl --out ../built_docs/ --doc docs/reference/index.asciidoc --open ``` That is, if you have checked out https://github.com/elastic/docs in `../docs`. Wait a minute or two and your browser will pop open in with all of Elasticsearch's reference documentation. If you go to `http://localhost:8000/painless-api-reference.html` you can see this list. Or you can get there by following the links to `Modules` and `Scripting` and `Painless` and then clicking the link in the paragraphs below titled `Appendix B. Painless API Reference`. I like having these in asciidoc because we can deep link to them from the rest of the guide with constructs like `<<painless-api-reference-Object-hashCode-0>>` and `<<painless-api-reference->>` and we get link checking. Then the only brittle link maintenance bit is the link generation for javadoc. Which sucks. But I think it is important that we link to the methods directly so they are easy to find.

It is confusing and break screen readers.

In elastic#28350, we fixed an endless flushing loop which may happen on replicas by tightening the relation between the flush action and the periodically flush condition. 1. The periodically flush condition is enabled only if it is disabled after a flush. 2. If the periodically flush condition is enabled then a flush will actually happen regardless of Lucene state. (1) and (2) guarantee that a flushing loop will be terminated. Sadly, the condition 1 can be violated in edge cases as we used two different algorithms to evaluate the current and future uncommitted translog size. - We use method `uncommittedSizeInBytes` to calculate current uncommitted size. It is the sum of translogs whose generation at least the minGen (determined by a given seqno). We pick a continuous range of translogs since the minGen to evaluate the current uncommitted size. - We use method `sizeOfGensAboveSeqNoInBytes` to calculate the future uncommitted size. It is the sum of translogs whose maxSeqNo at least the given seqNo. Here we don't pick a range but select translog one by one. Suppose we have 3 translogs `gen1={#1,#2}, gen2={}, gen3={#3} and seqno=#1`, `uncommittedSizeInBytes` is the sum of gen1, gen2, and gen3 while `sizeOfGensAboveSeqNoInBytes` is the sum of gen1 and gen3. Gen2 is excluded because its maxSeqno is still -1. This commit removes both `sizeOfGensAboveSeqNoInBytes` and `uncommittedSizeInBytes` methods, then enforces an engine to use only `sizeInBytesByMinGen` method to evaluate the periodically flush condition. Closes elastic#29097 Relates #elastic#28350

…uginFuncTest builds distribution from branches via archives extractedAssemble [bwcDistVersion: 8.1.3, bwcProject: bugfix2, expectedAssembleTaskName: extractedAssemble, #3] elastic#119261

nik9000 force-pushed the painless_links branch 2 times, most recently from ba015d9 to e0c8433 Compare January 23, 2017 22:57

jimczi and others added 12 commits January 24, 2017 08:34

Add BWC tests for field collapsing

868b12b

Field collapsing is supported from version 5.3

Remove unused field in CollapseBuilder

b0c2a5d

Make build Gradle 2.14 / 3.x compatible (elastic#22669)

36198e0

This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.

Use Gradle-version specific source sets

12b6ff5

Instead of using Gradle-version specific compilation options, use distinct source sets. This also allows compilation of buildSrc/build-tools under IDEs that don't understand the version-specific compilation options. Relates to elastic#22669

Add hack for Docker cgroups

bcffc6f

Docker cgroups are mounted in the wrong place (i.e., inconsistently with /proc/self/cgroup). This commit adds an undocumented hack for working around, for now. Relates elastic#22757

Docs: Made cross-cluster search experimental

12f5309

Fix BulkItemResponse serialization for 6.x <-> 5.3.x

304296e

Previously the behavior where the `OpType` byte was serialized was only in master, but it was recently backported to 5.x, so the serialization version checks need to be updated as well.

Remove leftover build files and simplify ProgressLogger usage

e37a7d7

These files should have been removed in an earlier commit. This commit also simplifies usage of ProgressLoggerWrapper by using the Groovy delegation instead of using explicit delegation.

nik9000 force-pushed the painless_links branch from 1b8d38b to a5ad02f Compare January 24, 2017 21:49

nik9000 force-pushed the painless_links branch from a5ad02f to 352d2d5 Compare January 24, 2017 21:52

nik9000 added 5 commits January 24, 2017 16:55

Remove useless import

dbab668

Drop the link to class level documentation

4fe5fee

It is confusing and break screen readers.

Switch to definition list

e86e87c

Move api reference to appendix A and make visible in TOC

72aaae5

Fix test

674f3af

nik9000 closed this Feb 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate reference links for painless API#3

Generate reference links for painless API#3
nik9000 wants to merge 18 commits intomasterfrom
painless_links

nik9000 commented Nov 29, 2016

Uh oh!

clintongormley commented Nov 30, 2016

Uh oh!

nik9000 commented Dec 1, 2016

Uh oh!

clintongormley commented Dec 1, 2016

Uh oh!

nik9000 commented Dec 2, 2016

Uh oh!

jdconrad commented Dec 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

nik9000 commented Nov 29, 2016

Uh oh!

clintongormley commented Nov 30, 2016

Uh oh!

nik9000 commented Dec 1, 2016

Uh oh!

clintongormley commented Dec 1, 2016

Uh oh!

nik9000 commented Dec 2, 2016

Uh oh!

jdconrad commented Dec 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants