@tengqm Travis failed:

travis_time:end:0971c169:start=1524018539424712844,finish=1524018543151904299,duration=3727191455
�[0Ktravis_fold:end:rvm
�[0K$ ruby --version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
$ rvm --version
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
$ bundle --version
Bundler version 1.16.1
$ gem --version
2.7.6
travis_time:start:01565834
�[0K$ rake
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)
/home/travis/.rvm/gems/ruby-2.4.1@global/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
(See full trace by running task with --trace)

travis_time:end:01565834:start=1524018543662891381,finish=1524018543816947599,duration=154056218
�[0K
�[31;1mThe command "rake" exited with 1.�[0m

Done. Your build exited with 1.

/assign

@chenopis: GitHub didn't allow me to assign the following users: chenopis.

Note that only kubernetes-incubator members and repo collaborators can be assigned.

@chenopis Travis has been failing for a long time I guess. It is a different issue to fix. The project needs a '.travis.yml' file for CI.

@tengqm What should we do w/ this PR then? How do we verify it so that it can be merged?

/assign janetkuo

@chenopis Will try fix travis CI.

@chenopis Added travis gate ci config. Now the gate is up and running. I think the PR is ready to go now. Will continue (in another PR) to stand up CI jobs for API doc and CLI doc generation.

@tengqm Ok, great! I don't have merge rights in this repo, so we'll have to wait for one of the kubernetes-incubator maintainers.

/cc foxish

@pwittrock can you help get this PR merged? We need it for kubeadm docs for 1.11 and it's currently a blocker on some docs PRs. Thanks!

I just started looking at this. But at first glance, I don't understand why we would want 20,000 files here in the kubernetes-incubator/reference-docs repository.

And I don't understand why we would shift the code that generates components like kube-apiserver from kubernetes/kubernetes into kubernetes-incubator/reference-docs.

I'm inclined to agree with Steve. I realized another thing, too, though, while reviewing this docs PR -- HTML formatting makes it impossible to diff with newer or older versions that don't have it. This is obvious, but until we get this PR sorted, we cannot diff generated files that contain new content. This is particularly a problem for the kubeadm docs, which are a mix of manually curated and generated content, and really should be reviewed in toto when a PR for manual content is submitted.

@steveperry-53 it looks to me as though most of those 20,000 files are the result of vendoring. But that's just a guess -- I certainly don't have the bandwidth even to spot-check the whole diff, assuming I knew how to get it which I don't.

@tengqm could we add just the code that formats the tables to the existing generators in k/k? And to other places wherever it's needed? (I'm not sure, for example, where the generator is for the kubeadm docs ... I did not build those for 1.10)

There is a long story behind this change. Let me try summarize the key points as Q&A below:

Q: Why this PR is changing more than 20 thousand files?
A: This PR removed all packages previously vendored. They are not necessary and they are not working as expected (explained below).

Q: Can we generate reference docs from within kubernetes/kubernetes project?
A: No. The reference docs are not generated from kubernetes code. They are generated from the package spf13/cobra or more specifically, code here: https://github.com/spf13/cobra/blob/master/doc/md_docs.go
The followings are make the doc generation logic more complicated:

• Usage of template language: https://github.com/spf13/cobra/blob/master/command.go#L411
• The actual printing of the usage is in yet another package: https://github.com/spf13/pflag/blob/master/flag.go#L665

Q: Why don't we try patch spf13/cobra package directly?
A: Markdown docs in general is meant to be convertible to different output formats, such as HTML, tex. The *.md file generated from spf13/cobra package, if I'm understanding it correctly, should be kept as generic as possible. We are not supposed to hijack that library for website generation.

Q: Is the revised doc generator making doc reviews difficult?
A: It depends. If we allow contributors to directly commit changes to generated docs in different format, we are asking for troubles ourselves. The docs generated are not supposed to generated out of the automation tool only, otherwise we will have a difficult time validating the docs against the component/API that is documented.

Q: Does the revised tool work or not? Is it only working on tengqm's workstation?
A: The revised tool is much simpler (no vendored package any more). It is working as proved by the Travis gate job. When the tool is broken, the travis gate will notice it immediately. From the revised .travis.yml file, you can see how easy the logic has become.

Q: Vendoring is a common practices in Go community, why are we killing the vendored packages?
A: Vendoring k8s.io/kubernetes no longer work now because the kuberenetes project has stopped checking in generated code. The impact to doc generation is that the vendored package won't build until we do a make generated_fiels in that project.

Q: How big a change this PR is?
Q: At its core, 69 lines of code, i.e. the flagUsages() function in the gen-compdocs/generators/doc.go file. The whole file is of 334 lines including license. If we include the changes to .travis.yml and Makefile, the whole change is about 500 lines.

cc @zacharysarah

@tengqm Thanks for the explanation. I understand what's going on now. And I totally agree that we should not keep vendored files here in the repository.

@tengqm Are there any updates required to https://kubernetes.io/docs/home/contribute/generated-reference/kubernetes-components/?

+1 to getting rid of vendored files.