-
Notifications
You must be signed in to change notification settings - Fork 25
Single binary #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Single binary #36
Conversation
.gitignore
Outdated
|
|
||
| # dependencies | ||
| /node_modules | ||
| /vendor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for me is a must upload the vendoring
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't see much profit from it. With proposed flow you have to compile frontend after go-get anyway. Running dep ensure there doesn't make it worse.
But not committing it allows using the tool with other frontend or whatever another crazy use case when people want to have their own vendoring.
Correct me if I'm wrong but before if your dependency has vendor directory you could have a conflict of dependencies.
Still if everybody wants to have vendor in repo, it's not such a big deal for me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I said is standard in go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@campoy you were a go evangelist, right? Can you give me more information about it, please? All I read on the internet, for example, https://golang.github.io/dep/docs/FAQ.html#should-i-commit-my-vendor-directory says it's up to project.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This says everything https://github.com/golang/dep/tree/master/vendor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a great question, actually. So good it's part of the FAQ of dep.
From the docs:
Should I commit my vendor directory?
It's up to you:
Pros
It's the only way to get truly reproducible builds, as it guards against upstream renames, deletes and commit history overwrites.
You don't need an extra dep ensure step to sync vendor/ with Gopkg.lock after most operations, such as go get, cloning, getting latest, merging, etc.Cons
Your repo will be bigger, potentially a lot bigger, though prune can help minimize this problem.
PR diffs will include changes for files under vendor/ when Gopkg.lock is modified, however files in vendor/ are hidden by default on Github.
So I'll let @mcuadros decide, since he's the CTO. I do think we should commit vendored dependencies, since this allows for anyone to simply do go get of our repo and everything will simply work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the problem is it wouldn't just work.
but no problem. I don't really have a strong opinion on it.
Just wanted some guideline for my projects outside of src-d.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I wrote, almost every single relevant project is committing the vendor. So please commit the vendor.
Makefile
Outdated
| bindata: | ||
| $(BINDATA) -o ./server/cmd/code-annotation/asset.go build/static/... build/*.json build/*.png build/index.html | ||
|
|
||
| release-build: build-frontend build-backend bindata packages docker-build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
release build should be packages something like:
packages: build-frontend dependencies-backend bindata packages docker-build
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, you need to force the order of the dependencies with rule: | one two three otherwise if someone use -j n will fail
README.md
Outdated
| ## Development | ||
|
|
||
| Backend: | ||
| Global dependencies: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This information should go int he CONTRIBUTING.md this is not the place for this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes. We have #29 for it.
I'll rebase later.
server/cmd/code-annotation/asset.go
Outdated
|
|
||
| // AssetNames returns the names of the assets. | ||
| func AssetNames() []string { | ||
| panic("shouldn't be used in dev mode") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually I should just remove it
Dockerfile
Outdated
| WORKDIR /go/src/code-annotation | ||
|
|
||
| RUN make build && \ | ||
| RUN make build-frontend && \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should copy the file from the bin folder like:
https://github.com/src-d/ci/blob/v1/examples/basic/Dockerfile#L2
And then you can remove almost the full Dockefile
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this way allows to build docker file on any machine without installing any dependencies like go or nodejs. It was requirement from @vmarkovtsev to be able to build docker file without correct version of nodejs and yarn
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I can confirm that: I even have no idea what is yarn and why it is better than npm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Err but if you are contributing you need to run the tests and other stuff. So you need yarn installed anyway.
Anyways you need to call make packages for having the proper version variables.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If contributing - agree 100%. If building the container to run the thing and forget about it - nopes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't mess up CI and release with local building of container.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
example use case: I'm a devops. I want to run deploy this tool on my cluster. There is a change in master I need. But no release yet. So I can just git clone && docker build -t my/tool . && docker push my/tool && kubectl <deploy>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Errr, no. You shouldn't, never do this, you should our standard image, tagged and tests. This Dockerfile is for the CI to build docker images.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But I am not understanding this discussion. We agreed and merged a CI, to be used, and not to ignore it two days after.
The goal of CI is exactly this one, having homogeneous repositories.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay, will remove build step from Dockerfile. I'm just used to be able to build images without external dependencies.
P.S. example of our app (merged by you):
https://github.com/src-d/github-reminder/blob/master/Dockerfile
| @@ -1,7 +1,8 @@ | |||
| # See https://help.github.com/ignore-files/ for more about ignoring files. | |||
|
|
|||
| # dependencies | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you should include .ci
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# ci files
Makefile.main
.ci
bin
Dockerfile
Outdated
| WORKDIR /root/ | ||
| COPY --from=0 /go/src/code-annotation/build ./build | ||
| COPY --from=0 /go/src/code-annotation/bin/code-annotation . | ||
| CMD ["./code-annotation"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The binary should be in the PATH and as an ENTRYPOINT and not as a CMD
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makefile
Outdated
| bindata: | ||
| $(BINDATA) -o ./server/cmd/code-annotation/asset.go build/static/... build/*.json build/*.png build/index.html | ||
|
|
||
| release-build: build-frontend build-backend bindata packages docker-build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, you need to force the order of the dependencies with rule: | one two three otherwise if someone use -j n will fail
README.md
Outdated
| cd $GOPATH/github.com/src-d/code-annotation | ||
| make serve | ||
| ``` | ||
| Download binary from [releases](releases) for you platform. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This link works? Or just links to a folder called releases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will fix
| docker run --env-file .env --rm -p 8080:8080 srcd/code-annotation | ||
| ``` | ||
|
|
||
| ### Non-docker |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should put a standard path to place the binary.
Something like:
wget https://github.com/mcuadros/passage/releases/download/v0.1.0/passage_v0.1.0_linux_amd64.tar.gz
tar -xvzf passage_v0.1.0_linux_amd64.tar.gz
cp passage_v0.1.0_linux_amd64/passage /usr/local/bin/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed. But I want to postpone it until we have our first release.
server/cmd/code-annotation/main.go
Outdated
| http.ServeFile(w, r, filepath) | ||
| return | ||
| b, err := Asset(filepath) | ||
| // fallback on index, will be handled by FE router |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a good idea? maybe is better return a 404 if is file not found and a 500 if not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is. Frontend has urls like /experiment/1/2, backend knows nothing about them. It's handled correctly if you click on a link (js will catch event), but if you open this url the first time response will be served by the server. That's why we have to return index.html. JS there will handle routing correctly (and show 404 if path doesn't exist). It's a limitation for single page application if you want to have nice urls without #.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
meeeh
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
frontend 😂
09f1a6c to
8303da6
Compare
|
All comments were addressed except documentation:
|
README.md
Outdated
| cd $GOPATH/github.com/src-d/code-annotation | ||
| make serve | ||
| ``` | ||
| Download binary from [releases](https://github.com/src-d/code-annotation/releases) for you platform. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for your platform
|
Now, that we have a single binary that we distribute I have 2 related concerns:
This is very different from before, when we could distribute only the source code as the main release artifact, so the user would be responsible for building the final binaries and combining all the potentially incompatible-licensed source code together. This change would imply necessity of some form of license audit, on both, backend and frontend before making a release. I belive it can be take care of in a separate PR, if we agree, what do you think @mcuadros ? |
|
I've left related comments here: Also, there is a discrepancy. This PR says |
|
Yes. Yarn will be replaced with make after rebase. Thanks! |
|
Rebase seems to be in order and feedback from @mcuadros and @dpordomingo is pending. |
|
@bzz This PR waits for documentation PR to be merged. Let's wait for it. Then I will rebase and it should resolve code review comments. |
|
Thanks for keeping the status posted! I assume that you mean waiting for #29 to be merged. |
|
Since this PR depends on #29 I'll review this one once its dependency has been approved/merged. |
4c84aca to
8c75da4
Compare
8c75da4 to
37e7f2b
Compare
|
Rebased. Documentation Please don't pay much attention to Information about how to run it from source has been moved to CONTRIBUTING. Changes in Makefile I don't redefine Please do another pass on review. |
|
\cc @mcuadros for another pass on it |
|
While waiting for @mcuadros feedback: as soon as this introduces a single binary deployment, @smacker do you think it would be reasonable to add a very simple end-to-end test to the CI in this PR as well? I.e one possible scenarios is
this would help checking that bindata is packaged as expected. |
|
@bzz I have a branch with e2e test against many endpoints ( |
|
@bzz yeah. we need it. but let's keep it out of this PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm using this PR as a dependency of #85 to fix some problems I spotted here during the deploy,
so yes: LGTM
| node_modules | ||
| build | ||
| bin | ||
| /node_modules |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
leftover?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why? now we don't do anything besides copying binary into docker. So we don't need to send node_modules (it can be huge and takes time)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
he is maybe asking about the extra / ??
| ### Docker | ||
|
|
||
| You need to satisfy all [project requirements](#requirements), and then run: | ||
| docker run --env-file .env --rm -p 8080:8080 srcd/code-annotation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find extremely odd having a .env file and not very devops friendly. If we are going ot have it, at least include the definition of each variable in the templete. But I prefer having it explained in the readme and having defaults.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the .env file is just an easy way to have in one place all the environmental values as expected by the app, but it is up to the developers to decide how they are providing them;
Environment values can be passed as regular vars (ie.UI_DOMAIN=http://127.0.0.1:3000 make gorun) or as regular docker envs (ie. docker run --env OAUTH_CLIENT_ID=id --env OAUTH_CLIENT_SECRET=secret --rm -p 8080:8080 srcd/code-annotation)
Most of them are documented in different parts of the README.md or the CONTRIBUTING.md.
@mcuadros Should we block this PR till everything is properly documented, or can we merge and handle this requirement in a separated issue?
These are the environment values recognized by the app;
most of them are not mandatory, but we can also make "optional" the JWT_SIGNING_KEY.
| key | required | actual default | default | meaning |
|---|---|---|---|---|
OAUTH_CLIENT_ID |
❗ | OAuth app client id | ||
OAUTH_CLIENT_SECRET |
❗ | OAuth app client secret | ||
JWT_SIGNING_KEY |
❗ | jwt_signing_key |
key to sign the user session JWT | |
DB_CONNECTION |
sqlite://internal.db |
sqlite:///var/code-annotation/internal.db |
DSN of the internal database | |
OAUTH_RESTRICT_ACCESS |
group or team of users that can access the app | |||
OAUTH_RESTRICT_REQUESTER_ACCESS |
group or team of users granted as Requesters | |||
REACT_APP_SERVER_URL |
127.0.0.1:8080 |
//code-annotation-staging.srcd.run |
host:port of the API |
|
UI_DOMAIN |
$(REACT_APP_SERVER_URL) |
$(REACT_APP_SERVER_URL) |
host:port of the frontend recieveing the user token after authentication |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.env file is just a simple way to see/edit environmental variables for local development. Most of the values have defaults. All variables must and will be documented. But this PR isn't the right place to do that.
About devops:
In my experience env vars are the most friendly and natural way to configure an application with kubernetes, thanks to configmap,secrets objects and valueFrom in container spec.
And The twelve-factor app also recommends it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dpordomingo thank you for great summary!
As on @mcuadros notice, agreed that we should have it clearly documented - have added it to the scope of #67
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@smacker where says that is recommended have a .env file? This is not recommended at all, and almost none use this.
Also is bad not having any prefix on your env vars.
If you are speaking about the usage of env vars, of course, is recommended, and actually is a MUST that all source{d} applications
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mcuadros no-no. I mean env vars, not .env file. We use .env file ONLY for local development. Agreed about prefix, we should add it to env vars without prefix! Thanks!
Didn't know it's must, https://github.com/bblfsh/dashboard uses flags instead. But glad to hear.
|
@mcuadros all feedback seems to be addressed. |
fix builds when there are no go files in top level
Fixes #34
Contains Gopkg.toml & Gopkg.yaml from #33
For easier understanding here is use cases:
For user:
make release(no .travis config yet, we aren't ready for release)So, user needs:
docker run src-d/code-annotationFor developers:
Global dependencies:
Start contributing:
go get -d -u github.com/src-d/code-annotation/...$GOPATH/github.com/src-d/code-annotationmake serveFor frontend developer who want to benefit from hot reloading:
go get -d -u github.com/src-d/code-annotation/...$GOPATH/github.com/src-d/code-annotationmake servemake dev-frontendFor crazy developers: (undocumented, unsupported but works)
go get github.com/src-d/code-annotation/...still works