Skip to content

Conversation

@smacker
Copy link
Contributor

@smacker smacker commented Jan 26, 2018

Fixes #34
Contains Gopkg.toml & Gopkg.yaml from #33

For easier understanding here is use cases:

For user:

  • CI react on a tag and run make release (no .travis config yet, we aren't ready for release)
  • It will produce single binary and docker image
  • binary will be uploaded to github for distribution

So, user needs:

  • download tarfile from releases
  • or use docker run src-d/code-annotation

For developers:

Global dependencies:

  • Go 1.9
  • Node 8
  • Yarn 1

Start contributing:

  • go get -d -u github.com/src-d/code-annotation/...
  • $GOPATH/github.com/src-d/code-annotation
  • make serve
  • no need for extra steps or commands before commit & push

For frontend developer who want to benefit from hot reloading:

  • go get -d -u github.com/src-d/code-annotation/...
  • $GOPATH/github.com/src-d/code-annotation
  • make serve
  • make dev-frontend

For crazy developers: (undocumented, unsupported but works)

  • go get github.com/src-d/code-annotation/... still works
  • but dependencies should be managed by the developer

@smacker smacker requested a review from mcuadros January 26, 2018 16:57
.gitignore Outdated

# dependencies
/node_modules
/vendor
Copy link

@mcuadros mcuadros Jan 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for me is a must upload the vendoring

Copy link
Contributor Author

@smacker smacker Jan 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't see much profit from it. With proposed flow you have to compile frontend after go-get anyway. Running dep ensure there doesn't make it worse.
But not committing it allows using the tool with other frontend or whatever another crazy use case when people want to have their own vendoring.
Correct me if I'm wrong but before if your dependency has vendor directory you could have a conflict of dependencies.

Still if everybody wants to have vendor in repo, it's not such a big deal for me.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said is standard in go.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@campoy you were a go evangelist, right? Can you give me more information about it, please? All I read on the internet, for example, https://golang.github.io/dep/docs/FAQ.html#should-i-commit-my-vendor-directory says it's up to project.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great question, actually. So good it's part of the FAQ of dep.

From the docs:

Should I commit my vendor directory?

It's up to you:

Pros

It's the only way to get truly reproducible builds, as it guards against upstream renames, deletes and commit history overwrites.
You don't need an extra dep ensure step to sync vendor/ with Gopkg.lock after most operations, such as go get, cloning, getting latest, merging, etc.

Cons

Your repo will be bigger, potentially a lot bigger, though prune can help minimize this problem.
PR diffs will include changes for files under vendor/ when Gopkg.lock is modified, however files in vendor/ are hidden by default on Github.

So I'll let @mcuadros decide, since he's the CTO. I do think we should commit vendored dependencies, since this allows for anyone to simply do go get of our repo and everything will simply work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is it wouldn't just work.
but no problem. I don't really have a strong opinion on it.
Just wanted some guideline for my projects outside of src-d.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I wrote, almost every single relevant project is committing the vendor. So please commit the vendor.

Makefile Outdated
bindata:
$(BINDATA) -o ./server/cmd/code-annotation/asset.go build/static/... build/*.json build/*.png build/index.html

release-build: build-frontend build-backend bindata packages docker-build

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

release build should be packages something like:

packages: build-frontend  dependencies-backend bindata packages docker-build

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, you need to force the order of the dependencies with rule: | one two three otherwise if someone use -j n will fail

README.md Outdated
## Development

Backend:
Global dependencies:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This information should go int he CONTRIBUTING.md this is not the place for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. We have #29 for it.
I'll rebase later.


// AssetNames returns the names of the assets.
func AssetNames() []string {
panic("shouldn't be used in dev mode")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I should just remove it

Dockerfile Outdated
WORKDIR /go/src/code-annotation

RUN make build && \
RUN make build-frontend && \

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should copy the file from the bin folder like:
https://github.com/src-d/ci/blob/v1/examples/basic/Dockerfile#L2

And then you can remove almost the full Dockefile

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this way allows to build docker file on any machine without installing any dependencies like go or nodejs. It was requirement from @vmarkovtsev to be able to build docker file without correct version of nodejs and yarn

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I can confirm that: I even have no idea what is yarn and why it is better than npm

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Err but if you are contributing you need to run the tests and other stuff. So you need yarn installed anyway.

Anyways you need to call make packages for having the proper version variables.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If contributing - agree 100%. If building the container to run the thing and forget about it - nopes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't mess up CI and release with local building of container.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

example use case: I'm a devops. I want to run deploy this tool on my cluster. There is a change in master I need. But no release yet. So I can just git clone && docker build -t my/tool . && docker push my/tool && kubectl <deploy>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Errr, no. You shouldn't, never do this, you should our standard image, tagged and tests. This Dockerfile is for the CI to build docker images.

Copy link

@mcuadros mcuadros Jan 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I am not understanding this discussion. We agreed and merged a CI, to be used, and not to ignore it two days after.

The goal of CI is exactly this one, having homogeneous repositories.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, will remove build step from Dockerfile. I'm just used to be able to build images without external dependencies.

P.S. example of our app (merged by you):
https://github.com/src-d/github-reminder/blob/master/Dockerfile

@@ -1,7 +1,8 @@
# See https://help.github.com/ignore-files/ for more about ignoring files.

# dependencies

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should include .ci

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dockerfile Outdated
WORKDIR /root/
COPY --from=0 /go/src/code-annotation/build ./build
COPY --from=0 /go/src/code-annotation/bin/code-annotation .
CMD ["./code-annotation"]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The binary should be in the PATH and as an ENTRYPOINT and not as a CMD

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makefile Outdated
bindata:
$(BINDATA) -o ./server/cmd/code-annotation/asset.go build/static/... build/*.json build/*.png build/index.html

release-build: build-frontend build-backend bindata packages docker-build

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, you need to force the order of the dependencies with rule: | one two three otherwise if someone use -j n will fail

README.md Outdated
cd $GOPATH/github.com/src-d/code-annotation
make serve
```
Download binary from [releases](releases) for you platform.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This link works? Or just links to a folder called releases

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix

docker run --env-file .env --rm -p 8080:8080 srcd/code-annotation
```

### Non-docker

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should put a standard path to place the binary.

Something like:

wget https://github.com/mcuadros/passage/releases/download/v0.1.0/passage_v0.1.0_linux_amd64.tar.gz
tar -xvzf passage_v0.1.0_linux_amd64.tar.gz
cp passage_v0.1.0_linux_amd64/passage /usr/local/bin/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed. But I want to postpone it until we have our first release.

http.ServeFile(w, r, filepath)
return
b, err := Asset(filepath)
// fallback on index, will be handled by FE router

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a good idea? maybe is better return a 404 if is file not found and a 500 if not

Copy link
Contributor Author

@smacker smacker Jan 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is. Frontend has urls like /experiment/1/2, backend knows nothing about them. It's handled correctly if you click on a link (js will catch event), but if you open this url the first time response will be served by the server. That's why we have to return index.html. JS there will handle routing correctly (and show 404 if path doesn't exist). It's a limitation for single page application if you want to have nice urls without #.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meeeh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

frontend 😂

@smacker
Copy link
Contributor Author

smacker commented Jan 29, 2018

All comments were addressed except documentation:

README.md Outdated
cd $GOPATH/github.com/src-d/code-annotation
make serve
```
Download binary from [releases](https://github.com/src-d/code-annotation/releases) for you platform.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for your platform

@bzz
Copy link
Contributor

bzz commented Jan 29, 2018

Now, that we have a single binary that we distribute I have 2 related concerns:

  1. Backend: for https://github.com/src-d/code-annotation/tree/master/vendor - as the source code would be part of release of this GPLv3 application and the file tree now contains the all the libs that are used

  2. Frontend: right now, the final artifact (single binary) that is going to be distributed as a part of release of this GPLv3 application will include all the frontend libraries and some of them might or might not even be an open source software

This is very different from before, when we could distribute only the source code as the main release artifact, so the user would be responsible for building the final binaries and combining all the potentially incompatible-licensed source code together.

This change would imply necessity of some form of license audit, on both, backend and frontend before making a release.

I belive it can be take care of in a separate PR, if we agree, what do you think @mcuadros ?

@carlosms
Copy link
Contributor

I've left related comments here:
#29 (review)

Also, there is a discrepancy. This PR says make serve, make dev-frontend
PR #29 says make serve, yarn start.
Just mentioning so you can make sure the right one ends up in the docs.

@smacker
Copy link
Contributor Author

smacker commented Jan 29, 2018

Yes. Yarn will be replaced with make after rebase. Thanks!

@bzz
Copy link
Contributor

bzz commented Jan 30, 2018

Rebase seems to be in order and feedback from @mcuadros and @dpordomingo is pending.

@smacker
Copy link
Contributor Author

smacker commented Jan 30, 2018

@bzz This PR waits for documentation PR to be merged. Let's wait for it. Then I will rebase and it should resolve code review comments.

@bzz
Copy link
Contributor

bzz commented Jan 31, 2018

Thanks for keeping the status posted!

I assume that you mean waiting for #29 to be merged.

@dpordomingo
Copy link
Contributor

Since this PR depends on #29 I'll review this one once its dependency has been approved/merged.

@bzz
Copy link
Contributor

bzz commented Feb 7, 2018

@smacker #29 is merged now, could you rebase please, so @mcuadros can take another pass on it

@smacker smacker force-pushed the single_binary branch 6 times, most recently from 4c84aca to 8c75da4 Compare February 8, 2018 11:10
@smacker
Copy link
Contributor Author

smacker commented Feb 8, 2018

Rebased.

Documentation

Please don't pay much attention to Installation section. We will improve it when we prepare a release. Currently, we don't have any but it should happen in a couple of days, so I kept it in README.

Information about how to run it from source has been moved to CONTRIBUTING.

Changes in Makefile

I don't redefine packages target anymore to avoid circular targets or anything else. KISS.
Rules for travis will be:

- make prepare-build
- make packages
- make docker-push

Please do another pass on review.

@bzz
Copy link
Contributor

bzz commented Feb 8, 2018

\cc @mcuadros for another pass on it

@bzz
Copy link
Contributor

bzz commented Feb 13, 2018

While waiting for @mcuadros feedback:

as soon as this introduces a single binary deployment, @smacker do you think it would be reasonable to add a very simple end-to-end test to the CI in this PR as well?

I.e one possible scenarios is

  • build single binary
  • start it
  • curl localhost:8080/index.html or even better a .js resource

this would help checking that bindata is packaged as expected.

@dpordomingo
Copy link
Contributor

@bzz I have a branch with e2e test against many endpoints (/index is one of them)
... so yes, it is feasible.

@smacker
Copy link
Contributor Author

smacker commented Feb 13, 2018

@bzz yeah. we need it. but let's keep it out of this PR.

Copy link
Contributor

@dpordomingo dpordomingo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using this PR as a dependency of #85 to fix some problems I spotted here during the deploy,
so yes: LGTM

node_modules
build
bin
/node_modules

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leftover?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why? now we don't do anything besides copying binary into docker. So we don't need to send node_modules (it can be huge and takes time)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

he is maybe asking about the extra / ??

### Docker

You need to satisfy all [project requirements](#requirements), and then run:
docker run --env-file .env --rm -p 8080:8080 srcd/code-annotation

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find extremely odd having a .env file and not very devops friendly. If we are going ot have it, at least include the definition of each variable in the templete. But I prefer having it explained in the readme and having defaults.

Copy link
Contributor

@dpordomingo dpordomingo Feb 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the .env file is just an easy way to have in one place all the environmental values as expected by the app, but it is up to the developers to decide how they are providing them;
Environment values can be passed as regular vars (ie.UI_DOMAIN=http://127.0.0.1:3000 make gorun) or as regular docker envs (ie. docker run --env OAUTH_CLIENT_ID=id --env OAUTH_CLIENT_SECRET=secret --rm -p 8080:8080 srcd/code-annotation)

Most of them are documented in different parts of the README.md or the CONTRIBUTING.md.

@mcuadros Should we block this PR till everything is properly documented, or can we merge and handle this requirement in a separated issue?

These are the environment values recognized by the app;
most of them are not mandatory, but we can also make "optional" the JWT_SIGNING_KEY.

key required actual default default meaning
OAUTH_CLIENT_ID OAuth app client id
OAUTH_CLIENT_SECRET OAuth app client secret
JWT_SIGNING_KEY jwt_signing_key key to sign the user session JWT
DB_CONNECTION sqlite://internal.db sqlite:///var/code-annotation/internal.db DSN of the internal database
OAUTH_RESTRICT_ACCESS group or team of users that can access the app
OAUTH_RESTRICT_REQUESTER_ACCESS group or team of users granted as Requesters
REACT_APP_SERVER_URL 127.0.0.1:8080 //code-annotation-staging.srcd.run host:port of the API
UI_DOMAIN $(REACT_APP_SERVER_URL) $(REACT_APP_SERVER_URL) host:port of the frontend recieveing the user token after authentication

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.env file is just a simple way to see/edit environmental variables for local development. Most of the values have defaults. All variables must and will be documented. But this PR isn't the right place to do that.

About devops:
In my experience env vars are the most friendly and natural way to configure an application with kubernetes, thanks to configmap,secrets objects and valueFrom in container spec.

And The twelve-factor app also recommends it.

Copy link
Contributor

@bzz bzz Feb 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dpordomingo thank you for great summary!

As on @mcuadros notice, agreed that we should have it clearly documented - have added it to the scope of #67

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smacker where says that is recommended have a .env file? This is not recommended at all, and almost none use this.

Also is bad not having any prefix on your env vars.

If you are speaking about the usage of env vars, of course, is recommended, and actually is a MUST that all source{d} applications

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mcuadros no-no. I mean env vars, not .env file. We use .env file ONLY for local development. Agreed about prefix, we should add it to env vars without prefix! Thanks!

Didn't know it's must, https://github.com/bblfsh/dashboard uses flags instead. But glad to hear.

@bzz
Copy link
Contributor

bzz commented Feb 14, 2018

@mcuadros all feedback seems to be addressed.
Please, let know if the plan from above looks good enough to proceed with merging this PR or you have other suggestions.

@bzz bzz mentioned this pull request Feb 15, 2018
@smacker smacker merged commit ec2e8e8 into src-d:master Feb 15, 2018
dpordomingo pushed a commit that referenced this pull request Mar 26, 2018
fix builds when there are no go files in top level
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants