Workflow: publish Argus, Colossus and QueryNode as docker images by mnaamani · Pull Request #4780 · Joystream/joystream

mnaamani · 2023-06-02T08:47:26Z

Running a storage-node and distributor-node currently requires

cloning the monorepo
building packages
start docker services with local repo mounted as a bind volume

This is not practical for many reasons, and although is great for development cycle it is not a proper production setup. So in this PR we are making the two nodes build-able into proper independent docker images that will get published to docker hub giving operators more flexibility and deployment options in cloud, k8 clusters etc.

proper docker images
fixes found when running nodes as docker images.
- missing direct dependencies
- TS compiler was not copying openapi spec files into compiled lib/ destination (why is not needed when running from the monorepo?) Is it because of not installing dev dependencies?
Update to use node18 -> better performance/security fixes
query-node generated schema.graphql is now placed in the storage-node, distributor-node package so we don't require building query-node ahead of the docker images.

The images can be tested by running integration tests with the docker-compose-no-bind-volumes.yml docker compose file.

Successful run:
https://github.com/Joystream/joystream/actions/runs/5437536353

you should be able to pull the images:

docker pull joystream/storage-node:3.4.0
docker pull joystream/distributor-node:1.2.0
docker pull joystream/query-node:1.2.1

I've prepared a standalone repo for running query-node from built docker image. https://github.com/mnaamani/query-node-standalone

vercel · 2023-06-02T08:47:31Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated (UTC)
pioneer-testnet	⬜️ Ignored (Inspect)		Jul 5, 2023 3:57pm

…o complete

mnaamani · 2023-06-09T08:43:01Z

After updating from master, storage-node as docker image is failing:

2023-06-09 12:40:58 2023-06-09 08:40:58:4058 error: Server error: TypeError: Cannot read properties of undefined (reading 'sync')
2023-06-09 12:40:58 error Command failed with exit code 105.

Looking into it.

mnaamani · 2023-06-09T16:26:29Z

Resolved it by reverting to older verision of mkdirp in storage-node. As there was a conflict with with version used in query-node/codegen. I think we really need to break out the query node into its own repo as it wrecks havock with package dependencies in the yarn worksapce.

…y-node in monorepo

mnaamani · 2023-06-20T07:00:11Z

For some reason in the integration test (content-directory scenario) with the docker images built the CLI is failing to connect to distributor

2023-06-20T06:39:40.309Z integration-tests:job:manage channels and videos through CLI flow 0 failed:
Error: connect ECONNREFUSED 127.0.0.1:3334
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 3334,
  config: {
    url: 'http://127.0.0.1:3334/api/v1/assets/0',
.
.

Tested locally without issues. Something specific to github runner maybe?

mnaamani · 2023-06-20T09:59:03Z

Tested locally without issues. Something specific to github runner maybe?

Again it was dependency issue on building monorepo before docker images.

 (node:28) [MODULE_NOT_FOUND] Error Plugin: @joystream/distributor-cli: Cannot find module '/joystream/node_modules/@joystream/storage-node-client/lib/index.js'. Please verify that the package.json has a valid "main" entry

Will add build step for the storage-node-client in distributor dockerfile.

zeeshanakram3

query-node generated schema.graphql is now placed in the storage-node, distributor-node package so we don't require building query-node ahead of the docker images.

I see that schema.graphql has also been committed to tests/network-tests package, since integration test has direct dependency on building & running the QN, so maybe the tests/network-tests/src/graphql/schema.graphql can be gitignored?
I think same can be said about cli package too, as during local development it has dependency on QN and most of the commands require running QN, so I think it does not make much sense to commit auto-generated code to the repo and it generally also aligns with the decision we made a while ago to un-commit auto generated files (#4640)

TS compiler was not copying openapi spec files into compiled lib/ destination (why is not needed when running from the monorepo?) Is it because of not installing dev dependencies?

First, tsc does not copy non .js/.d.ts files into the outDir directory defined in tsconfig.json (only exception to this is I think .json files).

Now, when running the storage/distributor nodes locally or with docker-compose's bind mount setup, the OPENAPI_SPEC_PATH resolves to the api-spec file in the src directory and not in the lib directory so the nodes bootstrap successfully.

This is because, by default oclif tries to run a typescript project directly using ts-node without compiling to js artifacts. From the oclif documentation:

We generate CLIs and plugins that use ts-node to make it fast to run the TypeScript code without a compilation step. You won't have to mess around with build configuration using oclif.

So why it does not work in the new docker steup? My hunch is that if oclif can't find ts-node in node_modules it would fallback to running the js artifacts In the /lib dir. And since ts-node is defined as devDependency in the package.json, and the docker image is only installing the dependencies (specified by --production flag to the yarn install), and not the dev dependencies.

Update to use node18 -> better performance/security fixes

Shouldn't we also update the compatible node versions in package.json file

joystream/package.json

Lines 67 to 74 in d61783a

    
           "engines": { 
        
             "node": ">=14.0.0", 
        
             "yarn": "^1.22.15" 
        
           }, 
        
           "volta": { 
        
             "node": "14.18.0", 
        
             "yarn": "1.22.15" 
        
           }

zeeshanakram3 · 2023-06-30T17:46:11Z

colossus.Dockerfile

 RUN yarn workspace @joystream/types build
 RUN yarn workspace @joystream/metadata-protobuf build
 RUN yarn workspace storage-node build
+RUN find . -name "node_modules" -type d -prune


Why this command is added here? It seems like a debug relic. This command is looking for all directories named "node_modules" starting from the current directory and printing it on console, the same command also exists in distributor-node.Dockerfile

Here is the response of the above command

./query-node/generated/graphql-server/node_modules ./query-node/node_modules ./query-node/codegen/node_modules ./query-node/mappings/node_modules ./devops/prettier-config/node_modules ./devops/eslint-config/node_modules ./types/node_modules ./node_modules ./tests/network-tests/node_modules ./utils/migration-scripts/node_modules ./utils/api-scripts/node_modules ./cli/node_modules ./metadata-protobuf/node_modules ./joystreamjs/node_modules ./storage-node/node_modules ./storage-node/client/node_modules ./distributor-node/node_modules ./distributor-node/client/node_modules

interesting, perhaps the output was supposed to be piped to xargs to apply rm -fr?
I'll remove it.

zeeshanakram3 · 2023-06-30T18:16:43Z

colossus.Dockerfile

+COPY ./package.json package.json
+COPY ./tsconfig.json ./tsconfig.json

 RUN yarn --frozen-lockfile


The way Dockerfile is structured, the npm dependencies will be installed even if only the code changed and, no new dependencies were added in the package.json files (in short all the subsequent layers needs to be build again after COPY ./storage-node storage-node even if there was only one liner change in storage-node package). Ideally the following build layers should be placed at the top of the Dockerfile,

COPY ./yarn.lock yarn.lock COPY ./types/package.json types/package.json COPY ./metadata-protobuf/package.json metadata-protobuf/package.json COPY ./storage-node/package.json storage-node/package.json COPY ./package.json package.json RUN yarn --frozen-lockfile

so that cached node modules layer can be used whenever there was no change in yarn.lock/package.json files, this approach considerably improves the build time of the images.

I think similar docker build optimisations can also be made in distributor-node.Dockerfile

Great, I'll apply your suggestions and test it out.

zeeshanakram3

Looks good. Just one thing, did you see the last comment from #4780 (review)

Shouldn't we also update the compatible node versions in package.json file
...

WDYT?

mnaamani · 2023-07-05T15:28:49Z

Looks good. Just one thing, did you see the last comment from #4780 (review)

Shouldn't we also update the compatible node versions in package.json file
...

WDYT?

Sorry I totally missed that.
I had actually prepared a separate PR to update for that #4778 which I was planning to finish after this.

Thanks for explaining the oclif/ts-node matter about the openapi spec file not being copied 👍

Regarding checking in the schema.graphql I take your point. I'll revert for cli and integration tests.

zeeshanakram3

LGTM

workflows: publish argus colossus docker images

3e7c288

mnaamani added 3 commits June 2, 2023 14:24

workflow: fix setting variable

c940d49

publish docker images: fix colossus.Dockerfile and use node18

ab0c74f

colossus argus docker builds

3da7158

mnaamani marked this pull request as ready for review June 5, 2023 05:31

mnaamani added colossus argus Argus distributor node labels Jun 5, 2023

fix build of images with buildx, add tests before publishing

d742314

mnaamani force-pushed the publish-docker-apps branch 2 times, most recently from 53e1765 to 5cd1bb4 Compare June 5, 2023 14:43

use correct context in docker/build-push action to reuse built artifacts

ce431f9

mnaamani force-pushed the publish-docker-apps branch from 5cd1bb4 to ce431f9 Compare June 5, 2023 16:56

mnaamani added 5 commits June 5, 2023 21:11

devops: fix rustup install ansible playbook

7c8d9cc

docker-compose fix docker image names for distributor-node

32cc846

fix auth when checking docker manifest

070b67e

skip query-node joystream/apps image

475ee8a

joystream/apps image published only from master branch

0fae438

mnaamani requested review from Lezek123, kdembler and zeeshanakram3 June 6, 2023 10:02

mnaamani added 5 commits June 7, 2023 11:44

common entrypoint in docker-compose, publish latest tag from master

b3cc70c

Merge branch 'master' into publish-docker-apps

3bd88e4

add missing mkdir dependency for storage-node

4eca28f

allow additional time for QN to catchup after 10K channel creating test

420ff2d

allow even more time for storageSync flow to wait for QN processing t…

a6e3a49

…o complete

mnaamani removed the request for review from Lezek123 June 9, 2023 07:04

storage sync wait for qn to sync

da3bdf3

mnaamani marked this pull request as draft June 9, 2023 08:41

mnaamani added 2 commits June 9, 2023 15:24

mkdirp missing in yarn.lock

2ec35de

fix mkdirp version

9eeda06

mnaamani marked this pull request as ready for review June 9, 2023 17:56

mnaamani added 4 commits June 19, 2023 23:04

place schema.graphql into dependant packages directory

08521b7

Merge branch 'master' into publish-docker-apps

f879264

checkin schema.graphql in cli

4077bca

joystream-apps-docker build without depending on pre-building of quer…

e7543fa

…y-node in monorepo

display storage node logs when starting test storage infra

889585e

fix build of storage-node-client in docker image for distributor

d61783a

zeeshanakram3 added the origo Origo Release label Jun 23, 2023

zeeshanakram3 requested changes Jun 30, 2023

View reviewed changes

mnaamani added 4 commits July 1, 2023 01:44

apps.Dockerfile -> query-node.Dockerfile

b7d709a

query-node.Dockerfile build @joystream/js

ff48294

better docker layer caching

873e6b1

missing deps in docker layer

489781b

mnaamani changed the title ~~Add workflow to publish argus and colossus docker images~~ Workflow: publish Argus, Colossus and QueryNode as docker images Jul 3, 2023

mnaamani requested a review from zeeshanakram3 July 3, 2023 12:26

zeeshanakram3 reviewed Jul 4, 2023

View reviewed changes

do not commit auto generated schema.graphql for cli,integration tests

1619781

zeeshanakram3 approved these changes Jul 6, 2023

View reviewed changes

mnaamani merged commit aee116a into master Jul 6, 2023

mnaamani deleted the publish-docker-apps branch July 19, 2023 21:37

mnaamani mentioned this pull request Jul 24, 2023

[Argus] Automated Docker publishing #4764

Closed

	"engines": {
	"node": ">=14.0.0",
	"yarn": "^1.22.15"
	},
	"volta": {
	"node": "14.18.0",
	"yarn": "1.22.15"
	}

Conversation

mnaamani commented Jun 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Jun 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mnaamani commented Jun 9, 2023

Uh oh!

mnaamani commented Jun 9, 2023

Uh oh!

mnaamani commented Jun 20, 2023

Uh oh!

mnaamani commented Jun 20, 2023

Uh oh!

zeeshanakram3 left a comment

Choose a reason for hiding this comment

Uh oh!

zeeshanakram3 Jun 30, 2023

Choose a reason for hiding this comment

Uh oh!

mnaamani Jul 1, 2023

Choose a reason for hiding this comment

Uh oh!

zeeshanakram3 Jun 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mnaamani Jul 1, 2023

Choose a reason for hiding this comment

Uh oh!

zeeshanakram3 left a comment

Choose a reason for hiding this comment

Uh oh!

mnaamani commented Jul 5, 2023

Uh oh!

zeeshanakram3 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mnaamani commented Jun 2, 2023 •

edited

Loading

vercel bot commented Jun 2, 2023 •

edited

Loading

zeeshanakram3 Jun 30, 2023 •

edited

Loading