Thanks for the delivery. We will look into it as soon as possible.

@randombishop thanks for your patience! I just took a first look at the deliverables and noticed that in the How we built on top of Subquery of your article you made 2 titles starting with "3." (3. Defining our dataset and 3. Mapping logic).

That being said, I'm going to start reviewing your other deliverables now and be back with more feedback soon.

@randombishop after following your build instructions I only have 4 docker containers up and running:

% docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED         STATUS                   PORTS                              NAMES
dc668fc317ab   onfinality/subql-query:v0.11.0           "/sbin/tini -- /usr/…"   2 minutes ago   Up 2 minutes             0.0.0.0:3000->3000/tcp             subql_wika_graphql-engine_1
88c627543a2c   onfinality/subql-node:v0.27.2            "/sbin/tini -- /usr/…"   2 minutes ago   Up 2 minutes (healthy)                                      subql_wika_subquery-node_1
dc3bbec77748   docker.elastic.co/kibana/kibana:7.16.3   "/bin/tini -- /usr/l…"   3 minutes ago   Up 3 minutes             0.0.0.0:5601->5601/tcp             subql_wika_kibana_1
9f9e8625cf9b   postgres:12-alpine                       "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes (healthy)   5432/tcp, 0.0.0.0:5433->5433/tcp   subql_wika_postgres_1

However, according to your Postgres, Neo4j and Elastic Search databases section the containers mentioned should also be running and accessible on their corresponding ports on localhost, correct? Am I missing something?

@randombishop thanks for your patience! I just took a first look at the deliverables and noticed that in the How we built on top of Subquery of your article you made 2 titles starting with "3." (3. Defining our dataset and 3. Mapping logic).

That being said, I'm going to start reviewing your other deliverables now and be back with more feedback soon.

Thanks a lot for the catch, just made the correction in the article

@randombishop after following your build instructions I only have 4 docker containers up and running:

% docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED         STATUS                   PORTS                              NAMES
dc668fc317ab   onfinality/subql-query:v0.11.0           "/sbin/tini -- /usr/…"   2 minutes ago   Up 2 minutes             0.0.0.0:3000->3000/tcp             subql_wika_graphql-engine_1
88c627543a2c   onfinality/subql-node:v0.27.2            "/sbin/tini -- /usr/…"   2 minutes ago   Up 2 minutes (healthy)                                      subql_wika_subquery-node_1
dc3bbec77748   docker.elastic.co/kibana/kibana:7.16.3   "/bin/tini -- /usr/l…"   3 minutes ago   Up 3 minutes             0.0.0.0:5601->5601/tcp             subql_wika_kibana_1
9f9e8625cf9b   postgres:12-alpine                       "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes (healthy)   5432/tcp, 0.0.0.0:5433->5433/tcp   subql_wika_postgres_1

However, according to your Postgres, Neo4j and Elastic Search databases section the containers mentioned should also be running and accessible on their corresponding ports on localhost, correct? Am I missing something?

There are a couple of possible causes:

• docker engine not having enough memory to run all services (someone from the team had this issue, they documented it in trouble_shooting.md
• also, i found an issue when installing it in a different os, where elastic search was failing because of the rights on its data folder. running sudo chown -R 1000:1000 .data/es fixed it for me. (Elasticsearch 5 won't start if data directory is owned by root, or not created pelias/docker#33)
  will add a note about this in trouble_shooting.md as well...
  Please let me know if one of those things works...
  Also, you can try docker logs in another terminal while the problematic container is starting to have more info about what is causing it to fail...
  Thanks again for reviewing and testing!

@takahser FYI, started a branch to work on the review feedback: https://github.com/randombishop/wika_etl/compare/review_w3f?expand=1

Hey @randombishop
I tried running chown within the elasticsearch container:

% docker exec -it e1529f197395 /bin/bash
bash-4.4$ sudo chown -R 1000:1000 .data/es

However sudo is not installed/activated which is why the command fails with:

bash: sudo: command not found

Regarding RAM, I still have ~2GB free (of 16GB) on my machine. But I can try on a different, more powerful machine. I'll get back to you in this regard after.

Hey @randombishop I tried running chown within the elasticsearch container:

% docker exec -it e1529f197395 /bin/bash
bash-4.4$ sudo chown -R 1000:1000 .data/es

However sudo is not installed/activated which is why the command fails with:

bash: sudo: command not found

Regarding RAM, I still have ~2GB free (of 16GB) on my machine. But I can try on a different, more powerful machine. I'll get back to you in this regard after.

You have to run the chown in the host machine, not inside the container (because that .data/es directory is mounted from host to container.)

@randombishop so on the more powerful machine I have more containers, probably RAM was actually an issue.

CONTAINER ID   IMAGE                                    COMMAND                  CREATED          STATUS                    PORTS                                                                                            NAMES
0003eba6a70e   onfinality/subql-query:v0.11.0           "/sbin/tini -- /usr/…"   15 minutes ago   Up 15 minutes             0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                        subql_wika_graphql-engine_1
f2965e6beba3   onfinality/subql-node:v0.27.2            "/sbin/tini -- /usr/…"   15 minutes ago   Up 15 minutes (healthy)                                                                                                    subql_wika_subquery-node_1
1f39c2d7e455   postgres:12-alpine                       "docker-entrypoint.s…"   15 minutes ago   Up 15 minutes (healthy)   5432/tcp, 0.0.0.0:5433->5433/tcp, :::5433->5433/tcp                                              subql_wika_postgres_1
3efa023bd3e8   neo4j                                    "tini -g -- /startup…"   15 minutes ago   Up 15 minutes (healthy)   0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp   subql_wika_neo4j_1
35f7f8671fc6   docker.elastic.co/kibana/kibana:7.16.3   "/bin/tini -- /usr/l…"   15 minutes ago   Up 15 minutes             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                        subql_wika_kibana_1

I started having a look at the postgres node and I can connect to it. However, there are only the following dbs available:

Unfortunately, I can't find any of the tables mentioned in your instructions (e.g. block_infos) - The postgres default db has only 1 table, the template0 db is not accessible and the template1 db has no tables at all:

What's the supposed db name and do you have any idea why the tables don't show up?

Hey!

• Looks like the elastic search one is still going down. there should be 6 services total.
• In your postgres, there should be 2 schemas, you want to look inside the app one

Also, once you have everything up and running, you can try submitting a like transaction to the test net and check its sync with your databases. You can do that by using the frontend at https://app.wika.network
You'll also need a polkadot.js extension with some Wikas, please send me your address and I'll send you testnet Wikas...

@randombishop thx, yes, after switching to the app schema I can see the tables.
My address is: 5CrXzGDramfy1QrfXpXC4k9eQb4ZtFkVLdNq1Rr2XYGSN63t

I was checking the other dbs:

• Kibana shows me only "Kibana server is not ready yet" (it's been ~1h since I started the containers)

• Neo4j started but I see no nodes/labels:

  

Any ideas?

• I guess Kibana is not ready because you still have an issue with Elastic Search container, it doesn't show as started in your latest docker ps
• Just sent you 100k Testnet Wikas for testing
• It's normal that you don't see anything in the tables, there's no traffic currently on test net. You can submit a like through app.wika.network to generate some data...
• You can also run the test suite to generate data (login to the subquery container, cd app, then yarn test)
• Also, I think you should fix the elastic search container issue before because the ETL will probably fail to sync if one of the databases is down.
  Hope this helps and looking forward to having you all set up with this repo!

@randombishop Thanks for your patience.

Just sent you 100k Testnet Wikas for testing

Thanks, I received them. 👍

Also, I think you should fix the elastic search container issue before because the ETL will probably fail to sync if one of the databases is down.

So I managed to resolve these issues however, now my graphql-engine wouldn't start up anymore.

$ sudo docker-compose up
Creating network "subql_wika_default" with the default driver
Creating subql_wika_kibana_1   ... done
Creating subql_wika_es_1       ... done
Creating subql_wika_neo4j_1    ... done
Creating subql_wika_postgres_1 ... done
Creating subql_wika_subquery-node_1 ... done

ERROR: for graphql-engine  Container "8c532658ae8a" is unhealthy.
ERROR: Encountered errors while bringing up the project.

In your Dockerfile I saw that it relies on the subquery-node container to be up and running which is not the case for me:

$ sudo docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS                    PORTS                                                                                            NAMES
f7ca9e0ecad5   postgres:12-alpine                                     "docker-entrypoint.s…"   25 seconds ago   Up 23 seconds (healthy)   5432/tcp, 0.0.0.0:5433->5433/tcp, :::5433->5433/tcp                                              subql_wika_postgres_1
f4dac8a2fd44   docker.elastic.co/elasticsearch/elasticsearch:7.16.3   "/bin/tini -- /usr/l…"   25 seconds ago   Up 23 seconds             0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp                                              subql_wika_es_1
927098d34d27   neo4j                                                  "tini -g -- /startup…"   25 seconds ago   Up 24 seconds (healthy)   0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp   subql_wika_neo4j_1
2d040d46ff65   docker.elastic.co/kibana/kibana:7.16.3                 "/bin/tini -- /usr/l…"   25 seconds ago   Up 24 seconds             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                        subql_wika_kibana_1

Have you ever experienced this error in the past? It's weird because these services used to be running. What I also tried was restarting the containers with docker-compose down and docker-compose up but that didn't help.

If it helps, here is the log data that also indicates that the connection (to the subquery-node I guess, although the PermissionError seems suspicious) is failing:

$ docker-compose logs 8c532658ae8a
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1277, in request
  File "http/client.py", line 1323, in _send_request
  File "http/client.py", line 1272, in endheaders
  File "http/client.py", line 1032, in _send_output
  File "http/client.py", line 972, in send
  File "docker/transport/unixconn.py", line 43, in connect
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 410, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1277, in request
  File "http/client.py", line 1323, in _send_request
  File "http/client.py", line 1272, in endheaders
  File "http/client.py", line 1032, in _send_output
  File "http/client.py", line 972, in send
  File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker/api/client.py", line 214, in _retrieve_server_version
  File "docker/api/daemon.py", line 181, in version
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 237, in _get
  File "requests/sessions.py", line 543, in get
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 81, in main
  File "compose/cli/main.py", line 200, in perform_command
  File "compose/cli/command.py", line 70, in project_from_options
  File "compose/cli/command.py", line 153, in get_project
  File "compose/cli/docker_client.py", line 43, in get_client
  File "compose/cli/docker_client.py", line 170, in docker_client
  File "docker/api/client.py", line 197, in __init__
  File "docker/api/client.py", line 222, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[561549] Failed to execute script docker-compose

takahser thanks for making progress and debugging together, never mind about the delay...
Could you try:

• docker-compose down everything
• delete .data/postgres folder
• reset the startBlock in project.yml file to a recent a block number
• docker-compose up again
  The subquery-node could be failing because the database has some old data and it can't sync up from there because the data is not available anymore in-chain...
  Also, in case it's still failing, can you please try docker logs subql_wika_subquery-node_1 -f in a different terminal as soon as you see Creating subql_wika_subquery-node_1 and before it fails. That way we can know what is going on in that service...

@randombishop thanks, removing the .data/postgres directory actually helped.
I think now all the containers are up and running:

$ docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                                            NAMES
cc59d3663e28   onfinality/subql-query:v0.11.0                         "/sbin/tini -- /usr/…"   9 minutes ago   Up 9 minutes             0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                        subql_wika-graphql-engine-1
55b5d99bb8ed   onfinality/subql-node:v0.27.2                          "/sbin/tini -- /usr/…"   9 minutes ago   Up 9 minutes (healthy)                                                                                                    subql_wika-subquery-node-1
2e5cd7b53711   docker.elastic.co/elasticsearch/elasticsearch:7.16.3   "/bin/tini -- /usr/l…"   9 minutes ago   Up 9 minutes             0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp                                              subql_wika-es-1
9b8db754a9d6   postgres:12-alpine                                     "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes (healthy)   5432/tcp, 0.0.0.0:5433->5433/tcp, :::5433->5433/tcp                                              subql_wika-postgres-1
37fb3fb02761   docker.elastic.co/kibana/kibana:7.16.3                 "/bin/tini -- /usr/l…"   9 minutes ago   Up 9 minutes             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                        subql_wika-kibana-1
cfb7dcd95281   neo4j                                                  "tini -g -- /startup…"   9 minutes ago   Up 9 minutes (healthy)   0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp   subql_wika-neo4j-1

I was also able to successfully submit some likes on your frontend, pretty cool!
I'll do another review iteration and give you feedback after.

@randombishop thanks, removing the .data/postgres directory actually helped. I think now all the containers are up and running:

$ docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                                                                                            NAMES
cc59d3663e28   onfinality/subql-query:v0.11.0                         "/sbin/tini -- /usr/…"   9 minutes ago   Up 9 minutes             0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                        subql_wika-graphql-engine-1
55b5d99bb8ed   onfinality/subql-node:v0.27.2                          "/sbin/tini -- /usr/…"   9 minutes ago   Up 9 minutes (healthy)                                                                                                    subql_wika-subquery-node-1
2e5cd7b53711   docker.elastic.co/elasticsearch/elasticsearch:7.16.3   "/bin/tini -- /usr/l…"   9 minutes ago   Up 9 minutes             0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp                                              subql_wika-es-1
9b8db754a9d6   postgres:12-alpine                                     "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes (healthy)   5432/tcp, 0.0.0.0:5433->5433/tcp, :::5433->5433/tcp                                              subql_wika-postgres-1
37fb3fb02761   docker.elastic.co/kibana/kibana:7.16.3                 "/bin/tini -- /usr/l…"   9 minutes ago   Up 9 minutes             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                        subql_wika-kibana-1
cfb7dcd95281   neo4j                                                  "tini -g -- /startup…"   9 minutes ago   Up 9 minutes (healthy)   0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp   subql_wika-neo4j-1

I was also able to successfully submit some likes on your frontend, pretty cool! I'll do another review iteration and give you feedback after.

Amazing, thanks a lot!
While it's still hot in your memory, can you please document the problems you had in https://github.com/randombishop/wika_etl/blob/master/trouble_shooting.md
Would be super helpful for the next users...
(you can just drop quick bullet points and I'll elaborate on them if needed...)

@randombishop here you go: randombishop/wika_etl#12

@randombishop Thanks for your patience! I just pushed my evaluation.
Please let me know if you have any questions.

@randombishop Thanks for your patience! I just pushed my evaluation. Please let me know if you have any questions.

@takahser Thanks a lot for the evaluation and feedback, good job!

We'll work right away on:

• Adding inline comments in the code for the mapping handlers, the plugins and other core functions.
• Adding tests for the mapping handlers

In the meantime, some comments about the other points:

• PluginElasticSearch tests failing. I suspect this only happened because the elastic search container was not up and running yet when you started the tests, it sometimes takes a minute before it's ready. Can you please try again after you make sure ES is running?

• PluginEmails failing. This one does fail indeed when EMAIL_ALERT_ENABLE is set to zero. I thought it would be a good signal that it's not configured. We could silence that test in that case but imo it's good to remind the user that it's not configured properly. That one is actually an integration test rather than a unit test so you need to have a real email sending service configured for it to work. Let me know how you would like us to fix that part...

• Configuration with environment variables. I added the following in the readme:

All configuration options are read from environment variables.
For dockerized users, they are all defined in the docker-compose file.
When not using docker, you can write a file to set the environment variables, or define them in .profile file.

We could also add an example linux script that sets the env variables with EXPORT but imo it would be redundant? But let me know and we can easily add it...

I think that's pretty much it, will get back to you with a PR by the end of week with the new comments and tests, excited to finalize this milestone 💯

PS: Also noted your recommendation to apply some refactoring and avoid duplicated code, working on it...

@randombishop

PluginElasticSearch tests failing

• I was able to fix the ES issues and added a PR to document how to fix it: Add instructions to fix graphql-engine error randombishop/wika_etl#13
• Now only 2 tests are failing, incl. the mail test:

35 passing (1s)
  2 failing

  1) PluginEmails
       should send a test email:
     TypeError: Only absolute URLs are supported
      at getNodeRequestOptions (node_modules/node-fetch/lib/index.js:1327:9)
      at /app/node_modules/node-fetch/lib/index.js:1440:19
      at new Promise (<anonymous>)
      at fetch (node_modules/node-fetch/lib/index.js:1437:9)
      at PluginEmails.sendError (dist/plugins/emails.js:35:57)
      at Context.<anonymous> (dist/tests/tests_emails.js:8:22)
      at processImmediate (node:internal/timers:464:21)

  2) fetchMetadata
       for valid urls
         should work for https://github.com/randombishop/wika_etl:

      AssertionError: expected 'https://opengraph.githubassets.com/4c96abea53e105751300eab9412a10d78636768326ffe58418e6dfc5667ec15b/randombishop/wika_etl' to equal 'https://opengraph.githubassets.com/1eb16d9d35b2a179a6b0c41c4fc97356852d3457a40196d1bf73dabc1717d5d8/randombishop/wika_etl'
      + expected - actual

      -https://opengraph.githubassets.com/4c96abea53e105751300eab9412a10d78636768326ffe58418e6dfc5667ec15b/randombishop/wika_etl
      +https://opengraph.githubassets.com/1eb16d9d35b2a179a6b0c41c4fc97356852d3457a40196d1bf73dabc1717d5d8/randombishop/wika_etl
      
      at Context.<anonymous> (dist/tests/tests_metadata.js:33:51)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)



error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

PluginEmails failing

Well, the expectation would certainly be that one could successfully run the tests without manually introducing additional infrastructure. Maybe you can use some kind of email sending mock service or spin up a container containing a mail server within the docker network? Just an idea.

Configuration with environment variables

In that case I'm fine with that. Just please make sure to update the README accordingly, as you suggested. 👍

Looking forward to your update!

@takahser Here is a big PR for you! :)
randombishop/wika_etl#14

This should address all open issues, except EmailsPlugin testing without the external API endpoint.
Also, I retested all features manually and with yarn test and it's all working after the refactoring.

The problem with EmailsPlugin is that I didn't find a standard email library that works inside the Subquery sandbox (anything that depends on events is forbidden in the sandbox.)
For this reason, it will require some work to add an email server docker image and adapt the plugin accordingly.
IMO, it's not a core feature of the project so we should simply keep it as an example implementation, and leave it to the users to adapt it depending on their environment, they could even rewrite that plugin to make it send alerts to slack or any other recipient.
Ideally, this should be a capability in Subquery itself, like airflow and other ETLs provide easy ways to configure alerting to email or slack.
Submitted a feature request here: subquery/subql#841

Finally, I added you as a contributor to the repo, feel free to comment or contribute directly to the review_w3f branch.

Looking forward to hearing back from you on this, cheers!

@randombishop thanks, I added my comments directly to your PR.

Thanks also for filing the feature request and elaborating on the email issue. Although the solution is imho not optimal, I'm fine with it, considering the circumstances. 👍

I retested all features manually and with yarn test and it's all working after the refactoring

I didn't run them yet, but I'll do soon and let you know the results.

Finally, I added you as a contributor to the repo

Thanks for putting so much trust in me, definitively appreciated! 🙏

Hey @takahser thanks for reviewing the PR, learnt a lot by going through it.
I addressed all your comments:

• Added eslint and fixed all its recommendations.
• Added prettier to apply standard formatting.
• Added a note about how to use lint and format commands in the readme
• Applied the little details improvements.
• Added index.ts in the tests to force disabling postgres.
• All tests PASS.

Thanks for putting so much trust in me, definitively appreciated! 🙏

Of course, you're welcome to join our discord as well to be part of the Wika community: https://discord.com/invite/sPEwJUmAwH

Cheers!

@randombishop I just accepted your delivery, thanks for all the work you put into this!
You can find my final evaluation here. It was fun working with you on this, I think you made a lot of significant improvements and I'm happy with the results.
Looking forward to your next milestone! 👍

Great news @takahser ! Thank you as well for all the time and guidance on this codebase. Looking forward to next milestones...

Also, is there any action we need to take to get the invoice processed or just wait? How long does it typically take?

@randombishop somebody from the invoice team will contact here as soon as they've processed the payment or if they have any questions, I've already forwarded the invoice.
Btw, I'd like to add your project to our tech stack but I'm not sure into which category your project belongs... Social networking is the closest I can think of. Introducing a new category would also be possible if it makes sense. So feel free to share your opinion on that. (:

@randombishop somebody from the invoice team will contact here as soon as they've processed the payment or if they have any questions, I've already forwarded the invoice.

Fantastic, thank you!

Btw, I'd like to add your project to our tech stack but I'm not sure into which category your project belongs... Social networking is the closest I can think of. Introducing a new category would also be possible if it makes sense. So feel free to share your opinion on that. (:

Social Networking fits, super glad to join the list!

hi @randombishop we transferred the payment today. Thanks!

@RouvenP This is the first tx ever received by the project and means a lot, big thanks!
💯