YPP backend by zeeshanakram3 · Pull Request #40 · Joystream/youtube-synch

zeeshanakram3 · 2022-08-30T09:52:11Z

implements

YoutubeSync

Required Stack

Docker
aws cli

Enter localhost:3001/docs to view all api endpoints and their descriptions

For user verification flow only two endpoints are needed

/users/verify

/users/verify-and-save

Running the app

Install dependencies
yarn
Deploy all the necessary infrastructure

cd packages/infrastructue
./deploy.sh

This script deploys the relevant dynamoDB tables for needed for the verifications flow.

Start the YPP backend server
yarn nx run api:serve

This will start the YPP backend service at localhost:3001

Start the frontend app
yarn nx run app:serve

Open the browser window at localhost:4200 to interact with the app. Click VERIFY YOUR YOUTUBE CHANNEL button to start the verification flow. If verification is successful the YPP will respond back with email and userId of the verified user. (See response type for /users/verify endpoint at localhost:3001/docs).

After that the frontend app need to prepare the request for save the verified channel to the YPP backend state. There is no UI for that so, you can use any api client(e.g. Postman) for making request to users/verify-and-save with provided values of userId, joystreamChannelId, email, and optional referrerId

kdembler

Quick first round of feedback after testing:

Error handling

I think we should introduce some kind of errorCode to make it easier to distinct between different error types. Currently if some validation is failed error is always Bad request, we could repurpose that to create some list of unique error codes to map to specific errors.

For example, if my channel doesn't meet some criteria I will see:

{
  "statusCode": 400,
  "message": "Channel UCCs9_brR2ADhDVuhrVTjKfQ with 0 subscribers does not \n          meet Youtube Partner Program requirement of 100 subscribers",
  "error": "Bad Request"
}

This is problematic because:
a) to understand what kind of error happened we need to do message string matching which can be risky if error message changes in the future
b) we need to parse the message to understand what is the requirement (100 subscribers)

Another issue is that when verifying the eligibility, code will return error on first error found. However, to properly present the status to the user we actually need to see all the failing criteria, not just the first one.

I would suggest error format along those lines:

{
  "errors": [
    {
      "errorCode": "CHANNEL_CRITERIA_UNMET_SUBSCRIBERS",
      "message": "Channel UCCs9_brR2ADhDVuhrVTjKfQ with 0 subscribers does not meet Youtube Partner Program requirement of 100 subscribers",
      "subscribers": 0,
      "requiredSubscribers": 100
    },
    {
        "errorCode": "CHANNEL_CRITERIA_UNMET_VIDEOS",
        "message": "Channel UCCs9_brR2ADhDVuhrVTjKfQ does not meet Youtube Partner Program requirement of at least 10 videos, each 1 month old",
        "videos": 3,
        "requiredVideos": 10
    }
  ]
}

Let me know how you think feasible this is.

Issues

When I query /users/{userId}/videos I get 500 server error
When I query /channels/{id}, what is the ID that should be passed here? Seems whatever I pass (I tried user ID, joystream channel ID) I get back

{
  "statusCode": 400,
  "message": "The number of conditions on the keys is invalid"
}

When I query /users I get 500 server error

packages/ytube/src/lib/youtubeClient.ts

zeeshanakram3 · 2022-09-07T07:42:24Z

I would suggest error format along those lines: ...

Addressed error changes as suggested in the review

When I query /users/{userId}/videos I get 500 server error

Fixed

When I query /channels/{id}, what is the ID that should be passed here? Seems whatever I pass (I tried user ID, joystream channel ID) I get back

Fixed. You should pass the Joystream ChannelId as parameter. I have updated the description

When I query /users I get 500 server error

Fixed

packages/ytube/src/lib/youtubeClient.ts

kdembler

The API still fails to start if it fails to connect to RPC endpoint. The worst thing about it is that it fails silently and doesn't really indicate that the app failed to start. We should do one of the following:
a) Make the app work without a successful RPC connection, since that isn't really needed at the moment (for the YPP scope)
b) Make it clear that the app failed to load because of failed RPC connection

kdembler · 2022-09-09T10:38:39Z

Also noticed that if I call verify (I will get my YT email back) and then call verify-and-save with different email, the different email is not saved and my user still has the YT email attached when I query /users

zeeshanakram3 · 2022-09-09T15:49:23Z

The API still fails to start if it fails to connect to RPC endpoint

Addressed. Now API won't crash on Node version > 14

Also noticed that if I call verify (I will get my YT email back) and then call verify-and-save with different email, the different email is not saved and my user still has the YT email attached when I query /users

Fixed.

kdembler

Additional points:

Seems this wasn't resolved? I still see the same behavior

Also noticed that if I call verify (I will get my YT email back) and then call verify-and-save with different email, the different email is not saved and my user still has the YT email attached when I query /users

Fixed.

We discussed changing the endpoints 👇 Did you decide to skip this after all?

/users/verify would be just POST to /users as this call creates a user (it also saves some data, not just verifies)
/users/verify-and-save would be POST to /channels as this call creates an entry in /channels list

kdembler · 2022-09-21T11:09:49Z

packages/ytube/src/lib/youtubeClient.ts

+
+// YPP induction criteria, each channel should meet following criteria
+const MINIMUM_SUBSCRIBERS_COUNT = 50
+const MINiMUM_VIDEO_COUNT = 10


Typo, should be MINIMUM, all caps

addressed in 5c475a3

kdembler · 2022-09-21T11:10:09Z

packages/ytube/src/lib/youtubeClient.ts

+        message:
+          `Channel ${channel.id} with ${channel.statistics.videoCount} videos does not meet Youtube ` +
+          `Partner Program requirement of at least ${MINiMUM_VIDEO_COUNT} videos, each 1 month old`,
+        result: channel.statistics.videoCount,


I think this should return videos.length instead, right?

addressed in 5c475a3

kdembler · 2022-09-21T11:11:09Z

packages/ytube/src/lib/youtubeClient.ts

+        errorCode: ExitCodes.CHANNEL_CRITERIA_UNMET_VIDEOS,
+        message:
+          `Channel ${channel.id} with ${channel.statistics.videoCount} videos does not meet Youtube ` +
+          `Partner Program requirement of at least ${MINiMUM_VIDEO_COUNT} videos, each 1 month old`,


each 1 month old should use MINIMUM_VIDEO_AGE_MONTHS

addressed in 5c475a3

kdembler · 2022-09-21T11:11:28Z

packages/ytube/src/lib/youtubeClient.ts

+        errorCode: ExitCodes.CHANNEL_CRITERIA_UNMET_CREATION_DATE,
+        message:
+          `Channel ${channel.id} with creation time of ${channel.publishedAt} does not ` +
+          `meet Youtube Partner Program requirement of channel being at least 3 months old`,


at least 3 months old should use MINIMUM_CHANNEL_AGE_MONTHS

addressed in 5c475a3

zeeshanakram3 · 2022-09-22T09:15:16Z

Seems this wasn't resolved? I still see the same behavior

Sorry, there was an issue in the previous fix. Now resolved

We discussed changing the endpoints 👇 Did you decide to skip this after all?

Overlooked that before. Now resolved, now verify-and-save is POST to /channels

kdembler

Looks good to me, great job!

zeeshanakram3 added 11 commits July 25, 2022 22:30

migrate from npm to yarn

4b4ff42

fix: prettier issues

15d2418

updated .edutorconfig max_line_length option

fc3b690

updated @joystream/types & @joystream/metadata-protobuf dependency

5607306

fix; prettier issues II

0ddc14e

added 'lines-between-class-members' eslint rule

c0f511b

fix: prettier issues III

f4b258d

added swagger api spec json file

0e0ae02

added email validation on verify-channel endpoint

ec30a97

bumped polkadot keyring version

6d257f6

updated dynamodb schema & youtube client for new verification flow

83c5fbf

kdembler suggested changes Sep 5, 2022

View reviewed changes

packages/ytube/src/lib/youtubeClient.ts Outdated Show resolved Hide resolved

address first review/feedback changes

abc4a62

add auth to /save-and-verify endpoint

4a5b180

kdembler reviewed Sep 9, 2022

View reviewed changes

packages/ytube/src/lib/youtubeClient.ts Outdated Show resolved Hide resolved

kdembler suggested changes Sep 9, 2022

View reviewed changes

address second review/feedback changes

25d7d69

zeeshanakram3 mentioned this pull request Sep 20, 2022

Youtube sync prototype #41

Merged

kdembler suggested changes Sep 21, 2022

View reviewed changes

address third review/feedback changes

5c475a3

updated POST endpoints channels & users

ab85467

kdembler approved these changes Sep 22, 2022

View reviewed changes

zeeshanakram3 merged commit 1474758 into Joystream:main Sep 22, 2022

This was referenced Oct 5, 2022

MVP BE: Receive Call from Atlas #28

Closed

Validation on Verification #29

Closed

Add Linting rules to youtube-synch codebase #35

Closed

zeeshanakram3 mentioned this pull request Oct 5, 2022

Visualisation of status and Tier for creators in Atlas #30

Closed

zeeshanakram3 mentioned this pull request Oct 14, 2022

Yt sync opt out #47

Merged

dmtrjsg mentioned this pull request Nov 16, 2022

YPP Test Script Joystream/atlas#3439

Closed

31 tasks

Conversation

zeeshanakram3 commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

YoutubeSync

Required Stack

Running the app

Uh oh!

kdembler left a comment

Choose a reason for hiding this comment

Error handling

Issues

Uh oh!

Uh oh!

zeeshanakram3 commented Sep 7, 2022

Uh oh!

Uh oh!

kdembler left a comment

Choose a reason for hiding this comment

Uh oh!

kdembler commented Sep 9, 2022

Uh oh!

zeeshanakram3 commented Sep 9, 2022

Uh oh!

kdembler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeeshanakram3 commented Sep 22, 2022

Uh oh!

kdembler left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zeeshanakram3 commented Aug 30, 2022 •

edited

Loading