646 lab 604 custom job selector label by thetechnocrat-dev · Pull Request #655 · labdao/plex

thetechnocrat-dev · 2023-09-19T00:50:40Z

Changes

Add selector over ride flag to plex init and plex run
Update Plex to use Bacalhau version 1.0.3

Details

Add the -s flag to plex run or plex init to directly pass selectors to Bacalhau. This will overwrite the default selector of owner=labdao.

Example

go run main.go init -s owner=josh -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test

Will produce the error

error submitting Bacalhau job: not enough nodes to run job. requested: 1, available: 0

Because there are no nodes with the selector owner = josh

linear · 2023-09-19T00:50:42Z

LAB-604 custom job selector label

be able to put custom labels on plex jobs that pass through to bacalhau as a label, will be useful as instance type selector for benchmarking

vercel · 2023-09-19T00:50:44Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 19, 2023 2:42pm

alabdao · 2023-09-19T02:49:14Z

If I read the code correctly, if --selector is specified with label=value, it is getting appended to hard-coded owner=labdao (or owner=labdaostaging for staging) making the final selector option to be label=value,owner=labdao. If that's correct, this would cause issues where if we want jobs submitted via plex to run on nodes without owner=labdao set. In short don't want any hard-coded selector specified.

acashmoney · 2023-09-19T14:23:52Z

With selector flag

Expected error case when passing in node selector which doesn't exist works.

go run main.go init -s owner=aakaash -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test
Plex version (v0.10.4) up to date.
Pinned IO JSON CID: QmWhpHj9qWwHETVwCJtK1o5wXvxF8h7oVBxQQKpp4HkDXK
Created working directory: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/fa205bcb-4fff-4875-9a04-4b17cbdc51e4
Initialized IO file at: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/fa205bcb-4fff-4875-9a04-4b17cbdc51e4/io.json
Processing IO Entries
Starting to process IO entry 0 
Error processing IO entry 0 
error submitting Bacalhau job: not enough nodes to run job. requested: 1, available: 0
Finished processing, results written to /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/fa205bcb-4fff-4875-9a04-4b17cbdc51e4/io.json
Completed IO JSON CID: QmQjaFxq4TpynLh7iLDjAAT96pzSL9T9S9TNd15sJqxSVY

Without selector flag

@thetechnocrat-dev however, I got a context deadline exceeded error 2/5 times running without the selector flag.

go run main.go init -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test
Plex version (v0.10.4) up to date.
Pinned IO JSON CID: QmWhpHj9qWwHETVwCJtK1o5wXvxF8h7oVBxQQKpp4HkDXK
Created working directory: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/abad6223-2cca-4aaf-827e-2f1154f5669b
Initialized IO file at: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/abad6223-2cca-4aaf-827e-2f1154f5669b/io.json
Processing IO Entries
Starting to process IO entry 0 
Job running...
Bacalhau job id: 422883ea-6648-4f77-98a0-691775da9d0f 
////_🌱___////
Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

Error processing IO entry 0 
error downloading Bacalhau results: failed to get ipfs cid 'QmdDEZZzjQNEFXeb3dGeS8ta2tbV5HuLMBgN6qsKqdF9fU': context deadline exceeded
Finished processing, results written to /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/abad6223-2cca-4aaf-827e-2f1154f5669b/io.json
Completed IO JSON CID: QmbpaTejgpAxXmu9qyhNMcbEnCHVDANt8oQuBBJ8ecAVqp

The CID QmdDEZZzjQNEFXeb3dGeS8ta2tbV5HuLMBgN6qsKqdF9fU does exist, but it appears DownloadBacalhauResults isn't able to fetch it? It's unclear to me why this works 3/5 times.

thetechnocrat-dev · 2023-09-19T14:50:15Z

@acashmoney thanks for the extra testing. I changed the timeout to 5 minutes, which is what we had before and what bacalhau has set as the default.

@alabdao, good catch I forgot I coded the selector to append last week. It now overrides.

acashmoney

With selector flag

go run main.go init -s owner=aakaash -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test
Plex version (v0.10.4) up to date.
Pinned IO JSON CID: QmWhpHj9qWwHETVwCJtK1o5wXvxF8h7oVBxQQKpp4HkDXK
Created working directory: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/fa3c55e8-9bf9-4d31-925f-9f8671a38315
Initialized IO file at: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/fa3c55e8-9bf9-4d31-925f-9f8671a38315/io.json
Processing IO Entries
Starting to process IO entry 0 
Error processing IO entry 0 
error submitting Bacalhau job: not enough nodes to run job. requested: 1, available: 0
Finished processing, results written to /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/fa3c55e8-9bf9-4d31-925f-9f8671a38315/io.json
Completed IO JSON CID: QmQjaFxq4TpynLh7iLDjAAT96pzSL9T9S9TNd15sJqxSVY

Without selector flag

go run main.go init -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test
Plex version (v0.10.4) up to date.
Pinned IO JSON CID: QmWhpHj9qWwHETVwCJtK1o5wXvxF8h7oVBxQQKpp4HkDXK
Created working directory: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/02a36ee7-ecd7-4aa6-87e7-ec11a6cf98fc
Initialized IO file at: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/02a36ee7-ecd7-4aa6-87e7-ec11a6cf98fc/io.json
Processing IO Entries
Starting to process IO entry 0 
Job running...
Bacalhau job id: 3ac8d6a5-e14e-4b1d-9657-f1d677bdfbd2 
////_🌱___////
Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

Success processing IO entry 0 
Finished processing, results written to /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/02a36ee7-ecd7-4aa6-87e7-ec11a6cf98fc/io.json
Completed IO JSON CID: QmbW5AqM8jdF8dcu2xe16g23Ykirc8TSjWWqESK4XNfbxe

thetechnocrat-dev added 5 commits September 12, 2023 17:16

almost finish bacalhau upgrade and selector patch

7878a7a

merge

2adebc0

go build with bacalhau 1.0.3 works

45cbe8b

new bacalhau version works

89eea59

add back verify cid function

cc1cca4

thetechnocrat-dev linked an issue Sep 19, 2023 that may be closed by this pull request

[LAB-604] custom job selector label #646

Closed

thetechnocrat-dev had a problem deploying to ci September 19, 2023 00:50 — with GitHub Actions Failure

add selector flag to init and fix bacalhau type in test

f0b624c

thetechnocrat-dev had a problem deploying to ci September 19, 2023 00:59 — with GitHub Actions Failure

fix log type

cf02a1c

thetechnocrat-dev temporarily deployed to ci September 19, 2023 01:05 — with GitHub Actions Inactive

thetechnocrat-dev requested review from acashmoney and alabdao September 19, 2023 01:11

make selector flag override instead of append

b59334c

thetechnocrat-dev temporarily deployed to ci September 19, 2023 13:58 — with GitHub Actions Inactive

change download to default ipfs of 5 minutes

f06cf4e

thetechnocrat-dev temporarily deployed to ci September 19, 2023 14:42 — with GitHub Actions Inactive

acashmoney approved these changes Sep 19, 2023

View reviewed changes

alabdao approved these changes Sep 19, 2023

View reviewed changes

thetechnocrat-dev merged commit 10656ed into main Sep 19, 2023

thetechnocrat-dev deleted the 646-lab-604-custom-job-selector-label branch September 19, 2023 15:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

646 lab 604 custom job selector label#655

646 lab 604 custom job selector label#655
thetechnocrat-dev merged 9 commits intomainfrom
646-lab-604-custom-job-selector-label

thetechnocrat-dev commented Sep 19, 2023 •

edited

Loading

Uh oh!

linear bot commented Sep 19, 2023

Uh oh!

vercel bot commented Sep 19, 2023 •

edited

Loading

Uh oh!

alabdao commented Sep 19, 2023

Uh oh!

acashmoney commented Sep 19, 2023 •

edited

Loading

Uh oh!

thetechnocrat-dev commented Sep 19, 2023 •

edited

Loading

Uh oh!

acashmoney left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

thetechnocrat-dev commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Details

Uh oh!

linear bot commented Sep 19, 2023

Uh oh!

vercel bot commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alabdao commented Sep 19, 2023

Uh oh!

acashmoney commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

With selector flag

Without selector flag

Uh oh!

thetechnocrat-dev commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acashmoney left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

With selector flag

Without selector flag

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thetechnocrat-dev commented Sep 19, 2023 •

edited

Loading

vercel bot commented Sep 19, 2023 •

edited

Loading

acashmoney commented Sep 19, 2023 •

edited

Loading

thetechnocrat-dev commented Sep 19, 2023 •

edited

Loading

acashmoney left a comment •

edited

Loading