Skip to content

Fixed getCPUCount | Issue #16268#16457

Merged
alalek merged 21 commits intoopencv:3.4from
ganesh-k13:bugfix/getCPUCount-fix
Feb 26, 2020
Merged

Fixed getCPUCount | Issue #16268#16457
alalek merged 21 commits intoopencv:3.4from
ganesh-k13:bugfix/getCPUCount-fix

Conversation

@ganesh-k13
Copy link
Copy Markdown
Contributor

@ganesh-k13 ganesh-k13 commented Jan 28, 2020

This pullrequest changes

Fixed getCPUCount to account for cgroups.
Added common code to parse CPU files into a generic function for android and Linux
Tested on Linux VMs, docker images, Kubernetes pods.

resolves #16268
resolves #16183

@ganesh-k13 ganesh-k13 requested a review from alalek January 28, 2020 19:12
@ganesh-k13
Copy link
Copy Markdown
Contributor Author

ganesh-k13 commented Jan 29, 2020

Hey @alalek, I do not have the environments of the failing builds, any idea how to solve? I'll fix Android now

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

ganesh-k13 commented Jan 29, 2020

In org.opencv.test.core.CoreTest.testGetNumberOfCPUUs(CoreTest.java:637) this line:

assertTrue(Runtime.getRuntime().availableProcessors() <= cpus);

I'm not sure if it is true.

[EDIT]
I added a check for this

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Checks pass now, open to suggestions.

@alalek
Copy link
Copy Markdown
Member

alalek commented Jan 29, 2020

Thank you for contribution!

Could you please squash commits and rebase onto 3.4 branch?

@ganesh-k13 ganesh-k13 changed the base branch from master to 3.4 January 29, 2020 12:43
Minor new line changes

Android fix | efficient linux checks

Android fix 2

Fixed cpu logic for non linux platforms

Android fix 3

Android fix 4
@ganesh-k13 ganesh-k13 force-pushed the bugfix/getCPUCount-fix branch from ffeca39 to f06c4be Compare January 29, 2020 12:52
@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Thanks @alalek, done.

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

@mshabunin, I have combined some of the android and linux code to reduce duplicates, I have also fixed the logic of cpus-sets by reusing android code.

@ganesh-k13 ganesh-k13 requested a review from mshabunin February 2, 2020 16:27
@mshabunin
Copy link
Copy Markdown
Contributor

Looks like this approach not work with --cpus Docker option, because it does not change cpuset-cpus. As I understand K8S mainly use this option to limit cpu load.

Perhaps we need to handle this case too by dividing cfs_quota to cfs_period and rounding the value, because this parameter supports fractional numbers.

$ docker run --cpus 2 -it ubuntu:18.04
# cat /sys/fs/cgroup/cpu/cpu.cfs_period_us 
100000
# cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us 
200000

$ docker run --cpus 1.3 -it ubuntu:18.04
# cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
# cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
130000

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Oh, thanks @mshabunin for finding this, I'll read up on this and get back to you

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

@mshabunin, I've handled --cpus case. I tested it on docker and a k8s pod as well.

Copy link
Copy Markdown
Contributor

@mshabunin mshabunin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several comments in general:

  • it is not clear which value have bigger priority and how they are chosen: platform-specific values should have bigger priority, if several valid values have been returned, minimum should be chosen
  • preprocessor macros should be made more readable, e.g. CV_HAVE_CGROUPS_V1, CV_HAVE_LINUX_SHED

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Thanks for the review @mshabunin, I'll address them. As for priority, I think we can take minimum most value if we have many options as that is always the most probable value.

@mshabunin
Copy link
Copy Markdown
Contributor

mshabunin commented Feb 4, 2020

@ganesh-k13 , yes minimum should be good. Maybe something like this will be more clear (variables and functions should have meaningful names ofc):

int result = 0;

#if CV_HAVE_METHOD_1
static const int m1 = getCPUS_method1();
if (result > 0 && m1 > 0 && result > m1 || result == 0 && m1 > 0)
    result = m1;
#endif

#if CV_HAVE_METHOD_2
...
#endif

return result > 0 ? result : 1;

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Yes, that is much clearer @mshabunin, I'll do like that. Thanks.

@ganesh-k13 ganesh-k13 requested a review from mshabunin February 7, 2020 13:59
@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Ok so Andriod is just hard when it comes to sched_getaffinity, seems like it's not directly supported due to missing cpu_set_t. Any pointers on how to proceed and should we even include this in Andriod code path? Open to other review comments as well. @alalek.

@asmorkalov asmorkalov added the Hackathon https://opencv.org/opencv-hackathon-starts-next-week/ label Feb 14, 2020
@ganesh-k13 ganesh-k13 requested a review from alalek February 17, 2020 04:15
@asmorkalov
Copy link
Copy Markdown
Contributor

@ganesh-k13 I propose to preserve original implementation for Andoird and do not use cgroups option there.

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

Sure @asmorkalov, will do.

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

@alalek / @asmorkalov / @mshabunin , can you please review?

Copy link
Copy Markdown
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great progress!

@ganesh-k13
Copy link
Copy Markdown
Contributor Author

@alalek and @asmorkalov, I have made some changes. Open to any changes, this was just my first thoughts on handling Linux and Andriod based stuff. Let me know what needs to be done.

Copy link
Copy Markdown
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: core Hackathon https://opencv.org/opencv-hackathon-starts-next-week/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cv::getCPUCount returns number of CPUs instead of available CPUs on Linux Memory leak on K8S

4 participants