Skip to content

Cache result of find_packages_allowing_duplicates to speed up find_packages#111

Closed
garaemon wants to merge 1 commit intoros-infrastructure:masterfrom
garaemon:cache-result-of-find_pakages_allowing_duplicates
Closed

Cache result of find_packages_allowing_duplicates to speed up find_packages#111
garaemon wants to merge 1 commit intoros-infrastructure:masterfrom
garaemon:cache-result-of-find_pakages_allowing_duplicates

Conversation

@garaemon
Copy link
Copy Markdown

@garaemon garaemon commented Dec 9, 2014

On my environment, roslaunch is super slow. (hydro, ubuntu 12.04, catkin 0.2.6)
Because I have a lot of packages under catkin workspace:

$ rospack list | wc -l
485
$ source /opt/ros/hydro/setup.zsh; rospack list| wc -l
256

For example, roslaunch openni2_launch openni2.launch takes 43 seconds.
After applying this patch, it takes 9 seconds.

@garaemon garaemon force-pushed the cache-result-of-find_pakages_allowing_duplicates branch from 5819c5a to 725edf7 Compare December 9, 2014 12:55
@dirk-thomas
Copy link
Copy Markdown
Member

You can not simply make this function call cache every previous result. This must allow the user calling the API to choose which behavior he wants. And for backward compatibility it must very likely be an opt-in for caching.

@garaemon
Copy link
Copy Markdown
Author

garaemon commented Dec 9, 2014

I think there is two ways to implement that.

  1. add optional argument to turn caching on
  2. add new variable like catkin_pkg.packages.enable_cache to toggle caching

I prefer 2 because it might be difficult to sync version of catkin_pkg, roslib and roslaunch and we can turn on caching without version problem.

@dirk-thomas
Copy link
Copy Markdown
Member

Enabling it globally will almost guarantee problems since no current code expects the calls to be cached.

@garaemon
Copy link
Copy Markdown
Author

garaemon commented Dec 9, 2014

Do you think optional argument is reasonable?

@dirk-thomas
Copy link
Copy Markdown
Member

The optional argument is trivial. The difficult part will be where to use it safely.

@dirk-thomas
Copy link
Copy Markdown
Member

Instead of try to fix it here you might want to dig in and figure out where this function is called multiple times in order to "fix" the caller side by use local caching there.

@garaemon
Copy link
Copy Markdown
Author

garaemon commented Dec 9, 2014

Instead of try to fix it here you might want to dig in and figure out where this function is called multiple times in order to "fix" the caller side by use local caching there.

  • speed of roslaunch depends on performance of roslib.packages.find_node and roslib.packages.find_resource because it calls them several times.
    They are used at here and here
  • roslib.packages.find_node and roslib.packages.find_resource uses catkin.find_in_workspaces.find_in_workspaces at here
  • catkin.find_in_workspaces.find_in_workspaces uses catkin_pkg.packages.find_package at here
  • find_packages_allowing_duplicates is used in catkin.find_in_workspaces.find_in_workspaces

catkin_pkg.packages.find_package lists up all the catkin packages in order to find one package every time. The list of catkin packages is consumed inside of find_package function and from the caller side, it's not revealed.

@dirk-thomas
Copy link
Copy Markdown
Member

The second location (https://github.com/ros/ros_comm/blob/indigo-devel/tools/roslaunch/src/roslaunch/rlutil.py#L228) looks interesting. Can you please modify it locally to pass the additional optional keyword argument rospack and report how that improves your timing?

@garaemon
Copy link
Copy Markdown
Author

I see.
Tomorrow I will send a series of patches

2014年12月10日水曜日、Dirk Thomasnotifications@github.comさんは書きました:

The second location (
https://github.com/ros/ros_comm/blob/indigo-devel/tools/roslaunch/src/roslaunch/rlutil.py#L228)
looks interesting. Can you please modify it locally and pass the additional
optional argument rospack and see how that affects your timing?


Reply to this email directly or view it on GitHub
#111 (comment)
.

from iPhone

@garaemon
Copy link
Copy Markdown
Author

This week I'm a little bit busy and I will dig this issue next week

@garaemon
Copy link
Copy Markdown
Author

I will arrange this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants