Skip to content

Optimize clip and downsample#2719

Merged
j9ac9k merged 2 commits intopyqtgraph:masterfrom
pijyoi:optimize-clip-downsample
May 11, 2023
Merged

Optimize clip and downsample#2719
j9ac9k merged 2 commits intopyqtgraph:masterfrom
pijyoi:optimize-clip-downsample

Conversation

@pijyoi
Copy link
Copy Markdown
Contributor

@pijyoi pijyoi commented May 11, 2023

This PR implements two of the ideas pointed out in #2710 that affect viewing of large datasets.

  1. Don't do finite check if it has already been performed elsewhere
  2. Use bisect_left instead of np.searchsorted

The changes done in this PR are conservative and should hopefully not break any existing use-cases.

pijyoi added 2 commits May 11, 2023 20:05
np.searchsorted performs poorly when the array.dtype does not
match the type of the value (float) being searched.
@j9ac9k
Copy link
Copy Markdown
Member

j9ac9k commented May 11, 2023

Thanks @pijyoi I agree that this is a conservative change and I would be more impressed than anything if this change broke things for someone.

@j9ac9k j9ac9k merged commit f3c77ac into pyqtgraph:master May 11, 2023
@pijyoi pijyoi deleted the optimize-clip-downsample branch May 11, 2023 16:53
@Mriv31
Copy link
Copy Markdown

Mriv31 commented May 11, 2023

Thanks a lot @pijyoi and @j9ac9k . In this version moving and zooming through a huge plot is now fluid once it is created.

However, the first plotting still takes several minutes of my computer. Two problems there :

  1. The property autoDownsample gets passed to the PlotDataItem by the PlotItem after a first call to _getDisplayDataset, so that the PlotDataItem itself needs to be created with the arguments autoDownsample = True to be sure that it computes the ds on the first call of _getDisplayDataset. (being created by a PlotItem which itself has autoDownsample = True is not enough)

  2. This right initialization of the PlotDataItem I suggest above is not enough. This is because the first time _getDisplayDataset gets called, view is None, so that the auto downsampling rate does not get computed and the software tries to draw all the points. view is None because the PlotDataItem is first created by the PlotItem.plot() method, and only after gets attributed to the view by the PlotItem.addItem method.

I think the easiest fix (it is what I am using) is to let the user correct the point 1 by correctly initializing large datasets, and then, for the point 2, in case of autoDownsample = true, default to a large ds in _getDisplayDataset (if view is None compute the parameter ds as if the whole dataset would be displayed), it will then be updated when the viewrange is set.

This is what I'm doing and it seems OK for me !

A better fix would be to remove useless repeated calls to _getDisplayDataset upon calling PlotItem.plot.

@aarpon
Copy link
Copy Markdown

aarpon commented Mar 14, 2024

Hello, @pijyoi, @j9ac9k and @Mriv31: in PyQtGraph version 0.13.4, coloring points in a ScatterPlotItem with brush set to a long index of integers (to rotate over the default number of colors) takes approximately twice as long as in PyQtGraph 0.13.3.

Example code:

...
self.scatter = pg.ScatterPlotItem(
    size=5,
    pen=pg.mkPen(None),
    brush=pg.mkBrush(255, 255, 255, 128),  # To be overridden in the next call depending on a setting
    hoverable=True,
    hoverSymbol="s",
    hoverSize=5,
    hoverPen=pg.mkPen("w", width=2),
    hoverBrush=None,
)
...
self.scatter.addPoints(x=x, y=y, data=tid, brush=brushes,)  # Here brushes is equal to tid

For x, y, and tid of about 200,000 elements, and np.unique(tid) of approximately 600, on my machine PyQtGrpah 0.1.13 takes approximately 10 seconds to plot all colored points, whereas 0.1.14 takes approximately 19 seconds.

Regarding the comment by @Mriv31, in my case I need to make sure to draw all the points.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants