Skip to content

Conversation

@stefanhaller
Copy link
Collaborator

@stefanhaller stefanhaller commented Mar 31, 2023

When pressing > in the commits panel to trigger loading all the remaining commits past the initial 300, memory consumption is a pretty big problem for larger repositories.

The two main causes seem to be

  1. the cell memory from rendering the entire list of commits into the gocui view
  2. the pipe sets when git.log.showGraph is on

This PR addresses only the first of these problems, by not rendering the entire view, but only the visible portion of it. Since we already re-render the visible portion of the view on every layout call, this was relatively easy to do.

Below are some measurements for our repository at work (261.985 commits):

master this PR
without Graph 855 MB 360 MB
with Graph 3.1 GB 770 MB

And for the linux kernel repo (1.170.387 commits):

master this PR
without Graph 5.8 GB 1.2G
with Graph Killed by the OS after it reached 86.9 GB 39.9 GB

The measurements were taken after scrolling all the way down in the list of commits. They have to be taken with a grain of salt, as memory consumption fluctuates quite a bit in ways that I find hard to make sense of.

As you can see, there's more work to do to reduce the memory usage for the graph, but for our repo at work this PR makes it usable already, which it wasn't really before.

The commits prefixed with [gocui] are meant to go into a PR on the gocui repo, and will be replaced with a single "bump gocui" commit here.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 31, 2023

Uffizzi Ephemeral Environment deployment-20904

☁️ https://app.uffizzi.com/github.com/jesseduffield/lazygit/pull/2533

📄 View Application Logs etc.

What is Uffizzi? Learn more!

@jesseduffield
Copy link
Owner

Nice work. I agree with the approach, though there is a problem: our search functionality (invoked by pressing forward-slash) depends on the content of the view. With this PR, we're only be able to search for content in our commits view within the current page of content.

I spent a bit of time trying to improve our search functionality a while back (so that rather than searching the view we're actually filtering down the list) but I didn't get far enough to commit anything and now it's too divergent to resurrect. More recently however, I did successfully get list filtering working in lazydocker, which does not depend on the contents of the view. A couple relevant files:
https://github.com/jesseduffield/lazydocker/blob/master/pkg/gui/panels/side_list_panel.go
https://github.com/jesseduffield/lazydocker/blob/master/pkg/gui/panels/filtered_list.go

So I'm thinking we first implement that filtering functionality, then go ahead with this PR afterwards. FWIW I'd wanna merge my big refactor PR before any filtering functionality is added. What do you think?

@stefanhaller
Copy link
Collaborator Author

As for filtering in general: yes, filtering is one the things that I miss the most in lazygit, so it would be very much appreciated; however, I want this only in the branches list (and tags, maybe), but not in the commits list. For the commits list I'd prefer to not filter it down when searching; the reason is that when hopping from one search result to the next it helps me see the context of the commit I arrived at, e.g. see what branch it is a part of, and maybe type up/down arrow a bit to look at the surrounding commits too.

But I suppose searching could still be implemented in terms of the model data rather than the view content. I haven't looked at the searching code yet to get a feeling for how much work that would be; let me know if you think it would make sense for me to look in that direction, and whether that should be done before or after the big refactor.

@jesseduffield
Copy link
Owner

I'm on board with that approach. Feel free to look into how things are currently done (there's code related to searching in vendor/github.com/jesseduffield/gocui/view.go in the searcher struct. But yes, I'd want those changes to wait for the big refactor first

@stefanhaller stefanhaller force-pushed the optimize-memory-consumption branch from bd22f04 to 719e69c Compare June 6, 2024 15:31
@stefanhaller stefanhaller changed the base branch from master to search-the-model-instead-of-the-view June 6, 2024 15:40
@stefanhaller stefanhaller added the enhancement New feature or request label Jun 6, 2024
@stefanhaller stefanhaller marked this pull request as ready for review June 6, 2024 15:43
Copy link
Collaborator Author

@stefanhaller stefanhaller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ready for review now!

It sits on top of #3642.

func (self *BaseContext) NeedsRerenderOnHeightChange() bool {
return self.needsRerenderOnHeightChange
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit unfortunate that this is a separate flag that clients must remember to set when they set renderOnlyVisibleLines on the ListContextTrait. I guess this could have been avoided by putting it on Context instead of IBaseContext, and then implementing it in ListContextTrait to return renderOnlyVisibleLines. I haven't tried this yet, let me know if you think it's worth trying.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable to me

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried, but it doesn't seem to be possible with the way ListContextTrait is implemented, since ListContextTrait embeds a context (usually a SimpleContext). It looks like we would first have to do something about the containment hierarchy of contexts, and this seems like too big of a change for this PR. Plus, I wouldn't actually know how. 😅

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the context hierarchy is indeed a pickle. Happy to leave for another PR

self.context.GetViewTrait().ScrollUp(scrollHeight)
if self.context.RenderOnlyVisibleLines() {
return self.context.HandleRender()
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unfortunate that we need to do this manually here (and in HandleFocus, see previous commit). It would be nice if we had a way to automatically call it whenever the origin changed (similar to what we do for the height in layout), but I haven't found a good way to do that.

SelectNextItem().
SelectNextItem().
SelectNextItem().
SelectPreviousItem().
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really ugly, and I wonder if there's a better way to deal with this. It is also the only test that happens to have this problem right now, but there could easily be others, for example for a t.Views().Commits().Lines() assertion that expects more lines than fit on the screen. This then also depends on the screen resolution, which is even worse.

I was thinking about disabling the renderOnlyVisibleLines optimization when running integration tests, but I wasn't sure that's a good idea either.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about a new function called NavigateDownToLine() which just calls SelectNextItem() until the selected line matches the matcher?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good suggestion, but it would be even better to add logic to NavigateToLine so that clients don't have to worry about which one to call. And it turns out that NavigateToLine already has this comment about exactly that:

// NOTE: this currently assumes that BufferLines returns all the lines that can be accessed.
// If this changes in future, we'll need to update this code to first attempt to find the item
// in the current page and failing that, jump to the top of the view and iterate through all of it,
// looking for the item.

I implemented this in a0427bb (and dropped the hack commit again), please have a look.

@stefanhaller
Copy link
Collaborator Author

Ah damn, this still has bugs. Setting back to draft for now.

@stefanhaller stefanhaller marked this pull request as draft June 6, 2024 16:49
@stefanhaller stefanhaller force-pushed the optimize-memory-consumption branch from 719e69c to dc96c70 Compare June 6, 2024 18:58
@stefanhaller stefanhaller marked this pull request as ready for review June 6, 2024 19:01
// those views that support it.
totalLength := self.list.Len()
if self.getNonModelItems != nil {
totalLength += len(self.getNonModelItems())
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are calling getNonModelItems twice now, here and inside of renderLines below. I couldn't come up with a good way of avoiding that, but I'm also not sure it's a problem.

@stefanhaller
Copy link
Collaborator Author

@jesseduffield The bug is fixed, this is back in review. There are several things here that I'm not very happy with, curious what you think about these.

@stefanhaller stefanhaller force-pushed the search-the-model-instead-of-the-view branch 2 times, most recently from 42901e9 to 9c561b8 Compare June 7, 2024 15:12
@stefanhaller stefanhaller force-pushed the optimize-memory-consumption branch from dc96c70 to 5da2942 Compare June 7, 2024 15:12
Copy link
Owner

@jesseduffield jesseduffield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, see comments

func (self *BaseContext) NeedsRerenderOnHeightChange() bool {
return self.needsRerenderOnHeightChange
}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable to me

SelectNextItem().
SelectNextItem().
SelectNextItem().
SelectPreviousItem().
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about a new function called NavigateDownToLine() which just calls SelectNextItem() until the selected line matches the matcher?

ListContextTrait.OnSearchSelect was introduced in 138be04, but it was never
called. I can only guess that a planned refactoring wasn't finished here.
Searching in the "Divergence from upstream" view would select the wrong lines.
The OnSearchSelect function gets passed a view index, and uses it to select a
model line. In most views these are the same, but not in the divergence view
(because of the Remote/Local section headers).
We do show the graph in the left/right view, as of b767357.
We want to add an additional method to ISearchableContext later in this branch,
and this will make sure that we don't forget to implement it in any concrete
context.
Just to be really sure that it not only contains the expected status text, but
also actually shows it.
…ighting

Previously, jumping to the next search result with 'n' would not show the search
result in the selected line. This is especially annoying when there is more than
one search result on a single line, because pressing 'n' seemingly does nothing.

Now we highlight the search result on top of the selection highlight, which
solves the problem nicely.
It is unnecessary to call it every time we draw, but we do need to call it every
time the view content changes.
We will need it again when searching the model; extracting the change as a
separate commit just to make the diffs of the following commits smaller.
When we add support for searching the model, the length of a highlighted search
result isn't necessarily the same as the length of the search string. As an
example, consider that you are searching for a full commit hash; the view only
renders an abbreviated hash, so when the hash is found in the model, we only
want to highlight the abbreviated version.

Prepare for this by adding an XEnd field to cellPos; when searching the view,
this is always XStart + len(searchString).

Also, rename cellPos to SearchPosition and uppercase its fields to expose them;
we will use the struct in the API to set a model search function in the next
commit.
Add a method View.SetModelSearchFunc; if set, it will be used to look for search
results instead of searching the view lines.

This will be needed when we optimize some list views to render only the visible
portion of the view instead of all lines of the list. However, it also has the
benefit that it allows searching for the full text of something that is only
rendered in an abbreviated form in the view; the most notable example being
commit hashes.

This is opt-in, so we still support searching the view as before. This is needed
for two reasons:
- adding a model search func can be a bit of work, so we should only do it when
  there's a benefit
- some views don't even have a model
…ruction

There is no need to overwrite things with spaces; since tcell clears the whole
surface before drawing, it is enough to just truncate the line at the current
write position.
It is similar to OverwriteLines, except that it also clears all lines outside of
the area that was rerendered. This is needed for views that only draw the
visible portion of their model, to ensure that cells for invisble lines don't
accumulate as you scroll.
When refreshViewportOnChange is true, we would refresh the viewport once at the
end of FocusLine, and then we would check at the end of AfterLayout if the
origin has changed, and refresh again if so. That's unnecessarily complicated,
let's just unconditionally refresh at the end of AfterLayout only.
@stefanhaller stefanhaller force-pushed the search-the-model-instead-of-the-view branch from 9c561b8 to d04cb9a Compare June 10, 2024 10:04
…dently

This is a commit only for testing; we'll drop it again before merging.

We now have two separate flags, refreshViewportOnChange and
renderOnlyVisibleLines, but we only have views that either use them both, or
neither. To test whether the logic around them is really independent, change two
other views to use them independently:

- the local branches view is changed to refresh viewport on change, and in order
  to make that testable, we change the branch render function so that the
  selected branch is rendered yellow. This is comparable to the graph node of
  the selected commit being rendered white.
- the remote branches view is changed to render only the visible lines.
  Conveniently, this is a view that uses filtering and not searching, so we
  don't have to bother implementing a model search func.
@stefanhaller stefanhaller force-pushed the optimize-memory-consumption branch from 5da2942 to f1540db Compare June 10, 2024 11:16
Copy link
Owner

@jesseduffield jesseduffield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@stefanhaller stefanhaller force-pushed the search-the-model-instead-of-the-view branch from d04cb9a to 6a6316c Compare June 23, 2024 09:44
stefanhaller added a commit that referenced this pull request Jun 23, 2024
- **PR Description**

This makes it possible to search the model data instead of the view when
pressing `/`, and uses this for the commits view.

This is mainly a preparation for #2533 which requires it, but it is also
useful on its own, because it makes it possible to search for full
commit hashes. It will highlight the abbreviated hash in that case.

- **Please check if the PR fulfills these requirements**

* [x] Cheatsheets are up-to-date (run `go generate ./...`)
* [x] Code has been formatted (see
[here](https://github.com/jesseduffield/lazygit/blob/master/CONTRIBUTING.md#code-formatting))
* [ ] Tests have been added/updated (see
[here](https://github.com/jesseduffield/lazygit/blob/master/pkg/integration/README.md)
for the integration test guide)
* [ ] Text is internationalised (see
[here](https://github.com/jesseduffield/lazygit/blob/master/CONTRIBUTING.md#internationalisation))
* [ ] Docs have been updated if necessary
* [x] You've read through your own file changes for silly mistakes etc
@stefanhaller stefanhaller deleted the branch jesseduffield:search-the-model-instead-of-the-view June 23, 2024 09:48
stefanhaller added a commit that referenced this pull request Jun 23, 2024
(Github decided to auto-close #2533, and I don't see any way to reopen
it, so opening a new one here. Please see there for discussion and
review.)

When pressing `>` in the commits panel to trigger loading all the
remaining commits past the initial 300, memory consumption is a pretty
big problem for larger repositories.

The two main causes seem to be
1. the cell memory from rendering the entire list of commits into the
gocui view
2. the pipe sets when git.log.showGraph is on

This PR addresses only the first of these problems, by not rendering the
entire view, but only the visible portion of it. Since we already
re-render the visible portion of the view on every layout call, this was
relatively easy to do.

Below are some measurements for our repository at work (261.985
commits):

|               | master | this PR |
| ------------- | ------ | ------- |
| without Graph | 855 MB | 360 MB  |
| with Graph    | 3.1 GB | 770 MB  |

And for the linux kernel repo (1.170.387 commits):

|               | master                                    | this PR |
| ------------- | ----------------------------------------- | ------- |
| without Graph | 5.8 GB                                    | 1.2G    |
| with Graph    | Killed by the OS after it reached 86.9 GB | 39.9 GB |

The measurements were taken after scrolling all the way down in the list
of commits. They have to be taken with a grain of salt, as memory
consumption fluctuates quite a bit in ways that I find hard to make
sense of.

As you can see, there's more work to do to reduce the memory usage for
the graph, but for our repo at work this PR makes it usable already,
which it wasn't really before.
stefanhaller added a commit that referenced this pull request Apr 29, 2025
- **PR Description**

As a followup to #2533, reduce the memory consumption some more by
optimizing the storage of the pipe sets used for the commit graph.

Some coarse measurements (taken by opening the respective repo, typing
`4 >`, and waiting for all commits to be loaded, and then reading the
Memory column in Activity Monitor):

|               | master  | this PR |
| ------------- | ------- | ------- |
| git           | 2.5 GB  | 1.0 GB  |
| our work repo | 2.3 GB  | 1.3 GB  |
| linux kernel  | 94.8 GB | 38.0 GB |

It's still not really usable for the linux kernel, but for all other
repos that I come across in my daily use of lazygit, it works quite well
now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants