Recreating summary on large folders takes forever

Ryan Kavanagh writes:

> Selecting a large IMAP folder (e.g., my archive of debian-devel with
> over 25000 messages) for the first time takes ages. I didn't check the
> time when I first selected the folder, but it must have been at least 25
> minutes ago and it's still "Recreating summary...".

I hope it came back eventually? If so, it isn't a bug.

For dealing with large IMAP folders, it is a good idea to use "external
messages". Please check the documentation for
`vm-enable-external-messages'.

Cheers,
Uday

Dear Uday,

On Fri, Mar 01, 2013 at 11:15:58PM -0000, Uday Reddy wrote:
> I hope it came back eventually? If so, it isn't a bug.

It did after a while.

> For dealing with large IMAP folders, it is a good idea to use
> "external messages". Please check the documentation for
> `vm-enable-external-messages'.

I've enabled vm-external-messages for imap, but loading large folders
still takes ages. Would it be worthwhile to look at how other MUAs do it
and try to optimize the code that handles IMAP folders? For comparison,
mutt loads my Debian.debian-devel folder in 3 seconds and records
changes almost instantly. Even after enabling
vm-enable-external-messages, it took vm over 40 minutes to load the
folder using 100% CPU on one of my cores. Loading a folder with 21290
messages took at least 35 minutes (my X session crashed thereabouts),
and emacs cycled between "Debian.debian-mentors: Generating summary...
NNNN", then sorting the messages, then recreating the summary, at least
twice.

Moreover, stuffing / saving the folder Debian.debian-devel (keybinding
'S') after I marked roughly 20 messages as read took over 14 minutes
(from 14:44:22 until 14:58:34).

Might it be worthwhile retitling this bug to "vm unusably slow on large
IMAP folders"? I unfortunately don't yet know any emacs lisp, but am
willing to help out where I can in debugging this issue.

For reference, you can find my .vm file on github[0].

Best wishes,
Ryan

[0] https://github.com/ryanakca/ryanakca-dotfiles/blob/94f915674d625b9c15e5518dd9c0fc54963b62a5/.vm

--
|_)|_/ Ryan Kavanagh | Debian Developer
| \| \ http://ryanak.ca/ | GPG Key 4A11C97A

Sorry, I should have also suggested that you should set `vm-imap-max-message-size' to 0, so that it doesn't load the message bodies. Then you are only limited by the speed at which the IMAP server can deliver the message headers and you would be approaching the operation of other mail clients like Mutt or Thunderbird.

Still, VM would be quite a bit slower than the others because it is running inside Emacs, which is after all a text editor! So, there would be overheads in piping the text through all the text editing infrastructure. Secondly, the other mail clients very likely use concurrent threads to download the messages so that the user interface continues to be responsive. We can't do that right now because Emacs VM doesn't have concurrent threads yet. Hopefully, it will have them some day and we can make VM more responsive.

The other problem you are running into is that your mail folder is quite large. Since VM folders are Emacs buffers, the entire folder would get read and written every time you visit or save the folder. Controlling the `vm-imap-max-message-size' is essential for cutting down the size of the folder. If you have already downloaded the message bodies of all messages, you can get rid of them using `vm-unload-message' (bound to `O'). To do this, you would need to go to the first message and type `M-nnnn O', where nnnn is the number of messages in your folder. That should speed up the loading and saving of folders. "Stuffing" takes a while the first time. But it only happens once.

VM has always been an in-memory mail client. So, VM users have gotten used to doing all the work-arounds needed to keep their folder sizes small. I am pushing it a bit by working with folders about 5000 messages. But I haven't tried it for folders approaching your size.

The only way to speed up VM beyond 1000 messages is to use out-of-memory folders. The external message facility allows us to do that. But, right now, it is only implemented for IMAP folders.

The next step is to support maildir. This has been talked about for a while:

https://bugs.launchpad.net/vm/+bug/107092

If you are able to help, then I would appreciate if you can working on coming up with a spec for what VM needs to do for supporting maildir.

Coming back to the original point of the bug report, the "recreating summary..." activity is definitely slow. But it is only supposed to be done for a few messages whose status has been changed. It is possible that the version of VM you are using (is it 8.2.0b) is not doing it right.

If you can download the trunk and build it, you might see an improvement.