Direct Connect: Just These Guys, Ya Know?

Boo(s)t up your DC++

While (especially the past entries of) this blog contains numerous great information about DC++ internals almost in a knowledge base manner, you can’t find any reference to an useful feature  introduced way back in DC++ 0.695. The feature itself is actually in the dcpp lilbrary and I believe it has been made in favour of  *nix ports. It allows the user to specify different folder than the program directory to store config files, temporary data and downloads (by default).  The custom folder can be set in a standalone config file called dcppboot.xml, specifying a path relative to the DC++ executable. If you open up the file you’ll see some usage instructions in the comments.

The configuration parameter can’t be set from DC++ GUI, it can be altered only by manually editing the file. The feature works well under Windows as well, however its rarely used in this platform.  In Windows XP there are no problems accessing (settings) files located besides the executable in the %PROGRAMFILES% folder, even though since the release of Windows 2000, Microsoft recommended using %APPDATA% folders for this purpose.

Problems started growing with the arrival of Windows Vista. According to the years long recommendations (which became expectations over the years), Microsoft introduced User State Virtualization which arose problems described eg. in this support FAQ. dcppboot.xml is (one of the many not too nice) existing solution for Vista users, still, DC++ definiately needed a better way to solve this.

The solution came with DC++ 0.75 as the available parameters of dcppboot.xml are extended in a simple way what suddenly solves all the problems  described earlier. Usage instructions are updated in the comments inside the file as you can see in the repostitory.

The new parameter is called LocalMode and when it is set to zero, all configuration files are stored in the appropriate user folder inside %APPDATA%.
For Windows 2000/XP, settings will be placed into %APPDATA%\DC++, temporary files to %APPDATA%\Local Settings\Application Data\DC++ and you’ll find your downloads in %APPDATA%\DC++\Downloads by default.
In Vista and Windows 7, these will be, respectively (assuming default Windows configs): %APPDATA%\DC++ (expands to C:\Users\[your-nick]\AppData\Roaming\DC++); C:\Users\[your-nick]\AppData\Local\DC++; My documents\Downloads.

The default mode is kept (DC++ runs in LocalMode) when you use the zipped binary archive. However, if you install DC++ with the installer, you’ll have a different dcppboot.xml file installed (existing dcppboot will be overridden) so DC++ will use the user profile folders for storing its settings per user.

The best thing is that these changes will be transparent to the average user even if  they upgrade DC++ using the installer. A nice automatic migrate function added and it takes care of moving settings files to the user profile if LocalMode set to false (zero). This is a one way migration only though; if your settings, etc… are already in the user profile and you want to switch back to local mode, you should edit dcppboot then move your settings files back to program folder manually.

DC++ 0.7091

We are happy to announce that the new version of DC++ is available for download. It’s been about 5 months since the last stable release and numerous improvements and bug fixes come with this new version.

The most interesting GUI changes are as follows:

• You can set the maximum width of the tabs as you like (no more truncated hub names)
• Finished Uploads & Downloads windows are refined, from now you can filter to show only 100% finished downloads.
• Added a popular option to always have a tray icon displayed for quick access
• Numerous small improvements in the search window
• DC++ should really remember the positions of all splitters, column orders and column sizes
• Up-down control in the main status bar to change the number of upload slots quickly

Important bugs fixed :

• Shell menus are restructured and loaded only if they needed. This also eliminates possible crashes when opening the shell menu under Vista.
• In some cases it was impossible to add all the found sources to the queue from the search window
• A possible crash if some ADL Search elements set to auto-download fixed
• Fixed a possible lockup at start when the download queue has too many directories on Vista
• Solved a problem which could cause crashes when tabs closed

Other changes and improvements :

• Automatch queue for all search results is always on from now
• Partial file list requests are visible in download queue so they can be deleted in case of connection problems
• The whole Help file became translatable and a few partially translated Help files are added to the release. The localized Help file and tooltips are shown automatically according to the current locale settings

For the complete list of changes refer to the changelog.

Unlike the previous unstable release ( 0.708 ) this new version is tested and found highly stable. As usual it will be marked stable only after the two weeks public test period but the upgrade is recommended even before – especially for Vista users.

The case of a missing tree

Recently when I downloaded some files from an user I came across a situation which seemed a bit confusing at first sight. The user (I knew him before) is from another country and while he has a fast connection with nice upload speeds, the connection to him is usually not fully trouble free: my downloads from him are usually disonnect in every 10-20 minutes with timeout. This is a small problem I know, but this will be a key of this story as you’ll see later…

So I queued up some folders with larger files (there were no other sources available) and left DC++ to download in the background. Sometime later when I came back, I realized that only a couple of smaller files are downloaded so far, despite that the speed of the file transfer was still as nice as usual. When I checked the current transfer in the Connections Tab I found a very suprising fact : the file has been downloaded in one chunk and the chunk size was equal to the file size! Cheking the Finished Downloads window made the problem more mysterious : it said that 150% (!) of the current file has already been transferred…

Since segmented download method introduced DC++ does not create one large chunkfor a download except when segmented downloads are disabled. The size of the chunks are automatically adjusted depending on how fast are the transfers and how many percent left from the current download. Faster transfers result bigger chunk sizes but a chunk never reaches the overall file size unless the file is very small. In this case (as always) I had segmented downloads enabled.

As I thought that this can be a bug, I tried to disconnect the download manually to see if the chunk sizes go normal after reconnect. There came another suprise then : the download wasn’t resumed at all, it started from the beginning and still with that huge chunk size! At this point I understood why this download didn’t finished at all. As I mentioned before there are plenty of disconnects happened during the download from this user so the actual file (which was a pretty large one) wasn’t able to finish. But… why ?!?

The user had a fairly new DC client so incompatibility was ruled out. I checked the download from other users – they worked as they should: normal chunk sizes and successful resume on reconnect. Then I thought I try to get more files from this problematic user and… would you believe or not: some of the files are worked well while others still didn’t! However, this last strangeness started to ring the bell at last…

I asked the user to rebuild his share and… voilà things started to work normally right away. The problem was with his hashdata file, it became partially corrupted. Hash trees of the shared (and queued) files are stored in the hashdata so the other client failed to provide the correct tree information.

Now we found the problem but you may ask: why the hash tree needed to resume an unfinished download? Or: why’s it needed to get smaller parts of a download from more sources at the same time?

Before segmented downloading there were two methods in DC++ for resume a download. Both became more or less obsolete when chunks arrived because since then, the downloaded part of an unfinished file isn’t a contigous data. There’s no certain point to resume the download from as before. Now the unfinished temporary file is fully allocated and it contains non-contigous segments of finished and unfinished data. The size of each segment is equal to or an exact multiply of the TTH leaf size (or block size) and when segments are just finished, their integrity checked at once using the hash(es) of the block(s). The offset and length of the already finished segments are stored in the download queue.

Now its clear that to be able to check the integrity of the finished segments DC++ needs the full Tiger tree of the download (faithful readers of this blog are already familiar with Tiger hashes and hash trees by a very explanatory earlier post). Since compatibility dropped with pre-TTH era DC clients, DC++ checks if the peer supports hashes and gets the full hash tree just before an actual download starts. (The only exception is when a file is smaller than the minimum leaf size – these files are downloaded in one go and checked by their TTH). One could think that if DC++ is unable to get the full tree then it won’t start to download the file. But actually this isn’t the case…

Instead, DC++ will start the download with the hope that it’ll find more sources, so it will be able to grab the full tree later from another source. Until then it uses the full file size as blocksize and TTH for checking the integrity – ofcourse its possible only when the whole download finished. This was a good strategy up until DC++ was able to resume a download without having the hash tree. As the good old rollback function is removed in 0.699 resume without the full tree became impossible.

Possibility of download without the tree is a nice feature for small or medium sized files. They are usually downloaded in short time, they can be checked by the TTH in the end, and thats it. However, it is a problem for huge files, especially if there’s no additional source to get the tree from. Even if they download all day long if it cannot finish in one go then its just a waste of time and bandwith. It will start all over again and again… And even if some other sources with free slots come around later (and the hash tree is successfully grabbed from them) these new sources won’t be used while the full size segment is running…