Last built commit: b6545c2

Given the sheer amount of time between the last two releases, there were bound to be a few things that didn't quite work the way I thought. That was the case with 1.2.0. Thanks to some very thorough testing by newuzer, a lot of things got fixed, corrected, and repaired. Serious thanks for the dedicated time testing, it makes the program way better. If you want more details, let's get to it:

Notable Changes:

• Experimental Code Disable - This one hit like a brick, mostly for large DATs. Turns out some of the experimental code I included that was running alongside the normal code was taking a massive performance hit. Disabling the majority of that code helped improve the worst of the performance regressions seen in the last release.
• Filtering Fixes - There were a few filters that didn't quite work the way that they should have, especially when it came to machine types like Bios and Device. A bunch of extra filtering code and framework went in and now things are working the way that a normal user would expect.
• More Parallelization and Performance - To improve performance further, multiple bits of the code have gotten parallelization, removed reflection calls, and even preemptive filtering and removal. All of these brought the final runtimes back in the realm of the less accurate code from the past.

Full Changelog: 1.2.0...1.2.1

It's been almost 4 years since v1.1.2 came out and a lot has changed. During this massive amount of time, the entire code base has been rewritten 3 separate times. Massive chunks of code were moved around, separated out into new projects, integrated into other places, and still used by a dedicated group of people. The absolute number of commits is over 1100, so recapping the changes effectively is going to be difficult. Compare against the last release if you want to see literally everything.

Notable Changes:

• Executable Cleanup - SabreTools as a suite included a bunch of different programs and it wasn't always obvious what each of them did and why. Half of them got wrapped into the main executable and, as of the last release, there were only two standalone applications left.
  • Headerer was wrapped in as extract and restore, dealing with copier headers. That project now lives with the standalone Skippers library. Newer versions of that library and Headerer can be found here.
  • RombaSharp was a fully standalone application that was meant to be a reimplementation of the file management tool romba. It ended up as a very thin wrapper over other SabreTools functionality and never gaining any traction. During this last cycle, the application was removed entirely from builds with no intention to create a replacement.
  • Standalone, Single-File - This wasn't an application, rather how the applications were packaged. Previous releases either relied on users having the correct .NET version installed on their machine or included all of the runtime files as separate DLLs so that it would be ready to go. Things are far easier now. All required files and libraries are now included in a single executable package. This has the advantage of a far cleaner application folder as well as not requiring users to have the newest .NET versions installed to use it.
• .NET Version Support - The last semi-stable release targeted both .NET Core 3.1 and .NET 5, both of which are extremely out of date at this point. During the intervening years, the versions were continued to be updated, with the primary target now being .NET 9. There are two notable differences this time around, though:
  1. Explicit builds for Windows, Linux, and macOS are provided now, instead of relying on running dotnet directly on alternative platforms. This should make running the program in scripts and the like far easier than before.
  2. Support back to .NET Framework 2.0. Wait, what? That's right, instead of just always moving forward, I made the decision to support as far back as humanly possible with the design and builds of the libraries and application. This means that a user can build a version of SabreTools that can run on run on Windows 98SE. These builds are not provided, but building for those ancient frameworks is tested on every commit.
• Testing - This affects end users far less than the other changes above, but there was a concerted effort made for this release to add unit testing around much of the basic functionality of SabreTools. This ended up helping find a bunch of subtle issues, but also ended up changing some functionality due to side effects. If you're comparing against last stable, things will have changed.

Less Notable Changes:

• Libraries - Many of the original bits of functionality in SabreTools have now been separated out into their own, semi-independent projects in the organization. This happened because some things locked to SabreTools ended up being used in other projects that I also maintain.
• Commandline - This didn't change as much as it may seem, but there are some changes to how the commandline is built since last stable. The majority of the flags stayed the same, but taking a look at the help output or README will be necessary to make sure things are still what are expected.
• Build Scripts - Automatic builds have moved away from AppVeyor and have moved to GitHub Actions. In this process, two build scripts are now provided for users to make semi-reproducible builds locally. One is a Powershell script and the other is a Bash script. Both provide identical functionality and are tested with every single rolling build.
• Nuget Packages - Even though they're not being published, nearly all core libraries are now being built as Nuget packages. In theory, this allows easier integration into other projects, but I'm not comfortable enough to have them published yet. The packages and their source packages are included in this release.
• Internal Beta - Internal to the existing libraries is a currently beta implementation of a new way of dealing with items. This is not enabled by default or by any flag, rather by direct source changes. It is mostly tested, but was not stable enough to be enabled for this release.

Full Changelog: 1.1.2...1.2.0

The few changes since the last prerelease do warrant having more people be able to use them and not have to rely on the AppVeyor builds sticking around. Going with another prerelease since the number of changes is relatively small.

Notable Changes:

• SMDB Size Support - Thanks to Teddy in the RomVault Discord, SMDB files can now have an optional size field. This brings SMDB closer to parity with other formats, and makes it more likely to convert nicely to common ones like Logiqx.
• DAT Identification Updates - As reported externally, one of the MAME software list DATs had a nonstandard way of including the license field before the second actual line. This necessitated overhauling how XML comments are identified and skipped as a part of the internal identification process.
• Consistency - Starting from the last prerelease, there has been at least one report of inconsistent behavior in DFD functionality. For some reason, all of the locks in the world didn't make it so that every item is actually processed, even if it's all done serially. As such, I introduced a thread-safe List implementation that all internal structures have been converted to, hopefully removing any issues with missing scanned files.

Since there have been some changes in the code base that warrant people taking a better look, I decided to put out another prerelease. I've opted to go with a prerelease instead of a full release since I have not gotten much feedback since v1.1.0 went out. Let's get to it!

Notable Changes:

• Internal Reorganization - It wouldn't be SabreTools without a near-constant stream of reorganization and rewrites. This time, the main focus was on splitting internal items for the Batch command and doing a near-total rewrite on the stats collection and writing. The latter took the most time but it ended up with a more DatFile-like internal structure that is more easily extensible to more formats in the future.
• Bug Fixes - There's a couple of bugs that snuck into 1.1.0, so those should have been taken care of. They're the dumbest little things, but at least they've been addressed.
• New / Updated Features - This is an umbrella for the new features that ended up in this prerelease, since none of them are really big enough for their own top-level comment:
  • Feature Flags Don't Need - - Minor change, but now all features don't need to have the - or -- prefix anymore. So instead of --d2d you can just write d2d. Small, but nice change for the future.
  • Split by Total Size - New feature requested by a user to allow for splitting a DAT into smaller DATs based on the total size of the items. This can help with partitioning larger sets.
  • "Include" Instead of "Skip" - This is a breaking change for anyone who was using the --skip-<hash> flags for D2D. Default behavior still uses CRC, MD5, and SHA-1, but now all hashes can be opt-in instead of opt-out.
  • Version - A quick new feature was added to output the current version to console. Requested by obiwantje.
  • Better Tool Version in Diff Outputs - This change makes the tool version more apparent in generated DATs such as those output by Diff. This should make tracking bugs and other issues way easier in the future. Requested by obiwantje.

After the last 3 prereleases, we finally have a new stable version! For anyone who has only been looking at stable releases, please check out 1.0.5, 1.0.6, and 1.0.7 release notes to get a better idea of the changes that have gone in since 1.0.4. This changelist will only mention things that have not been previously addressed in those other releases. So let's get to it!

Notable Changes:

• Code Quantization - After making sure that there were a sufficient amount of tests in place in the last release, I was able to go ahead and split up some functionality that were in monolithic files into better-organized bits. This was definitely the bulk of this release, since the effort to split out the code was non-trivial.
• Performance Tracking - As a request from a tester, more stopwatches have been added to more bits of the code in order to help both future debugging and testing efforts. Thanks to obiwantje for the suggestion!
• Universal Parameters - Internally, there were some issues with flags that were considered to be "universal" since they weren't actually handled across every feature equally. The addition of configurable log levels helped push this across the finish line. Thankfully, if you're using --script for any of your batch runs, this will not affect your workflow.

In addition to the above, I added one more unit test to ensure that hashing functioned properly and did a bunch of tiny fixes here and there for lesser used features like extract, restore, and batch. Thanks to everyone who gave feedback on the previous pre-releases! This ended up being a really good way to have a better stable release, so look forward to more pre-release releases in the future!

I decided that I needed at least one more prerelease before getting to a new stable version. The big thing this time around was the introduction of unit testing for large portions of the code and fixing those things that came up. Next release has a high chance of being a stable at this point, so that's exciting. Let's get to the changes, shall we?

Notable Changes:

• Unit Testing Framework - Similar to how the .NET Framework 4.8 removal was the reason that that 1.0.6 exists, this is the reason that 1.0.7 exists. Over the past month or so, I added a whole suite of tests covering large chunks of the library code. In writing these tests, a lot of underused functionality was able to be fixed, including some issues with parsing lesser-used formats as well as nearly everything having to do with reading skipper files. These changes spread quite a few commits, so please see the individual commits for more details.
• .NET Framework 4.8 Removal Followup - Going along with the changes made last time stripping out .NET Framework 4.8, some more things got to use new syntax. This included reducing the number of unnecessary summaries that got replaced by the inheritdoc tag as well as being able to use compounded nullable assignments for a lot of fields (??=). I also took the opportunity to use the catch / when syntax to remove some indirection issues for situations where the library was being called with the throwErrors flag. Before, they got wrapped and rethrown, which is bad form. I also dropped the idea of removing .NET Core 3.1 support for now, since it's an LTS version. When .NET 6.0 comes out, that's likely when I'll make the full switch.
• Deprecated Flag Removal - For those who like to use a lot of filters for the update command, you may notice that if you were using a certain set of parameters it would yell at you every time for the past year or so saying "this flag is deprecated". Well, I finally put the hammer down and removed those flags once and for all. They were already removed from both the README and the wiki, but this cements that they aren't to be used anymore. If you were still using those old flags up until now, you really need to move on to the new syntax. Please see the wiki for more details on how to properly use the filter flags.
• Stats Gathering Improvements - Due to a misunderstanding with one of my testers, I ended up making the entire stats gathering pipeline much more memory efficient. Honestly, that's really about all there is to it, though I wish my testers were a little more clear 😋
• History Tag Support - This came as a request from another tester. Apparently, some variants of the ListXML format can contain a machine-level tag called history that can include information about a machine. It's a neat tag, but I just didn't have support for it. Now there's support for it. How nice of me 😄

With the unit testing changes behind me, not a lot of functionality changed this time around either. There were definitely a fair share of fixes that resulted from the testing, but that was the biggest bit. Thanks to everyone who has been testing these prereleases so far, and I hope that in the near future, we can get a new stable version out the door.

This is the next pre-release that I had alluded to in 1.0.5. The big thing this time around is the .NET Framework is no longer supported as a platform for SabreTools. This frees up development to include a lot of new language features as well as focusing on actual, broad compatibility across platforms. This doesn't have much that takes advantage of the Framework removal, but I'll be taking a look at that for the next pre-release or stable release. Still haven't figured that out yet. So let's get on it:

Notable Changes:

• .NET Framework 4.8 Removal - This is the reason that this release exists. No more .NET Framework 4.8 and the special exclusions that were made for it. Old-style switch statements in code and the ability to use RIPEMD160 hashes have gone with this change.
• Console Color Fixes - For the longest time, if you were running the .NET Core build or you were running on mono, you didn't see a blue header when starting the program. Turns out, it was something dumb on my part completely unrelated to color that I had gated off. That's been fixed, so it should look identical cross-platform now.
• Better Field Logging - Starting in the 1.0.5 cycle and ending in this cycle, the generic Field type has gone away internally. This was an unwieldly beast that included reference to every single field in a DatFile. Now, that's been split up into 3 different pieces and the last remnants of this behemoth have gone away. With that change, a lot of "hacky" stuff went as well, such as supporting non-standard field names for filtering and exclusion. This caused some confusion with those who tested 1.0.5, so better logs have been put around these cases to make it clear that a field name was not accurate (anymore).

A lot of lines changed, but not a lot actually changed. Next round will be making things look prettier internally with the hopes that I might even be able to move to .NET 5.0 entirely. record types anybody?

It's been a seemingly quiet couple of months. Too quiet. So let's destroy that silence and make some broad, sweeping changes that might screw up everything! As such, this is being marked as one of my few pre-releases. It's stable in my eyes, but enough changed that I can't check every corner case. Alongside that, other releases are planned relatively soon that make other large changes as well. With that out of the way, let's get on it:

Notable Changes:

• Logging Overhaul - Due to some issues with how logging was implemented (i.e. it was global), the entire logging system was given an overhaul to have instance-based logs similar to the NLog library, except worse. This also brings about some progression logging that was missing before.
• .NET 5.0 (Initial) Support - This is a bit of a token thing, but now, there are .NET 5.0 builds. I wanted to add this now as things will be moving over toward that in the near future. Until then, nothing special here, really.
• Oh No - Yeah, this is the big item of this release, and the main reason that this is being made into a release. Without writing a novel, the gist of the changes that make up the majority of the release went into detangling the various namespaces in the former monolithic SabreTools.Library DLL and split them into logically grouped (and logically detangled) SabreTools.XXX DLLs. No logic should have changed as a part of this, as everything should be just shuffled. Now I've done shuffle-only changes before and it broke everything. Though my normal tests seem to be passing still, which makes me suspicious.

Unlike most of the time, not much other cleanup beside some SMDB fixes went in this last release. The next release might literally be out in the next week, but that one will likely be a prerelease too. Take a look at the full list of commits if you're curious about the amount of work that went into this. Thank you to everyone who decided to keep using this weird program of mine.

Another month, another release of SabreTools. I'm not sure I like this pattern, but here we are. This past release cycle was frontloaded with getting the last of the ListXML items promoted and then ended up with a whole bunch of bugfixes in the latter half. With only as much ado as is required by law, let's get on it:

Notable Changes:

• DatItem Promotions - As of this release, all of the different items in a ListXML DatFile (such as driver, control, and sound) are fully independent of the machine they're in. This means that each of them have filters that can apply across the board and can affect the output to relevant DatFile types. Please take a look at the wiki for the entirely overhauled set of filters that came along with this.
• Rom-ificiation - Now this doesn't really affect much outside of the inner workings, but it's worth calling out here. Previously, a lot of fields such as loadflag were included as settable on every single DatItem. Turns out, that was wrong. So a lot of these fields that are actually unique to the Rom DatItem type are now only on Rom. It's minor, but has an impact on filtering so it's cool to call out.
• Machine-Level Filters - This is an exciting change that hinged on the first item above to complete. The TL;DR of this one is that if a single item in a machine is filtered out, the entire machine can be marked as filtered out. This allows users to take advantage of, say, filtering out games that have more than 4 players (on DatFile types that support it (like ListXML)). Fun and exciting stuff that also ended up getting its own addition in batch as well. Check it out.
• Sabre... XML?? - This is another one that took me by surprise about how fun it was. Remember when I added the "serialize-it-all" JSON output? Yeah, I did that with XML as well. The completely underused (and admittedly neglected by me as well) format of SabreDAT is now SabreXML. SabreXML is the XML serialized equivalent and will contain everything you throw at it. It's super fun but really weird to look at. With this, JSON is now SabreJSON. Because I might like naming stuff after myself.
• Librarian - Thanks to some prompting (thanks claunia), I started a bit more work on getting the SabreTools.Library project to behave more like a library. A couple of things such as allowing throws on exceptions were added on top of the work already done to support batch. This might help future library users more than the current application frontend, but still neat.
• DFD/D2D improvements - Thanks to a combination of me breaking things massively while cleaning up large chunks of code and the input from a few trusted testers (looking at you: hardrider, edc, and Obi), DFD got a massive improvement to speed and accuracy. Yes, accuracy. I can't tell if I broke it during this last cycle or prior, but certain situations ended up with incorrect DatFiles on the other side. These have been fixed as of this release. If you ran any DFDs in the prior release or using one of the WIP builds, you should probably run them again.

As always, there's a bunch of cleanup work that went into this, including removing large chunks of redundant code, fixing the issues that arose from de-redundanting the code, etc. Take a look at the list of commits since the last release for more details. Thank you to everyone who contributed their time into helping make this an oddly enjoyable project to work on. Your feedback has been immensely valuable. Special thanks to Pucci, Claunia, hardrider, edc, and Obiwantje for their assistance in testing this time around.