Non-deterministic results on different platforms #60
Description
In the generate_hierarchical_codebook_threaded function, an std::unordered_map is used for counting unique training vectors.
Lines 1598 to 1608 in 6002320
Iteration over the unordered map is implementation-dependent, so this code builds group_quant differently on MSVC and GCC/Clang.
Lines 1634 to 1638 in 6002320
As a result, the encoder produces different selectors and codebooks on different platforms.
After switching group_hash to be an ordered map, files produced on different platforms are the same (likely with some performance cost):
diff --git a/basisu_enc.h b/basisu_enc.h
index 7d42121..7946c76 100644
--- a/basisu_enc.h
+++ b/basisu_enc.h
@@ -21,7 +21,7 @@
#include <condition_variable>
#include <functional>
#include <thread>
-#include <unordered_map>
+#include <map>
#ifndef _WIN32
#include <libgen.h>
@@ -1602,8 +1602,7 @@ namespace basisu
uint32_t max_threads, job_pool *pJob_pool)
{
typedef bit_hasher<typename Quantizer::training_vec_type> training_vec_bit_hasher;
- typedef std::unordered_map < typename Quantizer::training_vec_type, weighted_block_group,
- training_vec_bit_hasher> group_hash;
+ typedef std::map < typename Quantizer::training_vec_type, weighted_block_group> group_hash;
group_hash unique_vecs;Maybe, there is a better way to achieve deterministic results.