Use shared memory for network weights

This enables different Stockfish processes that use the same weights to use the
same memory. The approach establishes equivalence by memory content, and is
compatible with NUMA replication.  The benefit of sharing is reduced memory usage
and a speedup thanks to improved (inter-process) caching of the network in the
CPUs cache, and thus reduced bandwidth usage to main memory. Even though this
change doesn't benefit a user running a single process, this helps on fishtest
or e.g.  for Lichess, when multiple games run concurrently, or multiple
positions are analyzed in parallel.

This concept was probably first introduced in the Monty engine
(https://github.com/official-monty/Monty/pull/62), after a discussion in
https://github.com/official-stockfish/fishtest/issues/2077 on the issue of
memory pressure. Measurements based on Torch
(https://github.com/user-attachments/files/21386224/verbatim.pdf) further
suggested that large gains were possible. Multiple other engines have
adopted this 'verbatim' format as well.

The implementation here adds the flexibility needed for SF, for example, retains
the ability to bundle compressed networks with the binary, to load nets by uci
option, and to distribute the shared nets to the proper NUMA region. This
flexibility comes with a fair amount of complexity in the implementation, such
as OS specific code, and fallback code.

For most users this should be transparent. However, for example, those running
docker containers should ensure the `--ipc` flag is set correctly, and
`--shm-size` is sufficiently large.

The benefits of this patch significantly depend on hardware, with systems with
many cores and a large (O(150MB), the net size) L3 cache benefitting typically
most.  On such systems SF speedups (as measured via nps playing games with
large concurrency but just 1 thread) can be 38%, which results in master vs.
patch Elo which gains about 25 Elo.

```
   # PLAYER             :  RATING  ERROR   POINTS  PLAYED   (%)
   1 shared_memoryPR    :    24.8    1.9  39432.0   73728    53
   2 master             :     0.0   ----  34296.0   73728    47
```

In a multithreaded setup, where weights are already shared, that benefit is smaller,
for example on the same HW as above, but with 8t for each side.
```
   # PLAYER             :  RATING  ERROR  POINTS  PLAYED   (%)
   1 shared_memoryPR    :     5.2    3.5  9351.0   18432    51
   2 master             :     0.0   ----  9081.0   18432    49
```

On fishtest with a typical hardware mix of our contributors, the following was measured:

STC, 60k games
https://tests.stockfishchess.org/tests/view/69074a49ea4b268f1fac236c
Elo: 4.69 ± 1.4 (95%) LOS: 100.0%
Total: 60000 W: 16085 L: 15275 D: 28640
Ptnml(0-2): 154, 6440, 16053, 7148, 205
nElo: 9.38 ± 2.8 (95%) PairsRatio: 1.12

To verify correctness with a single process on a NUMA architecture,
speedtest was used, confirming near equivalence:
```
master:        Average (over 10):  296236186
shared_memory: Average (over 10):  295769332
```
Currently, using large pages for the shared network weights is not always possible,
which can lead to a small slowdown (1-2%), in case a single process is run.

closes https://github.com/official-stockfish/Stockfish/pull/6173

No functional change

Co-authored-by: disservin <disservin.social@gmail.com>
Co-authored-by: Joost VandeVondele <Joost.VandeVondele@gmail.com>

src/misc.h

@@ -23,11 +23,15 @@
 #include <array>
 #include <cassert>
 #include <chrono>
-#include <cstddef>
 #include <cstdint>
 #include <cstdio>
+#include <exception>  // IWYU pragma: keep
+// IWYU pragma: no_include <__exception/terminate.h>
+#include <functional>
 #include <iosfwd>
 #include <optional>
+#include <cstring>
+#include <memory>
 #include <string>
 #include <string_view>
 #include <vector>
@@ -291,6 +295,94 @@ inline uint64_t mul_hi64(uint64_t a, uint64_t b) {
 }
 
 
+template<typename T>
+inline void hash_combine(std::size_t& seed, const T& v) {
+    std::hash<T> hasher;
+    seed ^= hasher(v) + 0x9e3779b9 + (seed << 6) + (seed >> 2);
+}
+
+template<>
+inline void hash_combine(std::size_t& seed, const std::size_t& v) {
+    seed ^= v + 0x9e3779b9 + (seed << 6) + (seed >> 2);
+}
+
+template<typename T>
+inline std::size_t get_raw_data_hash(const T& value) {
+    return std::hash<std::string_view>{}(
+      std::string_view(reinterpret_cast<const char*>(&value), sizeof(value)));
+}
+
+template<std::size_t Capacity>
+class FixedString {
+   public:
+    FixedString() :
+        length_(0) {
+        data_[0] = '\0';
+    }
+
+    FixedString(const char* str) {
+        size_t len = std::strlen(str);
+        if (len > Capacity)
+            std::terminate();
+        std::memcpy(data_, str, len);
+        length_        = len;
+        data_[length_] = '\0';
+    }
+
+    FixedString(const std::string& str) {
+        if (str.size() > Capacity)
+            std::terminate();
+        std::memcpy(data_, str.data(), str.size());
+        length_        = str.size();
+        data_[length_] = '\0';
+    }
+
+    std::size_t size() const { return length_; }
+    std::size_t capacity() const { return Capacity; }
+
+    const char* c_str() const { return data_; }
+    const char* data() const { return data_; }
+
+    char& operator[](std::size_t i) { return data_[i]; }
+
+    const char& operator[](std::size_t i) const { return data_[i]; }
+
+    FixedString& operator+=(const char* str) {
+        size_t len = std::strlen(str);
+        if (length_ + len > Capacity)
+            std::terminate();
+        std::memcpy(data_ + length_, str, len);
+        length_ += len;
+        data_[length_] = '\0';
+        return *this;
+    }
+
+    FixedString& operator+=(const FixedString& other) { return (*this += other.c_str()); }
+
+    operator std::string() const { return std::string(data_, length_); }
+
+    operator std::string_view() const { return std::string_view(data_, length_); }
+
+    template<typename T>
+    bool operator==(const T& other) const noexcept {
+        return (std::string_view) (*this) == other;
+    }
+
+    template<typename T>
+    bool operator!=(const T& other) const noexcept {
+        return (std::string_view) (*this) != other;
+    }
+
+    void clear() {
+        length_  = 0;
+        data_[0] = '\0';
+    }
+
+   private:
+    char        data_[Capacity + 1];  // +1 for null terminator
+    std::size_t length_;
+};
+
 struct CommandLine {
    public:
     CommandLine(int _argc, char** _argv) :
@@ -335,4 +427,11 @@ void move_to_front(std::vector<T>& vec, Predicate pred) {
 
 }  // namespace Stockfish
 
+template<std::size_t N>
+struct std::hash<Stockfish::FixedString<N>> {
+    std::size_t operator()(const Stockfish::FixedString<N>& fstr) const noexcept {
+        return std::hash<std::string_view>{}((std::string_view) fstr);
+    }
+};
+
 #endif  // #ifndef MISC_H_INCLUDED