Files
stockfish/src/misc.h
T
Tomasz SobczykandJoost VandeVondele a169c78b6d Improve performance on NUMA systems
Allow for NUMA memory replication for NNUE weights.  Bind threads to ensure execution on a specific NUMA node.

This patch introduces NUMA memory replication, currently only utilized for the NNUE weights. Along with it comes all machinery required to identify NUMA nodes and bind threads to specific processors/nodes. It also comes with small changes to Thread and ThreadPool to allow easier execution of custom functions on the designated thread. Old thread binding (WinProcGroup) machinery is removed because it's incompatible with this patch. Small changes to unrelated parts of the code were made to ensure correctness, like some classes being made unmovable, raw pointers replaced with unique_ptr. etc.

Windows 7 and Windows 10 is partially supported. Windows 11 is fully supported. Linux is fully supported, with explicit exclusion of Android. No additional dependencies.

-----------------

A new UCI option `NumaPolicy` is introduced. It can take the following values:
```
system - gathers NUMA node information from the system (lscpu or windows api), for each threads binds it to a single NUMA node
none - assumes there is 1 NUMA node, never binds threads
auto - this is the default value, depends on the number of set threads and NUMA nodes, will only enable binding on multinode systems and when the number of threads reaches a threshold (dependent on node size and count)
[[custom]] -
  // ':'-separated numa nodes
  // ','-separated cpu indices
  // supports "first-last" range syntax for cpu indices,
  for example '0-15,32-47:16-31,48-63'
```

Setting `NumaPolicy` forces recreation of the threads in the ThreadPool, which in turn forces the recreation of the TT.

The threads are distributed among NUMA nodes in a round-robin fashion based on fill percentage (i.e. it will strive to fill all NUMA nodes evenly). Threads are bound to NUMA nodes, not specific processors, because that's our only requirement and the OS can schedule them better.

Special care is made that maximum memory usage on systems that do not require memory replication stays as previously, that is, unnecessary copies are avoided.

On linux the process' processor affinity is respected. This means that if you for example use taskset to restrict Stockfish to a single NUMA node then the `system` and `auto` settings will only see a single NUMA node (more precisely, the processors included in the current affinity mask) and act accordingly.

-----------------

We can't ensure that a memory allocation takes place on a given NUMA node without using libnuma on linux, or using appropriate custom allocators on windows (https://learn.microsoft.com/en-us/windows/win32/memory/allocating-memory-from-a-numa-node), so to avoid complications the current implementation relies on first-touch policy. Due to this we also rely on the memory allocator to give us a new chunk of untouched memory from the system. This appears to work reliably on linux, but results may vary.

MacOS is not supported, because AFAIK it's not affected, and implementation would be problematic anyway.

Windows is supported since Windows 7 (https://learn.microsoft.com/en-us/windows/win32/api/processtopologyapi/nf-processtopologyapi-setthreadgroupaffinity). Until Windows 11/Server 2022 NUMA nodes are split such that they cannot span processor groups. This is because before Windows 11/Server 2022 it's not possible to set thread affinity spanning processor groups. The splitting is done manually in some cases (required after Windows 10 Build 20348). Since Windows 11/Server 2022 we can set affinites spanning processor group so this splitting is not done, so the behaviour is pretty much like on linux.

Linux is supported, **without** libnuma requirement. `lscpu` is expected.

-----------------

Passed 60+1 @ 256t 16000MB hash: https://tests.stockfishchess.org/tests/view/6654e443a86388d5e27db0d8
```
LLR: 2.95 (-2.94,2.94) <0.00,10.00>
Total: 278 W: 110 L: 29 D: 139
Ptnml(0-2): 0, 1, 56, 82, 0
```

Passed SMP STC: https://tests.stockfishchess.org/tests/view/6654fc74a86388d5e27db1cd
```
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 67152 W: 17354 L: 17177 D: 32621
Ptnml(0-2): 64, 7428, 18408, 7619, 57
```

Passed STC: https://tests.stockfishchess.org/tests/view/6654fb27a86388d5e27db15c
```
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 131648 W: 34155 L: 34045 D: 63448
Ptnml(0-2): 426, 13878, 37096, 14008, 416
```

fixes #5253
closes https://github.com/official-stockfish/Stockfish/pull/5285

No functional change
2024-05-28 18:34:15 +02:00

274 lines
7.2 KiB
C++

/*
Stockfish, a UCI chess playing engine derived from Glaurung 2.1
Copyright (C) 2004-2024 The Stockfish developers (see AUTHORS file)
Stockfish is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Stockfish is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef MISC_H_INCLUDED
#define MISC_H_INCLUDED
#include <algorithm>
#include <cassert>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <iosfwd>
#include <memory>
#include <string>
#include <vector>
#include <optional>
#define stringify2(x) #x
#define stringify(x) stringify2(x)
namespace Stockfish {
std::string engine_info(bool to_uci = false);
std::string compiler_info();
// Preloads the given address in L1/L2 cache. This is a non-blocking
// function that doesn't stall the CPU waiting for data to be loaded from memory,
// which can be quite slow.
void prefetch(const void* addr);
void start_logger(const std::string& fname);
void* std_aligned_alloc(size_t alignment, size_t size);
void std_aligned_free(void* ptr);
// memory aligned by page size, min alignment: 4096 bytes
void* aligned_large_pages_alloc(size_t size);
// nop if mem == nullptr
void aligned_large_pages_free(void* mem);
size_t str_to_size_t(const std::string& s);
// Deleter for automating release of memory area
template<typename T>
struct AlignedDeleter {
void operator()(T* ptr) const {
ptr->~T();
std_aligned_free(ptr);
}
};
template<typename T>
struct LargePageDeleter {
void operator()(T* ptr) const {
ptr->~T();
aligned_large_pages_free(ptr);
}
};
template<typename T>
using AlignedPtr = std::unique_ptr<T, AlignedDeleter<T>>;
template<typename T>
using LargePagePtr = std::unique_ptr<T, LargePageDeleter<T>>;
struct PipeDeleter {
void operator()(FILE* file) const {
if (file != nullptr)
{
pclose(file);
}
}
};
#if defined(__linux__)
inline std::optional<std::string> get_system_command_output(const std::string& command) {
std::unique_ptr<FILE, PipeDeleter> pipe(popen(command.c_str(), "r"));
if (!pipe)
return std::nullopt;
std::string result;
char buffer[1024];
while (fgets(buffer, sizeof(buffer), pipe.get()) != nullptr)
result += buffer;
return result;
}
#endif
void dbg_hit_on(bool cond, int slot = 0);
void dbg_mean_of(int64_t value, int slot = 0);
void dbg_stdev_of(int64_t value, int slot = 0);
void dbg_correl_of(int64_t value1, int64_t value2, int slot = 0);
void dbg_print();
using TimePoint = std::chrono::milliseconds::rep; // A value in milliseconds
static_assert(sizeof(TimePoint) == sizeof(int64_t), "TimePoint should be 64 bits");
inline TimePoint now() {
return std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch())
.count();
}
inline std::vector<std::string> split(const std::string& s, const std::string& delimiter) {
size_t begin = 0;
std::vector<std::string> res;
for (;;)
{
const size_t end = s.find(delimiter, begin);
if (end == std::string::npos)
break;
res.emplace_back(s.substr(begin, end - begin));
begin = end + delimiter.size();
}
res.emplace_back(s.substr(begin));
return res;
}
enum SyncCout {
IO_LOCK,
IO_UNLOCK
};
std::ostream& operator<<(std::ostream&, SyncCout);
#define sync_cout std::cout << IO_LOCK
#define sync_endl std::endl << IO_UNLOCK
// Get the first aligned element of an array.
// ptr must point to an array of size at least `sizeof(T) * N + alignment` bytes,
// where N is the number of elements in the array.
template<uintptr_t Alignment, typename T>
T* align_ptr_up(T* ptr) {
static_assert(alignof(T) < Alignment);
const uintptr_t ptrint = reinterpret_cast<uintptr_t>(reinterpret_cast<char*>(ptr));
return reinterpret_cast<T*>(
reinterpret_cast<char*>((ptrint + (Alignment - 1)) / Alignment * Alignment));
}
// True if and only if the binary is compiled on a little-endian machine
static inline const union {
uint32_t i;
char c[4];
} Le = {0x01020304};
static inline const bool IsLittleEndian = (Le.c[0] == 4);
template<typename T, std::size_t MaxSize>
class ValueList {
public:
std::size_t size() const { return size_; }
void push_back(const T& value) { values_[size_++] = value; }
const T* begin() const { return values_; }
const T* end() const { return values_ + size_; }
const T& operator[](int index) const { return values_[index]; }
private:
T values_[MaxSize];
std::size_t size_ = 0;
};
// xorshift64star Pseudo-Random Number Generator
// This class is based on original code written and dedicated
// to the public domain by Sebastiano Vigna (2014).
// It has the following characteristics:
//
// - Outputs 64-bit numbers
// - Passes Dieharder and SmallCrush test batteries
// - Does not require warm-up, no zeroland to escape
// - Internal state is a single 64-bit integer
// - Period is 2^64 - 1
// - Speed: 1.60 ns/call (Core i7 @3.40GHz)
//
// For further analysis see
// <http://vigna.di.unimi.it/ftp/papers/xorshift.pdf>
class PRNG {
uint64_t s;
uint64_t rand64() {
s ^= s >> 12, s ^= s << 25, s ^= s >> 27;
return s * 2685821657736338717LL;
}
public:
PRNG(uint64_t seed) :
s(seed) {
assert(seed);
}
template<typename T>
T rand() {
return T(rand64());
}
// Special generator used to fast init magic numbers.
// Output values only have 1/8th of their bits set on average.
template<typename T>
T sparse_rand() {
return T(rand64() & rand64() & rand64());
}
};
inline uint64_t mul_hi64(uint64_t a, uint64_t b) {
#if defined(__GNUC__) && defined(IS_64BIT)
__extension__ using uint128 = unsigned __int128;
return (uint128(a) * uint128(b)) >> 64;
#else
uint64_t aL = uint32_t(a), aH = a >> 32;
uint64_t bL = uint32_t(b), bH = b >> 32;
uint64_t c1 = (aL * bL) >> 32;
uint64_t c2 = aH * bL + c1;
uint64_t c3 = aL * bH + uint32_t(c2);
return aH * bH + (c2 >> 32) + (c3 >> 32);
#endif
}
struct CommandLine {
public:
CommandLine(int _argc, char** _argv) :
argc(_argc),
argv(_argv) {}
static std::string get_binary_directory(std::string argv0);
static std::string get_working_directory();
int argc;
char** argv;
};
namespace Utility {
template<typename T, typename Predicate>
void move_to_front(std::vector<T>& vec, Predicate pred) {
auto it = std::find_if(vec.begin(), vec.end(), pred);
if (it != vec.end())
{
std::rotate(vec.begin(), it, it + 1);
}
}
}
} // namespace Stockfish
#endif // #ifndef MISC_H_INCLUDED