jirayu_sri/stockfish
1
0
Fork 0
mirror of https://github.com/official-stockfish/Stockfish.git synced 2026-07-22 20:57:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Improve performance on NUMA systems

Browse Source 
Allow for NUMA memory replication for NNUE weights.  Bind threads to ensure execution on a specific NUMA node.

This patch introduces NUMA memory replication, currently only utilized for the NNUE weights. Along with it comes all machinery required to identify NUMA nodes and bind threads to specific processors/nodes. It also comes with small changes to Thread and ThreadPool to allow easier execution of custom functions on the designated thread. Old thread binding (WinProcGroup) machinery is removed because it's incompatible with this patch. Small changes to unrelated parts of the code were made to ensure correctness, like some classes being made unmovable, raw pointers replaced with unique_ptr. etc.

Windows 7 and Windows 10 is partially supported. Windows 11 is fully supported. Linux is fully supported, with explicit exclusion of Android. No additional dependencies.

-----------------

A new UCI option `NumaPolicy` is introduced. It can take the following values:
```
system - gathers NUMA node information from the system (lscpu or windows api), for each threads binds it to a single NUMA node
none - assumes there is 1 NUMA node, never binds threads
auto - this is the default value, depends on the number of set threads and NUMA nodes, will only enable binding on multinode systems and when the number of threads reaches a threshold (dependent on node size and count)
[[custom]] -
  // ':'-separated numa nodes
  // ','-separated cpu indices
  // supports "first-last" range syntax for cpu indices,
  for example '0-15,32-47:16-31,48-63'
```

Setting `NumaPolicy` forces recreation of the threads in the ThreadPool, which in turn forces the recreation of the TT.

The threads are distributed among NUMA nodes in a round-robin fashion based on fill percentage (i.e. it will strive to fill all NUMA nodes evenly). Threads are bound to NUMA nodes, not specific processors, because that's our only requirement and the OS can schedule them better.

Special care is made that maximum memory usage on systems that do not require memory replication stays as previously, that is, unnecessary copies are avoided.

On linux the process' processor affinity is respected. This means that if you for example use taskset to restrict Stockfish to a single NUMA node then the `system` and `auto` settings will only see a single NUMA node (more precisely, the processors included in the current affinity mask) and act accordingly.

-----------------

We can't ensure that a memory allocation takes place on a given NUMA node without using libnuma on linux, or using appropriate custom allocators on windows (https://learn.microsoft.com/en-us/windows/win32/memory/allocating-memory-from-a-numa-node), so to avoid complications the current implementation relies on first-touch policy. Due to this we also rely on the memory allocator to give us a new chunk of untouched memory from the system. This appears to work reliably on linux, but results may vary.

MacOS is not supported, because AFAIK it's not affected, and implementation would be problematic anyway.

Windows is supported since Windows 7 (https://learn.microsoft.com/en-us/windows/win32/api/processtopologyapi/nf-processtopologyapi-setthreadgroupaffinity). Until Windows 11/Server 2022 NUMA nodes are split such that they cannot span processor groups. This is because before Windows 11/Server 2022 it's not possible to set thread affinity spanning processor groups. The splitting is done manually in some cases (required after Windows 10 Build 20348). Since Windows 11/Server 2022 we can set affinites spanning processor group so this splitting is not done, so the behaviour is pretty much like on linux.

Linux is supported, **without** libnuma requirement. `lscpu` is expected.

-----------------

Passed 60+1 @ 256t 16000MB hash: https://tests.stockfishchess.org/tests/view/6654e443a86388d5e27db0d8
```
LLR: 2.95 (-2.94,2.94) <0.00,10.00>
Total: 278 W: 110 L: 29 D: 139
Ptnml(0-2): 0, 1, 56, 82, 0
```

Passed SMP STC: https://tests.stockfishchess.org/tests/view/6654fc74a86388d5e27db1cd
```
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 67152 W: 17354 L: 17177 D: 32621
Ptnml(0-2): 64, 7428, 18408, 7619, 57
```

Passed STC: https://tests.stockfishchess.org/tests/view/6654fb27a86388d5e27db15c
```
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 131648 W: 34155 L: 34045 D: 63448
Ptnml(0-2): 426, 13878, 37096, 14008, 416
```

fixes #5253
closes https://github.com/official-stockfish/Stockfish/pull/5285

No functional change
This commit is contained in:
Tomasz Sobczyk
2024-05-28 18:34:15 +02:00
committed by Joost VandeVondele
parent b0287dcb1c
commit a169c78b6d
19 changed files with 1418 additions and 289 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/thread.cpp
+138 -54
View File
@@ -22,19 +22,17 @@
#include <cassert>
#include <deque>
#include <memory>
#include <string>
#include <unordered_map>
#include <utility>
#include <string>
#include "misc.h"
#include "movegen.h"
#include "search.h"
#include "syzygy/tbprobe.h"
#include "timeman.h"
#include "tt.h"
#include "types.h"
#include "ucioption.h"
#include "uci.h"
#include "ucioption.h"
namespace Stockfish {
@@ -42,13 +40,24 @@ namespace Stockfish {
// in idle_loop(). Note that 'searching' and 'exit' should be already set.
Thread::Thread(Search::SharedState& sharedState,
std::unique_ptr<Search::ISearchManager> sm,
size_t n) :
worker(std::make_unique<Search::Worker>(sharedState, std::move(sm), n)),
size_t n,
OptionalThreadToNumaNodeBinder binder) :
idx(n),
nthreads(sharedState.options["Threads"]),
stdThread(&Thread::idle_loop, this) {
wait_for_search_finished();
run_custom_job([this, &binder, &sharedState, &sm, n]() {
// Use the binder to [maybe] bind the threads to a NUMA node before doing
// the Worker allocation.
// Ideally we would also allocate the SearchManager here, but that's minor.
this->numaAccessToken = binder();
this->worker =
std::make_unique<Search::Worker>(sharedState, std::move(sm), n, this->numaAccessToken);
});
wait_for_search_finished();
}
@@ -66,12 +75,15 @@ Thread::~Thread() {
// Wakes up the thread that will start the search
void Thread::start_searching() {
mutex.lock();
searching = true;
mutex.unlock(); // Unlock before notifying saves a few CPU-cycles
cv.notify_one(); // Wake up the thread in idle_loop()
assert(worker != nullptr);
run_custom_job([this]() { worker->start_searching(); });
}
// Wakes up the thread that will start the search
void Thread::clear_worker() {
assert(worker != nullptr);
run_custom_job([this]() { worker->clear(); });
}
// Blocks on the condition variable
// until the thread has finished searching.
@@ -81,20 +93,20 @@ void Thread::wait_for_search_finished() {
cv.wait(lk, [&] { return !searching; });
}
void Thread::run_custom_job(std::function<void()> f) {
{
std::unique_lock<std::mutex> lk(mutex);
cv.wait(lk, [&] { return !searching; });
jobFunc = std::move(f);
searching = true;
}
cv.notify_one();
}
// Thread gets parked here, blocked on the
// condition variable, when it has no work to do.
void Thread::idle_loop() {
// If OS already scheduled us on a different group than 0 then don't overwrite
// the choice, eventually we are one of many one-threaded processes running on
// some Windows NUMA hardware, for instance in fishtest. To make it simple,
// just check if running threads are below a threshold, in this case, all this
// NUMA machinery is not needed.
if (nthreads > 8)
WinProcGroup::bind_this_thread(idx);
while (true)
{
std::unique_lock<std::mutex> lk(mutex);
@@ -105,9 +117,13 @@ void Thread::idle_loop() {
if (exit)
return;
std::function<void()> job = std::move(jobFunc);
jobFunc = nullptr;
lk.unlock();
worker->start_searching();
if (job)
job();
}
}
@@ -121,49 +137,82 @@ uint64_t ThreadPool::tb_hits() const { return accumulate(&Search::Worker::tbHits
// Creates/destroys threads to match the requested number.
// Created and launched threads will immediately go to sleep in idle_loop.
// Upon resizing, threads are recreated to allow for binding if necessary.
void ThreadPool::set(Search::SharedState sharedState,
void ThreadPool::set(const NumaConfig& numaConfig,
Search::SharedState sharedState,
const Search::SearchManager::UpdateContext& updateContext) {
if (threads.size() > 0) // destroy any existing thread(s)
{
main_thread()->wait_for_search_finished();
while (threads.size() > 0)
delete threads.back(), threads.pop_back();
threads.clear();
boundThreadToNumaNode.clear();
}
const size_t requested = sharedState.options["Threads"];
if (requested > 0) // create new thread(s)
{
auto manager = std::make_unique<Search::SearchManager>(updateContext);
threads.push_back(new Thread(sharedState, std::move(manager), 0));
// Binding threads may be problematic when there's multiple NUMA nodes and
// multiple Stockfish instances running. In particular, if each instance
// runs a single thread then they would all be mapped to the first NUMA node.
// This is undesirable, and so the default behaviour (i.e. when the user does not
// change the NumaConfig UCI setting) is to not bind the threads to processors
// unless we know for sure that we span NUMA nodes and replication is required.
const std::string numaPolicy(sharedState.options["NumaPolicy"]);
const bool doBindThreads = [&]() {
if (numaPolicy == "none")
return false;
if (numaPolicy == "auto")
return numaConfig.suggests_binding_threads(requested);
// numaPolicy == "system", or explicitly set by the user
return true;
}();
boundThreadToNumaNode = doBindThreads
? numaConfig.distribute_threads_among_numa_nodes(requested)
: std::vector<NumaIndex>{};
while (threads.size() < requested)
{
auto null_manager = std::make_unique<Search::NullSearchManager>();
threads.push_back(new Thread(sharedState, std::move(null_manager), threads.size()));
const size_t threadId = threads.size();
const NumaIndex numaId = doBindThreads ? boundThreadToNumaNode[threadId] : 0;
auto manager = threadId == 0 ? std::unique_ptr<Search::ISearchManager>(
std::make_unique<Search::SearchManager>(updateContext))
: std::make_unique<Search::NullSearchManager>();
// When not binding threads we want to force all access to happen
// from the same NUMA node, because in case of NUMA replicated memory
// accesses we don't want to trash cache in case the threads get scheduled
// on the same NUMA node.
auto binder = doBindThreads ? OptionalThreadToNumaNodeBinder(numaConfig, numaId)
: OptionalThreadToNumaNodeBinder(numaId);
threads.emplace_back(
std::make_unique<Thread>(sharedState, std::move(manager), threadId, binder));
}
clear();
main_thread()->wait_for_search_finished();
// Reallocate the hash with the new threadpool size
sharedState.tt.resize(sharedState.options["Hash"], requested);
}
}
// Sets threadPool data to initial values
void ThreadPool::clear() {
for (Thread* th : threads)
th->worker->clear();
if (threads.size() == 0)
return;
for (auto&& th : threads)
th->clear_worker();
for (auto&& th : threads)
th->wait_for_search_finished();
main_manager()->callsCnt = 0;
main_manager()->bestPreviousScore = VALUE_INFINITE;
main_manager()->bestPreviousAverageScore = VALUE_INFINITE;
@@ -172,6 +221,17 @@ void ThreadPool::clear() {
main_manager()->tm.clear();
}
void ThreadPool::run_on_thread(size_t threadId, std::function<void()> f) {
assert(threads.size() > threadId);
threads[threadId]->run_custom_job(std::move(f));
}
void ThreadPool::wait_on_thread(size_t threadId) {
assert(threads.size() > threadId);
threads[threadId]->wait_for_search_finished();
}
size_t ThreadPool::num_threads() const { return threads.size(); }
// Wakes up main thread waiting in idle_loop() and
// returns immediately. Main thread will wake up other threads and start the search.
@@ -216,31 +276,36 @@ void ThreadPool::start_thinking(const OptionsMap& options,
// be deduced from a fen string, so set() clears them and they are set from
// setupStates->back() later. The rootState is per thread, earlier states are shared
// since they are read-only.
for (Thread* th : threads)
for (auto&& th : threads)
{
th->worker->limits = limits;
th->worker->nodes = th->worker->tbHits = th->worker->nmpMinPly =
th->worker->bestMoveChanges = 0;
th->worker->rootDepth = th->worker->completedDepth = 0;
th->worker->rootMoves = rootMoves;
th->worker->rootPos.set(pos.fen(), pos.is_chess960(), &th->worker->rootState);
th->worker->rootState = setupStates->back();
th->worker->tbConfig = tbConfig;
th->run_custom_job([&]() {
th->worker->limits = limits;
th->worker->nodes = th->worker->tbHits = th->worker->nmpMinPly =
th->worker->bestMoveChanges = 0;
th->worker->rootDepth = th->worker->completedDepth = 0;
th->worker->rootMoves = rootMoves;
th->worker->rootPos.set(pos.fen(), pos.is_chess960(), &th->worker->rootState);
th->worker->rootState = setupStates->back();
th->worker->tbConfig = tbConfig;
});
}
for (auto&& th : threads)
th->wait_for_search_finished();
main_thread()->start_searching();
}
Thread* ThreadPool::get_best_thread() const {
Thread* bestThread = threads.front();
Thread* bestThread = threads.front().get();
Value minScore = VALUE_NONE;
std::unordered_map<Move, int64_t, Move::MoveHash> votes(
2 * std::min(size(), bestThread->worker->rootMoves.size()));
// Find the minimum score of all threads
for (Thread* th : threads)
for (auto&& th : threads)
minScore = std::min(minScore, th->worker->rootMoves[0].score);
// Vote according to score and depth, and select the best thread
@@ -248,10 +313,10 @@ Thread* ThreadPool::get_best_thread() const {
return (th->worker->rootMoves[0].score - minScore + 14) * int(th->worker->completedDepth);
};
for (Thread* th : threads)
votes[th->worker->rootMoves[0].pv[0]] += thread_voting_value(th);
for (auto&& th : threads)
votes[th->worker->rootMoves[0].pv[0]] += thread_voting_value(th.get());
for (Thread* th : threads)
for (auto&& th : threads)
{
const auto bestThreadScore = bestThread->worker->rootMoves[0].score;
const auto newThreadScore = th->worker->rootMoves[0].score;
@@ -272,26 +337,26 @@ Thread* ThreadPool::get_best_thread() const {
// Note that we make sure not to pick a thread with truncated-PV for better viewer experience.
const bool betterVotingValue =
thread_voting_value(th) * int(newThreadPV.size() > 2)
thread_voting_value(th.get()) * int(newThreadPV.size() > 2)
> thread_voting_value(bestThread) * int(bestThreadPV.size() > 2);
if (bestThreadInProvenWin)
{
// Make sure we pick the shortest mate / TB conversion
if (newThreadScore > bestThreadScore)
bestThread = th;
bestThread = th.get();
}
else if (bestThreadInProvenLoss)
{
// Make sure we pick the shortest mated / TB conversion
if (newThreadInProvenLoss && newThreadScore < bestThreadScore)
bestThread = th;
bestThread = th.get();
}
else if (newThreadInProvenWin || newThreadInProvenLoss
|| (newThreadScore > VALUE_TB_LOSS_IN_MAX_PLY
&& (newThreadMoveVote > bestThreadMoveVote
|| (newThreadMoveVote == bestThreadMoveVote && betterVotingValue))))
bestThread = th;
bestThread = th.get();
}
return bestThread;
@@ -302,7 +367,7 @@ Thread* ThreadPool::get_best_thread() const {
// Will be invoked by main thread after it has started searching
void ThreadPool::start_searching() {
for (Thread* th : threads)
for (auto&& th : threads)
if (th != threads.front())
th->start_searching();
}
@@ -312,9 +377,28 @@ void ThreadPool::start_searching() {
void ThreadPool::wait_for_search_finished() const {
for (Thread* th : threads)
for (auto&& th : threads)
if (th != threads.front())
th->wait_for_search_finished();
}
std::vector<size_t> ThreadPool::get_bound_thread_count_by_numa_node() const {
std::vector<size_t> counts;
if (!boundThreadToNumaNode.empty())
{
NumaIndex highestNumaNode = 0;
for (NumaIndex n : boundThreadToNumaNode)
if (n > highestNumaNode)
highestNumaNode = n;
counts.resize(highestNumaNode + 1, 0);
for (NumaIndex n : boundThreadToNumaNode)
counts[n] += 1;
}
return counts;
}
} // namespace Stockfish