[Passed STC](https://tests.stockfishchess.org/tests/live_elo/691fc321acb6dbdf23d07e4b):
```
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 215360 W: 55651 L: 55108 D: 104601
Ptnml(0-2): 520, 22399, 61290, 22960, 511
```
This PR is on top of ces42's work so I'll rebase if that PR changes.
Essentially, it comprises my adjustments to `make_index` that remained
unmerged before threat inputs was finalized.
The unsigned intermediate variables let us skip a bunch of useless sign
extensions (because Square and Piece inherit from `int8_t`, and C/C++
semantics require narrow integers to be promoted to `int` before doing
arithmetic on them). Additionally, by removing the special usage of
`offsets[][64]` and `[65]` the indexing becomes more efficient. (This
usage was a temporary hack from sscg anyway, so I think he'll like that
it's gone.) Finally, the `sf_assume` condition was fixed so that it
actually makes a difference to the compiler.
The speedup is fairly nice locally (combining both ces42's and these changes):
```
Result of 100 runs
==================
base (...kfish.master) = 1691982 +/- 1907
test (./stockfish ) = 1714349 +/- 1789
diff = +22367 +/- 2465
speedup = +0.0132
P(speedup > 0) = 1.0000
```
closes https://github.com/official-stockfish/Stockfish/pull/6445
No functional change
fetch_add and friends must produce a locking instruction because they guarantee
consistency in the presence of multiple writers. Because only one thread ever
writes to nodes and tbHits, we may use relaxed load and store to emit regular
loads and stores (on both x86-64 and arm64), while avoiding undefined behavior
and miscounting nodes.
Credit must go to Arseniy Surkov of Reckless (codedeliveryservice) for
uncovering this performance gotcha.
failed yellow as a gainer on fishtest:
LLR: -2.95 (-2.94,2.94) <0.00,2.00>
Total: 219616 W: 57165 L: 57105 D: 105346
Ptnml(0-2): 732, 24277, 59720, 24357, 722
https://tests.stockfishchess.org/tests/view/69086abfea4b268f1fac2809
potential small speedup on large core system:
```
==== master ====
1 Nodes/second : 294269736
2 Nodes/second : 295217629
Average (over 2): 294743682
==== patch ====
1 Nodes/second : 299003603
2 Nodes/second : 298111320
Average (over 2): 298557461
```
closes https://github.com/official-stockfish/Stockfish/pull/6392
No functional change
This PR allows for time checking within Tablebases::rank_root_moves(). The
principal application for now is syzygy_extend_pv().
In the past, Stockfish suffered some time losses at e.g. TCEC when the HDD with
the DTZ tables was too slow for the chosen move overhead setting, see #5894.
```
> ./fastchess -engine name="patch" cmd=./stockfish.patch -engine name="master" cmd=./stockfish.master -each tc=10+0.1 option.SyzygyPath=/disk1/syzygy/3-4-5-6/WDL:/disk2/syzygy/3-4-5-6/DTZ -draw movenumber=34 movecount=8 score=20 -openings file=UHO_Lichess_4852_v1.epd format=epd -rounds 50 -repeat -concurrency 16
...
Results of patch vs master (10+0.1, 1t, 16MB, UHO_Lichess_4852_v1.epd):
Elo: 59.64 +/- 35.35, nElo: 117.61 +/- 68.10
LOS: 99.96 %, DrawRatio: 56.00 %, PairsRatio: 4.50
Games: 100, Wins: 34, Losses: 17, Draws: 49, Points: 58.5 (58.50 %)
Ptnml(0-2): [0, 4, 28, 15, 3], WL/DD Ratio: 0.87
--------------------------------------------------
Player: master
Timeouts: 19
Crashed: 0
Finished match
```
closes https://github.com/official-stockfish/Stockfish/pull/6422
No functional change
During recent work on threat inputs, there was a hard-to-debug crash in tbprobe
caused by incorrectly computing st->materialKey. This PR adds correctness
checking to pos_is_ok and a faster check in tbprobe that will actually run in
CI (because pos_is_ok checking is disabled even in debug mode, for performance
purposes).
closes https://github.com/official-stockfish/Stockfish/pull/6410
No functional change
Normally, Stockfish outputs the compilation settings from advanced to less
advanced, e.g.: `AVX512ICL VNNI AVX512 BMI2 AVX2 SSE41 SSSE3 SSE2 POPCNT`
With NEON, however, POPCNT is printed first, followed by the more advanced NEON
options: `POPCNT NEON_DOTPROD`
This PR places POPCNT behind the NEON options as well.
closes https://github.com/official-stockfish/Stockfish/pull/6402
no functional change
The contents of DirtyThreats gets memcpyed twice for each call to do_move.
(Return value optimization doesn't apply to the do_move function itself because
it constructs a std::pair, so it gets copied; and the calls to reset also
require a copy.) This patch inserts the dirty info in place.
Sometimes the caller of do_move ignores the DirtyThreats info, so we pass in
scratch objects. I found that stack-allocating these scratch objects was bad on
Fishtest, so I begrudgingly put them in the Position struct. Both templating
the do_move function on whether dirty threats are needed and putting a
null-check branch for each use of dirty threats were slowdowns locally. Of
course, nothing prevents a future attempt at cleaning this up.
passed STC
LLR: 2.96 (-2.94,2.94) <0.00,2.00>
Total: 68448 W: 17770 L: 17418 D: 33260
Ptnml(0-2): 198, 7425, 18630, 7769, 202
https://tests.stockfishchess.org/tests/view/6914c01a7ca87818523318ba
closes https://github.com/official-stockfish/Stockfish/pull/6414
No functional change
This enables different Stockfish processes that use the same weights to use the
same memory. The approach establishes equivalence by memory content, and is
compatible with NUMA replication. The benefit of sharing is reduced memory usage
and a speedup thanks to improved (inter-process) caching of the network in the
CPUs cache, and thus reduced bandwidth usage to main memory. Even though this
change doesn't benefit a user running a single process, this helps on fishtest
or e.g. for Lichess, when multiple games run concurrently, or multiple
positions are analyzed in parallel.
This concept was probably first introduced in the Monty engine
(https://github.com/official-monty/Monty/pull/62), after a discussion in
https://github.com/official-stockfish/fishtest/issues/2077 on the issue of
memory pressure. Measurements based on Torch
(https://github.com/user-attachments/files/21386224/verbatim.pdf) further
suggested that large gains were possible. Multiple other engines have
adopted this 'verbatim' format as well.
The implementation here adds the flexibility needed for SF, for example, retains
the ability to bundle compressed networks with the binary, to load nets by uci
option, and to distribute the shared nets to the proper NUMA region. This
flexibility comes with a fair amount of complexity in the implementation, such
as OS specific code, and fallback code.
For most users this should be transparent. However, for example, those running
docker containers should ensure the `--ipc` flag is set correctly, and
`--shm-size` is sufficiently large.
The benefits of this patch significantly depend on hardware, with systems with
many cores and a large (O(150MB), the net size) L3 cache benefitting typically
most. On such systems SF speedups (as measured via nps playing games with
large concurrency but just 1 thread) can be 38%, which results in master vs.
patch Elo which gains about 25 Elo.
```
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 shared_memoryPR : 24.8 1.9 39432.0 73728 53
2 master : 0.0 ---- 34296.0 73728 47
```
In a multithreaded setup, where weights are already shared, that benefit is smaller,
for example on the same HW as above, but with 8t for each side.
```
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 shared_memoryPR : 5.2 3.5 9351.0 18432 51
2 master : 0.0 ---- 9081.0 18432 49
```
On fishtest with a typical hardware mix of our contributors, the following was measured:
STC, 60k games
https://tests.stockfishchess.org/tests/view/69074a49ea4b268f1fac236c
Elo: 4.69 ± 1.4 (95%) LOS: 100.0%
Total: 60000 W: 16085 L: 15275 D: 28640
Ptnml(0-2): 154, 6440, 16053, 7148, 205
nElo: 9.38 ± 2.8 (95%) PairsRatio: 1.12
To verify correctness with a single process on a NUMA architecture,
speedtest was used, confirming near equivalence:
```
master: Average (over 10): 296236186
shared_memory: Average (over 10): 295769332
```
Currently, using large pages for the shared network weights is not always possible,
which can lead to a small slowdown (1-2%), in case a single process is run.
closes https://github.com/official-stockfish/Stockfish/pull/6173
No functional change
Co-authored-by: disservin <disservin.social@gmail.com>
Co-authored-by: Joost VandeVondele <Joost.VandeVondele@gmail.com>
fixes https://github.com/official-stockfish/Stockfish/issues/6328
Before calling search(depth), the patch ensures that depth is at least 1
whenever we encounter a decisive score in the transposition table (TT). This
prevents search(depth) from being executed by qsearch, which would otherwise
ignore that information. Typically, decisive TT hits occur when analyzing a
mating sequence backward. Due to the nature of Iterative Deepening (IID), such
scores are usually first found at depth 0. Without this patch, valuable
information can be lost because qsearch may overwrite the TT entry by replacing
the value with a static evaluation, even though the node was already processed
at a higher depth. This is also why the engine sometimes loses track of an
already discovered mate.
Using ..\sf\patch.exe on matetrack.epd with --nodes 1000000
Engine ID: Stockfish dev-20251015-nogit
Total FENs: 6554
Found mates: 3437
Best mates: 2438
Using ..\sf\master.exe on matetrack.epd with --nodes 1000000
Engine ID: Stockfish dev-20251015-nogit
Total FENs: 6554
Found mates: 3337
Best mates: 2407
Passed STC
https://tests.stockfishchess.org/tests/view/68fa3fa7637acd2a11e72a79
LLR: 3.55 (-2.94,2.94) <0.00,2.00>
Total: 134144 W: 34960 L: 34471 D: 64713
Ptnml(0-2): 376, 14199, 37459, 14636, 402
Failed LTC
https://tests.stockfishchess.org/tests/view/68ffc1b5637acd2a11e73377
LLR: -3.10 (-2.94,2.94) <0.50,2.50>
Total: 75360 W: 19423 L: 19519 D: 36418
Ptnml(0-2): 38, 7553, 22605, 7435, 49
closes https://github.com/official-stockfish/Stockfish/pull/6386
bench: 2530552
The speedup seems to vary by machine. The indexing function can be changed w/o needing to understand intrinsics.
Result of 100 runs
==================
base (...ish_baseline) = 1719637 +/- 3233
test (./stockfish ) = 1734245 +/- 3534
diff = +14608 +/- 4868
speedup = +0.0085
P(speedup > 0) = 1.0000
closes https://github.com/official-stockfish/Stockfish/pull/6366
No functional change
Only a modest amount of work happens between the transposition table prefetch
and the probe, so the probe still often stalls waiting for DRAM. The vast
majority of the time (in particular, if !checkEP), the key is known much
earlier in the do_move function and the latency can be better hidden.
passed STC SMP
https://tests.stockfishchess.org/tests/view/68f337c528e6d77fcffa066a
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 65256 W: 16806 L: 16462 D: 31988
Ptnml(0-2): 76, 7386, 17362, 7726, 78
but failed to gain STC
https://tests.stockfishchess.org/tests/view/68f3378328e6d77fcffa0665
LLR: -2.94 (-2.94,2.94) <0.00,2.00>
Total: 109824 W: 28523 L: 28618 D: 52683
Ptnml(0-2): 311, 11799, 30788, 11702, 312
In local tests, the speedup grows with thread count
closes https://github.com/official-stockfish/Stockfish/pull/6372
No functional change
