This PR is a replacement for #6652 and takes into account the changes made to this part of the code since then.
In multiPV analysis master may suppress genuine proven mated-in scores, if the iteration for a secondary PV line is aborted. In very rare cases, it may also evict a proven mated-in score from the best PV line with a score from an incomplete iteration of a secondary PV line. This PR avoids both.
In addition, we allow best thread selection in multithreaded multiPV searches. This was originally disabled in #524.
Single PV searches are not affected by this patch at all.
Fixes#6642.
closes https://github.com/official-stockfish/Stockfish/pull/6728
No functional change.
LLR: 2.97 (-2.94,2.94) <0.00,2.00>
Total: 38400 W: 10057 L: 9749 D: 18594
Ptnml(0-2): 93, 4110, 10498, 4394, 105
https://tests.stockfishchess.org/tests/view/69d2073261a12cebe17edc04
compile time generation leads to sharing tables through the binary, resulting
in gain on fishtest, as well as reduced memory usage. The downside is slightly
longer compilation time.
Using pdep reduces table size 4x, and gains a little outside of fishtest as well.
closes https://github.com/official-stockfish/Stockfish/pull/6706
No functional change
`stockfish-ubuntu-x86-64-universal.tar` goes from 94.8MiB to 67.4MiB (tar.gz)
Note that the CI code is written such that it easy to select other compression
schemes in the future (e.g. tar.xz) by simply changing the `archive_ext`
parameter.
closes https://github.com/official-stockfish/Stockfish/pull/6746
No functional change
Pin every external action used by the root GitHub workflow set to a full-length
commit SHA and keep the selected release tag as a same-line comment.
Update the root workflows to the latest stable tagged action versions found
during the audit while preserving existing release guards, permissions, and
reusable workflow wiring.
closes https://github.com/official-stockfish/Stockfish/pull/6754
No functional change.
In setup_benchmark, the variables evaluating the current ply simply mirror the
exact iterative progress of the string arrays loop (since ply just increments
by 1 with every pass). We can delete the separate integer assignment and
execute it securely within the loop declaration.
closes https://github.com/official-stockfish/Stockfish/pull/6732
No functional change
Fix MultiArray::at() which was marked noexcept despite delegating to
std::array::at() that throws on out-of-range. Per review feedback, at() now
keeps standard bounds-checking semantics (only noexcept removed), and
operator[] gains an assert for debug-mode safety.
All eight correction history lookups in search.cpp (correction_value and
update_correction_history) are switched from .at(us) to [us].
Passed STC:
https://tests.stockfishchess.org/tests/view/69de7a735c67bee7d241ff92
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 77536 W: 20031 L: 19860 D: 37645
Ptnml(0-2): 221, 8411, 21338, 8572, 226
closes https://github.com/official-stockfish/Stockfish/pull/6734
No functional change
We maintain support for quite a few x86 ISA extensions but relied on the end
user to select the binary that is best for their system. We can detect at
runtime which architecture to use, and ship a single binary (per OS/ISA
combination). This PR does so, maintaining performance.
This is what I've landed on after quite a few iterations. Basically, we build
each arch separately, use `cpuid` to select one, and jump to that arch's main
function. Some details:
- The preprocessor macro `UNIVERSAL_BINARY` is defined when building for the universal binary.
- The Makefile target `universal-object-[no]pgo` is added, which produces a `stockfish.o` in each arch's build directory.
- To prevent symbol collisions between the multiple builds, we `#define Stockfish Stockfish_[arch]`. Furthermore we wrap the `main` function in `namespace Stockfish { }`.
- We can still get PGO data by linking to a per-arch binary, with `Stockfish_x86_64_avx2::main` as `main`, and running `bench`.
- For arches not supported by the host we can use Intel SDE – this is what we do in CI.
- When embedding the NNUE, we use C++26/C23 `#embed` instead of `INCBIN`. The issue with `INCBIN` is that it injects assembly directives that we have no control over.
- To ensure there's only one copy of the networks in the final binary, we define the network data as weak symbols in each per-arch build. Then, in the final link, we add a special nnue_embed.cpp which provides strong symbols.
- This does require GCC 15. Statically linking libstdc++ allows the binary to still run on older OSes though. And latest msys2 already does static linking + is based off GCC 15.2.0
building can be as simple as
make -j profile-build ARCH=x86-64-universal
or using the sde if the host doesn't support all architectures supported in the
universal binary, and the default host compiler is not recent enough.
make -j profile-build ARCH=x86-64-universal RUN_PREFIX="/path/to/sde -future --" CXX=g++-15
This change is also integrated in CI, so universal binaries are available as
downloadable artifacts.
Next we could also do ARM64, especially Android (all Apple silicon users use
the same binary). It also might be worth getting this to work with clang.
closes https://github.com/official-stockfish/Stockfish/pull/6740
No functional change
In C++, all standard input streams (including std::istringstream) are initialized with the skipws format flag enabled by default. Because these stream objects are scoped locally and freshly constructed on each loop iteration, we are guaranteed that skipws is already active.
closes https://github.com/official-stockfish/Stockfish/pull/6719
No functional change
Conditions for no pawn pushes, non-pawn material and last captured piece are already such restrictive that we don't need to artificially set checkers bitboard for legal move generation.
Also moving the check for available pawn pushes at first place, as it should already filter about 90% of cases.
Passed STC and LTC non-regression tests.
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 171232 W: 44202 L: 44131 D: 82899
Ptnml(0-2): 466, 18809, 47023, 18824, 494
https://tests.stockfishchess.org/tests/view/69d3623b61a12cebe17ededa
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 55662 W: 14424 L: 14247 D: 26991
Ptnml(0-2): 38, 5479, 16609, 5678, 27
https://tests.stockfishchess.org/tests/view/69d64cada07a818b0a25f95e
And also neutral on 10k fixed games test with custom stalemate book.
Elo: 0.24 ± 0.9 (95%) LOS: 69.9%
Total: 10000 W: 4681 L: 4674 D: 645
Ptnml(0-2): 0, 87, 4819, 94, 0
nElo: 1.81 ± 6.8 (95%) PairsRatio: 1.08
https://tests.stockfishchess.org/tests/view/69d7e4183ca80bdf151d43b8
closes https://github.com/official-stockfish/Stockfish/pull/6715
Bench: 2984258
A bit tricky to identify, but in `apple-silicon` builds, specifying a value for
`SyzygyPath` could still lead to crashes at certain depths from a
SIGBUS/SIGSEGV.
This PR replaces the `std::count_if()` adjustment in `tbprobe.cpp`'s
`do_probe_table()` with an annotated handwritten loop, avoiding the clang/LTO
miscompilation behind the crash.
closes https://github.com/official-stockfish/Stockfish/pull/6721
No functional change.
the assert checks for `bestThreadDecisive && newThreadDecisive`. However `newThreadDecisive` is not guaranteed at this point, leading to assertion failures. This PR rearranges the conditions to ensure correctness.
closes https://github.com/official-stockfish/Stockfish/pull/6716
No functional change
The uci pv suppression and PV roll-back logic in master is a bit
convoluted, which makes it hard to reason about the code. In fact,
subtle bugs that led to wrong mated-in scores in game play or a mismatch
between bestmove and first PV move were only recently fixed. Moreover,
in master the uci pv output through `pv()` is called in four different
places.
This PR proposes to simplify this logic. In this patch, the PV is sent
to the GUI from within `iterative_deepening()` only (a) for fail
highs/lows or (b) when an iteration for a root move is completed. All
other PV outputs are now handled by `start_searching()`, just before the
bestmove extraction. This easily ensures that bestmove (and ponder move)
will always be in sync with the final PV output.
The only noticeable change to master is for multi-threaded searches that
do not involve `limits.depth`. Here master would show both the PV from
the aborted search in main thread, as well as the new PV from the
selected best thread. This patch would only show the latter.
While at it, we also remove the requirement to finish at least a depth 1
search and simplify the logic around stalemates/checkmates at root and
in particular avoid any PVs that start with `Move::none()`.
Passed STC non-reg:
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 72416 W: 18784 L: 18609 D: 35023
Ptnml(0-2): 200, 7762, 20127, 7901, 218
https://tests.stockfishchess.org/tests/view/69d3ef0c33584dad27b3c85a
closes https://github.com/official-stockfish/Stockfish/pull/6704
No functional change
Instead of checking whether a threat index is valid and then writing to
a vector if it isn't, we can instead always write to the buffer of the
vector, and only increase the size of the vector if the index is valid.
This saves some branch mispredictions.
passed STC:
https://tests.stockfishchess.org/tests/view/69ceb1689f7a7e3fdfc9a44b
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 205760 W: 53109 L: 52561 D: 100090
Ptnml(0-2): 603, 22552, 56076, 22992, 657
local speedtest shows
```
Result of 100 runs (cycles)
===========================
base (...ockfish.orig) = 296886 +/- 1645
test (...kfish.df0822) = 298862 +/- 1662
speedup % = +0.67 +/- 0.19
[95% CI; t-statistic]
p(speedup > 0) = 1.0000
p(speedup > .5%) = 0.9562
CPU: 16 x Intel(R) Core(TM) Ultra 9 185H
Hyperthreading: on
```
closes https://github.com/official-stockfish/Stockfish/pull/6703
No functional change
[Passed
STC](https://tests.stockfishchess.org/tests/live_elo/69cf6a5d1668971c9da23ae1)
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 51104 W: 13376 L: 13180 D: 24548
Ptnml(0-2): 134, 5555, 13963, 5781, 119
[Passed
LTC](https://tests.stockfishchess.org/tests/live_elo/69d00a7ee2b443cb2670b5c6)
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 101256 W: 25867 L: 25732 D: 49657
Ptnml(0-2): 59, 10423, 29520, 10576, 50
When processing a promotion we add/remove nonsensical threats, namely,
piece threats of pawns on the 1st or 8th rank. This is not only
inefficient but also seems to lead to an obscure bug during a
promotion–capture + `double_inc_update` that causes a lingering invalid
feature. Currently – as an artifact of the training process – these
parts of the net are filled with random values in [-1,1], which is why
there is a bench change.
@ces42 did a helpful analysis:
> checking bench with depth 17 actually shows that the bug in has been
there all the time since 8e5392d7 (when threat inputs was merged into
master)
> i.e. double_inc_update for ThreatFeatureSet has always been slightly
bugged
After this patch is applied we'll be able to better compress the net by
filling the invalid features with 0, w/o changing bench. I expect a
15%-ish size savings.
closes https://github.com/official-stockfish/Stockfish/pull/6701
Bench: 2700393
After 415f2ef1. Now works again with CXX paths like `A:\path\to\g++`
under MSYS2. The cache files are no longer read in the rule itself and
only used as markers for change detection.
closes https://github.com/official-stockfish/Stockfish/pull/6699
No functional change
```
clang++: error: unknown argument: '-fno-ipa-cp-clone'
clang++: error: unknown argument: '-fno-ipa-cp-clone'
make[1]: *** [bitboard.o] Error 1
```
The flag is unsupported by Clang, which pretends to be GCC on macOS, so
only passing -fno-ipa-cp-clone to real GCC fixes macOS builds.
closes https://github.com/official-stockfish/Stockfish/pull/6698
No functional change
[STC](https://tests.stockfishchess.org/tests/live_elo/69cb991bc025c305b7daa219)
LLR: 2.97 (-2.94,2.94) <0.00,2.00>
Total: 47520 W: 12558 L: 12219 D: 22743
Ptnml(0-2): 190, 5474, 12102, 5795, 199
[LTC](https://tests.stockfishchess.org/tests/live_elo/69cc6b42731dfb72d2e92128)
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 123756 W: 31996 L: 31497 D: 60263
Ptnml(0-2): 81, 13512, 34221, 13955, 109
Adds features for pawns pushing against other pawns. For indexing, we
use the existing threats feature set by adding the forward square to the
squares threatened by a pawn. This threat is only enabled when another
pawn occupies that square. We then add appropriate logic changes in
`update_piece_threats` and `features/full_threats.cpp`.
The wonderful Ciekce (of Stormphrax) and peregrine (of Reckless) also
had similar ideas with adding forward pawn movements to TI, and I
appreciate their giving me motivation to figure out `nnue-pytorch` and
test it.
closes https://github.com/official-stockfish/Stockfish/pull/6694
Bench: 2926703
Makes eval calls more cache-friendly on a hot path.
Hints of a speedup found locally:
```
Result of 30 runs
==================
base (...ockfish-base) = 1754124 +/- 2887
test (...h-eval-order) = 1757848 +/- 2412
diff = +3724 +/- 1368
speedup = +0.0021
P(speedup > 0) = 1.0000
CPU: 16 x arm
Hyperthreading: off
```
Passed STC:
https://tests.stockfishchess.org/tests/view/69c84f3cf40d95b975597c21
LLR: 3.36 (-2.94,2.94) <0.00,2.00>
Total: 335840 W: 87022 L: 86257 D: 162561
Ptnml(0-2): 932, 36760, 91816, 37435, 977
closes https://github.com/official-stockfish/Stockfish/pull/6693
No functional change
Passed non-regression STC at
tests.stockfishchess.org/tests/view/69c9b8c3a54395a761335044 (set to
gainer bounds, *mea culpa*)
There's a major performance penalty when compiling the fish with GCC 16.
@Torom 's measurements (similar to mine, but more precise):
```
GCC 15 (master) vs. 16 (master)
sf_base = 2549174 +/- 2984 (95%)
sf_test = 2435456 +/- 2239 (95%)
diff = -113718 +/- 3599 (95%)
speedup = -4.46100% +/- 0.141% (95%)
GCC 15 (master) vs. 16 (patch)
sf_base = 2544107 +/- 3091 (95%)
sf_test = 2568837 +/- 3164 (95%)
diff = 24729 +/- 4359 (95%)
speedup = 0.97204% +/- 0.171% (95%)
```
Based on looking at the disassembly + profiling data of the builds, the
root cause appears to be overly aggressive code duplication from
constant propagation of various functions especially in `search.cpp`.
The worst offender is `Worker::search`, which I think is being cloned
and optimized based on the value of `cutNode`. `perf stat` indicates
that the instruction cache miss rate goes up by ~50% on the GCC 16
build.
closes https://github.com/official-stockfish/Stockfish/pull/6691
No functional change
Passed simp STC
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 29632 W: 7772 L: 7552 D: 14308
Ptnml(0-2): 76, 3155, 8147, 3349,
Id expect there to be a small speedup (bc the compiler can skip some
stores) but i haven’t tried to measure it
closes https://github.com/official-stockfish/Stockfish/pull/6688
No functional change
