The core idea here is to greatly speed up the backwards insertion loop of the insertion sort. We can store up to 16 sorted moves and their values/scores in vectors. We obtain all moves with a smaller score (i.e., sorted later than the move-to-insert) as a mask, giving us something like `1111111111000000`. If we subtract one this gives us `1111111110111111` which can be can be used as an expand mask, to insert the new move.
Passed STC:
https://tests.stockfishchess.org/tests/view/69e9c9190e9667dd5a765853
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 45024 W: 11867 L: 11550 D: 21607
Ptnml(0-2): 101, 4938, 12142, 5205, 126
Local test:
```
Result of 50 runs
==================
base (...kfish.master) = 2026071 +/- 5124
test (./stockfish ) = 2083770 +/- 5964
diff = +57699 +/- 8535
speedup = +0.0285
P(speedup > 0) = 1.0000
```
closes https://github.com/official-stockfish/Stockfish/pull/6768
No functional change
Passed STC
https://tests.stockfishchess.org/tests/view/69e82e200e9667dd5a765631
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 115424 W: 29950 L: 29822 D: 55652
Ptnml(0-2): 318, 12712, 31544, 12800, 338
Passed AVX2-only STC
https://tests.stockfishchess.org/tests/view/69e953080e9667dd5a7657c0
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 134720 W: 34773 L: 34665 D: 65282
Ptnml(0-2): 344, 14958, 36694, 14974, 390
Now that our L1 size is much smaller, the overhead of tracking diffs for
`double_inc_update` is a larger share of the cost.
Interestingly ces42 measured a -0.9% slowdown locally (avxvnni build).
Meanwhile Torom and I find this branch to be 2% faster. I'm not sure exactly
what's going on but more tests may be in order
closes https://github.com/official-stockfish/Stockfish/pull/6776
No functional change
TRACKED_TOP previously used a shell pipeline (xargs, awk, sort) to extract
unique top-level directory/file names from SRCS and HEADERS. When invoking make
from cmd.exe on Windows, $(shell ...) uses cmd.exe rather than sh.exe, so the
pipeline fails because cmd.exe does not support Unix pipe syntax. Other MSYS2
tools (e.g. the compiler) work fine because they are invoked directly as
commands, not through a shell pipeline. This caused TRACKED_TOP to be empty, so
ln -s never ran and the per-arch temp_builds subdirectories were left without a
Makefile symlink, breaking the universal build with: "No rule to make target
'universal-object-nopgo'". Fixed by replacing the shell pipeline with pure GNU
make built-in functions (sort, foreach, firstword, subst) which have no shell
or PATH dependencies and work identically on Windows, Linux and macOS.
closes https://github.com/official-stockfish/Stockfish/pull/6762
No functional change
For quite some time SF has reported a too low final search depth in rare cases
(in multi-threaded search if a non-main thread was selected). Unfortunately,
the recent refactoring in #6704 means that this now happens for almost every
stopped search.
This PR fixes this issue, which was first spotted and reported by joergoster,
by ensuring that `rootDepth` always holds the last depth for which a search was
started.
Fixes#6756.
closes https://github.com/official-stockfish/Stockfish/pull/6761
No functional change.
Piece promotion is generated, and then type_of(promotion) is used to retrieve
the underlying type. The class Move provides .promotion_type() natively,
meaning we can cache the type directly and avoid extracting it indirectly
afterward. By extracting pt = m.promotion_type(); up front, we don't need to
call type_of(promotion) iteratively inside the assertions or conditions below.
closes https://github.com/official-stockfish/Stockfish/pull/6759
No functional change
This PR is a replacement for #6652 and takes into account the changes made to this part of the code since then.
In multiPV analysis master may suppress genuine proven mated-in scores, if the iteration for a secondary PV line is aborted. In very rare cases, it may also evict a proven mated-in score from the best PV line with a score from an incomplete iteration of a secondary PV line. This PR avoids both.
In addition, we allow best thread selection in multithreaded multiPV searches. This was originally disabled in #524.
Single PV searches are not affected by this patch at all.
Fixes#6642.
closes https://github.com/official-stockfish/Stockfish/pull/6728
No functional change.
LLR: 2.97 (-2.94,2.94) <0.00,2.00>
Total: 38400 W: 10057 L: 9749 D: 18594
Ptnml(0-2): 93, 4110, 10498, 4394, 105
https://tests.stockfishchess.org/tests/view/69d2073261a12cebe17edc04
compile time generation leads to sharing tables through the binary, resulting
in gain on fishtest, as well as reduced memory usage. The downside is slightly
longer compilation time.
Using pdep reduces table size 4x, and gains a little outside of fishtest as well.
closes https://github.com/official-stockfish/Stockfish/pull/6706
No functional change
In setup_benchmark, the variables evaluating the current ply simply mirror the
exact iterative progress of the string arrays loop (since ply just increments
by 1 with every pass). We can delete the separate integer assignment and
execute it securely within the loop declaration.
closes https://github.com/official-stockfish/Stockfish/pull/6732
No functional change
Fix MultiArray::at() which was marked noexcept despite delegating to
std::array::at() that throws on out-of-range. Per review feedback, at() now
keeps standard bounds-checking semantics (only noexcept removed), and
operator[] gains an assert for debug-mode safety.
All eight correction history lookups in search.cpp (correction_value and
update_correction_history) are switched from .at(us) to [us].
Passed STC:
https://tests.stockfishchess.org/tests/view/69de7a735c67bee7d241ff92
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 77536 W: 20031 L: 19860 D: 37645
Ptnml(0-2): 221, 8411, 21338, 8572, 226
closes https://github.com/official-stockfish/Stockfish/pull/6734
No functional change
We maintain support for quite a few x86 ISA extensions but relied on the end
user to select the binary that is best for their system. We can detect at
runtime which architecture to use, and ship a single binary (per OS/ISA
combination). This PR does so, maintaining performance.
This is what I've landed on after quite a few iterations. Basically, we build
each arch separately, use `cpuid` to select one, and jump to that arch's main
function. Some details:
- The preprocessor macro `UNIVERSAL_BINARY` is defined when building for the universal binary.
- The Makefile target `universal-object-[no]pgo` is added, which produces a `stockfish.o` in each arch's build directory.
- To prevent symbol collisions between the multiple builds, we `#define Stockfish Stockfish_[arch]`. Furthermore we wrap the `main` function in `namespace Stockfish { }`.
- We can still get PGO data by linking to a per-arch binary, with `Stockfish_x86_64_avx2::main` as `main`, and running `bench`.
- For arches not supported by the host we can use Intel SDE – this is what we do in CI.
- When embedding the NNUE, we use C++26/C23 `#embed` instead of `INCBIN`. The issue with `INCBIN` is that it injects assembly directives that we have no control over.
- To ensure there's only one copy of the networks in the final binary, we define the network data as weak symbols in each per-arch build. Then, in the final link, we add a special nnue_embed.cpp which provides strong symbols.
- This does require GCC 15. Statically linking libstdc++ allows the binary to still run on older OSes though. And latest msys2 already does static linking + is based off GCC 15.2.0
building can be as simple as
make -j profile-build ARCH=x86-64-universal
or using the sde if the host doesn't support all architectures supported in the
universal binary, and the default host compiler is not recent enough.
make -j profile-build ARCH=x86-64-universal RUN_PREFIX="/path/to/sde -future --" CXX=g++-15
This change is also integrated in CI, so universal binaries are available as
downloadable artifacts.
Next we could also do ARM64, especially Android (all Apple silicon users use
the same binary). It also might be worth getting this to work with clang.
closes https://github.com/official-stockfish/Stockfish/pull/6740
No functional change
In C++, all standard input streams (including std::istringstream) are initialized with the skipws format flag enabled by default. Because these stream objects are scoped locally and freshly constructed on each loop iteration, we are guaranteed that skipws is already active.
closes https://github.com/official-stockfish/Stockfish/pull/6719
No functional change
Conditions for no pawn pushes, non-pawn material and last captured piece are already such restrictive that we don't need to artificially set checkers bitboard for legal move generation.
Also moving the check for available pawn pushes at first place, as it should already filter about 90% of cases.
Passed STC and LTC non-regression tests.
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 171232 W: 44202 L: 44131 D: 82899
Ptnml(0-2): 466, 18809, 47023, 18824, 494
https://tests.stockfishchess.org/tests/view/69d3623b61a12cebe17ededa
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 55662 W: 14424 L: 14247 D: 26991
Ptnml(0-2): 38, 5479, 16609, 5678, 27
https://tests.stockfishchess.org/tests/view/69d64cada07a818b0a25f95e
And also neutral on 10k fixed games test with custom stalemate book.
Elo: 0.24 ± 0.9 (95%) LOS: 69.9%
Total: 10000 W: 4681 L: 4674 D: 645
Ptnml(0-2): 0, 87, 4819, 94, 0
nElo: 1.81 ± 6.8 (95%) PairsRatio: 1.08
https://tests.stockfishchess.org/tests/view/69d7e4183ca80bdf151d43b8
closes https://github.com/official-stockfish/Stockfish/pull/6715
Bench: 2984258
A bit tricky to identify, but in `apple-silicon` builds, specifying a value for
`SyzygyPath` could still lead to crashes at certain depths from a
SIGBUS/SIGSEGV.
This PR replaces the `std::count_if()` adjustment in `tbprobe.cpp`'s
`do_probe_table()` with an annotated handwritten loop, avoiding the clang/LTO
miscompilation behind the crash.
closes https://github.com/official-stockfish/Stockfish/pull/6721
No functional change.
the assert checks for `bestThreadDecisive && newThreadDecisive`. However `newThreadDecisive` is not guaranteed at this point, leading to assertion failures. This PR rearranges the conditions to ensure correctness.
closes https://github.com/official-stockfish/Stockfish/pull/6716
No functional change
The uci pv suppression and PV roll-back logic in master is a bit
convoluted, which makes it hard to reason about the code. In fact,
subtle bugs that led to wrong mated-in scores in game play or a mismatch
between bestmove and first PV move were only recently fixed. Moreover,
in master the uci pv output through `pv()` is called in four different
places.
This PR proposes to simplify this logic. In this patch, the PV is sent
to the GUI from within `iterative_deepening()` only (a) for fail
highs/lows or (b) when an iteration for a root move is completed. All
other PV outputs are now handled by `start_searching()`, just before the
bestmove extraction. This easily ensures that bestmove (and ponder move)
will always be in sync with the final PV output.
The only noticeable change to master is for multi-threaded searches that
do not involve `limits.depth`. Here master would show both the PV from
the aborted search in main thread, as well as the new PV from the
selected best thread. This patch would only show the latter.
While at it, we also remove the requirement to finish at least a depth 1
search and simplify the logic around stalemates/checkmates at root and
in particular avoid any PVs that start with `Move::none()`.
Passed STC non-reg:
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 72416 W: 18784 L: 18609 D: 35023
Ptnml(0-2): 200, 7762, 20127, 7901, 218
https://tests.stockfishchess.org/tests/view/69d3ef0c33584dad27b3c85a
closes https://github.com/official-stockfish/Stockfish/pull/6704
No functional change
Instead of checking whether a threat index is valid and then writing to
a vector if it isn't, we can instead always write to the buffer of the
vector, and only increase the size of the vector if the index is valid.
This saves some branch mispredictions.
passed STC:
https://tests.stockfishchess.org/tests/view/69ceb1689f7a7e3fdfc9a44b
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 205760 W: 53109 L: 52561 D: 100090
Ptnml(0-2): 603, 22552, 56076, 22992, 657
local speedtest shows
```
Result of 100 runs (cycles)
===========================
base (...ockfish.orig) = 296886 +/- 1645
test (...kfish.df0822) = 298862 +/- 1662
speedup % = +0.67 +/- 0.19
[95% CI; t-statistic]
p(speedup > 0) = 1.0000
p(speedup > .5%) = 0.9562
CPU: 16 x Intel(R) Core(TM) Ultra 9 185H
Hyperthreading: on
```
closes https://github.com/official-stockfish/Stockfish/pull/6703
No functional change
[Passed
STC](https://tests.stockfishchess.org/tests/live_elo/69cf6a5d1668971c9da23ae1)
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 51104 W: 13376 L: 13180 D: 24548
Ptnml(0-2): 134, 5555, 13963, 5781, 119
[Passed
LTC](https://tests.stockfishchess.org/tests/live_elo/69d00a7ee2b443cb2670b5c6)
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 101256 W: 25867 L: 25732 D: 49657
Ptnml(0-2): 59, 10423, 29520, 10576, 50
When processing a promotion we add/remove nonsensical threats, namely,
piece threats of pawns on the 1st or 8th rank. This is not only
inefficient but also seems to lead to an obscure bug during a
promotion–capture + `double_inc_update` that causes a lingering invalid
feature. Currently – as an artifact of the training process – these
parts of the net are filled with random values in [-1,1], which is why
there is a bench change.
@ces42 did a helpful analysis:
> checking bench with depth 17 actually shows that the bug in has been
there all the time since 8e5392d7 (when threat inputs was merged into
master)
> i.e. double_inc_update for ThreatFeatureSet has always been slightly
bugged
After this patch is applied we'll be able to better compress the net by
filling the invalid features with 0, w/o changing bench. I expect a
15%-ish size savings.
closes https://github.com/official-stockfish/Stockfish/pull/6701
Bench: 2700393
After 415f2ef1. Now works again with CXX paths like `A:\path\to\g++`
under MSYS2. The cache files are no longer read in the rule itself and
only used as markers for change detection.
closes https://github.com/official-stockfish/Stockfish/pull/6699
No functional change
```
clang++: error: unknown argument: '-fno-ipa-cp-clone'
clang++: error: unknown argument: '-fno-ipa-cp-clone'
make[1]: *** [bitboard.o] Error 1
```
The flag is unsupported by Clang, which pretends to be GCC on macOS, so
only passing -fno-ipa-cp-clone to real GCC fixes macOS builds.
closes https://github.com/official-stockfish/Stockfish/pull/6698
No functional change
[STC](https://tests.stockfishchess.org/tests/live_elo/69cb991bc025c305b7daa219)
LLR: 2.97 (-2.94,2.94) <0.00,2.00>
Total: 47520 W: 12558 L: 12219 D: 22743
Ptnml(0-2): 190, 5474, 12102, 5795, 199
[LTC](https://tests.stockfishchess.org/tests/live_elo/69cc6b42731dfb72d2e92128)
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 123756 W: 31996 L: 31497 D: 60263
Ptnml(0-2): 81, 13512, 34221, 13955, 109
Adds features for pawns pushing against other pawns. For indexing, we
use the existing threats feature set by adding the forward square to the
squares threatened by a pawn. This threat is only enabled when another
pawn occupies that square. We then add appropriate logic changes in
`update_piece_threats` and `features/full_threats.cpp`.
The wonderful Ciekce (of Stormphrax) and peregrine (of Reckless) also
had similar ideas with adding forward pawn movements to TI, and I
appreciate their giving me motivation to figure out `nnue-pytorch` and
test it.
closes https://github.com/official-stockfish/Stockfish/pull/6694
Bench: 2926703
Makes eval calls more cache-friendly on a hot path.
Hints of a speedup found locally:
```
Result of 30 runs
==================
base (...ockfish-base) = 1754124 +/- 2887
test (...h-eval-order) = 1757848 +/- 2412
diff = +3724 +/- 1368
speedup = +0.0021
P(speedup > 0) = 1.0000
CPU: 16 x arm
Hyperthreading: off
```
Passed STC:
https://tests.stockfishchess.org/tests/view/69c84f3cf40d95b975597c21
LLR: 3.36 (-2.94,2.94) <0.00,2.00>
Total: 335840 W: 87022 L: 86257 D: 162561
Ptnml(0-2): 932, 36760, 91816, 37435, 977
closes https://github.com/official-stockfish/Stockfish/pull/6693
No functional change
Passed non-regression STC at
tests.stockfishchess.org/tests/view/69c9b8c3a54395a761335044 (set to
gainer bounds, *mea culpa*)
There's a major performance penalty when compiling the fish with GCC 16.
@Torom 's measurements (similar to mine, but more precise):
```
GCC 15 (master) vs. 16 (master)
sf_base = 2549174 +/- 2984 (95%)
sf_test = 2435456 +/- 2239 (95%)
diff = -113718 +/- 3599 (95%)
speedup = -4.46100% +/- 0.141% (95%)
GCC 15 (master) vs. 16 (patch)
sf_base = 2544107 +/- 3091 (95%)
sf_test = 2568837 +/- 3164 (95%)
diff = 24729 +/- 4359 (95%)
speedup = 0.97204% +/- 0.171% (95%)
```
Based on looking at the disassembly + profiling data of the builds, the
root cause appears to be overly aggressive code duplication from
constant propagation of various functions especially in `search.cpp`.
The worst offender is `Worker::search`, which I think is being cloned
and optimized based on the value of `cutNode`. `perf stat` indicates
that the instruction cache miss rate goes up by ~50% on the GCC 16
build.
closes https://github.com/official-stockfish/Stockfish/pull/6691
No functional change
