After merging the HalfKA and Threats accumulators (7c7fe322) and the
subsequent removal of the double-incremental/fused update, a number of
NNUE helpers and fields became unreachable. Each was verified to have
zero callers/readers across the source tree:
- FusedUpdateData logic in FullThreats: the fused-update branch of
append_changed_indices and the FusedUpdateData parameter are unused;
the accumulator update no longer passes fused data.
- FullThreats::requires_refresh: never called. The live king-bucket
refresh check is HalfKAv2_hm::requires_refresh (PSQFeatureSet), used
in nnue_accumulator.
- HalfKAv2_hm::append_active_indices: never called. The live
active-index builder is FullThreats::append_active_indices
(ThreatFeatureSet).
- DirtyThreats::us, prevKsq and ksq: written in do_move but only read by
the now-removed FullThreats::requires_refresh. Removing them also
drops three stores from the do_move path.
- Unused feature Name constants and the unused FtOneVal / HiddenMaxVal
constants in nnue_common.h.
- Two stale feature-header banner comments.
closes https://github.com/official-stockfish/Stockfish/pull/6898
No functional change
The FEN validation check intended to reject pawns on the first or eighth rank uses the `Rank` enum values in a bitwise OR operation:
`if (pieces(PAWN) & (RANK_1 | RANK_8))`
`RANK_1 | RANK_8` evaluates to the integer `0 | 7 == 7` instead of a bitboard, so the expression only tests squares A1, B1 and C1. As a result, unsupported positions with pawns elsewhere on the first or eighth rank are silently accepted. For instance, `position fen 3P3k/8/8/8/8/8/8/3K4 w - - 0 1` is accepted even though the pawn on d8 makes the position unsupported.
Use the `Rank1BB | Rank8BB` bitboard constants so any pawn on the first or eighth rank are correctly rejected.
closes https://github.com/official-stockfish/Stockfish/pull/6887
No functional change
In a recent CCC event, Stockfish (probably through no fault of its own), lost some games on time when it was winning and when it had already found the move that delivers checkmate.
This patch stops the search when TM is active, and when mainthread can be certain that it is impossible to find a better move. That is if (i) it has found mate-in-1, (ii) it has found a mate-in-2 or (iii) it has found a mated-in-1.
patch:
```
position fen 5K2/8/2qk4/2nPp3/3r4/6B1/B7/3R4 w - e6
go wtime 100000000 winc 100000000
info string Available processors: 0-7
info string Using 1 thread
info string NNUE evaluation using nn-71d6d32cb962.nnue (106MiB, (83248, 1024, 31, 32, 1))
info string Network replica 1: Shared memory.
info depth 1 seldepth 3 multipv 1 score mate 1 nodes 30 nps 30000 hashfull 0 tbhits 0 time 1 pv d5e6
bestmove d5e6
```
master:
```
position fen 5K2/8/2qk4/2nPp3/3r4/6B1/B7/3R4 w - e6
go wtime 100000000 winc 100000000
info string Available processors: 0-7
info string Using 1 thread
info string NNUE evaluation using nn-71d6d32cb962.nnue (106MiB, (83248, 1024, 31, 32, 1))
info string Network replica 1: Shared memory.
info depth 1 seldepth 3 multipv 1 score mate 1 nodes 30 nps 15000 hashfull 0 tbhits 0 time 2 pv d5e6
<snip>
info depth 245 seldepth 2 multipv 1 score mate 1 nodes 5886 nps 367875 hashfull 0 tbhits 0 time 16 pv d5e6
bestmove d5e6
```
Note: In MultiPV analysis (extremely rare with TM active), we take the point of view that the user would like to continue to search until none of the PVs can be improved anymore. This means we only stop if the worst searched line is at least a mate-in-2, or if the best searched line is a mated-in-1.
closes https://github.com/official-stockfish/Stockfish/pull/6879
No functional change
Passed STC
https://tests.stockfishchess.org/tests/view/6a2682ce351b79f679cc47c5
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 57248 W: 14730 L: 14399 D: 28119
Ptnml(0-2): 145, 6159, 15698, 6464, 158
Reordering operations in `do_move` allows us to effectively prefetch the TT entry earlier, since the piece moving helpers don't actually modify the position key. I suspect that with threat inputs, `put_piece` and friends got a lot more expensive, and so this helps us a lot.
vondele's machine:
==== master ====
1 Nodes/second : 294311526
2 Nodes/second : 297068312
3 Nodes/second : 297418763
Average (over 3): 296266200
==== pfearly ====
1 Nodes/second : 303986449
2 Nodes/second : 304221719
3 Nodes/second : 305302969
Average (over 3): 304503712 (+2.78%)
Locally, `bench`:
Result of 200 runs
speedup = +0.0158
P(speedup > 0) = 1.0000
As expected it helps even more in a large-hash, NUMA setting.
closes https://github.com/official-stockfish/Stockfish/pull/6891
No functional change
Passed STC (https://tests.stockfishchess.org/tests/view/6a2893ad7c758d82accea129):
LLR: 3.20 (-2.94,2.94) <0.00,2.00>
Total: 23328 W: 6145 L: 5838 D: 11345
Ptnml(0-2): 50, 2463, 6346, 2740, 65
Instead of repeatedly doing the sum HalfKA + threats at the end, it's profitable to simply store one accumulator per side that combines them. This also avoids an extra/load store of an accumulator, and halves the cache footprint of the accumulators.
For full refreshes, we always compute both halfka and threats simultaneously. Any threat full refresh is always a halfka refresh because it occurs when the king crosses the center line, while halfka refreshes are required for ANY king move, so we don't need a separate detection path for threats.
I get about a 2.5% speedup locally with this, but I'd appreciate other ppl's measurements.
closes https://github.com/official-stockfish/Stockfish/pull/6890
No functional change
Adds a `RelaxedAtomic` wrapper around either `T` or `std::atomic<T>` and `USE_SLOPPY_ATOMICS` preprocessor define. The intent of this flag is to allow easy disabling of atomics on WASM, where even relaxed atomics are expensive because all atomics have `seq_cst` semantics.
Passed non regression STC
LLR: 2.99 (-2.94,2.94) <-1.75,0.25>
Total: 50624 W: 12976 L: 12776 D: 24872
Ptnml(0-2): 112, 5445, 14005, 5631, 119
https://tests.stockfishchess.org/tests/view/6a1f690e818cacc1db0ad2c7
Passed non-regression STC SMP
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 163696 W: 41514 L: 41438 D: 80744
Ptnml(0-2): 162, 18272, 44904, 18348, 162
https://tests.stockfishchess.org/tests/view/6a21fb97351b79f679cc44b3
Using this class for the TT also allows us to remove the TSAN suppressions, since the UB is fixed.
closes https://github.com/official-stockfish/Stockfish/pull/6877
No functional change
This PR introduces the additional `RootMove` attribute `previousPV` so that scores and PVs we send to the GUI in MultiPV analysis always match. This allows us in particular to extend our guarantee of exact mate (and TB win/loss) scores having a complete PV (leading to checkmate in the correct number of plies) to all PV lines. Recall that master fails here, since partially searched root moves may send to the GUI the previous score with the current/modified PV. See #6784.
The PR also uses the new attribute to extend the followPV logic to the analysis of sidelines, building on the idea in #6813 by @joergoster.
Passed non-reg STC:
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 166880 W: 42357 L: 42282 D: 82241
Ptnml(0-2): 394, 18685, 45177, 18820, 364
https://tests.stockfishchess.org/tests/view/6a0dea55818cacc1db0abb6a
Failed non-reg LTC:
LLR: -2.97 (-2.94,2.94) <-1.75,0.25>
Total: 890520 W: 224168 L: 225282 D: 441070
Ptnml(0-2): 390, 91902, 261789, 90790, 389
https://tests.stockfishchess.org/tests/view/6a1143ad818cacc1db0ac14c
Opening as draft for discussion on how to proceed. In SinglePV analysis, the patch is completely nonfunctional. But it is maybe a (small?) slowdown because of the increased size of `RootMove`. I am not sure if there as an elegant way to enrich the class only for MultiPV analysis (but the switch can happen at any time through the UCI interface), or to mitigate the speed penalty in some other way.
A local speedup test shows only a small slowdown on my system (but still high error bars):
```
sf_base = 1156928 +/- 1459 (95%)
sf_test = 1155885 +/- 1283 (95%)
diff = -1043 +/- 1777 (95%)
speedup = -0.09021% +/- 0.154% (95%)
```
The PR also adds the new MultiPV mate PV correctness check to the CI.
closes https://github.com/official-stockfish/Stockfish/pull/6886
No functional change
Lichess maintains some patches on top of SF dev to get it working with Emscripten. This PR moves some of these patches into SF and adds WASM to CI. It also adds a few changes in places where the x86 intrinsics don't cleanly map onto WebAssembly SIMD instructions; otherwise, we use Emscripten's x86 compatibility layer and take SSE4.1 code paths.
Summary of the compatibility changes:
- Define `wasm32` and `wasm32-relaxed-simd` targets.
- We don't support wasm without SIMD; it'd be a waste of time.
- Add option to disable TBs
- This is required because `tbprobe.cpp` pulls in `mmap`. This option can be used on any target, of course, but it's only enabled by default for wasm.
- Add compilation job + test to CI
And the changes for performance:
- Disable atomics for shared history on wasm
- Atomics are always `seq_cst` there, which can be quite slow (even on the x86, stores are locked `xchg [mem], reg`)
- Add SSE code path to `get_changed_pieces`, modeled after the AVX2 path
- `_mm_mulhi_epi16` has a complicated emulation sequence, so for the pairwise multiplication, use an approach similar to the NEON impl.
- __int128 is gets lowered to runtime functions on wasm, so use the fallback impl for `mul_hi64`
- V8 does a poor job with the NNZ finding, so use a slightly different sequence there
- Add relaxed simd support for `m128_dpbusd`.
Some local perf figures (single-threaded speedtest):
```
wasm
Nodes/second : 902523
sse4.1
Nodes/second : 1155380
avx512icl
Nodes/second : 1676184
```
Further avenues to explore:
- Optimize for performance under V8's experimental AVX revectorizer (Currently it's about +10% in my testing, but could definittely be more)
- Branch hinting. For example, run bench while collecting branch frequency info, then inject it late in the WASM compilation pipeline. I tried this locally and it didn't help much, but maybe I'm missing something.
- PGO. Gives +1.5% NPS locally, but hard to integrate with WASM compilation wrokflows
closes https://github.com/official-stockfish/Stockfish/pull/6875
No functional change
example of using, to avoid mixed usage of std::uint/std::int and uint/int...
```cpp
using u64 = std::uint64_t;
using u32 = std::uint32_t;
using u16 = std::uint16_t;
using u8 = std::uint8_t;
using i64 = std::int64_t;
using i32 = std::int32_t;
using i16 = std::int16_t;
using i8 = std::int8_t;
using usize = std::size_t;
using isize = std::ptrdiff_t;
#if defined(__GNUC__) && defined(IS_64BIT)
__extension__ using u128 = unsigned __int128;
__extension__ using i128 = signed __int128;
#endif
```
closes https://github.com/official-stockfish/Stockfish/pull/6874
No functional change
Hyperbola Quintessence works out without the "... * 2" in r and rr. This then is cheap enough to compute at runtime, instead of looking it up, for a minor speedup.
Passed apple-silicon|armv8|loongarch64 STC:
https://tests.stockfishchess.org/tests/view/6a15f031818cacc1db0aca49
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 410208 W: 104369 L: 103556 D: 202283
Ptnml(0-2): 758, 43951, 114883, 44744, 768
Verified attacks_bb<ROOK>() and attacks_bb<BISHOP>() assembly remains unchanged on x86 (gcc 16.1.1).
closes https://github.com/official-stockfish/Stockfish/pull/6870
No functional change
- Add new target `macos-lipo`.
- Created by compiling a universal x86 binary (no PGO) and a standard Apple silicon binary (with PGO), then combining them into a Mach-O fat binary
- To keep only one copy of the net, we add custom loading logic in the x86 section. The executable reads its own path and mmaps the net that's in the ARM section.
- The offset and size (from the executable base) of the mapping is injected after compilation in `patch_x86_slice.sh`
- avx512 on macOS isn't advertised in the xcr0 register by default. The simple solution I came up with is to execute a dummy AVX512 instruction, which sets up the register, before calling `__builtin_cpu_init`.
Some housekeeping as well:
- Rename `armv8-universal` -> `arm64-universal`.
- Add standard copyright headers to the files we've added recently.
Potential follow-ups:
- Disservin's Makefile cleanup
- Alternative ideas for the net loading. In particular, this will error out if the user strips the binary (since that'll invalidate the offset).
closes https://github.com/official-stockfish/Stockfish/pull/6860
No functional change
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
passed STC (https://tests.stockfishchess.org/tests/view/6a1157dc818cacc1db0ac172):
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 29792 W: 7759 L: 7465 D: 14568
Ptnml(0-2): 75, 3206, 8033, 3514, 68
Also passed STC after cleanups https://tests.stockfishchess.org/tests/view/6a121beb818cacc1db0ac35e
vondele's local test:
Result of 100 runs
base (./stockfish.master ) = 1136025 +/- 2816
test (./stockfish.patch2 ) = 1171370 +/- 2848
diff = +35346 +/- 3275
speedup = +0.0311
P(speedup > 0) = 1.0000
Basically we just do hyperbola quintessence in parallel. AVX2 doesn't have efficient bit reversal so we only do it for file and bishop attacks, then do rank attacks separately with a lookup table. This LUT is much smaller which is why this seems to be faster than standard magics.
closes https://github.com/official-stockfish/Stockfish/pull/6845
No functional change
Potentially fixes https://github.com/official-stockfish/Stockfish/issues/6846
I'm not a Linux guru but here's my understanding of what's going on, from looking at strace logs.
If `/dev/shm` is undersized or too full, `MADV_POPULATE_WRITE` can partially fault in the pages and then return an error. The crash then happens in the second process, on the first read to `header_ptr_->initialized`. There's a TOCTOU between when the region is attachable (the second process can see it in `/dev/shm`) and when we can actually safely read from it. The SIGBUS happens because `/dev/shm` is so full from partially faulted pages that we can't even fault in a single additional page.
This didn't happen before because a failed `posix_fallocate` *rolls back any faulted pages*. Then the second process faults in one page (which succeeds), reads a `0` from `header_ptr_->initialized`, declares the region invalid, and continues to try (and fail) to set up its own region.
There's still a problem here, methinks, if the process gets killed/interrupted between the failed `madvise` and the `ftruncate`. I think a more robust approach is to create the file anonymously first (to prevent other processes from attaching to it), then `linkat` to publish it atomically, but that would be more implementation work. (Could create a follow-up issue for that as an nice first issue for new contribs?)
closes https://github.com/official-stockfish/Stockfish/pull/6852
No functional change
These days Stockfish at TCEC often plays trivial endgames badly, failing to find a mate score as well. This patch helps Stockfish in those situations, by forcing it to play an optimal DTM move if we can deduce it from the DTZ tables. That is, when the only zeroing move is the checkmate itself.
Hence this PR checks if there are no pawns left to push and if there are no captures or sacrifices possible that would maintain a win. Example positions where this applies are KQvK, KRvK, KBBvK and KNBvK.
The change will also lead to optimal Syzygy PV extensions once the PV enters the "known DTM" territory.
The patch should help mitigate issue #6742.
closes https://github.com/official-stockfish/Stockfish/pull/6843
No functional change
Over the past few months our attacks and bitboard logic got a bit more involved and target specific, so I think it makes sense to split this up a bit more into separate files.
While it's not in an ideal state I think it's first step and can be improved later, `pawn_attacks_bb` could also be moved into the attacks.h but this was more of an oversight from my side.
closes https://github.com/official-stockfish/Stockfish/pull/6833
No functional change
1GB pages for the TT give a small performance bump (`./stockfish speedtest 256 32000`):
master:
Nodes/second : 230514479
patch
Nodes/second : 237997728
With a large enough TT there is still dTLB thrashing with 2MB pages. 1GB pages reduces that and makes page table walks a bit faster (they're shorter, and there's fewer PTEs, which fits better in cache). `madvise` won't give you huge pages, only large pages, so we use mmap here.
We guard the huge page attempt to x86 + at least 8 huge pages per NUMA node, to avoid memory oversubscription when for example running with a 1.1 GB hash. Additionally, we change the TT clearing slightly so that the pages are better distributed across NUMA nodes.
Data from Torom:
master:
Nodes/second : 43524292
patch:
Nodes/second : 44080034
Data on an 8-thread Emerald Rapids VM:
./stockfish speedtest 8 65536
Baseline speedtest (2MB huge pages):
Nodes/second : 7366232
Nodes/second : 7262794
Nodes/second : 7355431
Nodes/second : 7318777
Baseline speedtest (1GB huge pages):
Nodes/second : 7541027
Nodes/second : 7572720
Nodes/second : 7550268
Nodes/second : 7538925
closes https://github.com/official-stockfish/Stockfish/pull/6681
No functional change
After the introduction of UnifiedCorrectionHistory, the Pawn, Minor and NonPawn entries in CorrHistType are dead - only PieceTo and Continuation are referenced via CorrectionHistory<T>. Drop them, along with the now unreachable default template and the NonPawn specialization. Reduce the primary CorrHistTypedef template to a forward declaration so that any future stray use of an unsupported tag fails to compile.
closes https://github.com/official-stockfish/Stockfish/pull/6828
No functional change
Follow-up to #6785
After completing an iteration, the engine verifies if a "mate in x" condition is met to stop searching. Currently, it checks is_mate() and is_mated() independently using two separate bulky logic branches. However, VALUE_MATE equals 32000, so both a mate (e.g., 31998) and a mated score (e.g., -31998) satisfy VALUE_MATE - std::abs(score) == 2. By utilizing is_mate_or_mated() and applying std::abs(), we mathematically combine these checks into a single concise expression.
Can someone please help run a matetrack with a --mate 100 limit for this?
closes https://github.com/official-stockfish/Stockfish/pull/6829
No functional change
- We add a new path for supporting universal builds without #embed . `xxd` causes the compiler to run out of memory, so instead we embed it as a string literal created at compile time.
- This file is created in `net.sh`, and only if `--embed-dir` isn't supported + we are doing a universal build.
- This was necessary because Android NDK seems to not support embed, idk why
- A couple other Makefile tweaks were necessary for the Android path.
- While we're at it, remove the non-universal arm64 windows and android binaries.
closes https://github.com/official-stockfish/Stockfish/pull/6827
No functional change
Passed avx2/bmi2 STC:
https://tests.stockfishchess.org/tests/view/69ff99eb9392f0c317213eda
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 20992 W: 5469 L: 5192 D: 10331
Ptnml(0-2): 38, 2197, 5751, 2470, 40
Passed avxvnni STC:
https://tests.stockfishchess.org/tests/view/6a00022b9392f0c317213f7c
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 47328 W: 12328 L: 12009 D: 22991
Ptnml(0-2): 112, 5148, 12847, 5423, 134
Passed NEON STC:
https://tests.stockfishchess.org/tests/view/69ff99d69392f0c317213ed8
LLR: 2.96 (-2.94,2.94) <0.00,2.00>
Total: 29600 W: 7664 L: 7376 D: 14560
Ptnml(0-2): 48, 3074, 8277, 3344, 57
Passed avx512icl non-regression:
https://tests.stockfishchess.org/tests/view/6a002b699392f0c317213f91
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 90112 W: 23130 L: 22975 D: 44007
Ptnml(0-2): 192, 9939, 24633, 10106, 186
Measurements from vondele (Neoverse V2, armv8-dotprod):
==== master ====
==== Bench: 2344696 ====
1 Nodes/second : 280724660
2 Nodes/second : 280647282
3 Nodes/second : 282055192
Average (over 3): 281142378
==== 50a44640a3 ====
==== Bench: 2344696 ====
1 Nodes/second : 284271937
2 Nodes/second : 285638071
3 Nodes/second : 284349426
Average (over 3): 284753144
The patch's benefit is non-uniform and is ~0 for avx512icl unfortunately -- although I think we should be able to find something there....
## Background/explanation
The idea here is to move the non-zero block computation back to the previous layer, and overlap the work better. Then, to avoid trying to emulate compress instructions on targets not supporting them (i.e., everything except for AVX512), we use a `pop_lsb` loop on a bitset enumerating the non-zero blocks, rather than first writing them out as indices.
An early AVX2 implementation worked on fishtest (https://tests.stockfishchess.org/tests/view/69c3410534b6988b1e472db4) and linrock demonstrated that it also worked for NEON (https://tests.stockfishchess.org/tests/view/69ce0dc73ddc0eccd617188f). Thanks to him for encouraging me to push this idea over the finish line.
To abstract over the particular NNZ representation (bitset or index list), we use `NNZInfo` and `NNZCursor`. The cursor is used to write to one or the other perspective of the NNZ list. I'd appreciate ideas on making the code cleaner/more readable as imo it's still a bit ugly.
## Fixing GCC 15 regression
Separately, vondele noticed a regression in ARM performance from GCC 15 caused by suboptimal codegen. See https://discord.com/channels/435943710472011776/813919248455827515/1502837886381461605 for more info, but the easiest fix that I could come up with was inserting a couple `asm` optimization barriers.
Single threaded data:
```
average: stockfish.master.13.3 1041619
average: stockfish.master.15.2 999214
average: stockfish.patched.15.2 1031369
```
There's definitely regressions elsewhere as well, which we shld chase down, but at least this should unblock the arm64 universal binary work.
closes https://github.com/official-stockfish/Stockfish/pull/6814
No functional change
This PR fixes an oversight in #6818. At least in multithreaded searches the TT scores can be corrupted, and accessing them is racy.
Passed non-reg STC:
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 61792 W: 15609 L: 15428 D: 30755
Ptnml(0-2): 134, 6534, 17373, 6727, 128
https://tests.stockfishchess.org/tests/view/6a0b15396524d21ee79b868b
We also increase the number of the threads in our instrumented CI tests to make these assert failures for buggy patches more likely in future.
closes https://github.com/official-stockfish/Stockfish/pull/6830
No functional change
Introduce the armv8-universal target. 🥴
The entry point is essentially the same as x86's, except we no longer have access to `__builtin_cpu_supports` so instead we need an OS-specific query for whether we support dotprod. The Makefile is modified to support both universal builds.
If in the future we add more ARM targets, such as SVE, we'll need to add qemu to the RUN_PREFIX in CI, because currently we assume (for PGO purposes) that the CI host supports all the used ARM instructions.
### clang/Windows
The painful part here is clang on Windows, which, until arm64 mingw is stabilized, is required for targeting arm64. This PR also gets it to work for x86. In the Makefile this setup corresponds to `use_lto_emit_asm=yes`.
In particular, `--defsym` and `--save-temps` are not supported by `lld-link`, and objcopy `--rename-section` doesn't work on COFF binaries because of how section names work there.
- `--defsym` is needed to define `main` for PGO purposes and assigns it to the namespaced, per-arch main function. Instead, we define `main` in `main.cpp` so that the compilation is successful, then delete it before the final link.
- Instead of `--save-temps` to get the LTO intermediate object, we pass `--lto-emit-asm` to the linker, which outputs `stockfish.exe.lto.s`.
- Finally, we have a small AWK script to find the `.ctors` section, neuter it, and put start/stop symbols around it with the same naming scheme as ELF (`__start_*_init`/`__stop_*_init`).
I'm lowk a Windows programming noob so if there's simpler ways of going about this, I'd appreciate a pointer. @PikaCat-OuO + Codex used an approach that involved going in and modifying the LLVM bitcode, but that felt more complicated to me.
closes https://github.com/official-stockfish/Stockfish/pull/6823
No functional change
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
