This network is trained by adding the following binpacks, relabeled by @vondele and totaling 74B positions, to the distillation fine-tuning stage:
```
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_0.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_1.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_2.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_3.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_4.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_0.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_1.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_2.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_3.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_4.relabel-BT4-tf13tune.binpack
```
The relabeling effort has been completed during the testing of this patch, and a full training run is on the way. Thanks to @vondele, @anematode, @Disservin, @yl25946, and all who've contributed to the process.
Passed STC:
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 24512 W: 6500 L: 6201 D: 11811
Ptnml(0-2): 69, 2772, 6300, 3021, 94
https://tests.stockfishchess.org/tests/view/6a40b7083036e45021aebbd6
Passed LTC:
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 15708 W: 4232 L: 3960 D: 7516
Ptnml(0-2): 8, 1572, 4420, 1848, 6
https://tests.stockfishchess.org/tests/view/6a413d293036e45021aebc84
nettest PR: https://github.com/vondele/nettest/pull/388
closes https://github.com/official-stockfish/Stockfish/pull/6932
Bench: 2102535
Co-authored-by: Joost VandeVondele <Joost.VandeVondele@gmail.com>
Deduplicating Color-Specific Piece Validation.
The validation checks for the number of pawns and additional promoted pieces are duplicated for WHITE and BLACK. We can combine this logic into a single range-based for loop over both colors.
closes https://github.com/official-stockfish/Stockfish/pull/6922
No functional change
do_null_move copies the whole StateInfo from the previous state, which leaves capturedPiece holding the piece captured by the last real move, so inside the null-move subtree captured_piece() reports a stale capture. The priorCapture consumers in search are all guarded by prevSq != SQ_NONE or (ss-1)->currentMove.is_ok(), which are false at the null-move child, but the stalemate verification gate in qsearch reads captured_piece() unguarded and can be spuriously triggered by the stale value.
Clear the field, since a null move captures nothing.
Passed non-regression STC:
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 82784 W: 21172 L: 21011 D: 40601
Ptnml(0-2): 194, 8976, 22923, 9073, 226
https://tests.stockfishchess.org/tests/view/6a2b5b356b4aa63ddbf31518
Passed non-regression LTC:
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 118134 W: 30007 L: 29891 D: 58236
Ptnml(0-2): 66, 11856, 35108, 11970, 67
https://tests.stockfishchess.org/tests/view/6a2c4c9c0d5d4b19d0805301
closes https://github.com/official-stockfish/Stockfish/pull/6910
No functional change
The following have zero call sites repo-wide:
SearchManager:🆔 never read or written (also never initialized, nor was it ever used).
Search::Worker::elapsed_time(): never called. PV output uses tm.elapsed_time() (TimeManager) directly. (removed callers on 25361e5)
MovePicker::begin()/end(): unused private accessors. (removed callers on 8c2d21f)
closes https://github.com/official-stockfish/Stockfish/pull/6909
No functional change
Worker::do_move computes the successor hash key via the new
Position::key_after(m) and prefetches the TT entry one full do_move
earlier than the existing prefetch in Position::do_move. key_after does
not model castling, en passant or promotion keys exactly; for rare
moves the prefetch lands on an unused line.
`key_after` has been around since 2014 (https://github.com/official-stockfish/Stockfish/commit/82d065b0) and was removed in (https://github.com/official-stockfish/Stockfish/pull/5770). Adding back `prefetch_key` helps in common, normal moves at the cost of extra compute.
Speedup (PGO vs PGO, interleaved paired bench, n=48 pairs, Apple M2
Pro / apple-silicon): +0.69% [0.47, 0.91]
Passed STC:
https://tests.stockfishchess.org/tests/view/6a291f8d7c758d82accea17f
LLR: 4.24 (-2.94,2.94) <0.00,2.00>
Total: 473504 W: 121250 L: 120228 D: 232026
Ptnml(0-2): 1112, 51137, 131251, 52121, 1131
No functional change
closes https://github.com/official-stockfish/Stockfish/pull/6911
No functional change
This PR fixes the remaining corner cases in the treatment of MultiPV
mated-in PVs, as well as an oversight in #6886. See the discussion in
In particular:
1. `previousScore` and `previousPV` can only be trusted, if that
rootmove was indeed fully searched in the previous iteration.
2. A move beyond `pvIdx` (that was hence not fully searched) may have an
exact loss score that cannot be trusted. So if a MultiPV search gets
aborted while searching `pvIdx`, we mark all the following loss
scores as bounds.
3. The forgotten mate logic also got broken in #6886, because the
`previousPV` of the forgotten mate's bestmove can only be trusted if
that move was fully searched in the previous iteration, something
that is not guaranteed. So we now store both `lastBestMoveScore` and
`lastBestMovePV`.
Here some scenarios for MultiPV = 8 that explain how master was broken:
1. Move A with an inexact mated-in-2 score from the previous iteration
(so outside the top8 moves) gets flushed into the top8 moves for the
current iteration, because the previous top8 move B is now scored as
a mated-in-1. Hence we cannot trust `previousScore` or `previousPV`
for move A, if the search gets aborted while it is being searched.
2. In the scenario above, move B has `Score != -VALUE_INFINITE` and a
mated-in-1 score, which cannot be trusted as it was not fully
searched.
3. Iteration N has bestmove A with mated-in-10, which gets recorded in
`lastBestMoveScore` (renamed from `lastIterationScore`). Iteration 11
forgets the mate and has bestmove B with a cp score, move A may have
an incomplete PV, and may even have a non-mate score. Iteration 12
gets aborted, and in trying to remember the forgotten mate, master
recovers the `previousScore` and `previousPV` of move A, which may be
neither mate nor complete.
Passed STC non-reg:
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 69728 W: 17748 L: 17573 D: 34407
Ptnml(0-2): 143, 7571, 19274, 7720, 156
https://tests.stockfishchess.org/tests/view/6a2c40c60d5d4b19d08052f2
closes https://github.com/official-stockfish/Stockfish/pull/6906
No functional change
After merging the HalfKA and Threats accumulators (7c7fe322) and the
subsequent removal of the double-incremental/fused update, a number of
NNUE helpers and fields became unreachable. Each was verified to have
zero callers/readers across the source tree:
- FusedUpdateData logic in FullThreats: the fused-update branch of
append_changed_indices and the FusedUpdateData parameter are unused;
the accumulator update no longer passes fused data.
- FullThreats::requires_refresh: never called. The live king-bucket
refresh check is HalfKAv2_hm::requires_refresh (PSQFeatureSet), used
in nnue_accumulator.
- HalfKAv2_hm::append_active_indices: never called. The live
active-index builder is FullThreats::append_active_indices
(ThreatFeatureSet).
- DirtyThreats::us, prevKsq and ksq: written in do_move but only read by
the now-removed FullThreats::requires_refresh. Removing them also
drops three stores from the do_move path.
- Unused feature Name constants and the unused FtOneVal / HiddenMaxVal
constants in nnue_common.h.
- Two stale feature-header banner comments.
closes https://github.com/official-stockfish/Stockfish/pull/6898
No functional change
This version handles aborting engine processes more gracefully. This
also test the engine prior to use, as the process is nevertheless not
fully robust.
closes https://github.com/official-stockfish/Stockfish/pull/6893
No functional change
The FEN validation check intended to reject pawns on the first or eighth rank uses the `Rank` enum values in a bitwise OR operation:
`if (pieces(PAWN) & (RANK_1 | RANK_8))`
`RANK_1 | RANK_8` evaluates to the integer `0 | 7 == 7` instead of a bitboard, so the expression only tests squares A1, B1 and C1. As a result, unsupported positions with pawns elsewhere on the first or eighth rank are silently accepted. For instance, `position fen 3P3k/8/8/8/8/8/8/3K4 w - - 0 1` is accepted even though the pawn on d8 makes the position unsupported.
Use the `Rank1BB | Rank8BB` bitboard constants so any pawn on the first or eighth rank are correctly rejected.
closes https://github.com/official-stockfish/Stockfish/pull/6887
No functional change
In a recent CCC event, Stockfish (probably through no fault of its own), lost some games on time when it was winning and when it had already found the move that delivers checkmate.
This patch stops the search when TM is active, and when mainthread can be certain that it is impossible to find a better move. That is if (i) it has found mate-in-1, (ii) it has found a mate-in-2 or (iii) it has found a mated-in-1.
patch:
```
position fen 5K2/8/2qk4/2nPp3/3r4/6B1/B7/3R4 w - e6
go wtime 100000000 winc 100000000
info string Available processors: 0-7
info string Using 1 thread
info string NNUE evaluation using nn-71d6d32cb962.nnue (106MiB, (83248, 1024, 31, 32, 1))
info string Network replica 1: Shared memory.
info depth 1 seldepth 3 multipv 1 score mate 1 nodes 30 nps 30000 hashfull 0 tbhits 0 time 1 pv d5e6
bestmove d5e6
```
master:
```
position fen 5K2/8/2qk4/2nPp3/3r4/6B1/B7/3R4 w - e6
go wtime 100000000 winc 100000000
info string Available processors: 0-7
info string Using 1 thread
info string NNUE evaluation using nn-71d6d32cb962.nnue (106MiB, (83248, 1024, 31, 32, 1))
info string Network replica 1: Shared memory.
info depth 1 seldepth 3 multipv 1 score mate 1 nodes 30 nps 15000 hashfull 0 tbhits 0 time 2 pv d5e6
<snip>
info depth 245 seldepth 2 multipv 1 score mate 1 nodes 5886 nps 367875 hashfull 0 tbhits 0 time 16 pv d5e6
bestmove d5e6
```
Note: In MultiPV analysis (extremely rare with TM active), we take the point of view that the user would like to continue to search until none of the PVs can be improved anymore. This means we only stop if the worst searched line is at least a mate-in-2, or if the best searched line is a mated-in-1.
closes https://github.com/official-stockfish/Stockfish/pull/6879
No functional change
Passed STC
https://tests.stockfishchess.org/tests/view/6a2682ce351b79f679cc47c5
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 57248 W: 14730 L: 14399 D: 28119
Ptnml(0-2): 145, 6159, 15698, 6464, 158
Reordering operations in `do_move` allows us to effectively prefetch the TT entry earlier, since the piece moving helpers don't actually modify the position key. I suspect that with threat inputs, `put_piece` and friends got a lot more expensive, and so this helps us a lot.
vondele's machine:
==== master ====
1 Nodes/second : 294311526
2 Nodes/second : 297068312
3 Nodes/second : 297418763
Average (over 3): 296266200
==== pfearly ====
1 Nodes/second : 303986449
2 Nodes/second : 304221719
3 Nodes/second : 305302969
Average (over 3): 304503712 (+2.78%)
Locally, `bench`:
Result of 200 runs
speedup = +0.0158
P(speedup > 0) = 1.0000
As expected it helps even more in a large-hash, NUMA setting.
closes https://github.com/official-stockfish/Stockfish/pull/6891
No functional change
Passed STC (https://tests.stockfishchess.org/tests/view/6a2893ad7c758d82accea129):
LLR: 3.20 (-2.94,2.94) <0.00,2.00>
Total: 23328 W: 6145 L: 5838 D: 11345
Ptnml(0-2): 50, 2463, 6346, 2740, 65
Instead of repeatedly doing the sum HalfKA + threats at the end, it's profitable to simply store one accumulator per side that combines them. This also avoids an extra/load store of an accumulator, and halves the cache footprint of the accumulators.
For full refreshes, we always compute both halfka and threats simultaneously. Any threat full refresh is always a halfka refresh because it occurs when the king crosses the center line, while halfka refreshes are required for ANY king move, so we don't need a separate detection path for threats.
I get about a 2.5% speedup locally with this, but I'd appreciate other ppl's measurements.
closes https://github.com/official-stockfish/Stockfish/pull/6890
No functional change
Adds a `RelaxedAtomic` wrapper around either `T` or `std::atomic<T>` and `USE_SLOPPY_ATOMICS` preprocessor define. The intent of this flag is to allow easy disabling of atomics on WASM, where even relaxed atomics are expensive because all atomics have `seq_cst` semantics.
Passed non regression STC
LLR: 2.99 (-2.94,2.94) <-1.75,0.25>
Total: 50624 W: 12976 L: 12776 D: 24872
Ptnml(0-2): 112, 5445, 14005, 5631, 119
https://tests.stockfishchess.org/tests/view/6a1f690e818cacc1db0ad2c7
Passed non-regression STC SMP
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 163696 W: 41514 L: 41438 D: 80744
Ptnml(0-2): 162, 18272, 44904, 18348, 162
https://tests.stockfishchess.org/tests/view/6a21fb97351b79f679cc44b3
Using this class for the TT also allows us to remove the TSAN suppressions, since the UB is fixed.
closes https://github.com/official-stockfish/Stockfish/pull/6877
No functional change
Fixes#6881.
`timeout_decorator()` used a `ThreadPoolExecutor` context manager around blocking output waits. When `future.result(timeout=...)` timed out, leaving the context manager still waited for the worker thread to finish, so a blocked stdout read could keep the instrumented tests hanging past the configured timeout.
This change removes that executor wrapper for interactive Stockfish output waits. The harness now drains process output on a daemon reader thread, queues received lines, and applies the deadline directly while waiting for the next queued line. `TimeoutException` also initializes the base exception message so failures show useful text.
Validation:
- `python3 -m py_compile tests/testing.py tests/instrumented.py`
- local timeout smoke test: a 0.2s no-output wait raises in ~0.204s
- Stockfish smoke test: startup/`uciok` read succeeds, deliberate no-output wait raises in ~0.205s, engine exits 0
- `make -C src -j4 build`
- `../tests/signature.sh` -> `2814421`
closes https://github.com/official-stockfish/Stockfish/pull/6882
No functional change
This PR introduces the additional `RootMove` attribute `previousPV` so that scores and PVs we send to the GUI in MultiPV analysis always match. This allows us in particular to extend our guarantee of exact mate (and TB win/loss) scores having a complete PV (leading to checkmate in the correct number of plies) to all PV lines. Recall that master fails here, since partially searched root moves may send to the GUI the previous score with the current/modified PV. See #6784.
The PR also uses the new attribute to extend the followPV logic to the analysis of sidelines, building on the idea in #6813 by @joergoster.
Passed non-reg STC:
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 166880 W: 42357 L: 42282 D: 82241
Ptnml(0-2): 394, 18685, 45177, 18820, 364
https://tests.stockfishchess.org/tests/view/6a0dea55818cacc1db0abb6a
Failed non-reg LTC:
LLR: -2.97 (-2.94,2.94) <-1.75,0.25>
Total: 890520 W: 224168 L: 225282 D: 441070
Ptnml(0-2): 390, 91902, 261789, 90790, 389
https://tests.stockfishchess.org/tests/view/6a1143ad818cacc1db0ac14c
Opening as draft for discussion on how to proceed. In SinglePV analysis, the patch is completely nonfunctional. But it is maybe a (small?) slowdown because of the increased size of `RootMove`. I am not sure if there as an elegant way to enrich the class only for MultiPV analysis (but the switch can happen at any time through the UCI interface), or to mitigate the speed penalty in some other way.
A local speedup test shows only a small slowdown on my system (but still high error bars):
```
sf_base = 1156928 +/- 1459 (95%)
sf_test = 1155885 +/- 1283 (95%)
diff = -1043 +/- 1777 (95%)
speedup = -0.09021% +/- 0.154% (95%)
```
The PR also adds the new MultiPV mate PV correctness check to the CI.
closes https://github.com/official-stockfish/Stockfish/pull/6886
No functional change
Lichess maintains some patches on top of SF dev to get it working with Emscripten. This PR moves some of these patches into SF and adds WASM to CI. It also adds a few changes in places where the x86 intrinsics don't cleanly map onto WebAssembly SIMD instructions; otherwise, we use Emscripten's x86 compatibility layer and take SSE4.1 code paths.
Summary of the compatibility changes:
- Define `wasm32` and `wasm32-relaxed-simd` targets.
- We don't support wasm without SIMD; it'd be a waste of time.
- Add option to disable TBs
- This is required because `tbprobe.cpp` pulls in `mmap`. This option can be used on any target, of course, but it's only enabled by default for wasm.
- Add compilation job + test to CI
And the changes for performance:
- Disable atomics for shared history on wasm
- Atomics are always `seq_cst` there, which can be quite slow (even on the x86, stores are locked `xchg [mem], reg`)
- Add SSE code path to `get_changed_pieces`, modeled after the AVX2 path
- `_mm_mulhi_epi16` has a complicated emulation sequence, so for the pairwise multiplication, use an approach similar to the NEON impl.
- __int128 is gets lowered to runtime functions on wasm, so use the fallback impl for `mul_hi64`
- V8 does a poor job with the NNZ finding, so use a slightly different sequence there
- Add relaxed simd support for `m128_dpbusd`.
Some local perf figures (single-threaded speedtest):
```
wasm
Nodes/second : 902523
sse4.1
Nodes/second : 1155380
avx512icl
Nodes/second : 1676184
```
Further avenues to explore:
- Optimize for performance under V8's experimental AVX revectorizer (Currently it's about +10% in my testing, but could definittely be more)
- Branch hinting. For example, run bench while collecting branch frequency info, then inject it late in the WASM compilation pipeline. I tried this locally and it didn't help much, but maybe I'm missing something.
- PGO. Gives +1.5% NPS locally, but hard to integrate with WASM compilation wrokflows
closes https://github.com/official-stockfish/Stockfish/pull/6875
No functional change
example of using, to avoid mixed usage of std::uint/std::int and uint/int...
```cpp
using u64 = std::uint64_t;
using u32 = std::uint32_t;
using u16 = std::uint16_t;
using u8 = std::uint8_t;
using i64 = std::int64_t;
using i32 = std::int32_t;
using i16 = std::int16_t;
using i8 = std::int8_t;
using usize = std::size_t;
using isize = std::ptrdiff_t;
#if defined(__GNUC__) && defined(IS_64BIT)
__extension__ using u128 = unsigned __int128;
__extension__ using i128 = signed __int128;
#endif
```
closes https://github.com/official-stockfish/Stockfish/pull/6874
No functional change
Hyperbola Quintessence works out without the "... * 2" in r and rr. This then is cheap enough to compute at runtime, instead of looking it up, for a minor speedup.
Passed apple-silicon|armv8|loongarch64 STC:
https://tests.stockfishchess.org/tests/view/6a15f031818cacc1db0aca49
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 410208 W: 104369 L: 103556 D: 202283
Ptnml(0-2): 758, 43951, 114883, 44744, 768
Verified attacks_bb<ROOK>() and attacks_bb<BISHOP>() assembly remains unchanged on x86 (gcc 16.1.1).
closes https://github.com/official-stockfish/Stockfish/pull/6870
No functional change
- Add new target `macos-lipo`.
- Created by compiling a universal x86 binary (no PGO) and a standard Apple silicon binary (with PGO), then combining them into a Mach-O fat binary
- To keep only one copy of the net, we add custom loading logic in the x86 section. The executable reads its own path and mmaps the net that's in the ARM section.
- The offset and size (from the executable base) of the mapping is injected after compilation in `patch_x86_slice.sh`
- avx512 on macOS isn't advertised in the xcr0 register by default. The simple solution I came up with is to execute a dummy AVX512 instruction, which sets up the register, before calling `__builtin_cpu_init`.
Some housekeeping as well:
- Rename `armv8-universal` -> `arm64-universal`.
- Add standard copyright headers to the files we've added recently.
Potential follow-ups:
- Disservin's Makefile cleanup
- Alternative ideas for the net loading. In particular, this will error out if the user strips the binary (since that'll invalidate the offset).
closes https://github.com/official-stockfish/Stockfish/pull/6860
No functional change
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
passed STC (https://tests.stockfishchess.org/tests/view/6a1157dc818cacc1db0ac172):
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 29792 W: 7759 L: 7465 D: 14568
Ptnml(0-2): 75, 3206, 8033, 3514, 68
Also passed STC after cleanups https://tests.stockfishchess.org/tests/view/6a121beb818cacc1db0ac35e
vondele's local test:
Result of 100 runs
base (./stockfish.master ) = 1136025 +/- 2816
test (./stockfish.patch2 ) = 1171370 +/- 2848
diff = +35346 +/- 3275
speedup = +0.0311
P(speedup > 0) = 1.0000
Basically we just do hyperbola quintessence in parallel. AVX2 doesn't have efficient bit reversal so we only do it for file and bishop attacks, then do rank attacks separately with a lookup table. This LUT is much smaller which is why this seems to be faster than standard magics.
closes https://github.com/official-stockfish/Stockfish/pull/6845
No functional change
* .tar -> .tar.gz: save a few more MB in distribution.
* drop android armv7 without NEON support: there should be almost no devices around that do not support this.
closes https://github.com/official-stockfish/Stockfish/pull/6850
No functional change
Potentially fixes https://github.com/official-stockfish/Stockfish/issues/6846
I'm not a Linux guru but here's my understanding of what's going on, from looking at strace logs.
If `/dev/shm` is undersized or too full, `MADV_POPULATE_WRITE` can partially fault in the pages and then return an error. The crash then happens in the second process, on the first read to `header_ptr_->initialized`. There's a TOCTOU between when the region is attachable (the second process can see it in `/dev/shm`) and when we can actually safely read from it. The SIGBUS happens because `/dev/shm` is so full from partially faulted pages that we can't even fault in a single additional page.
This didn't happen before because a failed `posix_fallocate` *rolls back any faulted pages*. Then the second process faults in one page (which succeeds), reads a `0` from `header_ptr_->initialized`, declares the region invalid, and continues to try (and fail) to set up its own region.
There's still a problem here, methinks, if the process gets killed/interrupted between the failed `madvise` and the `ftruncate`. I think a more robust approach is to create the file anonymously first (to prevent other processes from attaching to it), then `linkat` to publish it atomically, but that would be more implementation work. (Could create a follow-up issue for that as an nice first issue for new contribs?)
closes https://github.com/official-stockfish/Stockfish/pull/6852
No functional change
