Fixes#6881.
`timeout_decorator()` used a `ThreadPoolExecutor` context manager around blocking output waits. When `future.result(timeout=...)` timed out, leaving the context manager still waited for the worker thread to finish, so a blocked stdout read could keep the instrumented tests hanging past the configured timeout.
This change removes that executor wrapper for interactive Stockfish output waits. The harness now drains process output on a daemon reader thread, queues received lines, and applies the deadline directly while waiting for the next queued line. `TimeoutException` also initializes the base exception message so failures show useful text.
Validation:
- `python3 -m py_compile tests/testing.py tests/instrumented.py`
- local timeout smoke test: a 0.2s no-output wait raises in ~0.204s
- Stockfish smoke test: startup/`uciok` read succeeds, deliberate no-output wait raises in ~0.205s, engine exits 0
- `make -C src -j4 build`
- `../tests/signature.sh` -> `2814421`
closes https://github.com/official-stockfish/Stockfish/pull/6882
No functional change
This PR introduces the additional `RootMove` attribute `previousPV` so that scores and PVs we send to the GUI in MultiPV analysis always match. This allows us in particular to extend our guarantee of exact mate (and TB win/loss) scores having a complete PV (leading to checkmate in the correct number of plies) to all PV lines. Recall that master fails here, since partially searched root moves may send to the GUI the previous score with the current/modified PV. See #6784.
The PR also uses the new attribute to extend the followPV logic to the analysis of sidelines, building on the idea in #6813 by @joergoster.
Passed non-reg STC:
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 166880 W: 42357 L: 42282 D: 82241
Ptnml(0-2): 394, 18685, 45177, 18820, 364
https://tests.stockfishchess.org/tests/view/6a0dea55818cacc1db0abb6a
Failed non-reg LTC:
LLR: -2.97 (-2.94,2.94) <-1.75,0.25>
Total: 890520 W: 224168 L: 225282 D: 441070
Ptnml(0-2): 390, 91902, 261789, 90790, 389
https://tests.stockfishchess.org/tests/view/6a1143ad818cacc1db0ac14c
Opening as draft for discussion on how to proceed. In SinglePV analysis, the patch is completely nonfunctional. But it is maybe a (small?) slowdown because of the increased size of `RootMove`. I am not sure if there as an elegant way to enrich the class only for MultiPV analysis (but the switch can happen at any time through the UCI interface), or to mitigate the speed penalty in some other way.
A local speedup test shows only a small slowdown on my system (but still high error bars):
```
sf_base = 1156928 +/- 1459 (95%)
sf_test = 1155885 +/- 1283 (95%)
diff = -1043 +/- 1777 (95%)
speedup = -0.09021% +/- 0.154% (95%)
```
The PR also adds the new MultiPV mate PV correctness check to the CI.
closes https://github.com/official-stockfish/Stockfish/pull/6886
No functional change
Lichess maintains some patches on top of SF dev to get it working with Emscripten. This PR moves some of these patches into SF and adds WASM to CI. It also adds a few changes in places where the x86 intrinsics don't cleanly map onto WebAssembly SIMD instructions; otherwise, we use Emscripten's x86 compatibility layer and take SSE4.1 code paths.
Summary of the compatibility changes:
- Define `wasm32` and `wasm32-relaxed-simd` targets.
- We don't support wasm without SIMD; it'd be a waste of time.
- Add option to disable TBs
- This is required because `tbprobe.cpp` pulls in `mmap`. This option can be used on any target, of course, but it's only enabled by default for wasm.
- Add compilation job + test to CI
And the changes for performance:
- Disable atomics for shared history on wasm
- Atomics are always `seq_cst` there, which can be quite slow (even on the x86, stores are locked `xchg [mem], reg`)
- Add SSE code path to `get_changed_pieces`, modeled after the AVX2 path
- `_mm_mulhi_epi16` has a complicated emulation sequence, so for the pairwise multiplication, use an approach similar to the NEON impl.
- __int128 is gets lowered to runtime functions on wasm, so use the fallback impl for `mul_hi64`
- V8 does a poor job with the NNZ finding, so use a slightly different sequence there
- Add relaxed simd support for `m128_dpbusd`.
Some local perf figures (single-threaded speedtest):
```
wasm
Nodes/second : 902523
sse4.1
Nodes/second : 1155380
avx512icl
Nodes/second : 1676184
```
Further avenues to explore:
- Optimize for performance under V8's experimental AVX revectorizer (Currently it's about +10% in my testing, but could definittely be more)
- Branch hinting. For example, run bench while collecting branch frequency info, then inject it late in the WASM compilation pipeline. I tried this locally and it didn't help much, but maybe I'm missing something.
- PGO. Gives +1.5% NPS locally, but hard to integrate with WASM compilation wrokflows
closes https://github.com/official-stockfish/Stockfish/pull/6875
No functional change
example of using, to avoid mixed usage of std::uint/std::int and uint/int...
```cpp
using u64 = std::uint64_t;
using u32 = std::uint32_t;
using u16 = std::uint16_t;
using u8 = std::uint8_t;
using i64 = std::int64_t;
using i32 = std::int32_t;
using i16 = std::int16_t;
using i8 = std::int8_t;
using usize = std::size_t;
using isize = std::ptrdiff_t;
#if defined(__GNUC__) && defined(IS_64BIT)
__extension__ using u128 = unsigned __int128;
__extension__ using i128 = signed __int128;
#endif
```
closes https://github.com/official-stockfish/Stockfish/pull/6874
No functional change
Hyperbola Quintessence works out without the "... * 2" in r and rr. This then is cheap enough to compute at runtime, instead of looking it up, for a minor speedup.
Passed apple-silicon|armv8|loongarch64 STC:
https://tests.stockfishchess.org/tests/view/6a15f031818cacc1db0aca49
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 410208 W: 104369 L: 103556 D: 202283
Ptnml(0-2): 758, 43951, 114883, 44744, 768
Verified attacks_bb<ROOK>() and attacks_bb<BISHOP>() assembly remains unchanged on x86 (gcc 16.1.1).
closes https://github.com/official-stockfish/Stockfish/pull/6870
No functional change
- Add new target `macos-lipo`.
- Created by compiling a universal x86 binary (no PGO) and a standard Apple silicon binary (with PGO), then combining them into a Mach-O fat binary
- To keep only one copy of the net, we add custom loading logic in the x86 section. The executable reads its own path and mmaps the net that's in the ARM section.
- The offset and size (from the executable base) of the mapping is injected after compilation in `patch_x86_slice.sh`
- avx512 on macOS isn't advertised in the xcr0 register by default. The simple solution I came up with is to execute a dummy AVX512 instruction, which sets up the register, before calling `__builtin_cpu_init`.
Some housekeeping as well:
- Rename `armv8-universal` -> `arm64-universal`.
- Add standard copyright headers to the files we've added recently.
Potential follow-ups:
- Disservin's Makefile cleanup
- Alternative ideas for the net loading. In particular, this will error out if the user strips the binary (since that'll invalidate the offset).
closes https://github.com/official-stockfish/Stockfish/pull/6860
No functional change
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
passed STC (https://tests.stockfishchess.org/tests/view/6a1157dc818cacc1db0ac172):
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 29792 W: 7759 L: 7465 D: 14568
Ptnml(0-2): 75, 3206, 8033, 3514, 68
Also passed STC after cleanups https://tests.stockfishchess.org/tests/view/6a121beb818cacc1db0ac35e
vondele's local test:
Result of 100 runs
base (./stockfish.master ) = 1136025 +/- 2816
test (./stockfish.patch2 ) = 1171370 +/- 2848
diff = +35346 +/- 3275
speedup = +0.0311
P(speedup > 0) = 1.0000
Basically we just do hyperbola quintessence in parallel. AVX2 doesn't have efficient bit reversal so we only do it for file and bishop attacks, then do rank attacks separately with a lookup table. This LUT is much smaller which is why this seems to be faster than standard magics.
closes https://github.com/official-stockfish/Stockfish/pull/6845
No functional change
* .tar -> .tar.gz: save a few more MB in distribution.
* drop android armv7 without NEON support: there should be almost no devices around that do not support this.
closes https://github.com/official-stockfish/Stockfish/pull/6850
No functional change
Potentially fixes https://github.com/official-stockfish/Stockfish/issues/6846
I'm not a Linux guru but here's my understanding of what's going on, from looking at strace logs.
If `/dev/shm` is undersized or too full, `MADV_POPULATE_WRITE` can partially fault in the pages and then return an error. The crash then happens in the second process, on the first read to `header_ptr_->initialized`. There's a TOCTOU between when the region is attachable (the second process can see it in `/dev/shm`) and when we can actually safely read from it. The SIGBUS happens because `/dev/shm` is so full from partially faulted pages that we can't even fault in a single additional page.
This didn't happen before because a failed `posix_fallocate` *rolls back any faulted pages*. Then the second process faults in one page (which succeeds), reads a `0` from `header_ptr_->initialized`, declares the region invalid, and continues to try (and fail) to set up its own region.
There's still a problem here, methinks, if the process gets killed/interrupted between the failed `madvise` and the `ftruncate`. I think a more robust approach is to create the file anonymously first (to prevent other processes from attaching to it), then `linkat` to publish it atomically, but that would be more implementation work. (Could create a follow-up issue for that as an nice first issue for new contribs?)
closes https://github.com/official-stockfish/Stockfish/pull/6852
No functional change
These days Stockfish at TCEC often plays trivial endgames badly, failing to find a mate score as well. This patch helps Stockfish in those situations, by forcing it to play an optimal DTM move if we can deduce it from the DTZ tables. That is, when the only zeroing move is the checkmate itself.
Hence this PR checks if there are no pawns left to push and if there are no captures or sacrifices possible that would maintain a win. Example positions where this applies are KQvK, KRvK, KBBvK and KNBvK.
The change will also lead to optimal Syzygy PV extensions once the PV enters the "known DTM" territory.
The patch should help mitigate issue #6742.
closes https://github.com/official-stockfish/Stockfish/pull/6843
No functional change
Over the past few months our attacks and bitboard logic got a bit more involved and target specific, so I think it makes sense to split this up a bit more into separate files.
While it's not in an ideal state I think it's first step and can be improved later, `pawn_attacks_bb` could also be moved into the attacks.h but this was more of an oversight from my side.
closes https://github.com/official-stockfish/Stockfish/pull/6833
No functional change
1GB pages for the TT give a small performance bump (`./stockfish speedtest 256 32000`):
master:
Nodes/second : 230514479
patch
Nodes/second : 237997728
With a large enough TT there is still dTLB thrashing with 2MB pages. 1GB pages reduces that and makes page table walks a bit faster (they're shorter, and there's fewer PTEs, which fits better in cache). `madvise` won't give you huge pages, only large pages, so we use mmap here.
We guard the huge page attempt to x86 + at least 8 huge pages per NUMA node, to avoid memory oversubscription when for example running with a 1.1 GB hash. Additionally, we change the TT clearing slightly so that the pages are better distributed across NUMA nodes.
Data from Torom:
master:
Nodes/second : 43524292
patch:
Nodes/second : 44080034
Data on an 8-thread Emerald Rapids VM:
./stockfish speedtest 8 65536
Baseline speedtest (2MB huge pages):
Nodes/second : 7366232
Nodes/second : 7262794
Nodes/second : 7355431
Nodes/second : 7318777
Baseline speedtest (1GB huge pages):
Nodes/second : 7541027
Nodes/second : 7572720
Nodes/second : 7550268
Nodes/second : 7538925
closes https://github.com/official-stockfish/Stockfish/pull/6681
No functional change
After the introduction of UnifiedCorrectionHistory, the Pawn, Minor and NonPawn entries in CorrHistType are dead - only PieceTo and Continuation are referenced via CorrectionHistory<T>. Drop them, along with the now unreachable default template and the NonPawn specialization. Reduce the primary CorrHistTypedef template to a forward declaration so that any future stray use of an unsupported tag fails to compile.
closes https://github.com/official-stockfish/Stockfish/pull/6828
No functional change
Follow-up to #6785
After completing an iteration, the engine verifies if a "mate in x" condition is met to stop searching. Currently, it checks is_mate() and is_mated() independently using two separate bulky logic branches. However, VALUE_MATE equals 32000, so both a mate (e.g., 31998) and a mated score (e.g., -31998) satisfy VALUE_MATE - std::abs(score) == 2. By utilizing is_mate_or_mated() and applying std::abs(), we mathematically combine these checks into a single concise expression.
Can someone please help run a matetrack with a --mate 100 limit for this?
closes https://github.com/official-stockfish/Stockfish/pull/6829
No functional change
- We add a new path for supporting universal builds without #embed . `xxd` causes the compiler to run out of memory, so instead we embed it as a string literal created at compile time.
- This file is created in `net.sh`, and only if `--embed-dir` isn't supported + we are doing a universal build.
- This was necessary because Android NDK seems to not support embed, idk why
- A couple other Makefile tweaks were necessary for the Android path.
- While we're at it, remove the non-universal arm64 windows and android binaries.
closes https://github.com/official-stockfish/Stockfish/pull/6827
No functional change
Passed avx2/bmi2 STC:
https://tests.stockfishchess.org/tests/view/69ff99eb9392f0c317213eda
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 20992 W: 5469 L: 5192 D: 10331
Ptnml(0-2): 38, 2197, 5751, 2470, 40
Passed avxvnni STC:
https://tests.stockfishchess.org/tests/view/6a00022b9392f0c317213f7c
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 47328 W: 12328 L: 12009 D: 22991
Ptnml(0-2): 112, 5148, 12847, 5423, 134
Passed NEON STC:
https://tests.stockfishchess.org/tests/view/69ff99d69392f0c317213ed8
LLR: 2.96 (-2.94,2.94) <0.00,2.00>
Total: 29600 W: 7664 L: 7376 D: 14560
Ptnml(0-2): 48, 3074, 8277, 3344, 57
Passed avx512icl non-regression:
https://tests.stockfishchess.org/tests/view/6a002b699392f0c317213f91
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 90112 W: 23130 L: 22975 D: 44007
Ptnml(0-2): 192, 9939, 24633, 10106, 186
Measurements from vondele (Neoverse V2, armv8-dotprod):
==== master ====
==== Bench: 2344696 ====
1 Nodes/second : 280724660
2 Nodes/second : 280647282
3 Nodes/second : 282055192
Average (over 3): 281142378
==== 50a44640a3 ====
==== Bench: 2344696 ====
1 Nodes/second : 284271937
2 Nodes/second : 285638071
3 Nodes/second : 284349426
Average (over 3): 284753144
The patch's benefit is non-uniform and is ~0 for avx512icl unfortunately -- although I think we should be able to find something there....
## Background/explanation
The idea here is to move the non-zero block computation back to the previous layer, and overlap the work better. Then, to avoid trying to emulate compress instructions on targets not supporting them (i.e., everything except for AVX512), we use a `pop_lsb` loop on a bitset enumerating the non-zero blocks, rather than first writing them out as indices.
An early AVX2 implementation worked on fishtest (https://tests.stockfishchess.org/tests/view/69c3410534b6988b1e472db4) and linrock demonstrated that it also worked for NEON (https://tests.stockfishchess.org/tests/view/69ce0dc73ddc0eccd617188f). Thanks to him for encouraging me to push this idea over the finish line.
To abstract over the particular NNZ representation (bitset or index list), we use `NNZInfo` and `NNZCursor`. The cursor is used to write to one or the other perspective of the NNZ list. I'd appreciate ideas on making the code cleaner/more readable as imo it's still a bit ugly.
## Fixing GCC 15 regression
Separately, vondele noticed a regression in ARM performance from GCC 15 caused by suboptimal codegen. See https://discord.com/channels/435943710472011776/813919248455827515/1502837886381461605 for more info, but the easiest fix that I could come up with was inserting a couple `asm` optimization barriers.
Single threaded data:
```
average: stockfish.master.13.3 1041619
average: stockfish.master.15.2 999214
average: stockfish.patched.15.2 1031369
```
There's definitely regressions elsewhere as well, which we shld chase down, but at least this should unblock the arm64 universal binary work.
closes https://github.com/official-stockfish/Stockfish/pull/6814
No functional change
This PR fixes an oversight in #6818. At least in multithreaded searches the TT scores can be corrupted, and accessing them is racy.
Passed non-reg STC:
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 61792 W: 15609 L: 15428 D: 30755
Ptnml(0-2): 134, 6534, 17373, 6727, 128
https://tests.stockfishchess.org/tests/view/6a0b15396524d21ee79b868b
We also increase the number of the threads in our instrumented CI tests to make these assert failures for buggy patches more likely in future.
closes https://github.com/official-stockfish/Stockfish/pull/6830
No functional change
Introduce the armv8-universal target. 🥴
The entry point is essentially the same as x86's, except we no longer have access to `__builtin_cpu_supports` so instead we need an OS-specific query for whether we support dotprod. The Makefile is modified to support both universal builds.
If in the future we add more ARM targets, such as SVE, we'll need to add qemu to the RUN_PREFIX in CI, because currently we assume (for PGO purposes) that the CI host supports all the used ARM instructions.
### clang/Windows
The painful part here is clang on Windows, which, until arm64 mingw is stabilized, is required for targeting arm64. This PR also gets it to work for x86. In the Makefile this setup corresponds to `use_lto_emit_asm=yes`.
In particular, `--defsym` and `--save-temps` are not supported by `lld-link`, and objcopy `--rename-section` doesn't work on COFF binaries because of how section names work there.
- `--defsym` is needed to define `main` for PGO purposes and assigns it to the namespaced, per-arch main function. Instead, we define `main` in `main.cpp` so that the compilation is successful, then delete it before the final link.
- Instead of `--save-temps` to get the LTO intermediate object, we pass `--lto-emit-asm` to the linker, which outputs `stockfish.exe.lto.s`.
- Finally, we have a small AWK script to find the `.ctors` section, neuter it, and put start/stop symbols around it with the same naming scheme as ELF (`__start_*_init`/`__stop_*_init`).
I'm lowk a Windows programming noob so if there's simpler ways of going about this, I'd appreciate a pointer. @PikaCat-OuO + Codex used an approach that involved going in and modifying the LLVM bitcode, but that felt more complicated to me.
closes https://github.com/official-stockfish/Stockfish/pull/6823
No functional change
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
This patch improves mate finding results by protecting mate scores from being forgotten in a later iteration. This applies especially to aborted searches, but also to completed iterations in e.g. `go depth` scenarios. The only visible change to the user is the final UCI info line.
The case of TB wins/losses was excluded from this patch on purpose. The reason is that in some corner cases related to rounding in some DTZ Syzygy EGTBs, later searches may turn a TB loss into a draw, for example. See #5175 and the example
```
info depth 5 seldepth 5 multipv 1 score cp -20000 nodes 122 nps 8714 hashfull 0 tbhits 26 time 14 pv h1b1 a7b6 f6e5 b6b1 e5f4 c8c3 f4g5 d7h7 g5f4 b1f5 f4f5 c3e5 f5g6 e5f4 g6h7 f4g5 h7h8 b8c7 h8h7 c7d6 h7h8 d6e5 h8h7 e5f6 h7h8 g5g7
info depth 6 seldepth 9 multipv 1 score cp 0 nodes 255 nps 18214 hashfull 0 tbhits 26 time 14 pv h1h2 d7d6 f6g5 a7e7
```
from #5414.
Matetrack for patch:
```
Using ./stockfish on matetrack.epd with --nodes 1000000
Engine ID: Stockfish dev-20260510-dbbc3ed1
Total FENs: 6554
Found mates: 3054
Best mates: 2224
```
This is slightly more/equal compared to master (3040 and 2224), see https://github.com/vondele/matetrack/blob/faf7cf3be95ab7dfdcde6e035de28a30887cd8a5/matetrack1000000.csv#L4368.
Passed STC non-reg:
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 108224 W: 27565 L: 27429 D: 53230
Ptnml(0-2): 212, 11755, 30064, 11847, 234
https://tests.stockfishchess.org/tests/view/6a00899e9392f0c31721406c
closes https://github.com/official-stockfish/Stockfish/pull/6816
No functional change
Only delete the prerelease once all assets have been built and only need upload.
Also fix a link in prerelease notes to the official releases.
closes https://github.com/official-stockfish/Stockfish/pull/6812
No functional change
Thanks to recent commits, we can now also trust mate(d) scores from multiPV analysis for the best move, with complete PV etc. Locally I have also tested master extensively with the help of the matetrack script in multiPV analysis, with no issues reported. So I believe we can add this to our CI.
closes https://github.com/official-stockfish/Stockfish/pull/6809
No functional change
