Over the past few months our attacks and bitboard logic got a bit more involved and target specific, so I think it makes sense to split this up a bit more into separate files.
While it's not in an ideal state I think it's first step and can be improved later, `pawn_attacks_bb` could also be moved into the attacks.h but this was more of an oversight from my side.
closes https://github.com/official-stockfish/Stockfish/pull/6833
No functional change
1GB pages for the TT give a small performance bump (`./stockfish speedtest 256 32000`):
master:
Nodes/second : 230514479
patch
Nodes/second : 237997728
With a large enough TT there is still dTLB thrashing with 2MB pages. 1GB pages reduces that and makes page table walks a bit faster (they're shorter, and there's fewer PTEs, which fits better in cache). `madvise` won't give you huge pages, only large pages, so we use mmap here.
We guard the huge page attempt to x86 + at least 8 huge pages per NUMA node, to avoid memory oversubscription when for example running with a 1.1 GB hash. Additionally, we change the TT clearing slightly so that the pages are better distributed across NUMA nodes.
Data from Torom:
master:
Nodes/second : 43524292
patch:
Nodes/second : 44080034
Data on an 8-thread Emerald Rapids VM:
./stockfish speedtest 8 65536
Baseline speedtest (2MB huge pages):
Nodes/second : 7366232
Nodes/second : 7262794
Nodes/second : 7355431
Nodes/second : 7318777
Baseline speedtest (1GB huge pages):
Nodes/second : 7541027
Nodes/second : 7572720
Nodes/second : 7550268
Nodes/second : 7538925
closes https://github.com/official-stockfish/Stockfish/pull/6681
No functional change
After the introduction of UnifiedCorrectionHistory, the Pawn, Minor and NonPawn entries in CorrHistType are dead - only PieceTo and Continuation are referenced via CorrectionHistory<T>. Drop them, along with the now unreachable default template and the NonPawn specialization. Reduce the primary CorrHistTypedef template to a forward declaration so that any future stray use of an unsupported tag fails to compile.
closes https://github.com/official-stockfish/Stockfish/pull/6828
No functional change
Follow-up to #6785
After completing an iteration, the engine verifies if a "mate in x" condition is met to stop searching. Currently, it checks is_mate() and is_mated() independently using two separate bulky logic branches. However, VALUE_MATE equals 32000, so both a mate (e.g., 31998) and a mated score (e.g., -31998) satisfy VALUE_MATE - std::abs(score) == 2. By utilizing is_mate_or_mated() and applying std::abs(), we mathematically combine these checks into a single concise expression.
Can someone please help run a matetrack with a --mate 100 limit for this?
closes https://github.com/official-stockfish/Stockfish/pull/6829
No functional change
- We add a new path for supporting universal builds without #embed . `xxd` causes the compiler to run out of memory, so instead we embed it as a string literal created at compile time.
- This file is created in `net.sh`, and only if `--embed-dir` isn't supported + we are doing a universal build.
- This was necessary because Android NDK seems to not support embed, idk why
- A couple other Makefile tweaks were necessary for the Android path.
- While we're at it, remove the non-universal arm64 windows and android binaries.
closes https://github.com/official-stockfish/Stockfish/pull/6827
No functional change
Passed avx2/bmi2 STC:
https://tests.stockfishchess.org/tests/view/69ff99eb9392f0c317213eda
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 20992 W: 5469 L: 5192 D: 10331
Ptnml(0-2): 38, 2197, 5751, 2470, 40
Passed avxvnni STC:
https://tests.stockfishchess.org/tests/view/6a00022b9392f0c317213f7c
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 47328 W: 12328 L: 12009 D: 22991
Ptnml(0-2): 112, 5148, 12847, 5423, 134
Passed NEON STC:
https://tests.stockfishchess.org/tests/view/69ff99d69392f0c317213ed8
LLR: 2.96 (-2.94,2.94) <0.00,2.00>
Total: 29600 W: 7664 L: 7376 D: 14560
Ptnml(0-2): 48, 3074, 8277, 3344, 57
Passed avx512icl non-regression:
https://tests.stockfishchess.org/tests/view/6a002b699392f0c317213f91
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 90112 W: 23130 L: 22975 D: 44007
Ptnml(0-2): 192, 9939, 24633, 10106, 186
Measurements from vondele (Neoverse V2, armv8-dotprod):
==== master ====
==== Bench: 2344696 ====
1 Nodes/second : 280724660
2 Nodes/second : 280647282
3 Nodes/second : 282055192
Average (over 3): 281142378
==== 50a44640a3 ====
==== Bench: 2344696 ====
1 Nodes/second : 284271937
2 Nodes/second : 285638071
3 Nodes/second : 284349426
Average (over 3): 284753144
The patch's benefit is non-uniform and is ~0 for avx512icl unfortunately -- although I think we should be able to find something there....
## Background/explanation
The idea here is to move the non-zero block computation back to the previous layer, and overlap the work better. Then, to avoid trying to emulate compress instructions on targets not supporting them (i.e., everything except for AVX512), we use a `pop_lsb` loop on a bitset enumerating the non-zero blocks, rather than first writing them out as indices.
An early AVX2 implementation worked on fishtest (https://tests.stockfishchess.org/tests/view/69c3410534b6988b1e472db4) and linrock demonstrated that it also worked for NEON (https://tests.stockfishchess.org/tests/view/69ce0dc73ddc0eccd617188f). Thanks to him for encouraging me to push this idea over the finish line.
To abstract over the particular NNZ representation (bitset or index list), we use `NNZInfo` and `NNZCursor`. The cursor is used to write to one or the other perspective of the NNZ list. I'd appreciate ideas on making the code cleaner/more readable as imo it's still a bit ugly.
## Fixing GCC 15 regression
Separately, vondele noticed a regression in ARM performance from GCC 15 caused by suboptimal codegen. See https://discord.com/channels/435943710472011776/813919248455827515/1502837886381461605 for more info, but the easiest fix that I could come up with was inserting a couple `asm` optimization barriers.
Single threaded data:
```
average: stockfish.master.13.3 1041619
average: stockfish.master.15.2 999214
average: stockfish.patched.15.2 1031369
```
There's definitely regressions elsewhere as well, which we shld chase down, but at least this should unblock the arm64 universal binary work.
closes https://github.com/official-stockfish/Stockfish/pull/6814
No functional change
This PR fixes an oversight in #6818. At least in multithreaded searches the TT scores can be corrupted, and accessing them is racy.
Passed non-reg STC:
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 61792 W: 15609 L: 15428 D: 30755
Ptnml(0-2): 134, 6534, 17373, 6727, 128
https://tests.stockfishchess.org/tests/view/6a0b15396524d21ee79b868b
We also increase the number of the threads in our instrumented CI tests to make these assert failures for buggy patches more likely in future.
closes https://github.com/official-stockfish/Stockfish/pull/6830
No functional change
Introduce the armv8-universal target. 🥴
The entry point is essentially the same as x86's, except we no longer have access to `__builtin_cpu_supports` so instead we need an OS-specific query for whether we support dotprod. The Makefile is modified to support both universal builds.
If in the future we add more ARM targets, such as SVE, we'll need to add qemu to the RUN_PREFIX in CI, because currently we assume (for PGO purposes) that the CI host supports all the used ARM instructions.
### clang/Windows
The painful part here is clang on Windows, which, until arm64 mingw is stabilized, is required for targeting arm64. This PR also gets it to work for x86. In the Makefile this setup corresponds to `use_lto_emit_asm=yes`.
In particular, `--defsym` and `--save-temps` are not supported by `lld-link`, and objcopy `--rename-section` doesn't work on COFF binaries because of how section names work there.
- `--defsym` is needed to define `main` for PGO purposes and assigns it to the namespaced, per-arch main function. Instead, we define `main` in `main.cpp` so that the compilation is successful, then delete it before the final link.
- Instead of `--save-temps` to get the LTO intermediate object, we pass `--lto-emit-asm` to the linker, which outputs `stockfish.exe.lto.s`.
- Finally, we have a small AWK script to find the `.ctors` section, neuter it, and put start/stop symbols around it with the same naming scheme as ELF (`__start_*_init`/`__stop_*_init`).
I'm lowk a Windows programming noob so if there's simpler ways of going about this, I'd appreciate a pointer. @PikaCat-OuO + Codex used an approach that involved going in and modifying the LLVM bitcode, but that felt more complicated to me.
closes https://github.com/official-stockfish/Stockfish/pull/6823
No functional change
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
This patch improves mate finding results by protecting mate scores from being forgotten in a later iteration. This applies especially to aborted searches, but also to completed iterations in e.g. `go depth` scenarios. The only visible change to the user is the final UCI info line.
The case of TB wins/losses was excluded from this patch on purpose. The reason is that in some corner cases related to rounding in some DTZ Syzygy EGTBs, later searches may turn a TB loss into a draw, for example. See #5175 and the example
```
info depth 5 seldepth 5 multipv 1 score cp -20000 nodes 122 nps 8714 hashfull 0 tbhits 26 time 14 pv h1b1 a7b6 f6e5 b6b1 e5f4 c8c3 f4g5 d7h7 g5f4 b1f5 f4f5 c3e5 f5g6 e5f4 g6h7 f4g5 h7h8 b8c7 h8h7 c7d6 h7h8 d6e5 h8h7 e5f6 h7h8 g5g7
info depth 6 seldepth 9 multipv 1 score cp 0 nodes 255 nps 18214 hashfull 0 tbhits 26 time 14 pv h1h2 d7d6 f6g5 a7e7
```
from #5414.
Matetrack for patch:
```
Using ./stockfish on matetrack.epd with --nodes 1000000
Engine ID: Stockfish dev-20260510-dbbc3ed1
Total FENs: 6554
Found mates: 3054
Best mates: 2224
```
This is slightly more/equal compared to master (3040 and 2224), see https://github.com/vondele/matetrack/blob/faf7cf3be95ab7dfdcde6e035de28a30887cd8a5/matetrack1000000.csv#L4368.
Passed STC non-reg:
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 108224 W: 27565 L: 27429 D: 53230
Ptnml(0-2): 212, 11755, 30064, 11847, 234
https://tests.stockfishchess.org/tests/view/6a00899e9392f0c31721406c
closes https://github.com/official-stockfish/Stockfish/pull/6816
No functional change
Only delete the prerelease once all assets have been built and only need upload.
Also fix a link in prerelease notes to the official releases.
closes https://github.com/official-stockfish/Stockfish/pull/6812
No functional change
Thanks to recent commits, we can now also trust mate(d) scores from multiPV analysis for the best move, with complete PV etc. Locally I have also tested master extensively with the help of the matetrack script in multiPV analysis, with no issues reported. So I believe we can add this to our CI.
closes https://github.com/official-stockfish/Stockfish/pull/6809
No functional change
This does require the user to have set `/sys/kernel/mm/transparent_hugepage/shmem_enabled` to something other than `never` (unfortunately the default, at least on Ubuntu).
```
Result of 50 runs
==================
base (...kfish.master) = 1814244 +/- 4396
test (./stockfish ) = 1823756 +/- 3448
diff = +9512 +/- 5496
speedup = +0.0052
P(speedup > 0) = 0.9996
```
```
Performance counter stats for './stockfish bench':
237,561 dTLB-misses
5,783,003,231 cycles
Performance counter stats for './bins/stockfish.master bench':
2,219,832 dTLB-misses
5,932,779,255 cycles
```
closes https://github.com/official-stockfish/Stockfish/pull/6805
No functional change
I found these changes useful and natural when working on a patch to improve mate reporting. I can revert some of the changes if maintainers would prefer that. E.g. I am not sure whether `unset_bound_flags()` is worth defining.
closes https://github.com/official-stockfish/Stockfish/pull/6785
No functional change
fix Node.js version dependency, update SDE.
This upgrades to 9.58, upgrade to 10.8 failed with:
Invalid file name and base dir combination: Pin root DLL: pincrt4.dll
Also adjusts Makefile to show PGO bench output on failure
closes https://github.com/official-stockfish/Stockfish/pull/6781
No functional change
Failed VVLTC non-regression
https://tests.stockfishchess.org/tests/view/69d562b84088e069540a2288
LLR: -2.96 (-2.94,2.94) <-1.75,0.25>
Total: 386998 W: 99181 L: 99760 D: 188057
Ptnml(0-2): 35, 35792, 122429, 35203, 40
Failed STC non-regression
https://tests.stockfishchess.org/tests/view/69f3c6601e5788938e86a99e
LLR: -2.93 (-2.94,2.94) <-1.75,0.25>
Total: 33696 W: 8492 L: 8795 D: 16409
Ptnml(0-2): 124, 4209, 8504, 3868, 143
Many thanks to Dubslow, Torom, ces42, Shawn, vondele, Disservin and others for discussion.
## Summary
The venerable small net has been around for quite some time now, and while the big net architecture has substantially advanced with TI, the small net has stayed with plain HalfKA. It therefore presents a few burdens: multiple net architectures to maintain, multiple nets to train, and a whole lot of templates to deal with the variable L1 size.
Locally I measure a slowdown of -2.5% in NPS with this branch – and it's probably more on non-AVX512 architectures – but a pure slowdown of that magnitude would lead to more dramatic losses (even at VVLTC) than exhibited in the above tests, suggesting that the small net's lower eval quality is deleterious.
Bonus: Shawn found this interesting PGN among the VVLTC games: https://lichess.org/study/hvo8jflc/OeTOityv `master` seems to misevaluate the fortress because all positions go to small net (the material difference is larger than the threshold).
closes https://github.com/official-stockfish/Stockfish/pull/6796
Bench: 2877007
Recent changes to this part of the code mean that the variable `completedDepth` is no longer neeeded. The only *functional* part of this patch is the effect on the thread voting formula. Basically, compared to master this patch raises the depth weight in the formula uniformly by 1. For very, very low depths this may have a tiny effect on thread selection compared to master. But in general we expect this patch to have no effect on game play at STC and LTC.
Passed STC SMP non-reg:
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 284808 W: 72944 L: 72994 D: 138870
Ptnml(0-2): 351, 32192, 77370, 32138, 353
https://tests.stockfishchess.org/tests/view/69eddd011e5788938e86a075
closes https://github.com/official-stockfish/Stockfish/pull/6793
No functional change
Master in `pv()` incorrectly assumes that any root move with `i != pvIdx` has an exact score. This can lead to the suppression of bound flags for (a) root moves that have been partially searched in an iteration but do not end up to be best move, or (b) when an aborted search for `pvIdx > 0` leads to a new best move. This latter case is the cause for issue #6783.
Fixes#6783.
The bug in its incarnation (a) was independently discovered by @ciekce a couple of weeks ago, and we have jointly discussed a patch that fixes it. He is therefore a co-author of the bugfix part of this patch.
While at it, we also fix three minor oversights in the PV roll-back for aborted searches with a loss score:
* We only roll back the PV if the aborted search would report an exact score.
* Master inadvertently may have reported a bound for the rolled-back score+PV.
* We ensure a final UCI info line with the correct search depth and searched nodes statistics etc.
closes https://github.com/official-stockfish/Stockfish/pull/6784
No functional change
Co-Authored-By: Ciekce <44617491+Ciekce@users.noreply.github.com>
The core idea here is to greatly speed up the backwards insertion loop of the insertion sort. We can store up to 16 sorted moves and their values/scores in vectors. We obtain all moves with a smaller score (i.e., sorted later than the move-to-insert) as a mask, giving us something like `1111111111000000`. If we subtract one this gives us `1111111110111111` which can be can be used as an expand mask, to insert the new move.
Passed STC:
https://tests.stockfishchess.org/tests/view/69e9c9190e9667dd5a765853
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 45024 W: 11867 L: 11550 D: 21607
Ptnml(0-2): 101, 4938, 12142, 5205, 126
Local test:
```
Result of 50 runs
==================
base (...kfish.master) = 2026071 +/- 5124
test (./stockfish ) = 2083770 +/- 5964
diff = +57699 +/- 8535
speedup = +0.0285
P(speedup > 0) = 1.0000
```
closes https://github.com/official-stockfish/Stockfish/pull/6768
No functional change
Passed STC
https://tests.stockfishchess.org/tests/view/69e82e200e9667dd5a765631
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 115424 W: 29950 L: 29822 D: 55652
Ptnml(0-2): 318, 12712, 31544, 12800, 338
Passed AVX2-only STC
https://tests.stockfishchess.org/tests/view/69e953080e9667dd5a7657c0
LLR: 2.93 (-2.94,2.94) <-1.75,0.25>
Total: 134720 W: 34773 L: 34665 D: 65282
Ptnml(0-2): 344, 14958, 36694, 14974, 390
Now that our L1 size is much smaller, the overhead of tracking diffs for
`double_inc_update` is a larger share of the cost.
Interestingly ces42 measured a -0.9% slowdown locally (avxvnni build).
Meanwhile Torom and I find this branch to be 2% faster. I'm not sure exactly
what's going on but more tests may be in order
closes https://github.com/official-stockfish/Stockfish/pull/6776
No functional change
Add workflow-level concurrency to the direct GitHub Actions entrypoints so a
newer run for the same pull request or repository ref cancels the older
in-progress run.
Keep the reusable workflow graph, release guards, triggers, and permissions
unchanged while documenting the concurrency policy and the official GitHub
references used for the change.
closes https://github.com/official-stockfish/Stockfish/pull/6773
No functional change.
TRACKED_TOP previously used a shell pipeline (xargs, awk, sort) to extract
unique top-level directory/file names from SRCS and HEADERS. When invoking make
from cmd.exe on Windows, $(shell ...) uses cmd.exe rather than sh.exe, so the
pipeline fails because cmd.exe does not support Unix pipe syntax. Other MSYS2
tools (e.g. the compiler) work fine because they are invoked directly as
commands, not through a shell pipeline. This caused TRACKED_TOP to be empty, so
ln -s never ran and the per-arch temp_builds subdirectories were left without a
Makefile symlink, breaking the universal build with: "No rule to make target
'universal-object-nopgo'". Fixed by replacing the shell pipeline with pure GNU
make built-in functions (sort, foreach, firstword, subst) which have no shell
or PATH dependencies and work identically on Windows, Linux and macOS.
closes https://github.com/official-stockfish/Stockfish/pull/6762
No functional change
For quite some time SF has reported a too low final search depth in rare cases
(in multi-threaded search if a non-main thread was selected). Unfortunately,
the recent refactoring in #6704 means that this now happens for almost every
stopped search.
This PR fixes this issue, which was first spotted and reported by joergoster,
by ensuring that `rootDepth` always holds the last depth for which a search was
started.
Fixes#6756.
closes https://github.com/official-stockfish/Stockfish/pull/6761
No functional change.
Rename Linux universal release package from
`stockfish-ubuntu-x86-64-universal.tar.gz` to
`stockfish-linux-x86-64-universal.tar.gz` (and similar for the binary), as the
bundled binary should run on any Linux distro with glibc >= 2.35 (which is the
glibc version of the Ubuntu 22.04 build server).
Note that the Linux universal binary is built with static libstdc++ linkage, so
this adds no extra dependency.
This change will hopefully make it less confusing for users, and in line with
how we show it on stockfishchess.org (as Linux).
closes https://github.com/official-stockfish/Stockfish/pull/6760
No functional change
