Failed STC (https://tests.stockfishchess.org/tests/view/6a4f36355529b8472df7ff7e):
LLR: -2.94 (-2.94,2.94) <0.00,2.00>
Total: 144768 W: 38003 L: 38051 D: 68714
Ptnml(0-2): 521, 17154, 37050, 17170, 489
Passed LTC (https://tests.stockfishchess.org/tests/view/6a501dad5529b8472df80103):
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 189888 W: 49613 L: 48984 D: 91291
Ptnml(0-2): 122, 20406, 53256, 21041, 119
Passed VLTC (https://tests.stockfishchess.org/tests/view/6a52892c5529b8472df80481):
LLR: 2.94 (-2.94,2.94) <0.00,2.00>
Total: 46124 W: 12156 L: 11865 D: 22103
Ptnml(0-2): 12, 4523, 13696, 4824, 7
Passed VVLTC (https://tests.stockfishchess.org/tests/view/6a566a285529b8472df80b84):
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 44698 W: 11805 L: 11507 D: 21386
Ptnml(0-2): 9, 3819, 14395, 4117, 9
This PR adds pawn pair features, a new set of features that is indexed by pairs of pawns occupying the same or adjacent files (hence "3-wide"). Because these are a superset of pawn–pawn interactions in threat inputs (including the recently added pawn-pusher inputs), those are removed. HalfKA features remain unchanged.
Pawn-pair features were invented by Jonathan for Pawnocchio, based on his observation that in a net trained on *all* pairs of pawns, the most important pawn pairs were those differing by at most one file.
sscg13 made the necessary changes to the trainer (https://github.com/official-stockfish/nnue-pytorch/pull/502) and initial changes to inference. The slowdown was originally far too large (some 10%); following major optimizations from Jonathan and a bit from me, the slowdown is small enough for a comfortable LTC pass; on my machine I get a 3.5% slowdown overall. I expect there are further "easy" speedups in the new code, so this gap should narrow a bit.
Nettest PR: https://github.com/vondele/nettest/pull/359
closes https://github.com/official-stockfish/Stockfish/pull/6982
Bench: 2718396
Co-Authored-By: anematode <timothy.herchen@gmail.com>
Co-Authored-By: Jonathan Hallström <lmj.hallstrom@gmail.com>
Add -fno-ipa-cp-clone for mingw just as we do for gcc. I get about a 1% speedup. My machine is pretty unreliable for bench so maybe someone can verify.
closes https://github.com/official-stockfish/Stockfish/pull/6973
No functional change
With the latest master I started getting:
"A breakpoint instruction (__debugbreak() statement or a similar call) was executed in Stockfish.exe." when debugging in MSVC.
I don't really understand the issue but Fable fixed it and I verified it works.
Here is its summary:
Bug: Calling _get_wpgmptr() in CommandLine::get_binary_directory() triggered a debug CRT assertion failure (__debugbreak()) on Windows. The CRT only initializes the wide program name _wpgmptr when the application uses a wide entry point (wmain). Since Stockfish uses the narrow main(int, char**) entry point, _wpgmptr remains nullptr, and the debug CRT's parameter validation asserts when _get_wpgmptr is called.
Fix: Replaced _get_wpgmptr() with GetModuleFileNameW(), which queries the executable path directly from the OS and does not depend on the CRT entry point type. The implementation uses a grow-and-retry loop, so paths longer than MAX_PATH are handled correctly. If the API fails, the code falls back to the original argv0-based behavior. The #ifdef _MSC_VER guard was also removed, since the fix no longer relies on the MSVC CRT — non-MSVC Windows builds (e.g., MinGW) now benefit from the accurate path as well.
closes https://github.com/official-stockfish/Stockfish/pull/6972
No functional change
This network is trained by adding the following binpacks, relabeled by @vondele and totaling 74B positions, to the distillation fine-tuning stage:
```
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_0.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_1.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_2.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_3.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_1_relabel/leela96-filt-v2.min.split_4.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_0.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_1.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_2.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_3.relabel-BT4-tf13tune.binpack
- vondele/from_kaggle_2_relabel/T60T70wIsRightFarseerT60T74T75T76.split_4.relabel-BT4-tf13tune.binpack
```
The relabeling effort has been completed during the testing of this patch, and a full training run is on the way. Thanks to @vondele, @anematode, @Disservin, @yl25946, and all who've contributed to the process.
Passed STC:
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 24512 W: 6500 L: 6201 D: 11811
Ptnml(0-2): 69, 2772, 6300, 3021, 94
https://tests.stockfishchess.org/tests/view/6a40b7083036e45021aebbd6
Passed LTC:
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 15708 W: 4232 L: 3960 D: 7516
Ptnml(0-2): 8, 1572, 4420, 1848, 6
https://tests.stockfishchess.org/tests/view/6a413d293036e45021aebc84
nettest PR: https://github.com/vondele/nettest/pull/388
closes https://github.com/official-stockfish/Stockfish/pull/6932
Bench: 2102535
Co-authored-by: Joost VandeVondele <Joost.VandeVondele@gmail.com>
Passed STC
https://tests.stockfishchess.org/tests/view/6a2682ce351b79f679cc47c5
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 57248 W: 14730 L: 14399 D: 28119
Ptnml(0-2): 145, 6159, 15698, 6464, 158
Reordering operations in `do_move` allows us to effectively prefetch the TT entry earlier, since the piece moving helpers don't actually modify the position key. I suspect that with threat inputs, `put_piece` and friends got a lot more expensive, and so this helps us a lot.
vondele's machine:
==== master ====
1 Nodes/second : 294311526
2 Nodes/second : 297068312
3 Nodes/second : 297418763
Average (over 3): 296266200
==== pfearly ====
1 Nodes/second : 303986449
2 Nodes/second : 304221719
3 Nodes/second : 305302969
Average (over 3): 304503712 (+2.78%)
Locally, `bench`:
Result of 200 runs
speedup = +0.0158
P(speedup > 0) = 1.0000
As expected it helps even more in a large-hash, NUMA setting.
closes https://github.com/official-stockfish/Stockfish/pull/6891
No functional change
TRACKED_TOP previously used a shell pipeline (xargs, awk, sort) to extract
unique top-level directory/file names from SRCS and HEADERS. When invoking make
from cmd.exe on Windows, $(shell ...) uses cmd.exe rather than sh.exe, so the
pipeline fails because cmd.exe does not support Unix pipe syntax. Other MSYS2
tools (e.g. the compiler) work fine because they are invoked directly as
commands, not through a shell pipeline. This caused TRACKED_TOP to be empty, so
ln -s never ran and the per-arch temp_builds subdirectories were left without a
Makefile symlink, breaking the universal build with: "No rule to make target
'universal-object-nopgo'". Fixed by replacing the shell pipeline with pure GNU
make built-in functions (sort, foreach, firstword, subst) which have no shell
or PATH dependencies and work identically on Windows, Linux and macOS.
closes https://github.com/official-stockfish/Stockfish/pull/6762
No functional change
Conditions for no pawn pushes, non-pawn material and last captured piece are already such restrictive that we don't need to artificially set checkers bitboard for legal move generation.
Also moving the check for available pawn pushes at first place, as it should already filter about 90% of cases.
Passed STC and LTC non-regression tests.
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 171232 W: 44202 L: 44131 D: 82899
Ptnml(0-2): 466, 18809, 47023, 18824, 494
https://tests.stockfishchess.org/tests/view/69d3623b61a12cebe17ededa
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 55662 W: 14424 L: 14247 D: 26991
Ptnml(0-2): 38, 5479, 16609, 5678, 27
https://tests.stockfishchess.org/tests/view/69d64cada07a818b0a25f95e
And also neutral on 10k fixed games test with custom stalemate book.
Elo: 0.24 ± 0.9 (95%) LOS: 69.9%
Total: 10000 W: 4681 L: 4674 D: 645
Ptnml(0-2): 0, 87, 4819, 94, 0
nElo: 1.81 ± 6.8 (95%) PairsRatio: 1.08
https://tests.stockfishchess.org/tests/view/69d7e4183ca80bdf151d43b8
closes https://github.com/official-stockfish/Stockfish/pull/6715
Bench: 2984258
A bit tricky to identify, but in `apple-silicon` builds, specifying a value for
`SyzygyPath` could still lead to crashes at certain depths from a
SIGBUS/SIGSEGV.
This PR replaces the `std::count_if()` adjustment in `tbprobe.cpp`'s
`do_probe_table()` with an annotated handwritten loop, avoiding the clang/LTO
miscompilation behind the crash.
closes https://github.com/official-stockfish/Stockfish/pull/6721
No functional change.
the assert checks for `bestThreadDecisive && newThreadDecisive`. However `newThreadDecisive` is not guaranteed at this point, leading to assertion failures. This PR rearranges the conditions to ensure correctness.
closes https://github.com/official-stockfish/Stockfish/pull/6716
No functional change
Instead of checking whether a threat index is valid and then writing to
a vector if it isn't, we can instead always write to the buffer of the
vector, and only increase the size of the vector if the index is valid.
This saves some branch mispredictions.
passed STC:
https://tests.stockfishchess.org/tests/view/69ceb1689f7a7e3fdfc9a44b
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 205760 W: 53109 L: 52561 D: 100090
Ptnml(0-2): 603, 22552, 56076, 22992, 657
local speedtest shows
```
Result of 100 runs (cycles)
===========================
base (...ockfish.orig) = 296886 +/- 1645
test (...kfish.df0822) = 298862 +/- 1662
speedup % = +0.67 +/- 0.19
[95% CI; t-statistic]
p(speedup > 0) = 1.0000
p(speedup > .5%) = 0.9562
CPU: 16 x Intel(R) Core(TM) Ultra 9 185H
Hyperthreading: on
```
closes https://github.com/official-stockfish/Stockfish/pull/6703
No functional change
fixes#6663, #6664, and a million others issues raised over the years.
This is similar to
https://github.com/official-stockfish/Stockfish/pull/4563/changes but
more conservative and errors are split between "Unsupported position"
and "Invalid FEN".
The FEN parser needs to be strict as a foundation for safety. It does
not specify much of the semantics, so this step is fairly simple. Parts
after the ep square are optional, however, since it's common, for
example in EPD notation.
Errors arising from positional semantics that were previously bucketed
under invalid FENs are now reported as "Unsupported position". Only
positions that are potentially problematic are designated as
unsupported. It is NOT guided by illegality of the position, but instead
by the ability of the engine to handle them correctly. This means that
some checks from the previous PR were removed.
std::exit is used instead of std::terminate so atexit handles will be
called.
Probably wise to run copilot on this or smth because I scribbled it
without much thought.
With these small changes and reduced, less controversial, scope I hope
this PR will finally make it and we will be done with these weekly
issues. I'll wait with putting it on fishtest until there's approval.
closes https://github.com/official-stockfish/Stockfish/pull/6665
No functional change
The error is "C1001: Internal compiler error." in uci.cpp on line 370 of
master. For some reason the compiler can't handle the
std::size(hashfullAges) inside the lambda. Older MSVC versions had no
problem.
Most of the warnings are due to implicit type conversions "conversion
from type A to type B, possible loss of data"
many of which have been present for a while.
closes https://github.com/official-stockfish/Stockfish/pull/6464
No functional change
[Passed STC](https://tests.stockfishchess.org/tests/live_elo/691fc321acb6dbdf23d07e4b):
```
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 215360 W: 55651 L: 55108 D: 104601
Ptnml(0-2): 520, 22399, 61290, 22960, 511
```
This PR is on top of ces42's work so I'll rebase if that PR changes.
Essentially, it comprises my adjustments to `make_index` that remained
unmerged before threat inputs was finalized.
The unsigned intermediate variables let us skip a bunch of useless sign
extensions (because Square and Piece inherit from `int8_t`, and C/C++
semantics require narrow integers to be promoted to `int` before doing
arithmetic on them). Additionally, by removing the special usage of
`offsets[][64]` and `[65]` the indexing becomes more efficient. (This
usage was a temporary hack from sscg anyway, so I think he'll like that
it's gone.) Finally, the `sf_assume` condition was fixed so that it
actually makes a difference to the compiler.
The speedup is fairly nice locally (combining both ces42's and these changes):
```
Result of 100 runs
==================
base (...kfish.master) = 1691982 +/- 1907
test (./stockfish ) = 1714349 +/- 1789
diff = +22367 +/- 2465
speedup = +0.0132
P(speedup > 0) = 1.0000
```
closes https://github.com/official-stockfish/Stockfish/pull/6445
No functional change
This enables different Stockfish processes that use the same weights to use the
same memory. The approach establishes equivalence by memory content, and is
compatible with NUMA replication. The benefit of sharing is reduced memory usage
and a speedup thanks to improved (inter-process) caching of the network in the
CPUs cache, and thus reduced bandwidth usage to main memory. Even though this
change doesn't benefit a user running a single process, this helps on fishtest
or e.g. for Lichess, when multiple games run concurrently, or multiple
positions are analyzed in parallel.
This concept was probably first introduced in the Monty engine
(https://github.com/official-monty/Monty/pull/62), after a discussion in
https://github.com/official-stockfish/fishtest/issues/2077 on the issue of
memory pressure. Measurements based on Torch
(https://github.com/user-attachments/files/21386224/verbatim.pdf) further
suggested that large gains were possible. Multiple other engines have
adopted this 'verbatim' format as well.
The implementation here adds the flexibility needed for SF, for example, retains
the ability to bundle compressed networks with the binary, to load nets by uci
option, and to distribute the shared nets to the proper NUMA region. This
flexibility comes with a fair amount of complexity in the implementation, such
as OS specific code, and fallback code.
For most users this should be transparent. However, for example, those running
docker containers should ensure the `--ipc` flag is set correctly, and
`--shm-size` is sufficiently large.
The benefits of this patch significantly depend on hardware, with systems with
many cores and a large (O(150MB), the net size) L3 cache benefitting typically
most. On such systems SF speedups (as measured via nps playing games with
large concurrency but just 1 thread) can be 38%, which results in master vs.
patch Elo which gains about 25 Elo.
```
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 shared_memoryPR : 24.8 1.9 39432.0 73728 53
2 master : 0.0 ---- 34296.0 73728 47
```
In a multithreaded setup, where weights are already shared, that benefit is smaller,
for example on the same HW as above, but with 8t for each side.
```
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 shared_memoryPR : 5.2 3.5 9351.0 18432 51
2 master : 0.0 ---- 9081.0 18432 49
```
On fishtest with a typical hardware mix of our contributors, the following was measured:
STC, 60k games
https://tests.stockfishchess.org/tests/view/69074a49ea4b268f1fac236c
Elo: 4.69 ± 1.4 (95%) LOS: 100.0%
Total: 60000 W: 16085 L: 15275 D: 28640
Ptnml(0-2): 154, 6440, 16053, 7148, 205
nElo: 9.38 ± 2.8 (95%) PairsRatio: 1.12
To verify correctness with a single process on a NUMA architecture,
speedtest was used, confirming near equivalence:
```
master: Average (over 10): 296236186
shared_memory: Average (over 10): 295769332
```
Currently, using large pages for the shared network weights is not always possible,
which can lead to a small slowdown (1-2%), in case a single process is run.
closes https://github.com/official-stockfish/Stockfish/pull/6173
No functional change
Co-authored-by: disservin <disservin.social@gmail.com>
Co-authored-by: Joost VandeVondele <Joost.VandeVondele@gmail.com>
for positions such as:
position fen rr6/2q2p1k/2P1b1pp/bB2P1n1/R2B2PN/p4P1P/P1Q4K/1R6 b - - 2 38 moves f7f5
position fen 8/p2r1pK1/6p1/1kp1P1P1/2p5/2P5/8/4R3 b - - 0 43 moves f7f5
position fen 4k3/4p3/2b3b1/3P1P2/4K3/8/8/8 b - - moves e7e5
ep is not possible for various reasons. Do not set the ep square and the do not change the zobrist key, as if ep were possible.
This fixes 3-fold detection, which could in rare cases be observed as a warning generated by fastchess for PVs that continued beyond an actual 3-fold.
Also fixes wrong evals for certain moves e.g. a2a1 for
position fen 4k3/1b2p2b/8/3P1P2/4K3/8/8/r7 b - - 0 1 moves e7e5 e4e3 a1a2 e3e4 a2a1 e4f3 a1a2 f3e4
fixes https://github.com/official-stockfish/Stockfish/issues/6138
originally written and tested by rn5f107s2, with small buglet as
https://tests.stockfishchess.org/tests/view/685af90843ce022d15794400
failed STC
LLR: -2.93 (-2.94,2.94) <-1.75,0.25>
Total: 130688 W: 33986 L: 34362 D: 62340
Ptnml(0-2): 414, 13292, 38302, 12928, 408
https://tests.stockfishchess.org/tests/view/688bcb35502b34dd5e7111c9
passed LTC
LLR: 2.94 (-2.94,2.94) <-1.75,0.25>
Total: 99018 W: 25402 L: 25273 D: 48343
Ptnml(0-2): 54, 9097, 31089, 9204, 65
https://tests.stockfishchess.org/tests/view/688c613c502b34dd5e7112d3https://github.com/official-stockfish/Stockfish/pull/6211
Bench: 2946135
Co-authored-by: Joost VandeVondele <Joost.VandeVondele@gmail.com>
