Fixes problems with non-ASCII characters in paths. Fixes#3424.
Changes
- Bump the minimum supported Clang version to 11 to support std::filesystem
- Use std::optional for empty values instead of string sentinels like "None"
- Remove the FixedString class
- Move the EvalFile network path from the network struct to the engine
- Simplify parts of the command-line handling
- Remove the old dual-net logic from export_net
- And ofc we now support non-ascii paths for windows
Behavior Change
- Because the EvalFile is no longer part of the network, it won't be included in the hashing, thus the network sharing is no longer dependent on the EvalFile path
- As a result, if the binary version and network are the same, changing only the EvalFile location will now reuse the same shared network
This also fixes a master bug where:
```
export_net None
setoption name EvalFile value None
export_net None
Failed to export a net
```
would fail on the second export. With this patch, the second export succeeds.
Quick Fishtest Check that nets load like normal.
https://tests.stockfishchess.org/tests/view/6a455593f97ff95f787954da
<img width="833" height="1137" alt="image" src="https://github.com/user-attachments/assets/fffee63b-c63a-4d5e-b13d-87d58bc28c88" />
closes https://github.com/official-stockfish/Stockfish/pull/6937
No functional change
Co-authored-by: anematode <timothy.herchen@gmail.com>
Performance on Spacemit K3, thanks @edolnx for testing
master:
Total time (ms) : 65200
Nodes searched : 3493826
Nodes/second : 53586
riscv-scalable-port:
Total time (ms) : 15834
Nodes searched : 3493826
Nodes/second : 220653
Also thanks to @camel-cdr for guidance on RVV programming, and https://cloud-v.co for supplying an RVV instance to test with
passed STC:
LLR: 2.81 (-2.94,2.94) <0.00,2.00>
Total: 1152 W: 527 L: 108 D: 517
Ptnml(0-2): 0, 17, 167, 348, 44
https://tests.stockfishchess.org/tests/view/6a39895b3036e45021aeb368
## Summary
We've had a `riscv64` target for a while, but haven't really optimized for it, in particular the vector extension (RVV).
RVV, like SVE, is based on a scalable vector system where the vector length ranges from 128 to 65536. In practice implementations are between 128 and 2048, and 256 bits is quite common (e.g. the Spacemit K3 system above). Unfortunately this doesn't fit well into the rest of our code which assumes a fixed vector length, so what I've done is bypass the `VECTOR` ifdef (which now basically means "FIXED_LENGTH_VECTOR") and just have RVV-specific paths.
The ability to explicitly control `vl` makes the code quite readable, in my opinion. We use LMUL>1 in most places to take advantage of multi-vector instructions. Generally the LMULs were chosen to best support a 256-bit vlen, which is very common, but by virtue of how the vlen control works, the code works with any vlen. In a couple places, i.e., `get_changed_pieces` and `AffineTransformSparseInput::propagate`, we have separate implementations depending on the vlen, because the optimal LMUL varies a lot between implementations.
One little wrinkle is that `load_as` is compiled to a sequence of byte loads, because although unaligned loads are legal in RVA23, the spec says that they *may* be extremely slow (even though they usually aren't, in actual hw), so compilers are conservative. Thus I aligned the relevant buffers and made the semantics of `load_as` that the operand is aligned, by adding a runtime assertion.
### Universal binary
Adding a universal binary is pretty easy and we can just cross-compile. There are two targets: baseline rv64gc and riscv64-rva23, which is actually a smaller subset of RVA23 that also works on some older processors that don't support the full thing. We use clang because GCC, until recently, has a nasty bug with LTO and RVV.
Like the universal ARM and x86 builds, we check all the builds in CI. In this case we run bench with multiple vlens, 128 through 1024.
In the meantime I deleted the existing broken and unused riscv64 tests.
### Follow-ups
- Optimizations
- zvdot4a8i path
closes https://github.com/official-stockfish/Stockfish/pull/6920
No functional change
This version handles aborting engine processes more gracefully. This
also test the engine prior to use, as the process is nevertheless not
fully robust.
closes https://github.com/official-stockfish/Stockfish/pull/6893
No functional change
This PR introduces the additional `RootMove` attribute `previousPV` so that scores and PVs we send to the GUI in MultiPV analysis always match. This allows us in particular to extend our guarantee of exact mate (and TB win/loss) scores having a complete PV (leading to checkmate in the correct number of plies) to all PV lines. Recall that master fails here, since partially searched root moves may send to the GUI the previous score with the current/modified PV. See #6784.
The PR also uses the new attribute to extend the followPV logic to the analysis of sidelines, building on the idea in #6813 by @joergoster.
Passed non-reg STC:
LLR: 2.95 (-2.94,2.94) <-1.75,0.25>
Total: 166880 W: 42357 L: 42282 D: 82241
Ptnml(0-2): 394, 18685, 45177, 18820, 364
https://tests.stockfishchess.org/tests/view/6a0dea55818cacc1db0abb6a
Failed non-reg LTC:
LLR: -2.97 (-2.94,2.94) <-1.75,0.25>
Total: 890520 W: 224168 L: 225282 D: 441070
Ptnml(0-2): 390, 91902, 261789, 90790, 389
https://tests.stockfishchess.org/tests/view/6a1143ad818cacc1db0ac14c
Opening as draft for discussion on how to proceed. In SinglePV analysis, the patch is completely nonfunctional. But it is maybe a (small?) slowdown because of the increased size of `RootMove`. I am not sure if there as an elegant way to enrich the class only for MultiPV analysis (but the switch can happen at any time through the UCI interface), or to mitigate the speed penalty in some other way.
A local speedup test shows only a small slowdown on my system (but still high error bars):
```
sf_base = 1156928 +/- 1459 (95%)
sf_test = 1155885 +/- 1283 (95%)
diff = -1043 +/- 1777 (95%)
speedup = -0.09021% +/- 0.154% (95%)
```
The PR also adds the new MultiPV mate PV correctness check to the CI.
closes https://github.com/official-stockfish/Stockfish/pull/6886
No functional change
Lichess maintains some patches on top of SF dev to get it working with Emscripten. This PR moves some of these patches into SF and adds WASM to CI. It also adds a few changes in places where the x86 intrinsics don't cleanly map onto WebAssembly SIMD instructions; otherwise, we use Emscripten's x86 compatibility layer and take SSE4.1 code paths.
Summary of the compatibility changes:
- Define `wasm32` and `wasm32-relaxed-simd` targets.
- We don't support wasm without SIMD; it'd be a waste of time.
- Add option to disable TBs
- This is required because `tbprobe.cpp` pulls in `mmap`. This option can be used on any target, of course, but it's only enabled by default for wasm.
- Add compilation job + test to CI
And the changes for performance:
- Disable atomics for shared history on wasm
- Atomics are always `seq_cst` there, which can be quite slow (even on the x86, stores are locked `xchg [mem], reg`)
- Add SSE code path to `get_changed_pieces`, modeled after the AVX2 path
- `_mm_mulhi_epi16` has a complicated emulation sequence, so for the pairwise multiplication, use an approach similar to the NEON impl.
- __int128 is gets lowered to runtime functions on wasm, so use the fallback impl for `mul_hi64`
- V8 does a poor job with the NNZ finding, so use a slightly different sequence there
- Add relaxed simd support for `m128_dpbusd`.
Some local perf figures (single-threaded speedtest):
```
wasm
Nodes/second : 902523
sse4.1
Nodes/second : 1155380
avx512icl
Nodes/second : 1676184
```
Further avenues to explore:
- Optimize for performance under V8's experimental AVX revectorizer (Currently it's about +10% in my testing, but could definittely be more)
- Branch hinting. For example, run bench while collecting branch frequency info, then inject it late in the WASM compilation pipeline. I tried this locally and it didn't help much, but maybe I'm missing something.
- PGO. Gives +1.5% NPS locally, but hard to integrate with WASM compilation wrokflows
closes https://github.com/official-stockfish/Stockfish/pull/6875
No functional change
- Add new target `macos-lipo`.
- Created by compiling a universal x86 binary (no PGO) and a standard Apple silicon binary (with PGO), then combining them into a Mach-O fat binary
- To keep only one copy of the net, we add custom loading logic in the x86 section. The executable reads its own path and mmaps the net that's in the ARM section.
- The offset and size (from the executable base) of the mapping is injected after compilation in `patch_x86_slice.sh`
- avx512 on macOS isn't advertised in the xcr0 register by default. The simple solution I came up with is to execute a dummy AVX512 instruction, which sets up the register, before calling `__builtin_cpu_init`.
Some housekeeping as well:
- Rename `armv8-universal` -> `arm64-universal`.
- Add standard copyright headers to the files we've added recently.
Potential follow-ups:
- Disservin's Makefile cleanup
- Alternative ideas for the net loading. In particular, this will error out if the user strips the binary (since that'll invalidate the offset).
closes https://github.com/official-stockfish/Stockfish/pull/6860
No functional change
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* .tar -> .tar.gz: save a few more MB in distribution.
* drop android armv7 without NEON support: there should be almost no devices around that do not support this.
closes https://github.com/official-stockfish/Stockfish/pull/6850
No functional change
- We add a new path for supporting universal builds without #embed . `xxd` causes the compiler to run out of memory, so instead we embed it as a string literal created at compile time.
- This file is created in `net.sh`, and only if `--embed-dir` isn't supported + we are doing a universal build.
- This was necessary because Android NDK seems to not support embed, idk why
- A couple other Makefile tweaks were necessary for the Android path.
- While we're at it, remove the non-universal arm64 windows and android binaries.
closes https://github.com/official-stockfish/Stockfish/pull/6827
No functional change
Introduce the armv8-universal target. 🥴
The entry point is essentially the same as x86's, except we no longer have access to `__builtin_cpu_supports` so instead we need an OS-specific query for whether we support dotprod. The Makefile is modified to support both universal builds.
If in the future we add more ARM targets, such as SVE, we'll need to add qemu to the RUN_PREFIX in CI, because currently we assume (for PGO purposes) that the CI host supports all the used ARM instructions.
### clang/Windows
The painful part here is clang on Windows, which, until arm64 mingw is stabilized, is required for targeting arm64. This PR also gets it to work for x86. In the Makefile this setup corresponds to `use_lto_emit_asm=yes`.
In particular, `--defsym` and `--save-temps` are not supported by `lld-link`, and objcopy `--rename-section` doesn't work on COFF binaries because of how section names work there.
- `--defsym` is needed to define `main` for PGO purposes and assigns it to the namespaced, per-arch main function. Instead, we define `main` in `main.cpp` so that the compilation is successful, then delete it before the final link.
- Instead of `--save-temps` to get the LTO intermediate object, we pass `--lto-emit-asm` to the linker, which outputs `stockfish.exe.lto.s`.
- Finally, we have a small AWK script to find the `.ctors` section, neuter it, and put start/stop symbols around it with the same naming scheme as ELF (`__start_*_init`/`__stop_*_init`).
I'm lowk a Windows programming noob so if there's simpler ways of going about this, I'd appreciate a pointer. @PikaCat-OuO + Codex used an approach that involved going in and modifying the LLVM bitcode, but that felt more complicated to me.
closes https://github.com/official-stockfish/Stockfish/pull/6823
No functional change
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Only delete the prerelease once all assets have been built and only need upload.
Also fix a link in prerelease notes to the official releases.
closes https://github.com/official-stockfish/Stockfish/pull/6812
No functional change
Thanks to recent commits, we can now also trust mate(d) scores from multiPV analysis for the best move, with complete PV etc. Locally I have also tested master extensively with the help of the matetrack script in multiPV analysis, with no issues reported. So I believe we can add this to our CI.
closes https://github.com/official-stockfish/Stockfish/pull/6809
No functional change
fix Node.js version dependency, update SDE.
This upgrades to 9.58, upgrade to 10.8 failed with:
Invalid file name and base dir combination: Pin root DLL: pincrt4.dll
Also adjusts Makefile to show PGO bench output on failure
closes https://github.com/official-stockfish/Stockfish/pull/6781
No functional change
Add workflow-level concurrency to the direct GitHub Actions entrypoints so a
newer run for the same pull request or repository ref cancels the older
in-progress run.
Keep the reusable workflow graph, release guards, triggers, and permissions
unchanged while documenting the concurrency policy and the official GitHub
references used for the change.
closes https://github.com/official-stockfish/Stockfish/pull/6773
No functional change.
Rename Linux universal release package from
`stockfish-ubuntu-x86-64-universal.tar.gz` to
`stockfish-linux-x86-64-universal.tar.gz` (and similar for the binary), as the
bundled binary should run on any Linux distro with glibc >= 2.35 (which is the
glibc version of the Ubuntu 22.04 build server).
Note that the Linux universal binary is built with static libstdc++ linkage, so
this adds no extra dependency.
This change will hopefully make it less confusing for users, and in line with
how we show it on stockfishchess.org (as Linux).
closes https://github.com/official-stockfish/Stockfish/pull/6760
No functional change
`stockfish-ubuntu-x86-64-universal.tar` goes from 94.8MiB to 67.4MiB (tar.gz)
Note that the CI code is written such that it easy to select other compression
schemes in the future (e.g. tar.xz) by simply changing the `archive_ext`
parameter.
closes https://github.com/official-stockfish/Stockfish/pull/6746
No functional change
Pin every external action used by the root GitHub workflow set to a full-length
commit SHA and keep the selected release tag as a same-line comment.
Update the root workflows to the latest stable tagged action versions found
during the audit while preserving existing release guards, permissions, and
reusable workflow wiring.
closes https://github.com/official-stockfish/Stockfish/pull/6754
No functional change.
We maintain support for quite a few x86 ISA extensions but relied on the end
user to select the binary that is best for their system. We can detect at
runtime which architecture to use, and ship a single binary (per OS/ISA
combination). This PR does so, maintaining performance.
This is what I've landed on after quite a few iterations. Basically, we build
each arch separately, use `cpuid` to select one, and jump to that arch's main
function. Some details:
- The preprocessor macro `UNIVERSAL_BINARY` is defined when building for the universal binary.
- The Makefile target `universal-object-[no]pgo` is added, which produces a `stockfish.o` in each arch's build directory.
- To prevent symbol collisions between the multiple builds, we `#define Stockfish Stockfish_[arch]`. Furthermore we wrap the `main` function in `namespace Stockfish { }`.
- We can still get PGO data by linking to a per-arch binary, with `Stockfish_x86_64_avx2::main` as `main`, and running `bench`.
- For arches not supported by the host we can use Intel SDE – this is what we do in CI.
- When embedding the NNUE, we use C++26/C23 `#embed` instead of `INCBIN`. The issue with `INCBIN` is that it injects assembly directives that we have no control over.
- To ensure there's only one copy of the networks in the final binary, we define the network data as weak symbols in each per-arch build. Then, in the final link, we add a special nnue_embed.cpp which provides strong symbols.
- This does require GCC 15. Statically linking libstdc++ allows the binary to still run on older OSes though. And latest msys2 already does static linking + is based off GCC 15.2.0
building can be as simple as
make -j profile-build ARCH=x86-64-universal
or using the sde if the host doesn't support all architectures supported in the
universal binary, and the default host compiler is not recent enough.
make -j profile-build ARCH=x86-64-universal RUN_PREFIX="/path/to/sde -future --" CXX=g++-15
This change is also integrated in CI, so universal binaries are available as
downloadable artifacts.
Next we could also do ARM64, especially Android (all Apple silicon users use
the same binary). It also might be worth getting this to work with clang.
closes https://github.com/official-stockfish/Stockfish/pull/6740
No functional change
Ever since #5094 master only lets the main thread terminate `go mate`
searches, reverting the earlier improvement #1215.
This PR restores the old logic. So any thread that found a "mate in x"
can now stop the search.
To make this work robustly, we need to guard against inexact mate scores
in the best thread selection. In addition, in contrast to time limits,
the main thread may now not complete a d1 search for a mated-in
position.
In master an aborted d1 search may have a PV beginning with
`Move::none()`, in which case no thread selection is performed. See
#623. We fix this bug here by checking if `lastBestPV` is empty or not.
For interrupted d1 searches we now label mated-in scores as inexact.
While at it, we also simplify the logic for detecting if we can
terminate a go mate x search, using the fact that threads.stop can only
be false if we have a completed iteration with a valid score.
The PR has no effect on game play, but should slightly improve general
mate finding and speed up multi-threaded `go mate` searches.
We also add a corresponding matecheck run to the CI. This only involves
61 mates up to mate-in-2. Test runs with the first 50 or 100 mates from
`mates2000.epd` did at times not finish within 30 minutes on my fork or
in local tests, possibly due to search explosions for some mate-in-3
positions.
closes https://github.com/official-stockfish/Stockfish/pull/6668
No functional change
WINE_PATH started as a Wine-specific knob, but it’s now used more generally
as a command prefix to run the built engine under wrappers
like Intel SDE, qemu-user, etc.
- Add RUN_PREFIX as the supported “run wrapper/prefix” variable in Makefile
- Set WINE_PATH as a deprecated alias
- Update CI and scripts to use RUN_PREFIX
closes https://github.com/official-stockfish/Stockfish/pull/6500
No functional change
When vnni256 was first introduced #3038 it was very slightly faster than vnni512
on some machines. We have since sped up vnni512 significantly (#4796 over 10%
alone, #6139, and probably others I'm forgetting). Since any machine that can
run vnni256 can run vnni512 this arch is no longer useful.
Note, x86-64-avxvnni still covers targets that don't have AVX512 but do have
VNNI on 128- and 256-bit vectors.
closes https://github.com/official-stockfish/Stockfish/pull/6340
No functional change
Add runs with --threads 4 to our matetrack CI. These won't be deterministic, of
course. But they may catch early some multithreading bugs in our mate
reporting.
Motivated by #6293 and #6296.
https://github.com/official-stockfish/Stockfish/pull/6297
No functional change.
Previously the upload step used the os of the artifact to create the upload and
used an extra msys2 step. This is in fact not needed and we can do all required
changes on ubuntu instead.
closes https://github.com/official-stockfish/Stockfish/pull/6294
No functional change
