Lichess maintains some patches on top of SF dev to get it working with Emscripten. This PR moves some of these patches into SF and adds WASM to CI. It also adds a few changes in places where the x86 intrinsics don't cleanly map onto WebAssembly SIMD instructions; otherwise, we use Emscripten's x86 compatibility layer and take SSE4.1 code paths.
Summary of the compatibility changes:
- Define `wasm32` and `wasm32-relaxed-simd` targets.
- We don't support wasm without SIMD; it'd be a waste of time.
- Add option to disable TBs
- This is required because `tbprobe.cpp` pulls in `mmap`. This option can be used on any target, of course, but it's only enabled by default for wasm.
- Add compilation job + test to CI
And the changes for performance:
- Disable atomics for shared history on wasm
- Atomics are always `seq_cst` there, which can be quite slow (even on the x86, stores are locked `xchg [mem], reg`)
- Add SSE code path to `get_changed_pieces`, modeled after the AVX2 path
- `_mm_mulhi_epi16` has a complicated emulation sequence, so for the pairwise multiplication, use an approach similar to the NEON impl.
- __int128 is gets lowered to runtime functions on wasm, so use the fallback impl for `mul_hi64`
- V8 does a poor job with the NNZ finding, so use a slightly different sequence there
- Add relaxed simd support for `m128_dpbusd`.
Some local perf figures (single-threaded speedtest):
```
wasm
Nodes/second : 902523
sse4.1
Nodes/second : 1155380
avx512icl
Nodes/second : 1676184
```
Further avenues to explore:
- Optimize for performance under V8's experimental AVX revectorizer (Currently it's about +10% in my testing, but could definittely be more)
- Branch hinting. For example, run bench while collecting branch frequency info, then inject it late in the WASM compilation pipeline. I tried this locally and it didn't help much, but maybe I'm missing something.
- PGO. Gives +1.5% NPS locally, but hard to integrate with WASM compilation wrokflows
closes https://github.com/official-stockfish/Stockfish/pull/6875
No functional change
- Add new target `macos-lipo`.
- Created by compiling a universal x86 binary (no PGO) and a standard Apple silicon binary (with PGO), then combining them into a Mach-O fat binary
- To keep only one copy of the net, we add custom loading logic in the x86 section. The executable reads its own path and mmaps the net that's in the ARM section.
- The offset and size (from the executable base) of the mapping is injected after compilation in `patch_x86_slice.sh`
- avx512 on macOS isn't advertised in the xcr0 register by default. The simple solution I came up with is to execute a dummy AVX512 instruction, which sets up the register, before calling `__builtin_cpu_init`.
Some housekeeping as well:
- Rename `armv8-universal` -> `arm64-universal`.
- Add standard copyright headers to the files we've added recently.
Potential follow-ups:
- Disservin's Makefile cleanup
- Alternative ideas for the net loading. In particular, this will error out if the user strips the binary (since that'll invalidate the offset).
closes https://github.com/official-stockfish/Stockfish/pull/6860
No functional change
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- We add a new path for supporting universal builds without #embed . `xxd` causes the compiler to run out of memory, so instead we embed it as a string literal created at compile time.
- This file is created in `net.sh`, and only if `--embed-dir` isn't supported + we are doing a universal build.
- This was necessary because Android NDK seems to not support embed, idk why
- A couple other Makefile tweaks were necessary for the Android path.
- While we're at it, remove the non-universal arm64 windows and android binaries.
closes https://github.com/official-stockfish/Stockfish/pull/6827
No functional change
Only delete the prerelease once all assets have been built and only need upload.
Also fix a link in prerelease notes to the official releases.
closes https://github.com/official-stockfish/Stockfish/pull/6812
No functional change
Add workflow-level concurrency to the direct GitHub Actions entrypoints so a
newer run for the same pull request or repository ref cancels the older
in-progress run.
Keep the reusable workflow graph, release guards, triggers, and permissions
unchanged while documenting the concurrency policy and the official GitHub
references used for the change.
closes https://github.com/official-stockfish/Stockfish/pull/6773
No functional change.
Pin every external action used by the root GitHub workflow set to a full-length
commit SHA and keep the selected release tag as a same-line comment.
Update the root workflows to the latest stable tagged action versions found
during the audit while preserving existing release guards, permissions, and
reusable workflow wiring.
closes https://github.com/official-stockfish/Stockfish/pull/6754
No functional change.
do not upload some unneeded intermediate directories,
disable running authenticated git commands with the checkout action.
Thanks to Yaron A for the report.
closes https://github.com/official-stockfish/Stockfish/pull/5435
No functional change
this action plays games under fast-chess with a `debug=yes` compiled binary.
It checks for triggered asserts in the code, or generally for engine disconnects.
closes https://github.com/official-stockfish/Stockfish/pull/5403
No functional change
In the last couple of months we sometimes saw duplicated prereleases uploaded to GitHub, possibly due to some racy behavior when concurrent jobs create a prerelease. This now creates an empty prerelease at the beginning of the CI and the binaries are later just attached to this one.
closes https://github.com/official-stockfish/Stockfish/pull/5144
No functional change
- Update codeql to v3
- Switch from dev-drprasad to native github cli
- Update softprops/action-gh-release to node 20 commit
`thollander/actions-comment-pull-request` needs to
be bumped to node20 too, but the author hasnt done
so atm
closes https://github.com/official-stockfish/Stockfish/pull/5044
No functional change
This refactors the CI workflows to group some
logic and makes sure that all (pre)release
binaries are actually tested.
The screenshot below shows the execution logic of
the reworked ci,
https://github.com/Disservin/Stockfish/actions/runs/7773581379.
You can also hover over the cards to see the
execution flow.
The `matrix.json` and `arm_matrix.json` define the
binaries which will be uploaded to GitHub.
Afterwards a matrix is created and each job
compiles a profile guided build for that arch and
uploads that as an artifact to GitHub. The
Binaries/ARM_Binaries workflow's are called when
the previous step has been completed, and uploads
all artifacts to the (pre)release.
This also fixes some indentations and renames the
workflows, see
https://github.com/official-stockfish/Stockfish/actions,
where every workflow is called `Stockfish` vs
https://github.com/Disservin/Stockfish/actions. It
also increases the parallel compilation used for
make from `-j2 to -j4`.
It now also prevents the prerelease action from
running on forks.
A test release can be viewed here
https://github.com/Disservin/Stockfish/releases.
closes https://github.com/official-stockfish/Stockfish/pull/5035
No functional change
The commit adds a CI workflow that uses the included-what-you-use (IWYU)
tool to check for missing or superfluous includes in .cpp files and
their corresponding .h files. This means that some .h files (especially
in the nnue folder) are not checked yet.
The CI setup looks like this:
- We build IWYU from source to include some yet unreleased fixes.
This IWYU version targets LLVM 17. Thus, we get the latest release
candidate of LLVM 17 from LLVM's nightly packages.
- The Makefile now has an analyze target that just build the object
files (without linking)
- The CI uses the analyze target with the IWYU tool as compiler to
analyze the compiled .cpp file and its corresponding .h file.
- If IWYU suggests a change the build fails (-Xiwyu --error).
- To avoid false positives we use LLVM's libc++ as standard library
- We have a custom mappings file that adds some mappings that are
missing in IWYU's default mappings
We also had to add one IWYU pragma to prevent a false positive in
movegen.h.
https://github.com/official-stockfish/Stockfish/pull/4783
No functional change
using github actions, create a prerelease for the latest commit to master.
As such a development version will be available on github, in addition to the latest release.
closes https://github.com/official-stockfish/Stockfish/pull/4622
No functional change
Github Actions allows us to use up to 20 workers.
This way we can launch multiple different checks
at the same time and optimize the overall time
the CI takes a bit.
closes https://github.com/official-stockfish/Stockfish/pull/4223
No functional change
For development versions of Stockfish, the version will now look like
dev-20221107-dca9a0533
indicating a development version, the date of the last commit,
and the git SHA of that commit. If git is not available,
the fallback is the date of compilation. Releases will continue to be
versioned as before.
Additionally, this PR extends the CI to create binary artifacts,
i.e. pushes to master will automatically build Stockfish and upload
the binaries to github.
closes https://github.com/official-stockfish/Stockfish/pull/4220
No functional change
This commit generalizes the feature transform to use vec_t macros
that are architecture defined instead of using a seperate code path for each one.
It should make some old architectures (MMX, including improvements by Fanael) faster
and make further such improvements easier in the future.
Includes some corrections to CI for mingw.
closes https://github.com/official-stockfish/Stockfish/pull/3955
closes https://github.com/official-stockfish/Stockfish/pull/3928
No functional change
- set the variable only for the required tests to keep simple the yml file
- use NDK 21.x until will be fixed the Stockfish static build problem
with NDK 23.x
- set the test for armv7, armv7-neon, armv8 builds:
- use armv7a-linux-androideabi21-clang++ compiler for armv7 armv7-neon
- enforce a static build
- silence the Warning for the unused compilation flag "-pie" with
the static build, otherwise the Github workflow stops
- use qemu to bench the build and get the signature
Many thanks to @pschneider1968 that made all the hard work with NDK :)
closes https://github.com/official-stockfish/Stockfish/pull/3924
No functional change
move to github actions to replace travis CI.
First version, testing on linux using gcc and clang.
gcc build with sanitizers and valgrind.
No functional change
