It causes a serious regression hanging a simple fixed
depth search. Reproducible with:
position fen q1B5/1P1q4/8/8/8/6R1/8/1K1k4 w - - 0 1
go depth 13
The reason is a search tree explosion due to:
if (... && depth < 3 * ONE_PLY)
extension = ONE_PLY;
This is very dangerous code by itself because triggers **at the leafs**
and in the above position keeps extending endlessly. In normal games
time deadline makes the search to stop sooner or later, but in fixed
seacrch we just hang possibly for a very long time. This is not acceptable
because 'go depth 13' shall not be a surprise for any position.
This patch reverts commit 76f1807baa.
and fixes the issue https://github.com/official-stockfish/Stockfish/issues/2091
Bench: 3243738
Introduce a new search extension when pushing an advanced passed pawn is
also suggested by the first killer move. There have been previous tests
which have similar ideas, mostly about pawn pushes, but it seems to be
overkill to extend too many moves. My idea is to limit the extension to
when a move happens to be noteworthy in some other way as well, such as
in this case, when it is also a killer move.
STC:
LLR: 2.96 (-2.94,2.94) [0.50,4.50]
Total: 19027 W: 4326 L: 4067 D: 10634
http://tests.stockfishchess.org/tests/view/5cac2cde0ebc5925cf00c36d
LTC:
LLR: 2.94 (-2.94,2.94) [0.00,3.50]
Total: 93390 W: 15995 L: 15555 D: 61840
http://tests.stockfishchess.org/tests/view/5cac42270ebc5925cf00c4b9
For future tests, it looks like this will interact heavily with passed
pawn evaluation. It may be good to try more variants of some of the more
promising evaluations tests/tweaks.
Bench: 3666092
The current update only by main thread depends on the luck of
whether main thread sees any/many changes to the best move or not.
It then makes large, lumpy changes to the time to be
used (1x, 2x, 3x, etc) depending on that sample of 1.
Use the average across all threads to get a more reliable
number with a smoother distribution.
STC @ 5+0.05 th 4 :
LLR: 2.95 (-2.94,2.94) [0.50,4.50]
Total: 51899 W: 11446 L: 11029 D: 29424
http://tests.stockfishchess.org/tests/view/5ca32ff20ebc5925cf0016fb
STC @ 5+0.05 th 8 :
LLR: 2.96 (-2.94,2.94) [0.50,4.50]
Total: 13851 W: 2843 L: 2620 D: 8388
http://tests.stockfishchess.org/tests/view/5ca35ae00ebc5925cf001adb
LTC @ 20+0.2 th 8 :
LLR: 2.95 (-2.94,2.94) [0.00,3.50]
Total: 48527 W: 7941 L: 7635 D: 32951
http://tests.stockfishchess.org/tests/view/5ca37cb70ebc5925cf001cec
Further work:
Similar changes might be possible for the fallingEval and timeReduction calculations (and elsewhere?), using either the min, average or max values across all threads.
Bench 3506898
Shuffle detection procedure :
Shuffling positions are detected if
the last 36 moves are reversible (rule50_count() > 36),
the position have been already in the TT,
there is a still a pawn on the board (to avoid special endings like KBN vs K).
The position is then judged as a draw.
An extension is realized if we already made 14 successive reversible moves in PV to accelerate the detection of the eventual draw.
To go further : we can still improve the idea. The length of the tests need a lot of ressources.
the limit of 36 is logic but must be checked again for special zugzwang positions,
this limit can be decreased in special positions,
the limit of 14 moves for extension has not been tuned.
STC
LLR: -2.94 (-2.94,2.94) [0.50,4.50]
Total: 32595 W: 7273 L: 7275 D: 18047 Elo +0.43
http://tests.stockfishchess.org/tests/view/5c90aa330ebc5925cfff1768
LTC
LLR: 2.95 (-2.94,2.94) [0.00,3.50]
Total: 51249 W: 8807 L: 8486 D: 33956 Elo +1.85
http://tests.stockfishchess.org/tests/view/5c90b2450ebc5925cfff1800
VLTC
LLR: 2.96 (-2.94,2.94) [0.00,3.50]
Total: 137974 W: 20503 L: 19983 D: 97488 Elo +1.05
http://tests.stockfishchess.org/tests/view/5c9243a90ebc5925cfff2a93
Bench: 3548313
Adding a clamp function makes some of these range limitations a bit prettier and removes some #include's.
STC
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 28117 W: 6300 L: 6191 D: 15626
http://tests.stockfishchess.org/tests/view/5c9aa1df0ebc5925cfff8fcc
Non functional change.
always use the implementation of gives_check in position, no need to
hand-inline part of the implementation in search.
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 57895 W: 12632 L: 12582 D: 32681
http://tests.stockfishchess.org/tests/view/5c9eaa4b0ebc5925cfffc9e3
No functional change.
While looking at pruning using see_ge() (which is very valuable)
it became apparent that the !extension test is not adding any
value - simplify it away.
STC:
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 56843 W: 12621 L: 12569 D: 31653
http://tests.stockfishchess.org/tests/view/5c8588cb0ebc5925cffe77f4
LTC:
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 78622 W: 13223 L: 13195 D: 52204
http://tests.stockfishchess.org/tests/view/5c8611cc0ebc5925cffe7f86
Further work could be to optimize the remaining see_ge() test. The idea of less pruning at higher depths is valuable, but perhaps the test (-PawnValueEg * depth) can be improved.
Bench: 3188688
This is a functional simplification that removes the FutilityMoveCounts array with a simple equation using only ints.
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 14175 W: 3123 L: 2987 D: 8065
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 9900 W: 1735 L: 1597 D: 6568
Bench: 3380343
This is a non-functional patch which shrinks the reductions array.
This saves about 8Kb of memory.
The only slow part of master's reductions array is the calculation
of the log values, so using a separate array for those values and
calculating the rest real-time appears to be just as fast as master.
STC
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 63245 W: 13906 L: 13866 D: 35473
http://tests.stockfishchess.org/tests/view/5c7b571f0ebc5925cffdc104
No funcional change.
This removes the skipQuiets variable, as was done in an earlier round by
@protonspring, but fixes an oversight which led to wrong mate
announcements. Quiets can only be pruned when there is no mate score, so
set moveCountPruning at the right spot.
tested as a fix at STC:
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 66321 W: 14690 L: 14657 D: 36974
http://tests.stockfishchess.org/tests/view/5c74f3170ebc5925cffd4b3c
and as the full patch at LTC:
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 25903 W: 4341 L: 4228 D: 17334
http://tests.stockfishchess.org/tests/view/5c7540030ebc5925cffd506f
Bench: 3292342
This is a somewhat different patch. It fixes blindspots for
two knights vs pawn endgame.
With local testing starting from random KNNvKP positions where the
pawn has not advanced beyond the 4th rank (thanks @protonspring !)
at 15+0.15 (4 cores), this went +105=868-27 against master. All except
two losses were won in reverse.
The heuristic is simple but effective - the strategy in these endgames
is to push the opposing king to the corner, then move the knight that's
blocking the pawn in for the checkmate while the pawn is free to move
and prevents stalemate. This patch gives SF the little boost it needs
to search the relevant king-cornering mating lines.
See the discussion in pull request 1939 for some more good results for
this test in independant tests:
https://github.com/official-stockfish/Stockfish/pull/1939
Bench: 3310239
A simple idea, but it makes sense: in current master the search is extended
for checks that are considered somewhat safe, and for for this we use the
static exchange evaluation which only considers the `to_sq` of a move.
This is not reliable for discovered checks, where another piece is giving
the check and is arguably a more dangerous type of check. Thus, if the check
is a discovered check, the result of SEE is not relevant and can be ignored.
STC:
LLR: 2.96 (-2.94,2.94) [0.50,4.50]
Total: 29370 W: 6583 L: 6274 D: 16513
http://tests.stockfishchess.org/tests/view/5c5062950ebc593af5d4d9b5
LTC:
LLR: 2.95 (-2.94,2.94) [0.00,3.50]
Total: 227341 W: 37972 L: 37165 D: 152204
http://tests.stockfishchess.org/tests/view/5c5094fb0ebc593af5d4dc2c
Bench: 3611854
This patch removes line 875 of search.cpp, which was updating pvHit after IID.
Bench testing at depth 22 shows that line 875 of search.cpp never changes the
value of pvHit at NonPV nodes, while at PV nodes it often changes the value
from true to false (and never the reverse). This is because the definition of
pvHit at line 642 is :
```
pvHit = (ttHit && tte->pv_hit()) || (PvNode && depth > 4 * ONE_PLY);
```
while the assignment after IID omits the ` (PvNode && depth > 4 * ONE_PLY) `
condition. As such, unlike the other two post-IID tte reads, this line of code
does not make SF's state more consistent, but rather introduces an inconsistency
in the definition of pvHit. Indeed, changing line 875 read
```
pvHit = (ttHit && tte->pv_hit()) || (PvNode && depth > 4 * ONE_PLY);
```
to match line 642 is functionally equivalent to removing the line entirely, as
this patch does.
STC
LLR: 2.96 (-2.94,2.94) [-3.00,1.00]
Total: 62756 W: 13787 L: 13746 D: 35223
http://tests.stockfishchess.org/tests/view/5c446c850ebc5902bb5d4b75
LTC
LLR: 3.19 (-2.94,2.94) [-3.00,1.00]
Total: 61900 W: 10179 L: 10111 D: 41610
http://tests.stockfishchess.org/tests/view/5c45bf610ebc5902bb5d5d62
Bench: 3796134
stopOnPonderhit is used to stop search quickly on a ponderhit. It is set by mainThread as part of its time management. However, master employs it as a signal between mainThread and the UCI thread. This is not necessary, it is sufficient for the UCI thread to signal that pondering finished, and mainThread should do its usual time-keeping job, and in this case stop immediately.
This patch implements this, removing stopOnPonderHit as an atomic variable from the ThreadPool,
and moving it as a normal variable to mainThread, reducing its scope. In MainThread::check_time() the search is stopped immediately if ponder switches to false, and the variable stopOnPonderHit is set.
Furthermore, ponder has been moved to mainThread, as the variable is only used to exchange signals between the UCI thread and mainThread.
The version has been tested locally (as fishtest doesn't support ponder):
Score of ponderSimp vs master: 2616 - 2528 - 8630 [0.503] 13774
Elo difference: 2.22 +/- 3.54
which indicates no regression.
No functional change.
The new form is likely to trigger a bit more at LTC. Given that LTC
appears to be an improvement, I think that is fine.
The change is not very invasive: it does the same as before, use
potentially less time for moves that are very stable. Most of the
time, the full bonus was given if the bonus was given, so the gradual
part {3, 4, 5} didn't matter much. Whereas previously 'stable' was
expressed as the last 80% of iterations are the same, now I use a
fixed depth (10 iterations). For TCEC style TC, it will presumably
imply some more moves that are played quicker (and thus more time
on the clock when it potentially matters). Note that 10 iterations
of stability means we've been proposing that move for 99.9% of search
time.
passed STC
http://tests.stockfishchess.org/tests/view/5c30d2290ebc596a450c055b
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 70921 W: 15403 L: 15378 D: 40140
passed LTC
http://tests.stockfishchess.org/tests/view/5c31ae240ebc596a450c1881
LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
Total: 17422 W: 2968 L: 2842 D: 11612
No functional change.
- add cluster info line
- provides basic info on positions received/stored in a cluster run,
useful to judge performance.
- document most cluster functionality in the readme.md
No functional change
Introducing new concept, saving principal lines into the transposition table
to generate a "critical search tree" which we can reuse later for intelligent
pruning/extension decisions.
For instance in this patch we just reduce reduction for these lines. But a lot
of other ideas are possible.
To go further : tune some parameters, how to add or remove lines from the
critical search tree, how to use these lines in search choices, etc.
STC :
LLR: 2.94 (-2.94,2.94) [0.50,4.50]
Total: 59761 W: 13321 L: 12863 D: 33577 +2.23 ELO
http://tests.stockfishchess.org/tests/view/5c34da5d0ebc596a450c53d3
LTC :
LLR: 2.96 (-2.94,2.94) [0.00,3.50]
Total: 26826 W: 4439 L: 4191 D: 18196 +2.9 ELO
http://tests.stockfishchess.org/tests/view/5c35ceb00ebc596a450c65b2
Special thanks to Miguel Lahoz for his help in transposition table in/out.
Bench: 3399866
This was inspired after reading about
[Multi-Cut](https://www.chessprogramming.org/Multi-Cut).
We now do non-singular cut node pruning. The idea is to prune when we
have a "backup plan" in case our expected fail high node does not fail
high on the ttMove.
For singular extensions, we do a search on all other moves but the
ttMove. If this fails high on our original beta, this means that both
the ttMove, as well as at least one other move was proven to fail high
on a lower depth search. We then assume that one of these moves will
work on a higher depth and prune.
STC:
LLR: 2.96 (-2.94,2.94) [0.50,4.50]
Total: 72952 W: 16104 L: 15583 D: 41265
http://tests.stockfishchess.org/tests/view/5c3119640ebc596a450c0be5
LTC:
LLR: 2.95 (-2.94,2.94) [0.00,3.50]
Total: 27103 W: 4564 L: 4314 D: 18225
http://tests.stockfishchess.org/tests/view/5c3184c00ebc596a450c1662
Bench: 3145487
Fixes one TODO, by moving the IO related to bestmove to the root, even if this move is found by a different rank.
This is needed to make sure IO from different ranks is ordered properly. If this is not done it is possible that e.g. a bestmove arrives before all info lines have been received, leading to output that confuses tools and humans alike (see e.g. https://github.com/cutechess/cutechess/issues/472)
This generalizes exchange of signals between the ranks using a non-blocking all-reduce. It is now used for the stop signal and the node count, but should be easily generalizable (TB hits, and ponder still missing). It avoids having long-lived outstanding non-blocking collectives (removes an early posted Ibarrier). A bit too short a test, but not worse than before:
Score of new-r4-1t vs old-r4-1t: 459 - 401 - 1505 [0.512] 2365
Elo difference: 8.52 +/- 8.43
This reverts my earlier change that only the root node gets to output best move after fixing problem with MPI_Allreduce by our custom operator(BestMoveOp). This function is not commutable and we must ensure that its output is consistent among all nodes.
Previous behavior was to wait on all nodes to finish their search on their own TM and aggregate to root node via a blocking MPI_Allreduce call. This seems to be problematic.
In this commit a proper non-blocking signalling barrier was implemented to use TM from root node to control the cluster search, and disable TM on all non-root nodes.
Also includes some cosmetic fix to the nodes/NPS display.
