Tomasz Sobczyk
e371d133a7
Fix grouping and do dedup in registry.
2021-04-05 17:25:28 +02:00
Tomasz Sobczyk
e7b3803fd0
Add more counters
2021-04-05 17:00:27 +02:00
Tomasz Sobczyk
fcd53684b6
To/from move stats
2021-04-05 16:43:25 +02:00
Tomasz Sobczyk
b2a5bf4171
Deduplicate statistic gatherers. Fix King square counter compilation errors.
2021-04-05 16:36:27 +02:00
Tomasz Sobczyk
eda51f19a2
Add king square counter
2021-04-05 16:15:37 +02:00
Tomasz Sobczyk
570a0f6f3c
Per square stats utility
2021-04-05 16:12:47 +02:00
Tomasz Sobczyk
7d74185d0b
Add max_count parameter to limit the number of positions read.
2021-04-05 14:21:25 +02:00
Tomasz Sobczyk
f85dbc3fe3
Reorder code and add important comments.
2021-04-05 14:21:25 +02:00
Tomasz Sobczyk
8365109972
Revert "Add additional checks for en-passant possiblity when fixing the erroneus ep flag from a fen."
...
This reverts commit 6afcdaa928
2021-04-05 12:37:11 +02:00
Tomasz Sobczyk and nodchip
6afcdaa928
Add additional checks for en-passant possiblity when fixing the erroneus ep flag from a fen.
2021-04-03 23:17:55 +09:00
Tomasz Sobczyk and nodchip
876902070d
Add optional warmup step for training.
...
Specified with `warmup_epochs`, uses `warmup_lr`.
The purpose is to put the net into a somewhat stable state so that the gradients are not as high during the early stages of the training and don't "accidentally" break the net.
2021-03-26 00:26:41 +09:00
Tomasz Sobczyk and nodchip
bbe338b9fc
Add random move accuracy for comparison.
2021-03-25 22:06:46 +09:00
Tomasz Sobczyk and nodchip
5fdb48a7cb
Change some learn parameter naming. Update docs.
2021-03-14 22:15:16 +09:00
Tomasz Sobczyk and nodchip
591609c262
Fix relation between halfmove and fullmove clocks.
2021-03-14 22:01:01 +09:00
Tomasz Sobczyk and nodchip
0ddad45ab2
Add gather_statistics command that allows gathering statistics from a .bin or .binpack file. Initially only support position count.
2021-03-01 00:36:45 +09:00
Tomasz Sobczyk and nodchip
74774c36e1
Fix wrong multipv depth range. Fixes #291
2021-01-25 21:39:22 +09:00
Tomasz Sobczyk and nodchip
1f7e5d3861
Add thread sanitized run for instrumented_learn and fix races.
2020-12-28 16:08:34 +09:00
Tomasz Sobczyk and nodchip
acf95c7c98
Accumulate clipping statistics to a 64 bit integer to prevent overflow for larger batch sizes.
2020-12-25 10:04:28 +09:00
Tomasz Sobczyk and nodchip
1b560efabd
Correctly handle the last batch of data in sfen_reader
2020-12-25 10:03:24 +09:00
Tomasz Sobczyk and nodchip
6d28d97a91
Don't unload evalfile on set nnue false
2020-12-25 09:58:24 +09:00
Tomasz Sobczyk and nodchip
c1e69f450e
Prevent q_ in loss calculation from reaching values that would produce NaN
2020-12-25 00:41:31 +09:00
Tomasz Sobczyk and nodchip
4f6fdca31f
Reduce the amount of sfens buffered for the validation step.
...
Used to be 10M, now we bound it by a multiple of validation_count, and at most 1M. This reduces the RAM usage greatly.
2020-12-25 00:17:35 +09:00
Tomasz Sobczyk and nodchip
7636bcccd1
Correctly account for factors when computing the average absolute weight of the feature transformer.
2020-12-25 00:08:51 +09:00
Tomasz Sobczyk and nodchip
2061be4730
smart_fen_skipping at gensfen_nonpv level
2020-12-24 21:37:30 +09:00
Tomasz Sobczyk and nodchip
868b4e9421
add gensfen_nonpv docs
2020-12-24 21:37:30 +09:00
Tomasz Sobczyk and nodchip
96b377a90a
Add gensfen_nonpv
2020-12-24 21:37:30 +09:00
Tomasz Sobczyk and nodchip
3f73c40412
More deterministic move accuracy validation.
2020-12-24 10:16:59 +09:00
Tomasz Sobczyk and nodchip
8ca82646a9
Use plain nnue eval for validation loss calculation instead of first performing qsearch
2020-12-22 10:35:19 +09:00
Tomasz Sobczyk and nodchip
6853b4aac2
Simple filtering for validation data.
2020-12-22 09:40:25 +09:00
Tomasz Sobczyk and nodchip
50df3a7389
fix annoying warning
2020-12-22 09:24:26 +09:00
Tomasz Sobczyk and nodchip
994eb5e183
rescore_fen -> rescore. Make it work on .bin and .binpack inputs.
2020-12-21 10:48:20 +09:00
Tomasz Sobczyk and nodchip
ffae19b5a1
Add docs for rescore_fen
2020-12-21 10:48:20 +09:00
Tomasz Sobczyk and nodchip
a9cfaa4d98
Add a tool for rescoring fens from an epd file with fixed depth search
2020-12-21 10:48:20 +09:00
Tomasz Sobczyk and nodchip
f56613ebf6
Add 'validation_count' option for 'learn' that specifies how many positions to use for validation
2020-12-20 09:47:30 +09:00
Tomasz Sobczyk and nodchip
a7378f3249
Make next_fen in opening_book a critical section
2020-12-14 09:03:04 +09:00
Tomasz Sobczyk and nodchip
d99ba07b81
Fix incorrect enpassant flag for moves read from uci format in the binpack lib
2020-12-11 00:31:32 +09:00
Tomasz Sobczyk and nodchip
3a1bd1185f
Add binpack coarse shuffle tool.
2020-12-06 19:08:52 +09:00
Tomasz Sobczyk and nodchip
28d6d7cb03
Avoid computing gradient for validation loss.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
fafb9557a8
Get train loss from update_parameters.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
4eb0e77a2a
Store references instead of copying the results of intermediate autograd computations.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
6cd0b03098
Add some comments regarding the current state of autograd loss computation.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
99cb869db3
Reintroduce use_wdl.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
cf6bc7ecaf
Cleanup around get_loss
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
256c4b55ec
Properly apply gradient norm clipping after it's scaled in the update_parameters.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
de675e3503
Reintroduce optional scaling of the teacher signal.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
01ae7b1e2c
Simplify passing constants that may vary between calls.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
cbd973fdaa
Detect constant expressions in autograd and return 0 grad early.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
e975889132
Move cross_entropy calculation to a separate function.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
891abf5511
Make the autograd loss expression chain thread_local.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
8adf00ae6e
Identify a single evalation chain by ID in autograd to prevent cache reuse for subsequent evaluations of the same expression tree.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
cb812c742c
Add [[nodiscard]] attributes to autograd functions.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
26f19e1429
Make automatic differentiation node types constexpr.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
aec6017195
When forming an autograd expression only copy parts that are rvalue references, store references to lvalues.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
a5c20bee5b
Apply gradient clipping.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
d103867558
Add memoization to the autograd expression evaluator.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
aa55692b97
Cross entropy loss.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
539bd2d1c8
Replace the old loss/grad calculation completely.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
b71d1e8620
Pass the new loss function to update_parameters
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
5a58eb803a
Loss func with autograd
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
541fb8177a
More utility in autograd.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
6ce0245787
Basic autograd
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
1322a9a5fd
Prevent false sharing of num_calls counter in the shared input trainer. Fix current_operation not being local to the executing thread.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
2aa7f5290e
Fix comparison of integers with different signedness.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
a97b65eaef
Fix compilation error with USE_BLAS
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
622e0b14c2
Remove superfluous example shuffling. Shuffling now only happens on reading.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
34510dd08a
Remove used examples asyncronously.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
0bee8fef64
Don't unnecessarily copy the batch part.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
e954b14196
Prefetch weights for feature transformer backprop to shared cache.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
8009973381
Special case for alpha=1 in saxpy, slight performance increase.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
49b2dcb1f3
Preallocate memory for unique_features. Keep the training_features temporary buffer as a thread_local so we reuse the storage.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
1c8495b54b
Remove handwritten saxpy because compilers optimize the second look anyway.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
15c528ca7b
Prepare feature transformer learner.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
a3c78691a2
Prepare input slice trainer.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
401fc0fbab
Prepare clipped relu trainer.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
774b023641
Add chunked for each with workers.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
cc11375f6d
Skeleton for new evaluate learner
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
0d4b803b08
Prepare trainer affine transform.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
4ea8572b6d
Add single threaded sgemm.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
92b14a5ba2
Add docs for transform.
2020-11-27 09:16:22 +09:00
Tomasz Sobczyk and nodchip
89294e2e4f
Add transform command. Add transform nudged_static subcommand.
2020-11-27 09:16:22 +09:00
Tomasz Sobczyk and nodchip
45e3335ee8
Add missing docs.
2020-11-23 19:22:11 +09:00
Tomasz Sobczyk and nodchip
9030020a85
Add smart_fen_skipping option to learn.
2020-11-23 19:22:11 +09:00
Tomasz Sobczyk and nodchip
ee13cfce67
Fix result assigned for a psvector when the positions are not continuous.
2020-11-23 08:32:08 +09:00
Tomasz Sobczyk and nodchip
3cee6881ee
Move the terminal position check to after qsearch, otherwise qsearch may end up in a terminal position.
2020-11-23 08:29:38 +09:00
Tomasz Sobczyk and nodchip
5f18c88b3d
Docs for book in gensfen.
2020-11-17 09:43:23 +09:00
Tomasz Sobczyk and nodchip
e1dbad47ce
Add support for opening book to gensfen.
2020-11-17 09:43:23 +09:00
Tomasz Sobczyk and nodchip
d4350a16f3
Add representation of an opening book.
2020-11-17 09:43:23 +09:00
Tomasz Sobczyk and nodchip
d793663188
Add docs for max_grad option for learn
2020-11-16 10:08:56 +09:00
Tomasz Sobczyk and nodchip
3dbc45bdfc
Add gradient clipping.
2020-11-16 10:08:56 +09:00
Tomasz Sobczyk and nodchip
50358e26c7
Fix searching terminal nodes in gensfen.
2020-11-15 22:18:13 +09:00
Tomasz Sobczyk and nodchip
00bc80c3c4
Add assume_quiet option to the learner.
2020-11-15 22:18:13 +09:00
Tomasz Sobczyk and nodchip
00797a3d86
add option ensure_quiet for gensfen that makes the generated position quiet
2020-11-15 22:18:13 +09:00
Tomasz Sobczyk and nodchip
691da3bdad
Add more information for factorizers at the start of training.
2020-11-14 18:47:22 +09:00
Tomasz Sobczyk and nodchip
4e1653d53a
Fix reliance on transitive includes for factorizers in trainer feature transformer. Add a file that includes all factorizers.
2020-11-14 12:35:12 +09:00
Tomasz Sobczyk and nodchip
69bc3ef9be
Output loss more often.
2020-11-14 12:33:25 +09:00
Tomasz Sobczyk and nodchip
a71623f74c
Add explicit read head seek to the start of the binpack file. Otherwise on MACOS the read head is placed at the end when app is specified.
2020-11-13 19:56:36 +09:00
Tomasz Sobczyk and nodchip
2a8576b804
Fix compilation issues.
2020-11-10 10:21:09 +09:00
Tomasz Sobczyk and nodchip
8069963c56
Update convert docs.
2020-11-10 10:21:09 +09:00
Tomasz Sobczyk and nodchip
5d88e7bce8
Add optional move validation to training data conversion. No longer rely on static initialization order for magics initialization.
2020-11-10 10:21:09 +09:00
Tomasz Sobczyk and Joost VandeVondele
ba35c88ab8
AVX-512 for smaller affine and feature transforms.
...
For the feature transformer the code is analogical to AVX2 since there was room for easy adaptation of wider simd registers.
For the smaller affine transforms that have 32 byte stride we keep 2 columns in one zmm register. We also unroll more aggressively so that in the end we have to do 16 parallel horizontal additions on ymm slices each consisting of 4 32-bit integers. The slices are embedded in 8 zmm registers.
These changes provide about 1.5% speedup for AVX-512 builds.
Closes https://github.com/official-stockfish/Stockfish/pull/3218
No functional change.
2020-11-07 16:49:49 +01:00
Tomasz Sobczyk