Tomasz Sobczyk and nodchip
fafb9557a8
Get train loss from update_parameters.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
4eb0e77a2a
Store references instead of copying the results of intermediate autograd computations.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
6cd0b03098
Add some comments regarding the current state of autograd loss computation.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
99cb869db3
Reintroduce use_wdl.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
cf6bc7ecaf
Cleanup around get_loss
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
256c4b55ec
Properly apply gradient norm clipping after it's scaled in the update_parameters.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
de675e3503
Reintroduce optional scaling of the teacher signal.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
01ae7b1e2c
Simplify passing constants that may vary between calls.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
cbd973fdaa
Detect constant expressions in autograd and return 0 grad early.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
e975889132
Move cross_entropy calculation to a separate function.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
891abf5511
Make the autograd loss expression chain thread_local.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
8adf00ae6e
Identify a single evalation chain by ID in autograd to prevent cache reuse for subsequent evaluations of the same expression tree.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
cb812c742c
Add [[nodiscard]] attributes to autograd functions.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
26f19e1429
Make automatic differentiation node types constexpr.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
aec6017195
When forming an autograd expression only copy parts that are rvalue references, store references to lvalues.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
a5c20bee5b
Apply gradient clipping.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
d103867558
Add memoization to the autograd expression evaluator.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
aa55692b97
Cross entropy loss.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
539bd2d1c8
Replace the old loss/grad calculation completely.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
b71d1e8620
Pass the new loss function to update_parameters
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
5a58eb803a
Loss func with autograd
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
541fb8177a
More utility in autograd.
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
6ce0245787
Basic autograd
2020-12-02 08:56:20 +09:00
Tomasz Sobczyk and nodchip
1322a9a5fd
Prevent false sharing of num_calls counter in the shared input trainer. Fix current_operation not being local to the executing thread.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
2aa7f5290e
Fix comparison of integers with different signedness.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
a97b65eaef
Fix compilation error with USE_BLAS
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
622e0b14c2
Remove superfluous example shuffling. Shuffling now only happens on reading.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
34510dd08a
Remove used examples asyncronously.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
0bee8fef64
Don't unnecessarily copy the batch part.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
e954b14196
Prefetch weights for feature transformer backprop to shared cache.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
8009973381
Special case for alpha=1 in saxpy, slight performance increase.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
49b2dcb1f3
Preallocate memory for unique_features. Keep the training_features temporary buffer as a thread_local so we reuse the storage.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
1c8495b54b
Remove handwritten saxpy because compilers optimize the second look anyway.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
15c528ca7b
Prepare feature transformer learner.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
a3c78691a2
Prepare input slice trainer.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
401fc0fbab
Prepare clipped relu trainer.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
774b023641
Add chunked for each with workers.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
cc11375f6d
Skeleton for new evaluate learner
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
0d4b803b08
Prepare trainer affine transform.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
4ea8572b6d
Add single threaded sgemm.
2020-11-30 08:54:53 +09:00
Tomasz Sobczyk and nodchip
92b14a5ba2
Add docs for transform.
2020-11-27 09:16:22 +09:00
Tomasz Sobczyk and nodchip
89294e2e4f
Add transform command. Add transform nudged_static subcommand.
2020-11-27 09:16:22 +09:00
Tomasz Sobczyk and nodchip
45e3335ee8
Add missing docs.
2020-11-23 19:22:11 +09:00
Tomasz Sobczyk and nodchip
9030020a85
Add smart_fen_skipping option to learn.
2020-11-23 19:22:11 +09:00
Tomasz Sobczyk and nodchip
ee13cfce67
Fix result assigned for a psvector when the positions are not continuous.
2020-11-23 08:32:08 +09:00
Tomasz Sobczyk and nodchip
3cee6881ee
Move the terminal position check to after qsearch, otherwise qsearch may end up in a terminal position.
2020-11-23 08:29:38 +09:00
Tomasz Sobczyk and nodchip
5f18c88b3d
Docs for book in gensfen.
2020-11-17 09:43:23 +09:00
Tomasz Sobczyk and nodchip
e1dbad47ce
Add support for opening book to gensfen.
2020-11-17 09:43:23 +09:00
Tomasz Sobczyk and nodchip
d4350a16f3
Add representation of an opening book.
2020-11-17 09:43:23 +09:00
Tomasz Sobczyk and nodchip
d793663188
Add docs for max_grad option for learn
2020-11-16 10:08:56 +09:00
Tomasz Sobczyk and nodchip
3dbc45bdfc
Add gradient clipping.
2020-11-16 10:08:56 +09:00
Tomasz Sobczyk and nodchip
50358e26c7
Fix searching terminal nodes in gensfen.
2020-11-15 22:18:13 +09:00
Tomasz Sobczyk and nodchip
00bc80c3c4
Add assume_quiet option to the learner.
2020-11-15 22:18:13 +09:00
Tomasz Sobczyk and nodchip
00797a3d86
add option ensure_quiet for gensfen that makes the generated position quiet
2020-11-15 22:18:13 +09:00
Tomasz Sobczyk and nodchip
691da3bdad
Add more information for factorizers at the start of training.
2020-11-14 18:47:22 +09:00
Tomasz Sobczyk and nodchip
4e1653d53a
Fix reliance on transitive includes for factorizers in trainer feature transformer. Add a file that includes all factorizers.
2020-11-14 12:35:12 +09:00
Tomasz Sobczyk and nodchip
69bc3ef9be
Output loss more often.
2020-11-14 12:33:25 +09:00
Tomasz Sobczyk and nodchip
a71623f74c
Add explicit read head seek to the start of the binpack file. Otherwise on MACOS the read head is placed at the end when app is specified.
2020-11-13 19:56:36 +09:00
Tomasz Sobczyk and nodchip
2a8576b804
Fix compilation issues.
2020-11-10 10:21:09 +09:00
Tomasz Sobczyk and nodchip
8069963c56
Update convert docs.
2020-11-10 10:21:09 +09:00
Tomasz Sobczyk and nodchip
5d88e7bce8
Add optional move validation to training data conversion. No longer rely on static initialization order for magics initialization.
2020-11-10 10:21:09 +09:00
Tomasz Sobczyk and Joost VandeVondele
ba35c88ab8
AVX-512 for smaller affine and feature transforms.
...
For the feature transformer the code is analogical to AVX2 since there was room for easy adaptation of wider simd registers.
For the smaller affine transforms that have 32 byte stride we keep 2 columns in one zmm register. We also unroll more aggressively so that in the end we have to do 16 parallel horizontal additions on ymm slices each consisting of 4 32-bit integers. The slices are embedded in 8 zmm registers.
These changes provide about 1.5% speedup for AVX-512 builds.
Closes https://github.com/official-stockfish/Stockfish/pull/3218
No functional change.
2020-11-07 16:49:49 +01:00
Tomasz Sobczyk and Joost VandeVondele
3f6451eff7
Manually align arrays on the stack
...
as a workaround to issues with overaligned alignas() on stack variables in gcc < 9.3 on windows.
closes https://github.com/official-stockfish/Stockfish/pull/3217
fixes #3216
No functional change
2020-11-04 19:52:42 +01:00
Tomasz Sobczyk and Joost VandeVondele
75e06a1c89
Optimize affine transform for SSSE3 and higher targets.
...
A non-functional speedup. Unroll the loops going over
the output dimensions in the affine transform layers by
a factor of 4 and perform 4 horizontal additions at a time.
Instead of doing naive horizontal additions on each vector
separately use hadd and shuffling between vectors to reduce
the number of instructions by using all lanes for all stages
of the horizontal adds.
passed STC of the initial version:
LLR: 2.95 (-2.94,2.94) {-0.25,1.25}
Total: 17808 W: 1914 L: 1756 D: 14138
Ptnml(0-2): 76, 1330, 5948, 1460, 90
https://tests.stockfishchess.org/tests/view/5f9d516f6a2c112b60691da3
passed STC of the final version after cleanup:
LLR: 2.95 (-2.94,2.94) {-0.25,1.25}
Total: 16296 W: 1750 L: 1595 D: 12951
Ptnml(0-2): 72, 1192, 5479, 1319, 86
https://tests.stockfishchess.org/tests/view/5f9df5776a2c112b60691de3
closes https://github.com/official-stockfish/Stockfish/pull/3203
No functional change
2020-11-02 19:41:17 +01:00
Tomasz Sobczyk and nodchip
987b6c98d4
Move the observed feature collection to the threaded part now that it can be done safely.
2020-11-01 11:02:44 +09:00
Tomasz Sobczyk and nodchip
c53be1b23f
Add specialized bitset for use in the trainer for observed features tracking.
2020-11-01 11:02:44 +09:00
Tomasz Sobczyk and nodchip
e8907bcfc4
Replace omp in trainer_feature_transformer
2020-10-31 11:54:03 +09:00
Tomasz Sobczyk and nodchip
db1b33d4ac
Optimize trainer clipped relu propagate
2020-10-31 11:52:51 +09:00
Tomasz Sobczyk and nodchip
b5714c4084
Parallelize input slice trainer backprop.
2020-10-31 11:52:26 +09:00
Tomasz Sobczyk and nodchip
941897ff2c
Optimize trainer clipped relu backpropagate.
2020-10-31 11:50:12 +09:00
Tomasz Sobczyk and nodchip
c96743c5bd
Optimize feature transformer backpropagation stats.
2020-10-31 11:49:29 +09:00
Tomasz Sobczyk and nodchip
2c10b1babc
Optimize feature transformer clipped relu.
2020-10-31 11:48:02 +09:00
Tomasz Sobczyk and nodchip
7bedf6c5ab
Specify the whole evalsave message because otherwise the first evalsave/0 triggers it.
2020-10-31 08:36:58 +09:00
Tomasz Sobczyk and nodchip
8c81bbd3db
Fix the counter in for_each_index_with_workers going out of scope before workers finish.
2020-10-31 08:36:58 +09:00
Tomasz Sobczyk and nodchip
a56d8124d8
Replace non-blas parts of trainers with our own blas-like routines.
2020-10-31 08:36:58 +09:00
Tomasz Sobczyk and nodchip
c56a4a36eb
Add our own blas-like routines that use stockfish's thread pool for parallelization.
2020-10-29 23:57:51 +09:00
Tomasz Sobczyk and nodchip
ee0917a345
Pass ThreadPool to update_parameters, propagate, and backpropagate.
2020-10-29 09:21:19 +09:00
Tomasz Sobczyk and nodchip
f1e96cab55
Align trainer arrays to cache line.
2020-10-29 09:12:50 +09:00
Tomasz Sobczyk and nodchip
8fac468259
Add a cache line aligned allocator.
2020-10-29 09:12:50 +09:00
Tomasz Sobczyk and nodchip
ec9e49e875
Add a HalfKA architecture (a product of K - king, and A - any piece) along with all required infrastructure. HalfKA doesn't discriminate kings compared to HalfKP. Keep old architecture as the default one.
2020-10-29 09:10:01 +09:00
Tomasz Sobczyk and nodchip
317fda2516
Cleanup eval saving and lr scheduling.
2020-10-28 23:08:05 +09:00
Tomasz Sobczyk and nodchip
680654b254
Add dots to output every epoch for progress visualization.
2020-10-28 09:36:43 +09:00
Tomasz Sobczyk and nodchip
f81fa3d712
Replace global_learning_rate with learning_rate local to the learner and passed to update_parameters as a parameter.
2020-10-28 09:36:07 +09:00
Tomasz Sobczyk and nodchip
cde6ec2bf2
Make all grad related functions in learn static. Pass calc_grad as a parameter.
2020-10-27 14:47:50 +09:00
Tomasz Sobczyk and nodchip
e4868cb59e
Move setting learn search limits to learner.
2020-10-27 14:47:07 +09:00
Tomasz Sobczyk and nodchip
c229929d26
Remove the position parameter from learn.
2020-10-27 00:35:43 +09:00
Tomasz Sobczyk and nodchip
a8066cd4a9
Rename elmo lambdas
2020-10-27 00:33:58 +09:00
Tomasz Sobczyk and nodchip
f7de49eb66
Create a collective parameter struct for learner.
2020-10-27 00:33:58 +09:00
Tomasz Sobczyk and nodchip
ba390a7f9a
Print the used factorizer when intializing training.
2020-10-27 00:32:39 +09:00
Tomasz Sobczyk and nodchip
e01397c674
Remove multi_think
2020-10-26 19:40:40 +09:00
Tomasz Sobczyk and nodchip
e515f1f61f
Move SfenWriter to a separate file
2020-10-26 19:39:58 +09:00
Tomasz Sobczyk and nodchip
65e443954a
Update expected gensfen finished responses.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
03abfae41f
Reorder members, renaming.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
6d4d20c4be
Cleaner printing and some renaming.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
d77b3d176e
Always flush sfen writer at the end of gensfen and when it is destroyed.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
21fac7c53c
A collective struct for gensfen parameters.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
cb61dc9c9b
Make sfen writer a part of gensfen.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
3f289546da
Make some gensfen members private.
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
821b655bc6
Move gensfen progress reporting from sfen writer to gensfen
2020-10-26 09:37:59 +09:00
Tomasz Sobczyk and nodchip
af238fe132
Rewrite gensfen to use stockfish's thread pool.
2020-10-26 09:37:59 +09:00
