Skip to content

Fix Tensorflow 2.17 training #191

Merged
lfoppiano merged 30 commits into
tensorflow-2.17-updatesfrom
tensorflow-2.17-training
Feb 28, 2026
Merged

Fix Tensorflow 2.17 training #191
lfoppiano merged 30 commits into
tensorflow-2.17-updatesfrom
tensorflow-2.17-training

Conversation

@lfoppiano

@lfoppiano lfoppiano commented Feb 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This incremental PR fixes the issues I with the training:

  • configuration of nb_workers, it seemed causing problems with LInux and GPUs
  • Breaking change: removed pickled arrays in the LMDB storage
  • Fixed the LMDB access using multiple workers
  • Added scripts for running training and training/evaluation on a SLURM cluster
  • Fix issue with tensorflow and pytorch being built on different cuda versions. Now everything should work with cuda 12
  • Fixed classification classes and completed migration to tensorflow 2.17

@lfoppiano
fix: update macos requirements
35b93dc
@lfoppiano
fix: disable multiprocessing when training with mac
2edb191
@lfoppiano
fix: prevent error when creating existing embedding LMDB directory by…
f270733 
… allowing existing paths.
@lfoppiano
feat: store embeddings in float32 instead of pickle
b29108a
@lfoppiano
fix: make embeddings pickable
9b9a552
@lfoppiano
fix: remove disable of multiprocessing for mac os
df5de56
@lfoppiano
feat: reopen embeddings in multithread to enable concurrency in LMDB …
3fabecb 
…accesss
@lfoppiano
feat: add num_workers parameter for configurable data loading
e26239b
@lfoppiano
feat: add slurm scripts
73936b5 
Signed-off-by: Luca Foppiano <luca@foppiano.org>
@lfoppiano
fix: adjust paths
c45d121
@lfoppiano
feat: add distributed scripts for SLURM
b060464 
Signed-off-by: Luca Foppiano <luca@foppiano.org>
@lfoppiano
feat: train incrementally after the first round
1fcafc8
@lfoppiano
feat: add num_workers support for header model in training and evalua…
f0a268f 
…tion scripts
@lfoppiano
chore: update the data, use a sortable date in the name
a68e433
@lfoppiano
chore: remove segmentation model
ebe6836
@lfoppiano
feat: get the latest training data when no input file is specified
630e902
@lfoppiano
feat: include citation model in the one with num-workers = 6
a9146fc
@lfoppiano
Re-trained models with the new tensorflow version
3037c65 
Signed-off-by: Luca Foppiano <luca@foppiano.org>
@lfoppiano
fix: refine scripts
98b508e
@lfoppiano
fix: update scripts
f34fc27
@lfoppiano
chore: update imports to use the right keras version
a1db014
@lfoppiano
chore: cleanup scripts
dfbb116
@lfoppiano
fix: move from srun to sbatch
16be6f3
@lfoppiano
fix: add fake import to get the correct CUDA
66952db
@lfoppiano
chore: add proper communication
d1e97e2
@lfoppiano
fix dependencies
32ca206
@lfoppiano
fix: license and copyright distributed script
3674e43
@lfoppiano
feat: retrain models
2467d92 
Signed-off-by: Luca Foppiano <luca@foppiano.org>
@lfoppiano
update script
e49d0ee 
Signed-off-by: Luca Foppiano <luca@foppiano.org>
@lfoppiano
feat: add script for retrain the citation models
87a69dc
@lfoppiano lfoppiano merged commit e223d1d into tensorflow-2.17-updates Feb 28, 2026
2 checks passed
@lfoppiano lfoppiano deleted the tensorflow-2.17-training branch February 28, 2026 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lfoppiano