Training performance fluctuate

[davidshen84]

Hi,

I have a simple flax module. While training on my desktop with a GPU, the tqdm lib shows the iteration speed fluctuates between 30 it/s and 40 it/s.

I wonder if it is expected that the iteration speed will fluctuate. One guess is that my CPU is not pushing data fast enough to my GPU.

Any suggestion where should I investigate?

[davidshen84]

Here's a recording of the training. At the bottom of the frame, you can see the it/s value fluctuates between 60 and 100. I wonder if the tqdm lib makes the it/s value look like this or if there was a repetitive delay when I ran the training loop.