Hello, all
I am new to Pytorch and I meet a strange GPU memory behavior while training a CNN model for semantic segmentation. Batchsize = 1, and there are totally 100 image-label pairs in trainset, thus 100 iterations per epoch. However the GPU memory consumption increases a lot at the first several iterations while training.
[Platform] GTX TITAN X (12G), CUDA-7.5, cuDNN-5.0
torch.backends.cudnn.enabled = False
torch.backends.cudnn.benchmark = False
Then GPU memory consumption is 2934M -- 4413M -- 4433M -- 4537M -- 4537M -- 4537M at the first six iterations.
torch.backends.cudnn.enabled = True
torch.backends.cudnn.benchmark = True
Then GPU memory consumption is 1686M -- 1791M -- 1791M -- 1791M -- 1791M -- 1791M at the first six iterations.
Why GPU memory consumption increases while training, especially, increases so largely while no cuDNN? (In my opinion, GPU memory consumption won't increase while the CNN has been build and starts training)
Does anyone meet the same problem? Or could anyone give some help?
This is the code snippet:
def train(train_loader, model, criterion, optimizer, epoch):
batch_time = AverageMeter()
data_time = AverageMeter()
losses = AverageMeter()
# switch to train mode
model.train()
end = time.time()
for i, (input, target) in enumerate(train_loader):
# measure data loading time
data_time.update(time.time() - end)
target = target.long()
input = input.cuda(async=True)
target = target.cuda(async=True)
input_var = torch.autograd.Variable(input)
target_var = torch.autograd.Variable(target)
# compute output
output = model(input_var)
loss = criterion(output, target_var)
# record loss
losses.update(loss.data[0], input.size(0))
# compute gradient and do SGD step
optimizer.zero_grad()
loss.backward()
optimizer.step()
# measure elapsed time
batch_time.update(time.time() - end)
end = time.time()
if i % args.print_freq == 0:
print('Epoch: [{0}][{1}/{2}]\t'
'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
'Loss {loss.val:.4f} ({loss.avg:.4f})'.format(
epoch, i+1, len(train_loader),
batch_time=batch_time,
data_time=data_time,
loss=losses))
Hello, all
I am new to Pytorch and I meet a strange GPU memory behavior while training a CNN model for semantic segmentation. Batchsize = 1, and there are totally 100 image-label pairs in trainset, thus 100 iterations per epoch. However the GPU memory consumption increases a lot at the first several iterations while training.
[Platform] GTX TITAN X (12G), CUDA-7.5, cuDNN-5.0
Then GPU memory consumption is 2934M -- 4413M -- 4433M -- 4537M -- 4537M -- 4537M at the first six iterations.
Then GPU memory consumption is 1686M -- 1791M -- 1791M -- 1791M -- 1791M -- 1791M at the first six iterations.
Why GPU memory consumption increases while training, especially, increases so largely while no cuDNN? (In my opinion, GPU memory consumption won't increase while the CNN has been build and starts training)
Does anyone meet the same problem? Or could anyone give some help?
This is the code snippet: