I was performing adversarial attacks with pytorch and I was not able to reproduce my experiments.
Finally I learned about non deterministic cudnn operations Discussion Forum-1 Discussion Forum-2 and final post
You can simply try to reproduce output from the following script (Ubuntu OS, Nvidia 1080 Ti GPU).
If i change my model to a simple resnet (and appropirately the target and loss function) then it results are deterministic. (So I guess maybe Upsample is the culprit, I don't know!!).
Here is my complete script :
import torch
import torch.nn as nn
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
class simple(nn.Module):
def __init__(self):
super(simple, self).__init__()
self.seq = nn.Sequential(
nn.Conv2d(3,64,7,2,3),
nn.Conv2d(64,64,3,1,1),
nn.MaxPool2d(2,2),
nn.Conv2d(64,64,3,1,1),
nn.MaxPool2d(2,2),
nn.Upsample(scale_factor = 2),
nn.Conv2d(64,64,3,1,1),
nn.Upsample(scale_factor = 2),
nn.Conv2d(64,1,1)
)
def forward(self, x):
return self.seq(x)
gpuid = 0
###
model = simple().to(gpuid)
lossfn = nn.MSELoss()
img = torch.randn(1,3,256,256).float().to(gpuid)
###
out = model(img)
target = torch.randn(1,1,128,128).to(gpuid)
print(out.norm())
def test():
adv = torch.zeros_like(img).to(gpuid)
for i in range(100):
inpcopy = torch.clamp(img + adv, 0, 1)
inpcopy.requires_grad = True
loss = lossfn(model(inpcopy), target)
loss.backward()
adv = torch.clamp(torch.sign(inpcopy.grad) + adv, 0, 1)
inpcopy = torch.clamp(img + adv, 0, 1)
out = model(inpcopy)
outNorm = float(out.norm())
print(outNorm)
while True:
test() # Since there is no randomness the output perturbed image norm must be consistent
Can someone please look into it. And if it is problem of Upsample is there some hacky solution out of it??
I was performing adversarial attacks with pytorch and I was not able to reproduce my experiments.
Finally I learned about non deterministic cudnn operations Discussion Forum-1 Discussion Forum-2 and final post
You can simply try to reproduce output from the following script (Ubuntu OS, Nvidia 1080 Ti GPU).
If i change my model to a simple resnet (and appropirately the target and loss function) then it results are deterministic. (So I guess maybe Upsample is the culprit, I don't know!!).
Here is my complete script :
Can someone please look into it. And if it is problem of Upsample is there some hacky solution out of it??