We have removed this restriction. Please check out the code at head, or in the upcoming 1.4 RC. Please reopen if there are issues with it.

_Originally posted by @martinwicke in https://github.com/tensorflow/tensorflow/issues/12367#issuecomment-335320352_