Hello, I would really like the functionality to use a different activation function on the output of the RNNs. I have found ReLU more useful in classification models because, for instance, tanh output from an LSTM makes it easy for a subsequent softmax-linear layer to produce values near .999. Just throwing the idea out there incase someone wants to include that in an upcoming release.

cc @zou3519