That's a good question and I'm not sure. I'll give some context to make sure we're on the same page.

As it stands, if the learner's code raises an exception, then all the tests fail. Prior to this PR, only exceptions which prevented the code being evaluated generated a special message, because the test messages (e.g. "foo() should equal 5") would be misleading (the problem is you've got bad indentation or a syntax error).

The 'fix' here is a little sketchy because, in principle, you might want to write a test that fails with a NameError.

I think the best approach is to extend the test runner, so it can differentiate between "initial learner code raised an exception" and "at some point during user code evaluation or test evaluation an exception was raised" (which is how it currently behaves.). That's not something I will have time to finish today. If we want an immediate fix, we can use this, but if it can wait a week or so, I can update the runner.

Also, to directly answer your question, if we update the runner, then I think we'd want to show the special message whenever the learner's code raised any kind of python exception during initial evaluation.

@ilenia-magoni @Dario-DC any thoughts?

It can be a bit confusing seeing a special message just for some errors but probably it can wait. I think it would be nice to show a dedicated message any time the user's code raises an exception (and I would keep a different message for IndentationErrors).

Thanks, Dario. Just to confirm: you're okay with this PR as a temporary measure and the proposed test-runner updates that would replace it?