I had a grader config that was off by a level - I'd put config into the grader section, but got no error - it was just silently skipped:

graders:
  - name: "my grader"
    type: "code"
    # (this should be under `config`, not a root element)
    assertions:
      - "hello" 

We should have more validation in place to detect obvious mistakes like this (for instance, there's no reason to have a code grader without any assertions.