The final commit (4212e3d) has been tested based on:

• assuming we have evaluate module under /DeepSpeedExamples/BingBertSquad/ directory
• assuming we update the run_squad_deepspeed.sh script to accept a config file; following is what I used in my test:

"
if [ $# = 5 ]
then
config_json=$5
else
config_json=deepspeed_bsz24_config.json
fi
"
Even with the current version in prod, we should be able to run; however test may fail.
Further, the runner script is currently getting updated and we can do the final update after other updates are finalized.
Another idea could be to have a separate runner script under tests.

This looks great Arash, thanks! Let's fix the formatting so the tests will pass and then we can merge it and integrate it into our overnight tests.

See this for more details on getting the formatting to pass: https://github.com/microsoft/DeepSpeed/blob/master/CONTRIBUTING.md#prerequisites