Per discussion with several users, there are some possible renames we could do to clarify the internal architecture:

• Rename rllib.evaluation.PolicyGraph to rllib.policy.Policy. Similarly, rllib.evaluation.TFPolicyGraph becomes rllib.policy.TFPolicy and so on. The files are moved to a new rllib/policy dir.

• Move the rllib/agents directory to rllib/train.

• Add TF qualifier for consistency with Torch policies
  A3CPolicyGraph => A3CTFPolicy
A3CTorchPolicyGraph => A3CTorchPolicy

• Rename PolicyEvaluator to RolloutWorker
  deprecate compute_gradients() / apply_gradients()

• Consolidate [local_evaluator, remote_evaluators] to single WorkerSet object

  workers = WorkerSet(...)
  local_evaluators => workers.local_worker()
  remote_evaluators => workers.remote_workers()
  

For the most part, I think we can do these renames without breaking backwards compatibility by leaving aliases behind. Though, I'm not sure how easy moving an entire directory is.