Describe your feature request

The current behaviour of memory requests is very confusing:
https://docs.ray.io/en/master/memory-management.html#memory-aware-scheduling

Case A:

@ray.remote(memory=500 * 1024 * 1024, cpu=2)
def some_function(x):
    pass

In this case, both memory and cpu requests are used for scheduling, but are not enforced. The function can use more than 2 cpu or 500MB if it wishes.

Case A:

@ray.remote(memory=500 * 1024 * 1024, cpu=2)
class SomeActor:
    pass

In this case, both memory and cpu requests are used for scheduling, but memory requests are enforced. If the actor starts consuming more than 500MB RAM, following remote calls will result in an OOM.

The behaviour is not consistent in many ways: It is different for remote functions and actors. But also different between resource types (cpu, memory, ..).

I would propose to make the enforcing of memory optional or turn it off by default (as suggested by @ericl in chat)
To make sure the behaviour is consistent for all resource types.

Thanks for taking this into consideration! Pj