Skip to content

Raylet Memory usage growing  #4657

@ghost

Description

System information

  • OS Platform and Distribution: Ubuntu 16.04
  • Ray installed from: binary
  • Ray version: 0.6.5
  • Python version:3.6.8
  • Exact command to reproduce
import numpy as np
import ray
import time
ray.init(redis_max_memory=100000000)


@ray.remote
class Runner():
    def __init__(self, dataList):
        self.run(dataList)

    def run(self,dataList):
        while True:
            dataList.put.remote(np.ones(10))

@ray.remote
class Optimizer():
    def __init__(self, dataList):
        self.optimize(dataList)

    def optimize(self,dataList):
        while True:
            dataList.pop.remote()

@ray.remote
class DataServer():
    def __init__(self):
        self.dataList= []

    def put(self,data):
        self.dataList.append(data)

    def pop(self):
        if len(self.dataList) !=0:
            return self.dataList.pop()
    def get_size(self):
        return len(self.dataList)


dataServer = DataServer.remote()
runner = Runner.remote(dataServer)
optimizer1 = Optimizer.remote(dataServer)
optimizer2 = Optimizer.remote(dataServer)

while True:
    time.sleep(1)
    print(ray.get(dataServer.get_size.remote()))

Describe the problem

The memory usage is constantly increasing even though the list is cleared repeatedly until the program crashes. Does anybody know why this is happening ?

Source code / logs

Traceback (most recent call last):
File "/home/test.py", line 48, in

File "/home/anaconda3/envs/py3/lib/python3.6/site-packages/ray/worker.py", line 2310, in get
value = worker.get_object([object_ids])[0]
File "/home/anaconda3/envs/py3/lib/python3.6/site-packages/ray/worker.py", line 531, in get_object
self.current_task_id,
File "python/ray/_raylet.pyx", line 254, in ray._raylet.RayletClient.fetch_or_reconstruct
File "python/ray/_raylet.pyx", line 59, in ray._raylet.check_status
Exception: [RayletClient] Connection closed unexpectedly.

Process finished with exit code 1

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionJust a question :)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions