Ok, I think I got it to work on my system. Below code generates a 12MB text file and pipes it into a list without locking up. The trick is that the reader is in a separate thread (and defined as daemon so that it is killed with the process x):

#!/usr/bin/env python
import subprocess
from threading import Thread
import time

from random import choice
import string
def gen_line(length=8, chars=string.letters + string.digits):
   return ''.join([choice(chars) for i in range(length)])

print "generating text file"
with open('/tmp/text','w') as f:
  for i in range(200000):
    f.write("%i: %s\n" % (i, gen_line(50,string.ascii_letters)))
print "done"

def reader(f,buffer):
   while True:
     line=f.readline()
     if line:
        buffer.append(line)
     else:
        print "reader has finished"
        break

linebuffer=[]
x=subprocess.Popen(['cat', '/tmp/text'],stdout=subprocess.PIPE)
t=Thread(target=reader,args=(x.stdout,linebuffer))
t.daemon=True
t.start()

# below is for near real time printing the output, could just use t.join() here
while True:
  time.sleep(0.1)
  if len(linebuffer):
    print "line " + linebuffer.pop(0),
  else:
    if not t.isAlive() and len(linebuffer) == 0:
      print "finished"
      t.join()
      break
    print "waiting for output"
    time.sleep(1)

output:

generating text file
done
line 0: SsvDEOgNhRzYMlZpCCQnYPlQxxAToOgteSkwXNjPcfOAlCYxiq
line 1: fdvzHJDegCKHGvdJgrtTCVWZlUhMGTneApoouDMODqGWNcdMTZ
line 2: XxlLEvAFQwJhahrrAfRSuitvETdbuzJHCngmTvTMdkukdsxEEZ
line 3: RpwJNHMgsIFAYUCxrnwbthMwApAbPUtwcAFQjiJLCPdhMPNFNo
line 4: RMwHkEhXfHFmGslZUDupfYtwiUVCIvkBGkTjjiYbnKlCRELUeq
line 5: NVklgzEbrtkaCDQNjlHzPiOVQSBOFwtYUpWlcTzJwcvCzYImKF
line 6: ouILCjAlaBVTcHmEaiabWHwmfphJSmRgtbwrGeWeewvuQwYPmg
line 7: CJBvyLQGCDwuPccDxNXVdButPdPauRXhmZeSoDSnHTHVGopfml
line 8: UagaFjWCXgEIRmmfsMZdDgVgEzcBQiLoOPIErMliokxhGSrFLk
line 9: resxJZznLgOszSPrZZmHpQiicYHNxKPVusPLbhsLAYaARxnZGl
line 10: OdkvCivOsFzsTrbMOPAyrfzalfkXDdEjDzTpTBzJNMHkkWsOQC
line 11: UluadxkUPqhbBFOwAPxaPnDcCettfPvBZbtlTJOeLApOjGSmlt
line 12: pdTLsvPXJkOkuTdpoRfgreACtMXbXAnuwNzqcCPPFQarsuIdRg
line 13: foetrXhjLIibZqcXKlHPItSRQgEqgVaxBPdCtAwyoFgcumZhHh
line 14: bcxnlgYutRWIzjrQKfYUImrmVqtIWfmoflPAMCKKXLFwllfnEa
line 15: YQJHgSnMZXfwmEPIxByCrTCUlskIuLqZJiRkeiHqRdQpqJeFtW
line 16: jNqHOUFnxNYoMjRYcKLlJXKOAtlUxFoeXZlFFvAekbvfSUNVYn
line 17: BfVnCjNkVupSQJUeNtqYblDqRGSIDoAJJMQxnAKygLOBytwCyk
line 18: hGOiWNmFolRpPmNsUCCBvGbJfeWjfMiMSSqATMyNlQKcxLYqlP
line 19: KpQTCzGdCITluWIWnhCSKXPykOTtafAeWRcckLoFnstbKgWHqc
line 20: BVQNJxfRnDnCqzXhhfpdddtvJsNqOLbmKEniWhdMQaZoIaXBNc
line 21: pLAAVKGNJYIrQicHjueZQQvdEXnNsgcpUlofRKACFxuNrXhAKh
line 22: MzgXpCUEHHUpOkTHexaSYHcKIluIoQPbKHIdGhjtMVGkiSsVwj
reader has finished
line 23: LifDAUBpniilQCEDTvXJKNOPcadskMJHqbAqgSpsUjmGaYFIak
line 24: sQICwsfpHEYluHMnSbsziZdtwoMPcugReGySIxpgZQsIQatiYT
line 25: NhuyHWsPJlzlLCMzhziVroTJGkwwWFghssNiQVWPEsYbTtbdcY
line 26: fDMQiTPnxVoYvDjMgfPdgogQHyYlvburCxaSPCnFntzKwSuytF
line 27: epZEeEEXndsvKNQHfCREqusFaeuocXXDlEwjGLkvNjWMhczvrm
line 28: bfNfuUzjRqibwKXqtQYiyzYBkkVTjZMtvOqSbojHwKxGTVnWTs
line 29: bSzEMLZRtvyfmniZxkRLqWMzaTEQXyBYOQhjSIATYMLiYZSdqa
line 30: ZytWQDRMIroEqoSQjhretWdNSSPxkLCzPILZDLqkJiZnPEXmDS

Communicate also has issues, depending on the version of python.

scripts: run.command blocked if command produces lots of output #927

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions