Allowed memory size exhausted #1044
Description
Hello,
i am new at flow-php/etl and give it a try, because i had very very big CSV Files to import into database.
Till yet i import this via "LOAD DATA INFILE" - but this have some problems if one row is "buggy" it throw awawy the complet chunk.
With flow-php i ran with one of the smaller files already in a "Allowed memory size exhausted" Error. :-(
What i am doing wrong:
data_frame()
->read(from_csv('my.csv',
with_header: false
))
->rename("e00", "country")
->rename("e01", "postal_code")
->rename("e02", "name")
->rename("e03", "admin1_name")
->rename("e04", "admin1_code")
->rename("e05", "admin2_name")
->rename("e06", "admin2_code")
->rename("e07", "admin3_name")
->rename("e08", "admin3_code")
->rename("e09", "latitude")
->rename("e10", "longitude")
->rename("e11", "accuracy")
->batchSize(25000)
->load(to_dbal_table_insert($this->db, 'my_table_name'))
->run();Batch Size is choosen low - but still break the first batch.
Is there a possibilite to "stream" or chunk the read from_csv?
Had the hope that flow-php will take care about that.
Any help is really appreciated, because i can´t find something suitable in th Documentation :/