- memory_limit\n- post_max_size\n- upload_max_filesize\n- max_execution_time<\/code><\/pre>\n\n\n\nCode<\/h3>\n\n\n\n
Once our PHP configuration is done, you might to restart the server or PHP itself. The next step is our code. We have to write a proper way not to run out of memory.<\/p>\n\n\n\n
Normally we used to read the csv files like this.<\/p>\n\n\n\n
$handle = fopen($filepath, \"r\"); \/\/getting our csv file\nwhile ($csv = fgetcsv($handle, 1000, \",\")) { \/\/looping through each records\n\/\/making csv rows validation\n\/\/ inserting to database\n\/\/ etc.\n}<\/code><\/pre>\n\n\n\nThe above code might be ok for a few records like 1000 to 5000 and so on. But if you are working with 100K 500K records , the while loop will consume lot of memory. So we have to chunk and separate the loop to get some rest time for our program.<\/p>\n\n\n\n
$handle = fopen($filepath, \"r\"); \/\/getting our csv file\n$data = []; \nwhile ($csv = fgetcsv($handle, 1000, \",\")) { \/\/looping through each records\n $data[] = $csv;\/\/ you can customize the array as you want\n \/\/we will only collect each 1000 records and do the operations\n if(count($data) >= 1000){\n\u3000 \/\/ do the operations here\n \/\/ inserting to database (If you already prepared the array in above, can directly add to db, no need loops)\n \/\/ etc.\n \n \/\/resetting the data array\n $data = [];\n }\n\n \/\/if there is any rows less than 1000, keep going for it\n if(count($data) > 0){\n \/\/ do the operations here\n }\n}<\/code><\/pre>\n\n\n\nAbove one is a simple protype to run the program not to run out of the memory, our program will get rest time for each 1000records. <\/p>\n\n\n\n
Here is an another way using array_chuck<\/strong> and file<\/strong> function<\/p>\n\n\n\n$csvArray = file($filepath); \/\/this will output array of our csv file\n\/\/chunking array by 1000 records\n$chunks = array_chunk($csvArray,1000);\n\n\/\/ Then lets store the chunked data files in somewhere\nforeach ($chunks as $key => $chunk) {\n file_put_contents($path,$chunk);\n}\n\n\/\/get the files we have stored and can loop through it\nfiles = glob('path\/path'.\"\/*.csv\");\n\nforeach ($files as $key => $file) {\n $filer = fopen($file, \"r\");\n while ($csv = fgetcsv($filer, 1000, \",\")) {\n \/\/ do the operations here\n }\n\n \/\/delete the file back\n unlink($file);\n}<\/code><\/pre>\n\n\n\nPlease don’t forget to close the files back fclose<\/code> if you have done the file operations.<\/p>\n\n\n\nLooping content<\/h3>\n\n\n\n
One more thing to keep in mind is we have to take care of the codes we put inside loops. If there is any <\/p>\n\n\n\n
- Database calls or <\/li>
- Third party API calls,<\/li><\/ul>\n\n\n\n
it will surely slow down the performance and consume the memory more. So if you can put these calls outside of the loops, our program will be much more efficient.<\/p>\n\n\n\n
I am sure there might also be some other work arounds or some packages to handle about this issue behind the scence.<\/p>\n\n\n\n
Yuuma<\/p>\n
\n <\/g:plusone><\/div>\n
![\"\u3053\u306e\u30a8\u30f3\u30c8\u30ea\u30fc\u3092\u306f\u3066\u306a\u30d6\u30c3\u30af\u30de\u30fc\u30af\u306b\u8ffd\u52a0\"](\"\/\/b.hatena.ne.jp\/images\/entry-button\/button-only@2x.png\")
<\/a>