Simplified Way of Processing Large Data Using Chunk in Laravel
Uploaded by
Dinesh SutharSimplified Way of Processing Large Data Using Chunk in Laravel
Uploaded by
Dinesh SutharSimplified way of
processing large
data using chunk() in
laravel
Learn More
TABLE OF CONTENTS
What do we mean by large data ?
Problems with processing large data ?
What do we mean by data chunks ?
Array chunks in PHP
Best practices to free unused memory in PHP
Unset a php variable
Nullify a php variable
Close database connection in the end manually
https://decodeweb.in
Simplified way of processing large data using chunk() in
laravel
Source: https://decodeweb.in/php/simplified-way-of-processing-large-data-using-chunk-in-laravel/
What do we mean by large data ?
As a laravel developer, by large data I mean collection of 1000 or more rows from a single data model that is,
a database table; specifically on a MySQL / MariaDB server.
Problems with processing large data ?
There may be situations when a single table holds millions of records and you want to migrate those records to
another table after processing. As far as MySQL is concerned here in this context, PHP has some limitations
in terms of memory allocation and script execution time to process those rows.
PHP takes data into its allocated memory on server and then process further, also it automatically lends some
extra bytes if it thinks it would be sufficient for the script to run and exit successfully. All these actions are
handled automatically until the records are less 1000 but once it crosses this limit server resources become
deficient of memory.
In short, server stops responding to other requests and many script is exited abruptly, which we call it as a
memory leakage. In return, we get a Fatal error: Allowed memory size of <some_number> bytes
exhausted (tried to allocate <some_number> bytes)
On the other hand, when the calculation during data processing is way too much complex then script would
exit abruptly because each script has a definite execution time allocated.
Now you may wonder, that we can manually change these settings in php.ini file but I do not recommend to
mess with any configuration file on the production server unless it is the last resort.
What do we mean by data chunks ?
What if we divide whole collection into chunks, let's say a collection of 25K rows and we divide them in chunks
of 500 rows at a single instance, fair enough ? Yes, because now rows are 500 only at given time. The PHP
script will take only 500 rows out of 25K rows. Let's see some examples related to arrays first.
Array chunks in PHP
In PHP, there is an array helper function, array_chunk().
<?php
$arr_a = [1, 2, 3, ....., 5000];
foreach(array_chunk($arr_a, 500) as $x){
// array_chunk() will divide $arr_a into smaller array as [[1, 2, 3...,
500],[501, 502, .... , 1000] and so one till 5000]
//do some stuff with $x;
}
Laravel Eloquent chunks
In Laravel, collection can be chunked using chunk() method for example,
<?php
Namespace App\Http\Controller;
Use User;
Public function processUsers()
{
User::chunk(500, function($chunked_collection){
//do some stuff with $chunked_collection
});
}
Best practices to free unused memory in PHP
PHP provides many ways to flush unused memory so that the next line of code can be accommodated into
available memory.
Unset a php variable
Using unset() method, we can free up the memory a variable is holding up, for example,
<?php
$a = 500;
unset($a);
var_dump($a); // prints null
?>
Nullify a php variable
Another way to free up memory is to reset the variable’s value to null explicitly, but it is slower than unset(), I
do not recommend it.
Close database connection in the end manually
While using core PHP, always make sure to close database connection after processing has been done on
data. For example:
<?php
$con=mysqli_connect("localhost","my_user","my_password","my_db");
// ....some PHP code...
mysqli_close($con);
?>
