I have an import script that has to import ~500,000 records from and old system of mine to my new one, but the import requires a bit of PHP processing (for field validation mostly, such as phone numbers, so I cannot use MySQL to do my processing).
Two things I need are the ability to watch the progress of the import, and for the PHP script not to run out of memory (so I can't just select all the rows at once).
So I've accomplished this by creating a resolution variable and a counter that increases (sort of like a page for paginating data) that will grab the first $Resolution amount of rows, do the processing, import the rows to the new system, and then loop back around and grab the next set of rows.
Now the problem lies when I have:
select `Column` from `table` where `Something`=1 limit 0, 1000;
That takes say, 5 seconds. But so now on the next loop I have:
select `Column` from `table` where `Something`=1 limit 1000, 2000;
Which takes 10 seconds because it has to recalculate the rows from the first loop to be able to get the next set of rows. As you can see, this gets super slow super fast, so I would I go about this without incurring the extra calculations from previous loops?
答案 0 :(得分：0)
I would suggest adding an ORDER BY xxx clause, with an index on the (Something, xxx) columns.
A LIMIT without an ORDER BY really has no value here since you would have no control over which rows will appear in each set.