Recently changed jobs, changed into the software establishment of the station, and our establishment of the station software is a problem with the template system, suggesting that the memory beyond the maximum. Memory problems are most simple way is to modify the php.ini to improve memory_limit, but with the increasing number of template required memory goes beyond the upper limit, so the boss let me take a look at this issue, look at me what a good way.
I got this problem, first read the original code, ideas, analysis of the main causes of the problem, problem page is simply a template display page 16 per page templates, to do page processing, and the background treatment is not simple, the first template data is not read from the database, but from the server to obtain a serialized string, remove all of the template is an array of information, information is then cached (after the first reading of the cache) and then array to filter, and paging operations, remove the template display 16.
Then analyze the problem, it is clear where the problem is with all of the templates out of the array of information on this issue is a huge number of templates, but in fact only 16 per needs only, but here it all added to achieve memory.
Problems found, how to solve it? So I asked the former employees, listen to their views. They provide a lot of views dealing with the template into the database, paging and other operations have to put all of our servers processing, but programs need to expand our back-end server processing improvements and changes things too much, the problem of the introduction of more, how do?
1. Lazy loading
Then I thought, since the problem lies in removing all the serialized data, if we can in the file, just remove the data we need good, because I have java experience, I first thought was XML SAX processing methods, But I'm not familiar with PHP, XML processing, to no avail.
But the right gave me the idea, a lot of PHP memory is too large problems due to PHP developers for the convenience, all required data pre-loaded into memory (array), then by addressing such issues is the result of memory required far exceeded its actual needs, has led to excessive memory problems. The solution is lazy loading, so that the use of data loaded into memory when processing.
The problem here is clearly to the issue of sequence, PHP's serialize mechanism is a strategy of pre-loaded, does not support lazy loading. Need to find a support serialization of lazy loading approach, a template can be a time to read the contents of treatment, and soon appeared in CSV format, CSV to support this approach to read in units, and for the original system mechanism does not require any major changes, so will the original system csv serialization changes.
2.Iterator interface uses
csv use while reading to traverse, while the original system uses an array to traverse, the system code filter conditions, involving a lot of code changes, and the previous array using the key => value pairs to achieve, and csv can only use 0, 1,2 digital access. To minimize the original code changes, we first made a deal on the csv, it's the first line array of key values, starting from the second line shows the actual value of the template value, location and key correspondence, which also meets the future scalability.
Foreach order and style of the original code as far as possible match, we use the operation of Iterator encapsulates the csv and csv completed inside the key-> value of the merger. Thus, for code migration is almost negligible, the greater the advantage of csv file size than the file size of the serialized under a lot of, some people may worry about performance issues.
After testing our existing template 2000, the speed does not exceed 1 second, the next number to be increased to 20,000 templates, traverse speed of no more than 2s, are also acceptable range. So the problem is solved.
Recently, we made a simple ORM framework of a memory exceeds the maximum, also owing to the value of all the databases into the memory array, so I lazy load the spear in one hand, shield in one hand Iterator once again into solve the problem of war.