Reported by Nicolas Ochem, Dec 23, 2008
Hello again, Now when I click on " source " I got the message : Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 35 bytes) in /home/www/indefero/src/IDF/Scm.php on line 62 My git repository is local. I already changed the max memory for php from 16mb to 64mb so the issue is somewhere else. The config file mention a debug mode but I couldn't find the output.
Comment 1 by Nicolas Ochem, Dec 23, 2008
I can browse individual commits through "latest updates", but "source tree" provokes this memory exhaustion.
Comment 2 by George McLachlan, Dec 23, 2008
Can you check your memory setting in php.ini. I had the same issue and had to increase it to 32m. I believe the default is 8 (maybe 16)
Comment 3 by Nicolas Ochem, Dec 23, 2008
The default is 16 on my system (ubuntu server with apache2 and php). I increased it to 64mb but same result. I assume a higher memory consumption should be abnormal anyway.
Comment 4 by George McLachlan, Dec 23, 2008
Hi I am running hardy and apache2. I checked the value in /etc/php5/apache2/php.ini and it does have a default of 16 when I changed it to 32 it works, although you need to restart apache for the changes to take effect
Comment 5 by Nicolas Ochem, Dec 23, 2008
I increased the size to 128mb and now it works, but it's very slow. That is a lot of memory consumption. I'm using a pretty old machine with 256mb of RAM as my dev server. Thanks for the help !
Comment 6 by Loïc d'Anterroches, Dec 23, 2008
I will create a special mode to log all the commands and allow us to debug such cases. You have most likely a large git tree. I will checkout the linux tree to perform some testing.
Comment 7 by Loïc d'Anterroches, Dec 23, 2008
I am checking out the Linux git tree to do hard testing on it. The goal is to be able to display the Linux git tree with only 8MB memory, so this will be a nice intensive optimization work. I may not be able to commit the changes before the end of the month because I am on holidays at the moment and I do not have all my SSH keys with me.
Comment 8 by Julien Issler, Dec 23, 2008
Hello, had the same issue with a Mercurial repository. If this helps...
Comment 9 by Nicolas Ochem, Dec 23, 2008
Good luck with that. My git repository is maybe bloated (it has several other repositories checked out inside it) but I assume it's still not as big as the linux kernel.
Comment 10 by Loïc d'Anterroches, Dec 23, 2008
Ok, the bad player is the one I expected, I am running this command: git log --raw --abbrev=40 --pretty=oneline 'master' This gives me a log with all the changes in the current branches. This is used to find the commit corresponding to each file so I can give for each file the corresponding change log in the tree. This can be huge and for the efficiency of the regular expressions, I am reverting it in memory and then parsing it with regex. This means that this way is not sustainable. Maybe I will need to cache all these details directly at each new commit and create a special script to populate the cache. At the moment, I am just limiting the log to the latest 3000 commits. Not perfect, but at least it is working. The only problem is that I am then missing stats for some files.
Comment 11 by Loïc d'Anterroches, Dec 23, 2008
Ok, I am not able to display the tree within 8MB while looking at the corresponding timestamp and message of each file, so I am limiting the number of commits in the log. This still requires sometimes more memory when looking at the details of each commit. I will need to take a look at a way to extract the memory profile with php xdebug to see in which conditions the memory usage goes up to 35MB. See commit 0a02916e812c8c76f12 for details. Could you please test and tell me if it is working for you?
Comment 12 by Loïc d'Anterroches, Dec 30, 2008
Ok, I am closing this ticket as it looks like everybody is happy. Do not hesitate to open it again or a new one if needed.
Comment 13 by Nicolas Ochem, Jan 5, 2009
I checked out the latest version and it still won't show the git tree under 64mb of memory, even if it feels more responsive. Not a big deal, I will leave my server configured this way (128mb)
Comment 14 by Loïc d'Anterroches, Jan 5, 2009
Nicolas, can you tell me how many files you have in your repository? The linux kernel has about 25000, so I would be pleased to know how many you have.
Comment 15 by Nicolas Ochem, Jan 5, 2009
Right now 1,112 items, totalling 32.9 MB. I believe the history is not huge either (I am far from 3000 commits). It's a ruby on rails app.
Comment 16 by Loïc d'Anterroches, Jan 6, 2009
Interesting. Thank you, I have done some more optimization in commit a6c4212, can you possibly checkout the latest version and try with 64mb of memory? It does not make sense to have a 33MB repository with less than 3000 commits requiring 128MB of RAM to display it.
Comment 17 by Loïc d'Anterroches, Jan 11, 2009
Nicolas, I think I have an idea about your problem. Have you created your git repository in one go, adding 1000 files in one commit? If so, it means that the first commit definition is huge, so huge that this can require a lot of memory to parse it.
Comment 18 by Julien Issler, Jan 11, 2009
Loïc, I think you're right, as I have the same issue with a Mercurial repository, with a huge initial commit. maybe you should do like in Trac, just displaying content of files that are modified, and not for those being created/removed. What about this behavior ?
Comment 19 by Loïc d'Anterroches, Jan 11, 2009
I will need to check, I have not yet tested this case, I will create a git repository with a huge import and see how to optimize its output.
Comment 20 by Nicolas Ochem, Jan 11, 2009
I can confirm that. There must have been something like 1000 files in my initial commit.
Comment 21 by Loïc d'Anterroches, Jan 12, 2009
Great, I will be able to reproduce the problem, thanks a lot for the help!
Comment 22 by Roger Sylte, Jan 15, 2009
I have the same issue, but my initial commit is just one file (.gitignore). My second commit is all the files in Wordpress 2.7. A few files, but not thousands...
Comment 23 by Loïc d'Anterroches, Jan 15, 2009
Thanks for the extra details, I will do some further testing, most likely next week.
Comment 24 by Patrick Georgi, Jan 17, 2009
I rewrote the tree parser to use popen, to use less memory. Downside: it's a bit slower. On the other hand, a _huge_ tree (34000 files since base rev, 9000 revs) showed without problems here.
Comment 25 by Patrick Georgi, Jan 17, 2009
The patch above is not enough get. getCommit() in the same file takes way too much memory, too (it has to cope with 51mb of "git show" output, after all). And when I changed it to use popen, as in the patch, it's way too slow, and _still_ takes too much memory (and I already allow 128mb)
Comment 26 by Loïc d'Anterroches, Jan 17, 2009
Partial fix in commit 61bc7a7 thanks to good tips from Patrick aka. Oxygene.
Comment 27 by Loïc d'Anterroches, Jan 17, 2009
Fixed in commit 835eab9c2419492ec3a39f445, see issue 103 for the performance one.
Comment 28 by Loïc d'Anterroches, Jan 17, 2009