Out of memory when recursive viewing 20.000+ videos
Moderators: XnTriq, helmut, xnview
Out of memory when recursive viewing 20.000+ videos
When doing thumbnailing for videos with recursive folder view anabled and "create thumbnails for whole folder" enabled, memory usage spiked and causes a lot of unwanted behavior including file system errors.
So the implementation should be improved. Thumbnails should be created one by one one file after another and once it is done released from memory. Despite a small queue to know which file is next, I don't see the need for large memory consumption even with a large number of files being thumbnailed.
So it looks like the implementation should be improved.
So the implementation should be improved. Thumbnails should be created one by one one file after another and once it is done released from memory. Despite a small queue to know which file is next, I don't see the need for large memory consumption even with a large number of files being thumbnailed.
So it looks like the implementation should be improved.
Re: Running out of memory
It's done like that. Video thumbnail with ffmpeg? do you see the gdi object count growing in process manager?
Pierre.
Re: Running out of memory
Yes, video thumbnail with ffmpeg. I don't know about a gdi object, But what I saw is that the XnViewMP process requested 3.5 GB of memory and there is absolutely no need for that.xnview wrote:It's done like that. Video thumbnail with ffmpeg? do you see the gdi object count growing in process manager?
Re: Running out of memory
so here it's ffmpeg that need memory. Are you able to see which video file is processed?
Pierre.
Re: Running out of memory
But this does not make sense. When I switch on "create thumbnails for whole folder" it takes up all memory, when I don't switch it on, it does not take all memory.xnview wrote:so here it's ffmpeg that need memory. Are you able to see which video file is processed?
There is non reason to consume more memory with that switch on and off as the task is absolutely the same: Process video after video and create thumbs for it. The only difference is that with that switch on there are more videos to process. But again, this cannot take all that memory.
Re: Running out of memory
yes, when you activate "create thumbnails for whole folder", all thumbnails will be created but one after one. It seems that there is a leak somewhere.jadO wrote:But this does not make sense. When I switch on "create thumbnails for whole folder" it takes up all memory, when I don't switch it on, it does not take all memory.xnview wrote:so here it's ffmpeg that need memory. Are you able to see which video file is processed?
There is non reason to consume more memory with that switch on and off as the task is absolutely the same: Process video after video and create thumbs for it. The only difference is that with that switch on there are more videos to process. But again, this cannot take all that memory.
Are you able to see which video file is processed when memory grow up?
Pierre.
Re: Running out of memory
No.xnview wrote:Are you able to see which video file is processed when memory grow up?
Re: Running out of memory
is it possible to create a folder with some of these video files, and reproduce this memory leak?jadO wrote:No.xnview wrote:Are you able to see which video file is processed when memory grow up?
Pierre.
Re: Running out of memory
I don't think it is a specific video. I believe it is the way the queue is handled at a large number of video files. Currently 20.000+ in muliple folders.xnview wrote:is it possible to create a folder with some of these video files, and reproduce this memory leak?jadO wrote:No.xnview wrote:Are you able to see which video file is processed when memory grow up?
Re: Running out of memory
if you have 20000+ video files, so there are 20000+ thumbnails in memory!
Pierre.
Re: Running out of memory
The video files are located in several folders. Perhaps the thumbnail creation could handle one folder by another. Just an idea...xnview wrote:if you have 20000+ video files, so there are 20000+ thumbnails in memory!
Re: Running out of memory
This does not make sense. Why should there be 20000+ thumbnails in memory? With 4 thumbnails per video this would mean 80000+ thumbnails.xnview wrote:if you have 20000+ video files, so there are 20000+ thumbnails in memory!
This way of implementation does not make sense. I don't neet more thumbnails in memory than I can see in XnViewMP displayed.
Again: If I don't turn on thumbnailing for whole folder then thumbnails get created as I scroll through a folder one by one. No memory problems then. But with that switch turn on, suddenly we have memory problems when there shouldn't be any. Thumbnailing should happen exactly the same way one by one without causing a memory leak.
Re: Running out of memory
You mean in XnViewMP explorer view? No there it only displays like 5 columns and 3 rows of thumbnails depending on size.xnview wrote:in thumbnail view, you have not 20000+ files?
What I mean is the following: When I switch off thumbnailing for whole folder, XnViewMP creates thumbnail after thumbnail according to what is displayed in the explorer. It only loads, into memory what is currently visible thus needing only very little memory.
Now with thumbnailing whole folder turned on it consumes more memory. But all it has to do is to prepare a file list of all files and go through them one by one like I do when I am browsing it create the thumbnails one by one. There is no change needed except that it has to have a list of files in memorey. But this cannot be 3.5 GB in size. There is no reason to even try to have the thumbails of all files in memory. This is what does not make sense.
When thumbnailing of a video file has been done, the video file and the thumbnails should be discarded from memory unless they are currently displayed by XnViewMP browser or they are prefetched for faster scrolling. But again 80000+ thumbnails for prefetching does not make sense.
Re: Running out of memory
It's not the case, when thumbnail is creating (whole folder option on or off), the thumbnail is kept in memory.jadO wrote: When thumbnailing of a video file has been done, the video file and the thumbnails should be discarded from memory unless they are currently displayed by XnViewMP browser or they are prefetched for faster scrolling. But again 80000+ thumbnails for prefetching does not make sense.
If you have 20000 files in thumbnail view, you'll have 20000 thumbnails in memory if 'create thumbnails for whole folder' is on. And when off, the thumbnails in memory will grow following the scroll...
Pierre.