Out of memory when recursive viewing 20.000+ videos

Ask for help and post your question on how to use XnView MP.

Moderators: XnTriq, xnview

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Out of memory when recursive viewing 20.000+ videos

Post by jadO » Sun Oct 23, 2016 11:16 am

When doing thumbnailing for videos with recursive folder view anabled and "create thumbnails for whole folder" enabled, memory usage spiked and causes a lot of unwanted behavior including file system errors.
So the implementation should be improved. Thumbnails should be created one by one one file after another and once it is done released from memory. Despite a small queue to know which file is next, I don't see the need for large memory consumption even with a large number of files being thumbnailed.
So it looks like the implementation should be improved.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Tue Oct 25, 2016 3:13 pm

It's done like that. Video thumbnail with ffmpeg? do you see the gdi object count growing in process manager?
Pierre.

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Re: Running out of memory

Post by jadO » Wed Oct 26, 2016 3:10 am

xnview wrote:It's done like that. Video thumbnail with ffmpeg? do you see the gdi object count growing in process manager?
Yes, video thumbnail with ffmpeg. I don't know about a gdi object, But what I saw is that the XnViewMP process requested 3.5 GB of memory and there is absolutely no need for that.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Wed Oct 26, 2016 7:54 am

so here it's ffmpeg that need memory. Are you able to see which video file is processed?
Pierre.

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Re: Running out of memory

Post by jadO » Wed Oct 26, 2016 12:10 pm

xnview wrote:so here it's ffmpeg that need memory. Are you able to see which video file is processed?
But this does not make sense. When I switch on "create thumbnails for whole folder" it takes up all memory, when I don't switch it on, it does not take all memory.

There is non reason to consume more memory with that switch on and off as the task is absolutely the same: Process video after video and create thumbs for it. The only difference is that with that switch on there are more videos to process. But again, this cannot take all that memory.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Wed Oct 26, 2016 12:13 pm

jadO wrote:
xnview wrote:so here it's ffmpeg that need memory. Are you able to see which video file is processed?
But this does not make sense. When I switch on "create thumbnails for whole folder" it takes up all memory, when I don't switch it on, it does not take all memory.

There is non reason to consume more memory with that switch on and off as the task is absolutely the same: Process video after video and create thumbs for it. The only difference is that with that switch on there are more videos to process. But again, this cannot take all that memory.
yes, when you activate "create thumbnails for whole folder", all thumbnails will be created but one after one. It seems that there is a leak somewhere.
Are you able to see which video file is processed when memory grow up?
Pierre.

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Re: Running out of memory

Post by jadO » Wed Oct 26, 2016 12:16 pm

xnview wrote:Are you able to see which video file is processed when memory grow up?
No.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Wed Oct 26, 2016 12:19 pm

jadO wrote:
xnview wrote:Are you able to see which video file is processed when memory grow up?
No.
is it possible to create a folder with some of these video files, and reproduce this memory leak?
Pierre.

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Re: Running out of memory

Post by jadO » Wed Oct 26, 2016 12:23 pm

xnview wrote:
jadO wrote:
xnview wrote:Are you able to see which video file is processed when memory grow up?
No.
is it possible to create a folder with some of these video files, and reproduce this memory leak?
I don't think it is a specific video. I believe it is the way the queue is handled at a large number of video files. Currently 20.000+ in muliple folders.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Wed Oct 26, 2016 2:02 pm

if you have 20000+ video files, so there are 20000+ thumbnails in memory!
Pierre.

User avatar
helmut
Posts: 8174
Joined: Sun Oct 12, 2003 6:47 pm
Location: Frankfurt, Germany

Re: Running out of memory

Post by helmut » Wed Oct 26, 2016 2:07 pm

xnview wrote:if you have 20000+ video files, so there are 20000+ thumbnails in memory!
The video files are located in several folders. Perhaps the thumbnail creation could handle one folder by another. Just an idea...

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Re: Running out of memory

Post by jadO » Wed Oct 26, 2016 2:48 pm

xnview wrote:if you have 20000+ video files, so there are 20000+ thumbnails in memory!
This does not make sense. Why should there be 20000+ thumbnails in memory? With 4 thumbnails per video this would mean 80000+ thumbnails.

This way of implementation does not make sense. I don't neet more thumbnails in memory than I can see in XnViewMP displayed.

Again: If I don't turn on thumbnailing for whole folder then thumbnails get created as I scroll through a folder one by one. No memory problems then. But with that switch turn on, suddenly we have memory problems when there shouldn't be any. Thumbnailing should happen exactly the same way one by one without causing a memory leak.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Wed Oct 26, 2016 3:52 pm

in thumbnail view, you have not 20000+ files?
Pierre.

jadO
Posts: 413
Joined: Wed Apr 29, 2015 6:36 am

Re: Running out of memory

Post by jadO » Thu Oct 27, 2016 4:13 am

xnview wrote:in thumbnail view, you have not 20000+ files?
You mean in XnViewMP explorer view? No there it only displays like 5 columns and 3 rows of thumbnails depending on size.

What I mean is the following: When I switch off thumbnailing for whole folder, XnViewMP creates thumbnail after thumbnail according to what is displayed in the explorer. It only loads, into memory what is currently visible thus needing only very little memory.

Now with thumbnailing whole folder turned on it consumes more memory. But all it has to do is to prepare a file list of all files and go through them one by one like I do when I am browsing it create the thumbnails one by one. There is no change needed except that it has to have a list of files in memorey. But this cannot be 3.5 GB in size. There is no reason to even try to have the thumbails of all files in memory. This is what does not make sense.
When thumbnailing of a video file has been done, the video file and the thumbnails should be discarded from memory unless they are currently displayed by XnViewMP browser or they are prefetched for faster scrolling. But again 80000+ thumbnails for prefetching does not make sense.

User avatar
xnview
Author of XnView
Posts: 31938
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Running out of memory

Post by xnview » Fri Oct 28, 2016 7:41 am

jadO wrote: When thumbnailing of a video file has been done, the video file and the thumbnails should be discarded from memory unless they are currently displayed by XnViewMP browser or they are prefetched for faster scrolling. But again 80000+ thumbnails for prefetching does not make sense.
It's not the case, when thumbnail is creating (whole folder option on or off), the thumbnail is kept in memory.

If you have 20000 files in thumbnail view, you'll have 20000 thumbnails in memory if 'create thumbnails for whole folder' is on. And when off, the thumbnails in memory will grow following the scroll...
Pierre.

Post Reply