0.78: Search function - possible memory leak

Reported bugs that have been closed and/or resolved

Moderators: XnTriq, xnview, Dreamer

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

0.78: Search function - possible memory leak

Post by CameronD » Wed Mar 30, 2016 1:00 pm

XnviewMP v0.78 x64 under Windows 7 with 8GB physical RAM.

DB has about 40000 images, nearly all jpeg.
XnView.db is 308MB in size.

I operate with IPTC and XMP duplicated as much as possible.
DB categories should match between categories, IPTC:keywords and XMP:Subject.
I have 71 images with "butterfly" as a category/keyword.

Start XnViewMP.
Memory allocated is around 50MB (I display working set, peak working set and private working set and for these tests they are all approximately the same as each other)
I search from a top folder, use catalog, recursive.
search IPTC:Caption contains: "butterfl"
After a few seconds if lists all entries, and all memory values read ~ 110MB

search IPTC:keyword contains: "butterfl"
After a few seconds if lists all 71 entries, and all memory values remain at ~ 110MB

Shut down and restart XnViewMP.
using the same search options,
search XMP:description contains: "butterfl"
After several minutes it lists the same files as during the IPTC:caption search, and memory used is shown as around 400MB.
The memory use increased steadily during the search duration, with lots of page faults per second.

search XMP:Subject contains: "butterfl"
Again it takes several minutes, produces the correct list of files, but now memory has increased by another 400MB.

If I do not use catalog, then the search takes perhaps a bit longer, but more importantly the memory consumed is only half that when I use the DB.
If I repeat the search (without even exiting the search dialog box) the memory used keeps incrementing by the same amount each time.

User avatar
xnview
Author of XnView
Posts: 31887
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Search function - possible memory leak.

Post by xnview » Fri Apr 01, 2016 2:13 pm

:shock: A big one, a leak in all XMP functions!!

See Issue 830 for current status and details.
Pierre.

User avatar
helmut
Posts: 8174
Joined: Sun Oct 12, 2003 6:47 pm
Location: Frankfurt, Germany

Re: Search function - possible memory leak.

Post by helmut » Sun Feb 26, 2017 4:52 pm

This problem is supposed to be fixed in XnView MP 0.80 or later. Could you please try and confirm the bug fix, CameronD?

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: Search function - possible memory leak.

Post by CameronD » Mon Feb 27, 2017 2:42 pm

version 0.84 - problem seems to be about the same.

There is perhaps a minor improvement in the amount of memory consumed, but not much.
Previously I said 400MB at the end of first XMP search, this time the memory consumed sat at 350MB after the first search, then 650MB, then 860MB after a 3rd search for a different XMP:description.

It also revealed a second bug - if I ever pressed "abort" or "cancel" during a search then the program hangs - becomes unresponsive and further attempts to close the sub-window asks if I want to kill the whole program.
I will post this separately
Last edited by CameronD on Tue Feb 28, 2017 2:56 am, edited 1 time in total.

User avatar
xnview
Author of XnView
Posts: 31887
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Search function - possible memory leak.

Post by xnview » Mon Feb 27, 2017 4:32 pm

CameronD wrote:version 0.84 - problem seems to be about the same.
could you send me sample file for the search? and a screenshot of the search dialog?
Pierre.

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: Search function - possible memory leak.

Post by CameronD » Tue Feb 28, 2017 1:53 am

xnview wrote:
CameronD wrote:version 0.84 - problem seems to be about the same.
could you send me sample file for the search? and a screenshot of the search dialog?
A sample file? no. This is searching a folder hierarchy. There are now 44k images in 930 folders.

Last night I selected the folder, and did view->update catalog from files just to make sure everything was correctly catalogued.

I repeated the tests this morning.
I used various search strings that gave results between zero and 200 matches - this does not noticeably change the main memory leak.
  • right click on the required folder in the folder window. select search.
  • search,
    • filename blank,
    • match ANY,
    • case sensitive: no;
    • use catalog: yes;
    • recurse: yes
  • Add single condition an XMP:Description-> contains -> the search word (or fragment)
  • search then takes about 8 minutes, with much thrashing of disc and paging rates between 100 and 1000 per second
  • the count of matching files is returned. (the correct matches seem to be found)
  • the current and peak working sets have increased by somewhere between 250 and 300 MB.
  • cancel search windows back to main screen and search again.
  • repeating with the same conditions, or any other XMP condition does not seem to matter. It always results in another increment of 200 to 300 MB in each of the memory allocation numbers
  • other searches based on IPTC or EXIF strings can be interposed, and memory allocation at the end of the search returns to what it was at the start of the search.
Every XMP-based search that I tried increased allocated memory by a similar amount - a little bit is released at the end of each search but most is never released and so it keeps ratchetting upwards until the program is exited.

A screen shot is attached:
xnviewmp-v084-search-memory-leak.png
XnViewMP search windows and Win7 task manager during XMP search (chosen to return zero matches). Memory allocation had been around 1.17GB at the start of the search.

User avatar
xnview
Author of XnView
Posts: 31887
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Search function - possible memory leak.

Post by xnview » Tue Feb 28, 2017 9:48 am

i can't reproduce on folder with jpeg files, which file format do you have?
Pierre.

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: Search function - possible memory leak.

Post by CameronD » Tue Feb 28, 2017 2:13 pm

xnview wrote:i can't reproduce on folder with jpeg files, which file format do you have?
In the large target I have 130 tiffs, the rest are jpeg.

I tested with a single folder, turned recursive off. 100 images and in fact only 4 of them have an XMP block.
The working set increments by a few hundred kB each time.

Even in a folder with only 10 files, of which only one has an XMP block, the memory increments by about 50kB each search.

I have a second installation from the zip file, with its own DB and ini files - it shows the same problem.

User avatar
xnview
Author of XnView
Posts: 31887
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Search function - possible memory leak.

Post by xnview » Tue Feb 28, 2017 2:24 pm

CameronD wrote: In the large target I have 130 tiffs, the rest are jpeg.
is it possible to have a TIFF with XMP sample used?
Pierre.

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: Search function - possible memory leak.

Post by CameronD » Tue Feb 28, 2017 3:18 pm

xnview wrote:
CameronD wrote: In the large target I have 130 tiffs, the rest are jpeg.
is it possible to have a TIFF with XMP sample used?
They tend to be panoramas, and rather large.

In any case I can see it with all-jpeg folders.

It is getting late - I'll try another test tomorrow.

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: Search function - possible memory leak.

Post by CameronD » Sun Mar 05, 2017 11:22 am

It took a bit longer than "tomorrow" :shock:

I have attached a zip file with two tiny test images and a shell script.

Image: nometa.jpg has no embedded metadata beyond what it required to describe the jpeg itself.
Image: hasmeta.jpg is the same image with an IPTC caption and XMP description added.
script: create-files.sh will create a folder structure with 1000 copies of nometa and another with 10000 of the image with metadata. Subfolders are used with a limit of 100 files per folder.

To reproduce:
  • unzip contents into a fresh folder.
  • using a suitable shell program, cd to the new folder and run the script.
  • in xnview, update catalog from files: Note buglet: the progress bar stays on zero for 99% of the time it takes to process the files. :bug:
  • run some utility that shows the memory used (such as windows task manager)
  • perform search for "IPTC:caption contains qwerty" or any string that will fail the search. repeat the search several times and note various memory allocations do not increment steadily
  • stop and restart xnview (just in case)
  • perform search for "XMP:description contains qwerty" or any string that will fail the search. Repeat the search several times and note various memory allocations increment steadily. Exiting search window returns relatively little memory.
  • repeating searching continues to increment memory usage.
Things I have noticed so far:
  • searching the folder of images with no metadata does not seem to increment memory
  • The memory increment for the sample file with metadata is about 300 to 350 bytes per file per search. The number scales correctly searching 1000 files or for 10000 files at a time.
  • The memory increment value is larger if the images have more metadata, but I have not quantified it.
To get my numbers I ran the search 10 times and returned to the main screen (closed the search window)
Read the memory usage as the starting baseline.
ran the same search another 10 times, returned to the main screen and read final memory usage.
There can be a significant jump in permanent memory allocation the first time a search is run, so I wanted to avoid counting that.
Attachments
search-bug-test.zip
(17.13 KiB) Downloaded 48 times

User avatar
helmut
Posts: 8174
Joined: Sun Oct 12, 2003 6:47 pm
Location: Frankfurt, Germany

Re: 0.78: Search function - possible memory leak

Post by helmut » Sun Mar 05, 2017 9:56 pm

Thank you very much, CameronD, for this very detailed description to reproduce the possible memory leak. I've created a folder structure with 10 subfolders and 1.000 files by copying your file hasmeta.jpg over and over, again. Pretty much like in your script.

I'm not too familiar with the memory info of Windows task manager, did you check the physical memory (total, available)?

@Pierre: From what I can see, VC++ offers various possibilities, e.g. CRT Debug Library Use. Could you please check and try CameronD's steps?

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: 0.78: Search function - possible memory leak

Post by CameronD » Mon Mar 06, 2017 1:15 am

helmut wrote: I'm not too familiar with the memory info of Windows task manager, did you check the physical memory (total, available)?...
Hi Helmut,

In windows task manager, select "Process" tab. Sort by process name, reverse will keep xnview at or near the top.
"View-> select columns" and enable:
* Memory - Working set
* Memory - Peak Working set
* Memory - Private Working set

I don't really understand the differences myself (peak is obvious and less useful) but all 3 generally show the increment in allocation.

User avatar
xnview
Author of XnView
Posts: 31887
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: 0.78: Search function - possible memory leak

Post by xnview » Mon Mar 06, 2017 8:39 am

O.k., thank you, I can also reproduce the problem. Issue 1152 is fixed in next version.
Pierre.

User avatar
helmut
Posts: 8174
Joined: Sun Oct 12, 2003 6:47 pm
Location: Frankfurt, Germany

Re: 0.78: Search function - possible memory leak

Post by helmut » Mon Mar 06, 2017 8:54 am

xnview wrote:O.k., thank you, I can also reproduce the problem. Issue 1152 is fixed in next version.
Very good! :)

@CameronD: Again, thank you very much for your detailed descriptions which allowed to tracked this down. Would be good if you could verify when the 0.85 test version is available.

Post Reply