Thumbnails and database size

Ideas for improvements and requests for new features in XnView MP

Moderators: XnTriq, xnview

Post Reply
soulky
Posts: 133
Joined: Tue Dec 06, 2011 9:20 am

Thumbnails and database size

Post by soulky » Sun Dec 09, 2012 11:04 pm

Some of my databases are getting a ridiculous size of over 1 GB. When I check the cache settings I can see that thumbnails get stored multiple time. In case of a portable volume with changing drive letter, new thumbnails get created and stored every time the drive letter changes.
So I have the same hundreds of folders stored in the database multiple times with drive letters F, G, T, U, K and L.
When I delete the duplicates it won´t help as the next time a new drive letter gets assigned to the drive, it will recreate the thumbnails and store them again with new drive letter.
This does not make sense to me.

User avatar
xnview
Author of XnView
Posts: 31002
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Thumbnails and database size

Post by xnview » Mon Dec 10, 2012 8:21 am

thumbnails are sorted by pathname, so if drive is changed, xnview can gets thumbnails :(
Pierre.

soulky
Posts: 133
Joined: Tue Dec 06, 2011 9:20 am

Re: Thumbnails and database size

Post by soulky » Tue Dec 11, 2012 7:03 am

xnview wrote:thumbnails are sotre by pathname, so if drive is changed, xnview can gets thumbnails :(
I cannot offer advice how to solve this. But from my idea paths should never be stored absolutely but relatively. As drive letter assignement is pretty much beyond users control.
xnviewmp should store a drive id (i don´t know, something like drive serial no.) and based on this it should store the paths as relative paths.
So no matter what drive letter the os assignst to the drive, xnviemp could always identify the drive and load/store the thumbnails only once.
This would have a major effect on controling database size and imrpive performance which gets out of control with the current implementation easily.

User avatar
JohnFredC
XnThusiast
Posts: 2010
Joined: Wed Mar 17, 2004 8:33 pm
Location: Sarasota Florida

Re: Thumbnails and database size

Post by JohnFredC » Tue Dec 11, 2012 5:14 pm

What sort of maintenance is done on the cache when XnView enters a folder?

Are entries for missing files removed from the cache automatically upon entry to a folder? This would seem to be good policy.

Or, if performance is an issue, perhaps the user could ad hoc choose a menu item to initiate a "rationalize" or "validate" cache contents for the current folder.

Or another approach, prompt the user: "The following images are no longer in this folder. Purge them from the cache?"

Perhaps XnView could reference a user-maintained list of "watched" folders when entering a folder...
John

soulky
Posts: 133
Joined: Tue Dec 06, 2011 9:20 am

Re: Thumbnails and database size

Post by soulky » Tue Dec 11, 2012 7:29 pm

JohnFredC wrote:What sort of maintenance is done on the cache when XnView enters a folder?

Are entries for missing files removed from the cache automatically upon entry to a folder? This would seem to be good policy.
No I don´t think this would be a good policy. It is absolutely legit to have files in the cache which are currently not present. Just think of files on a portable drive or usbstick. It would not be a good idea to remove those.
Also it can be absolutely legit to have files stored multiple times on multiple drives even using the same folder hierachy like it would happen when maing backups. Basically there is only one case where it does NOT make sense to keep those files in the database. And this is the case I have mentioned when the same drive gets assigned all different drive letters and xnviewmp caches the files again and again. By using some kind of drive id, this could be prevented.

User avatar
JohnFredC
XnThusiast
Posts: 2010
Joined: Wed Mar 17, 2004 8:33 pm
Location: Sarasota Florida

Re: Thumbnails and database size

Post by JohnFredC » Tue Dec 11, 2012 7:58 pm

I just meant:

If the user navigates to a folder of images, and some of the thumbed files listed in the cache for that folder are no longer there, then the cache holds those thumbs in error for that folder and the entries should be deleted, either automatically or via the user's response to a prompt.
John

soulky
Posts: 133
Joined: Tue Dec 06, 2011 9:20 am

Re: Thumbnails and database size

Post by soulky » Wed Dec 12, 2012 12:00 pm

JohnFredC wrote:I just meant:

If the user navigates to a folder of images, and some of the thumbed files listed in the cache for that folder are no longer there, then the cache holds those thumbs in error for that folder and the entries should be deleted, either automatically or via the user's response to a prompt.
The implementation of such a feature must be well thought. Still without a proper unique id this can cause unwanted deletion. I believe it still should be possible to attach different portable drives to a computer all having the same drive letter and not have the caches wiped. So even for your suggestion some kind of unique drive identifier would be required to make sure only files for the appropriate drive get deleted.
Still this would not be a solution for my initial feedback as it can be desired to have the files in cache even if they are currently not present. The solution is to prevent multiple thumbnailing and caching of 100% identical files only because the assigned drive letter has been changed.

User avatar
XnTriq
Moderator & Librarian
Posts: 5369
Joined: Sun Sep 25, 2005 3:00 am
Location: Ref Desk

Re: Thumbnails and database size

Post by XnTriq » Wed Dec 12, 2012 9:30 pm

Before the release of the first version of XnView for Wintel-based systems, I was using a shareware program called ThumbsPlus, which relies on volume labels for cataloging on- & offline / local & removable media.
Cerious Software (ThumbsPlus v3 Manual » [url=http://www.cerious.com/sp/manual3/faq.shtml]FAQ[/url]) wrote:
Something important everyone needs to know about ThumbsPlus:
  • So that ThumbsPlus can know which disk is what, you need to give your disks unique volume labels. If you've already made thumbnails on a particular disk, you should do this within ThumbsPlus (File | Volumes | Label Disk).

vommie
Posts: 108
Joined: Sat Apr 14, 2007 2:36 am
Location: Berlin

Re: Thumbnails and database size

Post by vommie » Sat Dec 15, 2012 11:56 am

XnView could use serial numbers of devides/hard disks to use relative paths or change the drive letter of them.

But the most important would be to be able to clean the database. But when I remember correctly this already is a discussed an planned feature.

epicedium
Posts: 4
Joined: Fri Dec 28, 2012 5:39 pm

Re: Thumbnails and database size

Post by epicedium » Fri Dec 28, 2012 5:48 pm

Can it not store thumbnail keyed by a hash of .... [filename + crc of first 10K] ... or [ filename + filesize + moddate ] or something? Ignoring paths completely?

All you need to do is avoid "false matches" and some variation on the above hashes should achieve that just as well (..or better) than by path

epicedium
Posts: 4
Joined: Fri Dec 28, 2012 5:39 pm

Re: Thumbnails and database size

Post by epicedium » Fri Dec 28, 2012 5:58 pm

In terms of "pruning" thumbs for deleted files, I don't see why a removable drive causes a problem. If folder "X:/a/b/" is online but file "X:/a/b/c.jpg" isn't, then ........

A side-stepping approach might be to store the cache at the root of each volume, instead of per-system. This way you're only ever evaluating on files which really should "exist" at the time of evaluation.

If you stored for each thumb ...

[identifying hash from filename+crc+moddate] ; [last known location] ; [ thumbnail raw binary ]

Then moving a file to a different location would still work,

And if you entered a folder, you could search for all entries with matching "last known location" and prune accordingly?

soulky
Posts: 133
Joined: Tue Dec 06, 2011 9:20 am

Re: Thumbnails and database size

Post by soulky » Fri Dec 28, 2012 10:53 pm

epicedium wrote:In terms of "pruning" thumbs for deleted files, I don't see why a removable drive causes a problem. If folder "X:/a/b/" is online but file "X:/a/b/c.jpg" isn't, then ........
It is a problem if folder "X:/a/b/" is not unique.
This can happen easily if files are not stored in a particular folder but in root. So for xnviewmp drive letter O: assigned to drive 1 is as good as drive letter O: which has been assigned to drive 2.
So without a unique identifer xnviemp cannot tell if a file is not present because it has been deleted or if a file is not present because it is on a different drive.

User avatar
budz45
XnThusiast
Posts: 1597
Joined: Sun Jun 03, 2007 6:05 pm
Location: UK

Re: Thumbnails and database size

Post by budz45 » Sat Dec 29, 2012 8:30 pm

soulky wrote: So no matter what drive letter the os assignst to the drive, xnviemp could always identify the drive and load/store the thumbnails only once.
I like the sound of this :) Yes optimized XnView database support for relative paths and disk volume device ID
All My Topics || my 'MP' Topics
My own Bookmarked topics--->for me only

Danny
Posts: 564
Joined: Sat Sep 04, 2004 5:09 pm

Re: Thumbnails and database size

Post by Danny » Sat Jan 19, 2013 11:17 pm

It should use the mounted volume's GUID then (like 47ba2efc-2db1-11e0-88f8-806e6f6e6963).
Get the bugs fixed, THEN start adding features. It sucks, but someone has to do it.

User avatar
oops66
XnThusiast
Posts: 1999
Joined: Tue Jul 17, 2007 1:17 am
Location: France

Re: Thumbnails and database size

Post by oops66 » Thu Feb 07, 2013 3:28 pm

budz45 wrote:
soulky wrote: So no matter what drive letter the os assignst to the drive, xnviemp could always identify the drive and load/store the thumbnails only once.
I like the sound of this :) Yes optimized XnView database support for relative paths and disk volume device ID
I like the sound of this too :) Yes optimized XnView database support for relative paths (as option) (but without to take care of the disk volume device ID)

Code: Select all

Setting\Cache DB\Thumbnails:
a "base path of your pictures"
and a new base path of your pictures (for relative path)
So, with the new definition of a single point(relative to this point) "a root images directory", it will be possible and easy to copy the "structure of all images/vidéo/mp3 folders" to an other support (a clone to a new: hard disk, or else) then only by changing the Xnview setting of this "single point" no need to recreate the DB (locations, thumbnails, etc...).
XnViewMP 0.82 Linux X64 - Ubuntu 16.04 LTS - X64

Post Reply