Thumbnails and database size
Moderators: XnTriq, helmut, xnview
Thumbnails and database size
Some of my databases are getting a ridiculous size of over 1 GB. When I check the cache settings I can see that thumbnails get stored multiple time. In case of a portable volume with changing drive letter, new thumbnails get created and stored every time the drive letter changes.
So I have the same hundreds of folders stored in the database multiple times with drive letters F, G, T, U, K and L.
When I delete the duplicates it won´t help as the next time a new drive letter gets assigned to the drive, it will recreate the thumbnails and store them again with new drive letter.
This does not make sense to me.
So I have the same hundreds of folders stored in the database multiple times with drive letters F, G, T, U, K and L.
When I delete the duplicates it won´t help as the next time a new drive letter gets assigned to the drive, it will recreate the thumbnails and store them again with new drive letter.
This does not make sense to me.
Re: Thumbnails and database size
thumbnails are sorted by pathname, so if drive is changed, xnview can gets thumbnails
Pierre.
Re: Thumbnails and database size
I cannot offer advice how to solve this. But from my idea paths should never be stored absolutely but relatively. As drive letter assignement is pretty much beyond users control.xnview wrote:thumbnails are sotre by pathname, so if drive is changed, xnview can gets thumbnails
xnviewmp should store a drive id (i don´t know, something like drive serial no.) and based on this it should store the paths as relative paths.
So no matter what drive letter the os assignst to the drive, xnviemp could always identify the drive and load/store the thumbnails only once.
This would have a major effect on controling database size and imrpive performance which gets out of control with the current implementation easily.
Re: Thumbnails and database size
What sort of maintenance is done on the cache when XnView enters a folder?
Are entries for missing files removed from the cache automatically upon entry to a folder? This would seem to be good policy.
Or, if performance is an issue, perhaps the user could ad hoc choose a menu item to initiate a "rationalize" or "validate" cache contents for the current folder.
Or another approach, prompt the user: "The following images are no longer in this folder. Purge them from the cache?"
Perhaps XnView could reference a user-maintained list of "watched" folders when entering a folder...
Are entries for missing files removed from the cache automatically upon entry to a folder? This would seem to be good policy.
Or, if performance is an issue, perhaps the user could ad hoc choose a menu item to initiate a "rationalize" or "validate" cache contents for the current folder.
Or another approach, prompt the user: "The following images are no longer in this folder. Purge them from the cache?"
Perhaps XnView could reference a user-maintained list of "watched" folders when entering a folder...
John
Re: Thumbnails and database size
No I don´t think this would be a good policy. It is absolutely legit to have files in the cache which are currently not present. Just think of files on a portable drive or usbstick. It would not be a good idea to remove those.JohnFredC wrote:What sort of maintenance is done on the cache when XnView enters a folder?
Are entries for missing files removed from the cache automatically upon entry to a folder? This would seem to be good policy.
Also it can be absolutely legit to have files stored multiple times on multiple drives even using the same folder hierachy like it would happen when maing backups. Basically there is only one case where it does NOT make sense to keep those files in the database. And this is the case I have mentioned when the same drive gets assigned all different drive letters and xnviewmp caches the files again and again. By using some kind of drive id, this could be prevented.
Re: Thumbnails and database size
I just meant:
If the user navigates to a folder of images, and some of the thumbed files listed in the cache for that folder are no longer there, then the cache holds those thumbs in error for that folder and the entries should be deleted, either automatically or via the user's response to a prompt.
If the user navigates to a folder of images, and some of the thumbed files listed in the cache for that folder are no longer there, then the cache holds those thumbs in error for that folder and the entries should be deleted, either automatically or via the user's response to a prompt.
John
Re: Thumbnails and database size
The implementation of such a feature must be well thought. Still without a proper unique id this can cause unwanted deletion. I believe it still should be possible to attach different portable drives to a computer all having the same drive letter and not have the caches wiped. So even for your suggestion some kind of unique drive identifier would be required to make sure only files for the appropriate drive get deleted.JohnFredC wrote:I just meant:
If the user navigates to a folder of images, and some of the thumbed files listed in the cache for that folder are no longer there, then the cache holds those thumbs in error for that folder and the entries should be deleted, either automatically or via the user's response to a prompt.
Still this would not be a solution for my initial feedback as it can be desired to have the files in cache even if they are currently not present. The solution is to prevent multiple thumbnailing and caching of 100% identical files only because the assigned drive letter has been changed.
Re: Thumbnails and database size
Before the release of the first version of XnView for Wintel-based systems, I was using a shareware program called ThumbsPlus, which relies on volume labels for cataloging on- & offline / local & removable media.
Cerious Software (ThumbsPlus v3 Manual » [url=http://www.cerious.com/sp/manual3/faq.shtml]FAQ[/url]) wrote:
Something important everyone needs to know about ThumbsPlus:
- So that ThumbsPlus can know which disk is what, you need to give your disks unique volume labels. If you've already made thumbnails on a particular disk, you should do this within ThumbsPlus (File | Volumes | Label Disk).
- ThumbsPlus v8 SP1 Help: ThumbsPlus Volume Matching
- Wikipedia: Volume (computing) » Volume label
- About.com: Volume Label
- Digital Detective: Volume Serial Numbers and Format Date/Time Verification
Re: Thumbnails and database size
XnView could use serial numbers of devides/hard disks to use relative paths or change the drive letter of them.
But the most important would be to be able to clean the database. But when I remember correctly this already is a discussed an planned feature.
But the most important would be to be able to clean the database. But when I remember correctly this already is a discussed an planned feature.
Re: Thumbnails and database size
Can it not store thumbnail keyed by a hash of .... [filename + crc of first 10K] ... or [ filename + filesize + moddate ] or something? Ignoring paths completely?
All you need to do is avoid "false matches" and some variation on the above hashes should achieve that just as well (..or better) than by path
All you need to do is avoid "false matches" and some variation on the above hashes should achieve that just as well (..or better) than by path
Re: Thumbnails and database size
In terms of "pruning" thumbs for deleted files, I don't see why a removable drive causes a problem. If folder "X:/a/b/" is online but file "X:/a/b/c.jpg" isn't, then ........
A side-stepping approach might be to store the cache at the root of each volume, instead of per-system. This way you're only ever evaluating on files which really should "exist" at the time of evaluation.
If you stored for each thumb ...
[identifying hash from filename+crc+moddate] ; [last known location] ; [ thumbnail raw binary ]
Then moving a file to a different location would still work,
And if you entered a folder, you could search for all entries with matching "last known location" and prune accordingly?
A side-stepping approach might be to store the cache at the root of each volume, instead of per-system. This way you're only ever evaluating on files which really should "exist" at the time of evaluation.
If you stored for each thumb ...
[identifying hash from filename+crc+moddate] ; [last known location] ; [ thumbnail raw binary ]
Then moving a file to a different location would still work,
And if you entered a folder, you could search for all entries with matching "last known location" and prune accordingly?
Re: Thumbnails and database size
It is a problem if folder "X:/a/b/" is not unique.epicedium wrote:In terms of "pruning" thumbs for deleted files, I don't see why a removable drive causes a problem. If folder "X:/a/b/" is online but file "X:/a/b/c.jpg" isn't, then ........
This can happen easily if files are not stored in a particular folder but in root. So for xnviewmp drive letter O: assigned to drive 1 is as good as drive letter O: which has been assigned to drive 2.
So without a unique identifer xnviemp cannot tell if a file is not present because it has been deleted or if a file is not present because it is on a different drive.
Re: Thumbnails and database size
I like the sound of this Yes optimized XnView database support for relative paths and disk volume device IDsoulky wrote: So no matter what drive letter the os assignst to the drive, xnviemp could always identify the drive and load/store the thumbnails only once.
Re: Thumbnails and database size
It should use the mounted volume's GUID then (like 47ba2efc-2db1-11e0-88f8-806e6f6e6963).
Get the bugs fixed, THEN start adding features. It sucks, but someone has to do it.
Re: Thumbnails and database size
I like the sound of this too Yes optimized XnView database support for relative paths (as option) (but without to take care of the disk volume device ID)budz45 wrote:I like the sound of this Yes optimized XnView database support for relative paths and disk volume device IDsoulky wrote: So no matter what drive letter the os assignst to the drive, xnviemp could always identify the drive and load/store the thumbnails only once.
Code: Select all
Setting\Cache DB\Thumbnails:
a "base path of your pictures"
and a new base path of your pictures (for relative path)
XnViewMP Linux X64 - Debian - X64