I have currently thumbnailed about 40,000 files in around 1,000 folders: selecting the Cache tab in the Options dialog, after more than FOURTEEN minutes has yet to yield any result, the dialog has not even been repainted, on a 2.2GHz P4 with 768MB RAM. I need to manually terminate the process to be able to rerun XnView and use it.
The thumbs, as long as I browse the single folders one by one, are correctly displayed, but I cannot use the Cache page and consequently am forced to thumbnail folders one by one since I cannot even get at the "Add folder" button.
As it is now, the cache management is simply impossible to use (note that I probably have at least as many files and folders still to thumbnail...): Pierre, at least move the "Add folder" option and all other options that don't require the folder list to be displayed to a menu option or elsewhere.
V 1.90 B1
cache management
Moderators: helmut, XnTriq, xnview
cache management
Last edited by robc on Sun Nov 26, 2006 2:19 pm, edited 1 time in total.
Re: B1: cache management utterly impossible
It would be better to fix the cache database, IMHO (I even wonder whether it could be 20x faster and more efficient...robc wrote:Pierre, at least move the "Add folder" option and all other options that don't require the folder list to be displayed to a menu option or elsewhere.
).Olivier
-
foxyshadis
- Posts: 395
- Joined: Sat Nov 18, 2006 8:57 am
You can use view->show files in subfolders, which will collect everything into one huge view and thumbnail all at once.
I'm pretty sure the listbox contributes; adding 1000 items to a listbox can become unbearably slow, but I believe it's due to the database design. It seems that a background thread to update a (temp?) table with the relevant information is required. At the moment whenever you load the cache tab, it has to scan the entire database once for each folder, adding up any matching files, to get the count & size for each.
It's still more of a feature request than a bug, though.
It takes over 40 seconds to load for mine, on a core2duo, with only 350 folders and 17000 files.
I'm also checking whether defragmenting the tables leads to any performance boost (I'm guessing probably not, though, except for my absolute largest folders.)
Edit: Olivier, the reads would probably be entirely within cache, so there's no disk read penalty; at least, they always are on all my systems. Maybe not if the hard drive has only 2MB cache and the system has only 128, though...
I'm pretty sure the listbox contributes; adding 1000 items to a listbox can become unbearably slow, but I believe it's due to the database design. It seems that a background thread to update a (temp?) table with the relevant information is required. At the moment whenever you load the cache tab, it has to scan the entire database once for each folder, adding up any matching files, to get the count & size for each.
It's still more of a feature request than a bug, though.
It takes over 40 seconds to load for mine, on a core2duo, with only 350 folders and 17000 files.
I'm also checking whether defragmenting the tables leads to any performance boost (I'm guessing probably not, though, except for my absolute largest folders.)
Edit: Olivier, the reads would probably be entirely within cache, so there's no disk read penalty; at least, they always are on all my systems. Maybe not if the hard drive has only 2MB cache and the system has only 128, though...
Re: B1: cache management utterly impossible
Do you have already tried on other software with a DB??robc wrote:I have currently thumbnailed about 40,000 files in around 1,000 folders: selecting the Cache tab in the Options dialog, after more than FOURTEEN minutes has yet to yield any result, the dialog has not even been repainted, on a 2.2GHz P4 with 768MB RAM. I need to manually terminate the process to be able to rerun XnView and use it.
The thumbs, as long as I browse the single folders one by one, are correctly displayed, but I cannot use the Cache page and consequently am forced to thumbnail folders one by one since I cannot even get at the "Add folder" button.
As it is now, the cache management is simply impossible to use (note that I probably have at least as many files and folders still to thumbnail...): Pierre, at least move the "Add folder" option and all other options that don't require the folder list to be displayed to a menu option or elsewhere.
Could you send me a mail, i'll send you a version without checking number of files and total Mb to test...
Pierre.
I've been working with databases for the past several (~15) years 
with many more records than this, it's always a matter of a few seconds in the worst case (and it should be). I believe foxyshadis is about right: I don't know how you implemented the db, but I'd blame more the dialog than the db. First, a thousand lines added to a listview can be very "heavy"; it depends on the implementation of the list control and on the development environment you're using (I did some tests, for example, with a treeview displaying a huge folder tree and the difference among, say, VB, VB.net, C# and C++ was overwhelming), but in many cases you can fill the list in several instances, loading just two-three "listfuls" (i.e. how many lines the list control can display at once) and loading more data in the list only if and when the user starts scrolling it. I believe the standard MS implementation can do something like that.
Also, if you scan all cached folders to build up the list and then sum up all files, that may also be very intensive: why not store that value in the db, folder by folder? when you first thumb the folder, you compute the number of files within and store it; whenever a modification is made to the folder, you simply upgrade the total and when you list everything no computation is made. From a philosophical point of view, why generate the list at all? I believe that everything should be made on a "folder" basis: you right click, say, on a folder in the tree (or in an empty space in thumbnail view) and get a context menu with the items "Thumbnail folder" and "Remove thumbnails"; the first option also allows the user to process just that folder or all the tree below it. It would also be easier than going to Options|Cache|Add folder and then search the folder to add or browse the list (which may have thousands of entries) to find the folder you need to remove.
Just my .02, of course. The only real problem is the time it takes to have the cached folder list appear. I've PM'd you my email address.
with many more records than this, it's always a matter of a few seconds in the worst case (and it should be). I believe foxyshadis is about right: I don't know how you implemented the db, but I'd blame more the dialog than the db. First, a thousand lines added to a listview can be very "heavy"; it depends on the implementation of the list control and on the development environment you're using (I did some tests, for example, with a treeview displaying a huge folder tree and the difference among, say, VB, VB.net, C# and C++ was overwhelming), but in many cases you can fill the list in several instances, loading just two-three "listfuls" (i.e. how many lines the list control can display at once) and loading more data in the list only if and when the user starts scrolling it. I believe the standard MS implementation can do something like that.Also, if you scan all cached folders to build up the list and then sum up all files, that may also be very intensive: why not store that value in the db, folder by folder? when you first thumb the folder, you compute the number of files within and store it; whenever a modification is made to the folder, you simply upgrade the total and when you list everything no computation is made. From a philosophical point of view, why generate the list at all? I believe that everything should be made on a "folder" basis: you right click, say, on a folder in the tree (or in an empty space in thumbnail view) and get a context menu with the items "Thumbnail folder" and "Remove thumbnails"; the first option also allows the user to process just that folder or all the tree below it. It would also be easier than going to Options|Cache|Add folder and then search the folder to add or browse the list (which may have thousands of entries) to find the folder you need to remove.
Just my .02, of course. The only real problem is the time it takes to have the cached folder list appear. I've PM'd you my email address.
As you may have noticed, I've removed the "utterly impossible" part from the topic title, since XnView doesn't deserve such a "definitive" bad remark, especially after we've found out that the problem only lies in the computations (number of files, size etc.) and not in the folder list itself.
Apart from this, there's still something to say concerning performance: I'm currently making several trials to find out the thumb size and compression that best suits my needs, so I'm adding folders to the cache, removing several folders at once, optimizing and then restarting another time; while I may agree that such a pattern of use is far from normal, I believe there's too much disk activity going on. As I said in another post, I'm quite accustomed to working with databases, though not really image DBs, and even using much bigger datasets and removing or adding thousands of records at a time I never encounter so much disk churning and so long operations. What DB engine are you using, Pierre? what kind of operations are taking place when you add, remove and optimize?
Apart from this, there's still something to say concerning performance: I'm currently making several trials to find out the thumb size and compression that best suits my needs, so I'm adding folders to the cache, removing several folders at once, optimizing and then restarting another time; while I may agree that such a pattern of use is far from normal, I believe there's too much disk activity going on. As I said in another post, I'm quite accustomed to working with databases, though not really image DBs, and even using much bigger datasets and removing or adding thousands of records at a time I never encounter so much disk churning and so long operations. What DB engine are you using, Pierre? what kind of operations are taking place when you add, remove and optimize?
I was about to post... surely it depends to the performance of the disk where the cache is stored (I was testing on a notebook), I'm currently testing on another disk and performance is definitely better. With big caches, though (around 2GB, lossy high quality db) it takes a lot to optimize or remove folders and it takes somewhat to display thumbs in the browser.
While it's difficult to remove all external variables, it seems that with my setup the move to a "centralized" db has worsened performance with respect to the previous approach. I'm currently testing the use of TWO XnViews, one in a duplicate folder with a different INI, to allow having TWO separate cache DBs, one for the most frequently used images, the other for the archive.
While it's difficult to remove all external variables, it seems that with my setup the move to a "centralized" db has worsened performance with respect to the previous approach. I'm currently testing the use of TWO XnViews, one in a duplicate folder with a different INI, to allow having TWO separate cache DBs, one for the most frequently used images, the other for the archive.
-
foxyshadis
- Posts: 395
- Joined: Sat Nov 18, 2006 8:57 am
When browsing a folder with 500 JPG images (altogether 1 GB) it takes about 31 seconds till all thumbnails are created (cached and drawn).
When adding the same folder to the cache using "Add folder...", it takes 145 seconds.
That means manually triggered caching is more than four times slower than simply browsing a folder.
Perhaps the problem is the immediate write to the cache database.
When adding the same folder to the cache using "Add folder...", it takes 145 seconds.
That means manually triggered caching is more than four times slower than simply browsing a folder.
Perhaps the problem is the immediate write to the cache database.
-
foxyshadis
- Posts: 395
- Joined: Sat Nov 18, 2006 8:57 am
Re: B1: cache management
Regardless of the performance of caching in general, in beta3 the cache tab is now totally usable; all folders are listed immediately, then the folder sizes are slowly added in the background.robc wrote:I have currently thumbnailed about 40,000 files in around 1,000 folders: selecting the Cache tab in the Options dialog, after more than FOURTEEN minutes has yet to yield any result, the dialog has not even been repainted, on a 2.2GHz P4 with 768MB RAM. I need to manually terminate the process to be able to rerun XnView and use it.