Page 1 of 2

Database performance

Posted: Thu Jun 13, 2013 3:30 pm
by xnview
<!--// Moderator's note: This topic was split off from “0.56/Win 8/64: Adjustment previews issues”. //-->
JohnFredC wrote: Perhaps do the same for XnView Classic? I still use it more than MP (for various reasons).
Can i know for which reasons do you use XnView Classic?

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Sat Jun 15, 2013 2:55 pm
by JohnFredC
Hi Pierre

I use the "Paint" filter, the "Effects" dialog, and the "Strip of Images" function quite a bit for a variety of "quick" tasks.

Plus. Sad to say, MP still "stops working" frequently and even on my new hot rod is slow to open some folders, even after they have been cached. See this topic.

Win8/64, I7-3770, 8Gb.

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Mon Jun 17, 2013 7:27 am
by m.Th.
xnview wrote:
JohnFredC wrote: Perhaps do the same for XnView Classic? I still use it more than MP (for various reasons).
Can i know for which reasons do you use XnView Classic?
Folders Tree:
- No drag & drop support http://newsgroup.xnview.com/viewtopic.php?f=60&t=28147
- Right Click on a folder selects the item. Leads to a very convoluted workflow. Use case: I have the folder 100CANON downloaded from my card. I need to stay on this folder and create new subfolders in another subfolders (eg. Portraits) and drag and drop (ideally) from my 100CANON the appropriate photos to the newly created folders. Now I have to go back and forth between the folders because when I try to create a new subfolder in 'Portraits', MP automatically selects it.

Engine:
- on a cold start (IOW the program runs for the first time since boot), if I want to see the tagged files (or any other category) the program freezes for a long period of time, most probably because it loads the SQLite, or something similar. A long time ago I sent you an email about the very same problem in XnView and you fixed it there.
- Paint. It was very nice to have the Paint plugin in order to draw some lines (arrows), circles to highlight something, comments etc. The 'Spotlight' feature many times isn't clear enough to underline something.

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Mon Jun 17, 2013 8:07 am
by xnview
JohnFredC wrote: Plus. Sad to say, MP still "stops working" frequently and even on my new hot rod is slow to open some folder
Strange i'll contact you.

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Mon Jun 17, 2013 8:08 am
by xnview
m.Th. wrote: - on a cold start (IOW the program runs for the first time since boot), if I want to see the tagged files (or any other category) the program freezes for a long period of time, most probably because it loads the SQLite, or something similar. A long time ago I sent you an email about the very same problem in XnView and you fixed it there.
do you have a way to reproduce this freeze? how many files do you have in the folder? Which format?

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 7:47 am
by m.Th.
xnview wrote:
m.Th. wrote: - on a cold start (IOW the program runs for the first time since boot), if I want to see the tagged files (or any other category) the program freezes for a long period of time, most probably because it loads the SQLite, or something similar. A long time ago I sent you an email about the very same problem in XnView and you fixed it there.
do you have a way to reproduce this freeze? how many files do you have in the folder? Which format?
Since you asked for the data, here it is:

First directory to show in XnViewMP when it opens (ie. the last dir of the prev session): 926 photos summing 26 GB.
I want to show 2 Tagged files from a folder with 248 photos summing 6.8 GB.

All files are 22.6 MPix CR2 files - the Canon 5D Mk3's RAW files.

However I think that the problem lies not in the folders above but elsewhere.

First, how to reproduce:


I think that the problem resides in having a big database on a slow drive. (non-SSD, having already other tasks from OS or from other programs etc.)

When I open the program, it opens quite fast (the thumbnails were generated in the previous session), but for the first time when I want to change the tab of the Tree Pane from the 'Folders' to 'Categories' it freezes for a very long time. In this time, the CPU (an Intel i5 3550 @ 3,3 GHz) is used somewhere at 6% (ie. is more or less waiting), while the disk activity on C:\...\AppData\Roaming\XnViewMP\XnView.DB (a 4.67 GB file) is quite high. As an aside, I tried to change the database directory to another drive with less activity from Tools > Options > Integration... but a Windows message appeared saying "XnView has stopped working ...etc.... Press 'Close' to terminate the program".

Back to debugging: I noticed that TWO processes - "System" and "XnView" - accessed the XnView's database during the XnViewMP's freeze. Because my antivirus is configured to _exclude_ the .DB files from its real time scan, I think that the main problem is with the extension .DB which is a system one (see thumbs.db etc.), and most probably the OS fiddles with the file, thing which is unnoticeable if the file is small but becames a big problem when the file grows. I remember now a similar situation with another extension (.GDB). In that case it wasn't SQLite, but Interbase/Firebird which for many many years had .GDB as default extension (comming from Gordon DB - Gordon Database System was Jim Starkey's company before Ashton Tate buyout) and everything was fine till Windows (XP I think) "declared" that .GDB is the extension of some of the their dbs and hence caused a bunch of delays. It was quite a discussion upon the matter in the Firebird forums and finally they settled to .FDB as default extension for their files.

Hence, I propose some things:

1. Fix the 'Custom Folder for the .db' thing in 'Tools | Options | Integration'. Anyway is a bug. Be sure, when the folder is changed to offer the posibility to copy/move the old db to the new folder.
2. Be careful, istm that SQLite tries to load the entire db in memory. Hmmmm... I am not sure that this is a very good thing. I think that this must be done 'on demand'. Usually such things happens when a query forces a natural scan over the database. Perhaps you do an aggregate (COUNT, SUM, MAX etc.) on a non-indexed field? Or perhaps a search ('Where...') and/or sort ('ORDER BY') on such a field? Usually such things trigger this behavior. I have 16GB RAM, the file is 4.67 GB and I wait a LOT at the first time.
3. Change the .DB extension of your file! Put something like .xndb or similar! You sure don't want to have other processes interefering with your IO engine (ok, SQLite's).

Just my2c & HTH

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 8:09 am
by oops66
xnview wrote:
m.Th. wrote: - on a cold start (IOW the program runs for the first time since boot), if I want to see the tagged files (or any other category) the program freezes for a long period of time, most probably because it loads the SQLite, or something similar. A long time ago I sent you an email about the very same problem in XnView and you fixed it there.
do you have a way to reproduce this freeze? how many files do you have in the folder? Which format?
Hello,
I also have this kind of issue (with some *.ppt files ) ... freeze + heats the CPU at 100% ... so the bug here seems to come when XnViewMP try to build the 4 (or 1) images per folder with some problematics files.
http://newsgroup.xnview.com/viewtopic.p ... 17#p111517

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 8:19 am
by m.Th.
oops66 wrote:
xnview wrote:
m.Th. wrote: - on a cold start (IOW the program runs for the first time since boot), if I want to see the tagged files (or any other category) the program freezes for a long period of time, most probably because it loads the SQLite, or something similar. A long time ago I sent you an email about the very same problem in XnView and you fixed it there.
do you have a way to reproduce this freeze? how many files do you have in the folder? Which format?
Hello,
I also have this kind of issue (with some *.ppt files ) ... freeze + heats the CPU at 100% ... so the bug here seems to come when XnViewMP try to build the 4 (or 1) images per folder with some problematics files.
http://newsgroup.xnview.com/viewtopic.p ... 17#p111517

The *.ppt files tend to be rather big. These are located on the same disk with the database? (usually XnViewMP's db is located on the system disk - C:\ on most installations).

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 8:39 am
by oops66
m.Th. wrote:... These are located on the same disk with the database? (usually XnViewMP's db is located on the system disk - C:\ on most installations).
Hello,
Yes, same disk and same partition (an ext4 partition - because I am under Linux here - XnViewMP 0.56 Linux X64 - Ubuntu 12.04 LTS - X64 )

Database performance

Posted: Tue Jun 18, 2013 9:09 am
by m.Th.
oops66 wrote:
m.Th. wrote:... These are located on the same disk with the database? (usually XnViewMP's db is located on the system disk - C:\ on most installations).
Hello,
Yes, same disk and same partition (an ext4 partition - because I am under Linux here - XnViewMP 0.56 Linux X64 - Ubuntu 12.04 LTS - X64 )
...hmmm... ...mhmmm... ...hummm...

There are several things here.

1st, Pierre should give us, the ones who use mechanical HDDs (but not only) the possibility to change the location of db on another drive (not only partition - drive). But of course, we need another HDD. I have, but I understand the ones who doesn't.
2nd, (aside of bugs in ppt processing and such) perhaps is better to have a way to speed up the db loading by eliminating some possible unnecessary queries (see my previous post)
3rd, tweak the thumbnail generation - for example commit on SQLite db after 100, 500 or even 1000 thumbnails. Anyway is a data generation which if the system crashes exactly when this is happening, it can be manually regenerated without too much hassle. I think that we could achieve important performance gains here without too much hassle. But perhaps this is already implemented?
4rd, perhaps move the blocking part of generation of thumbs in a separate thread? ...but this, even if will solve the problem of 'freezing' could be a complicated change, depending of the program's internal architecture. Also, here one can gain speed by having a high-priority thread etc.

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 10:24 am
by m.Th.
m.Th. wrote: ....
2. Be careful, istm that SQLite tries to load the entire db in memory. Hmmmm... I am not sure that this is a very good thing. I think that this must be done 'on demand'. Usually such things happens when a query forces a natural scan over the database. Perhaps you do an aggregate (COUNT, SUM, MAX etc.) on a non-indexed field? Or perhaps a search ('Where...') and/or sort ('ORDER BY') on such a field? Usually such things trigger this behavior. I have 16GB RAM, the file is 4.67 GB and I wait a LOT at the first time.
....
OH, PIERRRRREEE.... :shock:

Had a look at your db. Sorry... {big sheepish grin}

You have the 'Images' table with the following DDL:

Code: Select all

CREATE TABLE Images(ImageID INTEGER PRIMARY KEY AUTOINCREMENT, FolderID INTEGER NOT NULL, Filename TEXT NOT NULL, Size INTEGER, ModifiedDate DATE, Rating INTEGER DEFAULT 0, Color INTEGER DEFAULT 0, Tag INTEGER DEFAULT 0, Meta BLOB, UNIQUE (Filename, FolderID));

CREATE UNIQUE INDEX Images_idx ON Images(FolderID, Filename);
Well, you have here SOME problems. In fact, at least in my use case of freezing (see my previous posts), the full table scan is (IMHO) triggered by 'Tag' field because you show in the Categories Tree the number of tagged photos (see Special Items\Tagged Files). That number is obtained by having a "Select count(*) from Images where Tag=1" or similar (for ex. "Select sum(tag) from Images"), isn't it? Also, in order to display the files you need to have a query (most probably a join with 'Datas') in which you have somewhere 'Where Tag=1', isn't it? :D

Well, the above SQL queries (an aggregate on a non-indexed field, a search condition on a non-indexed field) triggers a full table scan. Hence the slowdown on big databases.

(of course 'big' in the statement above is relative to HDD response time, seek time, waiting queue etc.)

The solution(s) is/are obvious:

1. Index the Tag field. Also, while we're here, index all the other management fields: Rating, Color. This will speedup all display operations and also the clearing engine (the code which clears all the Tags when the program closes).

2. Also, if you want to be really fast, since you display the number of tagged files you can have this number as an maintained aggregate in a special field in a special table, for example in DBInfo. This will avoid having even an natural scan on an indexed subset.

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 12:33 pm
by xnview
m.Th. wrote: When I open the program, it opens quite fast (the thumbnails were generated in the previous session), but for the first time when I want to change the tab of the Tree Pane from the 'Folders' to 'Categories' it freezes for a very long time
Do you have a lot of categories? Images in categories? Tagged files?
m.Th. wrote: Well, you have here SOME problems. In fact, at least in my use case of freezing (see my previous posts), the full table scan is (IMHO) triggered by 'Tag' field because you show in the Categories Tree the number of tagged photos (see Special Items\Tagged Files). That number is obtained by having a "Select count(*) from Images where Tag=1" or similar (for ex. "Select sum(tag) from Images"), isn't it? Also, in order to display the files you need to have a query (most probably a join with 'Datas') in which you have somewhere 'Where Tag=1', isn't it? :D

Well, the above SQL queries (an aggregate on a non-indexed field, a search condition on a non-indexed field) triggers a full table scan. Hence the slowdown on big databases.
Ok, i'm not expert in DB :)

So you think that i must add

Code: Select all

CREATE UNIQUE INDEX Tags_idx ON Images(ImageID, Tag);
, right?

And could you send me your email by PM

Re: 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 12:58 pm
by JohnFredC
Recommended practice is to index all fields used in db filtering mechanisms ("where" etc.) as well as all fields used in joins. In my experience, the benefits are beyond huge.

But "unique"? The relationship between tags and file names could be many-to-many. That requires three tables:

Files Table
[FID], [Filename], other attributes
[FID] as Long Integer
Unique indexes on [FID], [Filename]

Tags table
[TID], [Tagname], other attributes
[TID] as Long Integer
Unique indexes on [FID], [Tagname]

FilesTags table
[TID], [FID]
[TID], [FID] as long integers
Non-unique Indexes on [FID], [TID]

Then a query seeking files with tag="Red" looks something like this:

Code: Select all

SELECT Files.Filename
FROM Files RIGHT JOIN FilesTags ON Files.FID = FilesTags.FID LEFT JOIN Tags ON FilesTags.TID = Tags.TID
WHERE Tags.Tagname="Red";
This should be quite speedy! Integer joins are fast. (If possible never use a string in a join!)

If the argument is that files may only have one tag value, this model still works correctly. In that case, the constraint would be applied at the UI level (via combo box or whatever), not the [FID] index in FilesTags.

Re: MP 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 2:04 pm
by oops66
m.Th. wrote:...There are several things here...
First, thanks for your expertise !

"1st, Pierre should give us, the ones who use mechanical HDDs (but not only) the possibility to change the location of db on another drive (not only partition - drive). But of course, we need another HDD. I have, but I understand the ones who doesn't."
... Right, to have the ability to improve the performance of the new "XnView.xndb" file. ;-)

"2nd, (aside of bugs in ppt processing and such) perhaps is better to have a way to speed up the db loading by eliminating some possible unnecessary queries (see my previous post)"
...Very interesting, some tests must be done first probably...

"3rd, tweak the thumbnail generation - for example commit on SQLite db after 100, 500 or even 1000 thumbnails. Anyway is a data generation which if the system crashes exactly when this is happening, it can be manually regenerated without too much hassle. I think that we could achieve important performance gains here without too much hassle. But perhaps this is already implemented?"
... I support this kind of "db custom tweak".

"4rd, perhaps move the blocking part of generation of thumbs in a separate thread? ...but this, even if will solve the problem of 'freezing' could be a complicated change, depending of the program's internal architecture. Also, here one can gain speed by having a high-priority thread etc."
... I think than some tests must be done too ?

If the DB is redesigned, maybe it's the time to also add some interesting news fields like for example:
- EXIF date taken (for a new functionality, tree by date - as option)

Code: Select all

2012(years)
  -- 01(month)
       -- 30(days)
              -- 23(hours)
- and some others fields, ... etc ... ?

Re: 0.56/Win 8/64: Adjustment previews issues

Posted: Tue Jun 18, 2013 2:23 pm
by xnview
JohnFredC wrote: But "unique"? The relationship between tags and file names could be many-to-many. That requires three tables:

Files Table
[FID], [Filename], other attributes
[FID] as Long Integer
Unique indexes on [FID], [Filename]
No Tag is for tagged files, not categories (categories use such tables)