XnView Software

XnView: MP 0.84
OS: Windows - 32 bit

When updating the catalogue from files, heaps of disk accesses are performed. Catalogue database resides on a hard disk while the image files are on a SDD. I hear heavy hard disk activity so I suspect the catalogue updates are responsible for the many disk accesses. Processor load is very low (<3%).

Effect: Updating of catalogue takes longer than needed (5 min for 1.000 files). Many disk access my give the user the feeling that something is going wrong.

To reproduce:
1. Create a directory structure with 10 subfolders with 100 image files in each subfolder as described in topic Search function - possible memory leak.
2. Update XnView's catalogue from the image files (View » Update catalog from files).

Actual behaviour (bug): While updating there is heavy disk access which is generated by the catalogue updates
Windows Task Manager:
I/O (Read): 6.022
I/O (Write): 18.020

Expected behaviour: Sure eneough there must be disk accesses to update the catalog, but there should be much less disk activity. I don't know the technical details behind XnView, but less frequent database commits (e.g. every 100 image files, only), might help here.

3. Repeat step 2.)
Actual behaviour (bug): Although nothing has changed, there is again heavy disk access.

Exptected behaviour: When running "Update catalog from files" for a second time on the same images files, there should be much less disk activity.

Note: I notice the many disk accesses because my hard disk of my PC is slow and fairly loud.

3. Repeat step 2.)
Actual behaviour (bug): Although nothing has changed, there is again heavy disk access.

The Update process does not check if the data to be updated is the same with the data at destination. Depending on the internal architecture of the program this check can be a serious slow-down.

Did you try to give more memory to your database engine?

In Settings | Catalog what value do you have?

Put at least 100-200 MB. It does the same?

Thank you for your reply, m.Th. I've just increased the memory for the database to 600 MB - no change: Lots of disk accesses, figures are same as before (or even worse:) I/O read: 6208 I/O write: 30745

I suspect that after each single update a commit is performed which forces the database manager to write all changes to the database. This will result in bad performance and should be changed.

I did a few tests and was surprised.
For my 1000 file system with minimal metadata, the total file size is 8MB on disc.
The DB size has reached 17MB (due to real images being scanned).
The "update catalog from files" performs a total of 66MB of written data - this must be all associated with the DB, since the images are not written.

sqlite3 has several journal and synchronous modes, and I think the default is DELETE/FULL - safest and slowest.
This saves the old data to journal, writes the new data directly to the DB, and deletes the journal on commit.
FULL sync I think is a commit and flush to physical disc every transaction.

I tried using sqlitebrowser to change the journalling and syncing but xnview must reset them when it opens the file.
The exception is WAL mode, which can be set and then xnview runs with the write-ahead log. sync mode is still full.
time to scan drops from 1 min 40 s for DELETE journalling, to 23 sec for WAL mode.

The total amount of written data drops from 66 to 25MB.

No matter how I arrange things, the I/O rate is low and the CPU load is low. I don't understand what is holding it up - maybe waiting for the disc flush.

Thank you, CameronD, for your tests and your analysis. I think we agree that there is something going wrong: Not really a functional bug but a non-functional one (slow performance / disk access).

Reproduced

Issue 1160 is fixed in next version.

xnview wrote:Issue 1160 is fixed in next version.

Verg good. I hope that CameronD (you) can perform a retest once a test version is available.

This problem is supposed to be fixed in XnView MP 0.85 beta 1. Please check and confirm the bug fix here.

dramatic improvement in speed as well as reduction of number of bytes written.

As an example, I could no longer measure my test for 1000 files.

So I scanned 11000 files - in 9 seconds, and a total of about 9MB written.

CameronD wrote:dramatic improvement in speed as well as reduction of number of bytes written.

Great! Well done Pierre! And thank you very much for your detailed analysis and testing, CameronD, which helped much to solve this quickly.