0.84: identifying uncataloged files/folders

[CameronD]

I think I know of a number of ways to update catalog DB contents, after new files are added, or file contents have been changed by other software:

1. manually browse to a folder and XnViewMP will add any new files and will somehow identify changed files and decide if any need updating
2. menu View->update catalog from files will recursively scan every file from a user-selected folder and unconditionally update DB with current file contents. Every file is scanned.
3. From menu Tools->Settings the tab Browser->Catalog->Catalog has a button "Add Folder...". This allows user to select a folder, which is recursively scanned for files. The action seems to add info to catalog DB and it also builds thumbnails.

The problem is - I need to scan the entire photo library to allow xnview to catalog newly added folders and files.
The second option is no good as it unconditionally recans new and old images, which makes it far too slow.
Is the 3rd option the efficient way to do this? NO - it builds thumbnails for every file where it does not already have one cached.

[helmut]

Good thoughts and aspects, CameronD. A smart(er) mechanism to synchronize files could be both fast and effective.

Synchronizing catalog with files
When creating catalog entries, XnView could store last-modified date and file size for each single file. When synchronizing next time, XnView compares the info stored in the database and the current info of the file and updates the metadata only if a change was detected.

Synchronizing files with catalog
The same could work the other way round: Whenever changing metadata in the catalog, only, the current system date will be stored per file in the catalog. When synchronizing the files with the catalog, XnView compares the last modified date and file size and updates only if there was a change.

Detecting a change
Path, File name, file size, and last modified should be sufficient to detect a change in most cases. A "Forced" update might be needed or alternatively a checksum of the file data could be created. But the building a checksum might slow down performance, again.

[CameronD]

I came across this while debugging the search using DB.

The search status gives you no information as to how many files and folders were checked, so you have no idea how many files were not even in the catalogue.
I got suspicious when the search was going really fast.

[CameronD]

I think I worked out the best way with current software:

• Under settings Browser->catalog... disable thumbnail caching
• exit xnviewmp and restart
• Under settings Browser->catalog... select the folder to scan from using the "Add Folder..." button
• repeat last step for all folders of interest
• Reinstate thumbnail caching.

And I have confirmed that, for any image already in the DB, if the File:modified date and the file size stored in the DB match the value during the scan then the metadata in the DB is not updated.

On the other hand, the "update catalog from files" will unconditionally rewrite the metadata in the DB (which after all is the main purpose of that command)

[helmut]

I think I worked out the best way with current software: ...

A good workaround, CameronD. It confirms what I suspected: Currently, cataloging and thumbnail caching are combined but in fact and practice, these are two different things. (On the technical level it might be better to combine catalog and thumbnails but from user perspective these are separate things).

[m.Th.]

Hic sunt leones ( https://en.wikipedia.org/wiki/Here_be_dragons )

Synchronizing catalog with files (...) Synchronizing files with catalog

There really must be NO discussion here. These are super-standard features in any DAM.

Detecting a change

Full file name+ Date time stamp in this order. (IF we have seconds in Date Time stamp... - dunno what Pierre saves)

I would add iSCSI's CRC because is already in CPUs but mainly for other reasons: for a much more detailed discussion on hashing & other goodies, see here: http://newsgroup.xnview.com/viewtopic.php?f=60&t=28396

However for this to work, the date time stamp should be updated accordingly - hence the correct options must be set up in Settings.

I think I worked out the best way with current software:

Under settings Browser->catalog... disable thumbnail caching
exit xnviewmp and restart
Under settings Browser->catalog... select the folder to scan from using the "Add Folder..." button
repeat last step for all folders of interest
Reinstate thumbnail caching.

I am really sad because it should really have an option just for that. When I click on 'Add Folder...' a dialog should appear saying "Do you want to add the data?" WITH a check box saying "Build the thumbnails also". Very, very important because sometimes we want to do this, other times we don't.

[CameronD]

...

Detecting a change

Full file name+ Date time stamp in this order. (IF we have seconds in Date Time stamp... - dunno what Pierre saves)

I would add iSCSI's CRC because is already in CPUs but mainly for other reasons: for a much more detailed discussion on hashing & other goodies, see here: http://newsgroup.xnview.com/viewtopic.php?f=60&t=28396

However for this to work, the date time stamp should be updated accordingly - hence the correct options must be set up in Settings.

If any part of the full file path is changed, then it is treated as a completely different file - the orphaned info remains until DB is "cleaned".
A corollary of this is that the same file referenced by different paths (UNC vs drive letter) is regarded as two different files.

The date stamp is stored to seconds.

I then discovered cases where it was still updating files even when the file dates were unchanged. These turned out to be due to changes in file size (Windows 7 likes to write extra padding in the metadata blocks when you use it to change content).

I think I worked out the best way with current software:

Under settings Browser->catalog... disable thumbnail caching
exit xnviewmp and restart
Under settings Browser->catalog... select the folder to scan from using the "Add Folder..." button
repeat last step for all folders of interest
Reinstate thumbnail caching.

I am really sad because it should really have an option just for that. When I click on 'Add Folder...' a dialog should appear saying "Do you want to add the data?" WITH a check box saying "Build the thumbnails also". Very, very important because sometimes we want to do this, other times we don't.

Agreed - more flexibility and clarity of operation would be good. I was just offering my findings as a record of what can be done with the current version.

[m.Th.]

If any part of the full file path is changed, then it is treated as a completely different file

Sure. ...but we speak here about metadata update.

Hence we need a PK which is the full-path and a way to detect the change which is the DateTime stamp (we must compare the value from DB with the value from disk) because we assume a friendly environment in which nobody fiddles with date time stamps.