Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC etc.)

Ideas for improvements and requests for new features in XnView MP

Moderators: XnTriq, xnview

User avatar
xnview
Author of XnView
Posts: 31881
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by xnview » Thu Apr 21, 2016 6:33 am

so which metadata will be the best?
Pierre.

User avatar
m.Th.
XnThusiast
Posts: 1557
Joined: Wed Aug 16, 2006 6:31 am
Contact:

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by m.Th. » Thu Apr 21, 2016 4:49 pm

xnview wrote:so which metadata will be the best?
I think that we can start the implementation with this layout:

https://docs.google.com/spreadsheet/pub ... utput=html

Beware that you have TWO sheets there: Fields and Unique Metadata Space which is in fact the table model of the Fields.

Details in the previous posts.
m. Th.

The Ascetic Experience - The best photos and texts from Holy Mountain (Athos)

- Dark Themed XnViewMP 0.90 64bit & XnView 2.00 x64 on Win7 x64 -

User avatar
oops66
XnThusiast
Posts: 1999
Joined: Tue Jul 17, 2007 1:17 am
Location: France

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by oops66 » Thu Apr 21, 2016 7:14 pm

Hello,
I have a question:
... For me the Xnview.db file is becoming too big ... initially the xnview.db was designed to be splitted in two parts (the big one, for blobs and thumbnails images cache: the Thumbs.db, and a small one Xnview.db without blobs and only for the users extra datas like categories, labels colors, IPTC keywords and so) to be able to make a small backup with only users extra data and to have the search in db function faster.
... So why Xnview.db has now some blobs that consumes data space, is it really necessary?
XnViewMP 0.82 Linux X64 - Ubuntu 16.04 LTS - X64

User avatar
m.Th.
XnThusiast
Posts: 1557
Joined: Wed Aug 16, 2006 6:31 am
Contact:

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by m.Th. » Fri Apr 22, 2016 8:16 am

oops66 wrote:Hello,
I have a question:
... For me the Xnview.db file is becoming too big ... initially the xnview.db was designed to be splitted in two parts (the big one, for blobs and thumbnails images cache: the Thumbs.db, and a small one Xnview.db without blobs and only for the users extra datas like categories, labels colors, IPTC keywords and so)
Too big??? :) :) :D

With what do you compare? Today storage options? Other databases? Which ones? (please do NOT look at Lightroom's DBs... ;) )

Ok, you can Optimize it through Tools | Settings | Catalog | Optimize...

The main reason is exactly the topic of this thread: Large binary blobs (read: black-boxes) which holds ALL the metadata (see Images.Meta field).

However, the main target is NOT to make the db small but to make it FAST. Ok, „small” has its influence in „fast” but not it isn't the only factor.
to be able to make a small backup with only users extra data
Yep. Do it. Zip on the Db. Mine from 1,07 GB goes to @ 300 MB. I am pretty sure that when the metadata will be split, the compression ratio will be bigger. However, here the size is mostly reduced by Optimization (see above) and leaving off the noise which sometimes is put in the Meta field.

and to have the search in db function faster.
... So why Xnview.db has now some blobs that consumes data space, is it really necessary?
Sure. This is the main reason to split the metadata from Images.Meta blob in Fields.

However you have a point here: I think that is better to have 3 (three) DBs: XnView.db, XnMeta.Db, and Thumbs.Db. THE fastest program on the market (AfterShotPro) has this layout (there, the DBs are called Base, Thumbs, Meta respectively).

This will solve the backup problem of the only user-generated DB (XnView.db) and will keep the speed at top.

Of course, the new DB (XnMeta.Db) will be attached in the same manner in which is used now Thumbs.db.
m. Th.

The Ascetic Experience - The best photos and texts from Holy Mountain (Athos)

- Dark Themed XnViewMP 0.90 64bit & XnView 2.00 x64 on Win7 x64 -

User avatar
oops66
XnThusiast
Posts: 1999
Joined: Tue Jul 17, 2007 1:17 am
Location: France

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by oops66 » Fri Apr 22, 2016 1:16 pm

( Xniew.db ... too big in relative point of view ... only half of the Thumb.db size here.)
m.Th. wrote: ... However you have a point here: I think that is better to have 3 (three) DBs: XnView.db, XnMeta.Db, and Thumbs.Db.
Hello, ... In fact, that was my subliminal request too ;-)

PS-edit: It would be probably also interesting to have this third database (Xnview-user.db) reserved only for: "Not already embedded data" (into each pics) ... to have an easy way to save, check the difference, or to synchronize data, between metadata pics and this database, if required. So this Xnview-user.db must be blank if all data are already synchronized (no needed backup, because the backup already exists into each pics/jpegs).
XnViewMP 0.82 Linux X64 - Ubuntu 16.04 LTS - X64

User avatar
m.Th.
XnThusiast
Posts: 1557
Joined: Wed Aug 16, 2006 6:31 am
Contact:

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by m.Th. » Fri Apr 22, 2016 5:00 pm

PS-edit: It would be probably also interesting to have this third database (Xnview-user.db) reserved only for: "Not already embedded data" (into each pics) ... to have an easy way to save, check the difference, or to synchronize data, between metadata pics and this database, if required. So this Xnview-user.db must be blank if all data are already synchronized (no needed backup, because the backup already exists into each pics/jpegs).
Nope, sorry. :) We need all the (meta)data in databases (in XnMeta.Db in our case) because of Search. We need to search/filter this data. The diff. checks / sync. will be done based on Last Modified Dates.
m. Th.

The Ascetic Experience - The best photos and texts from Holy Mountain (Athos)

- Dark Themed XnViewMP 0.90 64bit & XnView 2.00 x64 on Win7 x64 -

User avatar
oops66
XnThusiast
Posts: 1999
Joined: Tue Jul 17, 2007 1:17 am
Location: France

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by oops66 » Fri Apr 22, 2016 5:30 pm

... Basically, it's not the same last based modified date, because these not synchronized data are not already embedded to the metadata of the pics. Then a simple duplication is only needed for these few fields to be synchronized or not (colors label, categories, rattings, set comment, etc)... I don't think than it is a so big deal. If not, this third user-database is not very interesting/necessary. (so generally only few x* KB is needed instead of X*100 MB to save this "not already synchronized user data DB" )
XnViewMP 0.82 Linux X64 - Ubuntu 16.04 LTS - X64

User avatar
m.Th.
XnThusiast
Posts: 1557
Joined: Wed Aug 16, 2006 6:31 am
Contact:

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by m.Th. » Sat Apr 23, 2016 7:25 am

oops66 wrote:... Basically, it's not the same last based modified date, because these not synchronized data are not already embedded to the metadata of the pics. Then a simple duplication is only needed for these few fields to be synchronized or not (colors label, categories, rattings, set comment, etc)... I don't think than it is a so big deal. If not, this third user-database is not very interesting/necessary. (so generally only few x* KB is needed instead of X*100 MB to save this "not already synchronized user data DB" )
Let's cool a little. We have the following facts:
  • all DAMs on the market have as one central feature the Real-Time Filtering on Metadata by presenting to the user the values already present in DB. Every. program. And I have installed 7-8 of them. Besides this, there are several programs which provide analytics/graphs (via a plugin like Lr) or natively (Zoner).
  • the industry standard for sync between Db and Files is Real-Time sync: when you change a value (being it RCK or other/image metadata), this value is immediately written on-disk in embedded EXIF/XMP or in a XMP sidecar. Due of OS caching and slowness of human user, no performance hit is taken. This system works not only for us (which basically have only RCK values) but also for DAMs which support non-destructive editing - hence much, much, much more values (edits etc.) to write in XMP/embedded metadata.
  • if one really wants an off-line sync, internally the code anyway will load the entire EXIF/metadata chunk (mind you, disks nowadays have big sectors) from file and taking in account the paged structure of SQL file, you will load in memory the entire page (many rows) even if you will read just one row or just one field. So, you will gain nothing, except code complexity. That's why off-line syncs in the way in which you want are in 99,9999% implemented by using change logs (lists of changes). However, this is an overkill for us since, anyway, we will have a metadata Db. Taking in account that this scenario is rare and the performance is the same, I say that a LMDate is enough.
Again, the main reason for XnMeta.Db is search/filter, not sync. which usually is a real-time sync from prg. to disk. In the case of conflicts and in the case of the off-line syncs definitely should be triggered by a decisive user action - the press of a button.
m. Th.

The Ascetic Experience - The best photos and texts from Holy Mountain (Athos)

- Dark Themed XnViewMP 0.90 64bit & XnView 2.00 x64 on Win7 x64 -

User avatar
oops66
XnThusiast
Posts: 1999
Joined: Tue Jul 17, 2007 1:17 am
Location: France

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by oops66 » Sat Apr 23, 2016 8:39 am

... This is already cool because technically (about Find Vs Sync): "One does not preclude the other" (a French saying) "L'un n'empêche pas l'autre".
XnViewMP 0.82 Linux X64 - Ubuntu 16.04 LTS - X64

CameronD
Posts: 303
Joined: Wed Aug 01, 2007 1:28 pm
Location: Australia

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by CameronD » Mon Apr 25, 2016 12:40 am

oops66 wrote:... Basically, it's not the same last based modified date, because these not synchronized data are not already embedded to the metadata of the pics....
Synchronisation can be either:
  • required by the user
  • not wanted by the user
  • not possible with certain file types
Only the first case is important to this bit of discussion, and synchronisation should always occur as quickly as possible.
For example, a user makes some changes to a selection of one or more files, the DB is changed and then the changes are immediately applied to the files. This will take a finite, but small, length of time.
The only time the DB and files should be out of sync is if the program or computer crashes in that brief time interval. If that happened then the DB itself would be unlikely to be correct either.

I cannot see much value in splitting the DB. At one extreme, which I prefer, all metadata is with the image files and the db can be safely deleted and rebuilt from file content. At the other extreme, the DB integrity would be critical, and I don't think it would be safe to lose the connection between the split DB parts.

In relation to relative speed of search, we are talking about many orders of magnitude difference. Think in the following very approximate timescales:
  • less than one second: DB entry, indexed
  • several seconds: DB entry, not indexed
  • several more seconds: DB entry currently in the blob structure
  • many minutes: not in DB - read every file for metadata
Potential for saving space would be in:
  • limiting what fields get stored in DB: I think this would be quite messy to program, and prone to user choices that would be regretted later
  • limiting what fields get indexed: This might offer scope for some reduction, although in the current structure there do not seem to be many indices that could be safely removed. Maybe just color and rating. I don't think that difference would be noticeable in db size

User avatar
oops66
XnThusiast
Posts: 1999
Joined: Tue Jul 17, 2007 1:17 am
Location: France

Re: Very important: Metadata V2 for 0.70 (EXIF, XMP, IPTC et

Post by oops66 » Mon Apr 25, 2016 1:21 pm

CameronD wrote:
oops66 wrote:... Basically, it's not the same last based modified date, because these not synchronized data are not already embedded to the metadata of the pics....
Synchronisation can be either:
  • 1-
  • required by the user
    2-
  • not wanted by the user
    3-
  • not possible with certain file types
...
Hello,
Yes, it's that why I think than this eventual xnview-user.db must be smaller as possible (especially for case 2 and 3) in fact, only few fields/data are concerned.
... I prefer, all metadata is with the image files and the db can be safely deleted and rebuilt from file content...
Me too when it is possible, (so excepted for case 3, via an eventual extra xnview-user.db file ?)
XnViewMP 0.82 Linux X64 - Ubuntu 16.04 LTS - X64

Post Reply