v0.72 DB-Category <-> IPTC keyword problems.

[CameronD]

I am having some problems getting good results between DB categories and IPTC keywords. I am not sure if they are bugs or I have some wrong settings.
I will use "category" here to refer to the DB category and "keyword" to refer to the IPTC keywords/XMP-Subject.

Background:
In Options => Metadata => IPTC&XMP, in the keywords box, all options are ticked. In other words I want complete synchronization between DB categories and IPTC/XMP metadata.
I also want the parent keywords automatically included.

My layout in the browser window has the file list view as set to details, and I have enabled keywords and captions in the view.
I have the Categories pane visible for clicking to apply a category.
When I apply keywords, I use the ctrl-I window, and the selection list in the keywords tab is set to "Categories (database)"

1. When I apply a category to the selected file, it does not immediately update the keywords - in fact it only seems to update the keywords after I browse to another folder or exit the program. (Edit: This is not fully correct - see more detail in next post )There are two problems with this:
   • I would like to see the list updated so I know which images have been categorized. (The "categories" column is the basically useless IPTC categories)
   • If I then edit image caption using ctrl-I, there sometimes are no IPTC keywords visible in the keywords tab, and when I save-all then XnView sees no keywords and removes the categories that I have recently applied (see next post for more details).
     It would fix these issues if you were able to update the files immediately, or perhaps when the categories pane loses focus
2. The program uses different parent-child separators depending on the process - this causes problems.
   • copying over from categories to keywords I get parent,child in the keywords
   • copying over from keywords to categories I get parent|child in the keywords
   It seems that when the program reads the second form, it imports either the child category only, or else creates a new category named "parent|child". But it does not do the second version very often, so I have not understood the situation that causes it. Today, for example, I got a total of 6 of these composite categories after processing maybe 200-300 images.
3. The children of the last category in the list never have the parent applied, so I have worked around that by creating a "zz_blank" category to make sure it is the last one.

[CameronD]

Updated observations:
When I assign a category to an image, iptc keyword does not get immediately updated in the Keywords column under "Details" view. Even if I refresh and switch between views the keywords are not updated to reflect the newly applied categories.

However in the following places it is updated immediately:

• The "Thumbnails + labels" view (I think I added IPTC:Keywords to the Thumbnails" labels view option.).
• the tooltip does show the new categories, customised as {ITPC:Keywords}
• If I ctrl-I to edit IPTC, then the new keywords are visible (I think previously I said they were not)

But this was only true sometimes!. The next time, when I assigned a category to a selection of 8 images, nothing else was updated. After I left the folder and returned, the new categories were remembered but the keywords were still blank. The same applied when I exited the program and restarted.
Update: when I examined these "unchanged" files in more detail using exiftool, I discovered that the XMP:subject and XMP:Hierarchicalsubject had been assigned the keyword, but there was no IPTC block created at all.

Losing keywords/categories:
Tests with images starting with no keywords or categories:

• If I select a single image, tick a category and immediately ctrl-I, then the IPTC keyword edit tab shows the copied category;
• If I select multiple images, tick a category (apply to both) and immediately ctrl-I, then the IPTC keyword panel is blank;
• If I apply a category to one image, then add another (later image) to the selection then the IPTC keyword edit tab shows the copied category for the correct image. If I edit the caption and save-all, then, provided that I was currently on the image with the new keyword, the new keywords are applied to all selected images. However, the DB category is not updated;

For these tests the IPTC option for keywords was set to "keep and add new value":enabled

[CameronD]

Having worked on a couple of folders using the mixture of ticking categories and entering IPTC keywords, I went back and examined with exiftool and find the image file metadata to be very inconsistent.
In this summary I will just refer to parent as P and child as C. An entry with no child means I only categorised it at the top level.

The situation shown first (missing IPTC:keywords) occurs when I select a category at the top level and lhen browse to another folder. It happens in cases where the top level has child categories and also for a category with no children.
For many of the other combinations I have not determined how they were created, but it seems like every pathway will produce a different outcome.

These are the combinations I have seen in a single folder:

IPTC:keywords - not present
XMP-dc:Subject - P
XMP-lr:HierarchicalSubject - P

IPTC:keywords - P, C
XMP-dc:Subject - P, C
XMP-lr:HierarchicalSubject - P, P|C

IPTC:keywords - P, C
XMP-dc:Subject - P, C
XMP-lr:HierarchicalSubject - not present

IPTC:keywords - P|C
XMP-dc:Subject - C
XMP-lr:HierarchicalSubject - P|C

IPTC:keywords - P
XMP-dc:Subject - P
XMP-lr:HierarchicalSubject - P

IPTC:keywords - P
XMP-dc:Subject - P
XMP-lr:HierarchicalSubject - not present

IPTC:keywords - P, C
XMP-dc:Subject - P, C
XMP-lr:HierarchicalSubject - not present

IPTC:keywords - P1|C1, P2|C2
XMP-dc:Subject - C1, C2
XMP-lr:HierarchicalSubject - P1|C1, P2|C2

[danjcla]

Having worked on a couple of folders using the mixture of ticking categories and entering IPTC keywords, I went back and examined with exiftool and find the image file metadata to be very inconsistent.

Just thought I'd comment that I'm also having problems with this stuff to the point it isn't really usable, so you know at least you aren't alone

I have nothing but admiration for the detail of your bug report. I have nothing to add really - I am trying to do only one-way DB-Category -> IPTC keyword sync; the problem is that it seems to be somewhat random in all aspects of working or not working. Where randomness at least in part likely means I haven't been nearly as methodical as you are in mapping out all the cases.

I'm currently looking into handling the one-way sync out-of-band; you can look at my list of recent posts for more on that in several different threads.

[CameronD]

Thanks,
I was wondering if it was just me.

I have been tracking down my many hundreds of badly categorised files, and find that for batch processing running exiftool is best.

For small numbers of files, it is possible to IPTC-edit and delete the bad keywords, then reassign the categories and exit the folder. You might need to unassign first then reassign, because I think you need to make a change to categories in order to force XnView to update the keywords. Make sure the IPTC options are not set to update unconditionally, and not add to existing values.

None of the following is useful for comparing the database categories, but only for checking for errors embedded in the files, and repairing some.

For example, to find cases where the XMP subject and IPTC keywords do not match, I use

Code: Select all

exiftool  -r  -IPTC:keywords -XMP-dc:Subject -XMP-lr:HierarchicalSubject  -if '$IPTC:keywords ne $XMP-dc:Subject'  -ext jpg -ext jpeg <folder-name>

The optional "-r" is to descend recursively. This format is for bash scripts, so windows command line might need slightly different quoting. In particular, the Unix shells need single quotes around entries with '$', but cmd.exe lines seem to need double quotes. I have no idea about powershell.

But before that, I generally check for missing keywords...

Code: Select all

exiftool  -r  -IPTC:keywords -XMP-dc:Subject -XMP-lr:HierarchicalSubject  -if 'not $IPTC:keywords'  -if '$XMP-dc:Subject'  -ext jpg -ext jpeg <folder-name>

Having checked the list of files output above I then fix them using:

Code: Select all

exiftool  -r  "-IPTC:keywords<XMP-dc:Subject"  -if 'not $IPTC:keywords'  -if '$XMP-dc:Subject'  -ext jpg -ext jpeg <folder-name>

Another situation where the check is easy is when the separator symbol appears in the keyword or subject list. This defaults to "|".

Code: Select all

exiftool  -r  -IPTC:keywords -XMP-dc:Subject -XMP-lr:HierarchicalSubject  -if '$IPTC:keywords =~ /\|/ or $XMP-dc:Subject  =~ /\|/'  -ext jpg -ext jpeg <folder-name>

The way to fix these is more difficult, as the parent and child categories can be difficult to extract. Substitute real values for parent and child in the following example:

Code: Select all

exiftool  -r -sep ", "  -"IPTC:keywords=parent, child" -"XMP-dc:Subject=parent, child"   -if '$IPTC:keywords =~ /\|child/ or $XMP-dc:Subject  =~ /\|child/'  -ext jpg -ext jpeg <folder-name>

Update: My original version was wrong, as exiftool only splits the items if the separator is explicitly specified (I think).

[danjcla]

How are you comparing the exiftool output to what's in the XnView MP database? If you are spot checking via eyeballs (instead of using some export feature of XnView MP I'm unaware of, or doing a direct XnView MP SQLite database query) and aren't familiar with SQL, I can get you command to dump all of XnView MP state in a format you can use to compare to all file state, might be useful to get exact info. (I don't have that large or varied of a corpus of photos, so wouldn't be as useful as your output most likely.)

If you are doing sync based on XnView MP database -> exiftool -> image file, want to share your script? Would save me a bit of time

[CameronD]

How are you comparing the exiftool output to what's in the XnView MP database? ...

I was probably a bit vague in my comments - In the exiftool commands listed above, the categories DB is completely ignored, I am just looking for errors and inconsistencies within the image files.
However, the file modification time gets updated for any files I fix and the next time XnView visits that folder it seems to re-sync the keywords to categories by itself.

For all the instances that I have seen where the DB and file do not agree, it also seems to have left the image metadata in an inconsistent state, so I think at the moment my checks are sufficient without needing the added complexity of cross-checking the db.

In reality my checks are part of a much larger bash function, because the corrections and permutations are so many that I need to hand-select files for correction. They need to be done in a specific order when images might have multiple categories.

In the long term, I would like to develop a consistency checker (if it does not appear in XnView) that spans captions and keywords, and also interrogates the XnView DB, but at the moment the captions are stored within an opaque (to me) blob in the DB. At the moment my exiftool solutions are adequate for me.

[danjcla]

I must be cursed; touching files didn't work for me either. In fact I can close the app, manually delete the .xmp files, and yet the GUI still shows XMP info.

I'm giving up on this the file-based to DB-based metadata synchronization; for me it was just going to be part of a workaround to get what would basically be computed categories - but I think I'm going to just say F-it and write computed categories directly to XnViewMP's SQLite DB via an outside script.

I've read elsewhere that having multiple XnView MP instances across a network accessing the same DB shouldn't be a problem for a small number of users, so having a script run periodically shouldn't pose any issues as long as the script itself doesn't do stupid things. And making new category associations to a file looks like really straight forward SQL.

[CameronD]

In fact I can close the app, manually delete the .xmp files, and yet the GUI still shows XMP info.

So you have embedded XMP as well as XMP sidecars. I wonder if they were even consistent.

[CameronD]

Here is a suggested process to reproduce the issues I have identified above:
Setup

• create a new local user on Windows 7-64bit - to ensure I have no old settings lying around
• start XnviewMP Windows x64 V0.72, with initial config as default; selecting language:English might be necessary.
• Starting from default MP settings, change the following :
  1. General->File operations: untick each item under "Keep original date/time"
  2. Browser->Metadata->IPTC&XMP: tick/enable "Export DB Category to XMP ..." and "Automatically apply parent keywords..."
• In the Edit IPTC/XMP options tab confirm the following (should be default):
  1. The 3 IPTC options for multiple selection are all ticked.
  2. Mode: set to "IPTC-IIM, update or create XMP".
• Switch browser main window to "details" view
• right click on the details title bar and select "None" to delete all columns, then enable "Info, Caption, Keywords, Categories"
• shutdown program, save copy of APPDATA folder and restart program - allows restoration to known state.
• prepare folders with test images: folder 1 needs a single image with no embedded keywords or XMP:subjects. Folder 2 has sample including images with various new keywords. This complexity is because of bug number 1 and to make bug 2 start from a consistent place. Make backups of the image files.

Bugs/misfeatures:

1. The "Categories" sub-window is completely greyed out unless I have an image selected. I cannot even import or export category lists.
2. Browse to the starting folder 1 described above and select a file. Under categories management, export the current category list. Then also under "categories management" select "clear list" and then re-import the same list. The ordering seems odd - they are almost alphabetical, except "Audios" is now at the bottom.
3. Because of this automatic sorting, I can see no way to sort the list to my liking, putting common categories to the top. I found no UI for it, so was hoping to manually sort them via file.
4. Browse to another directory, containing images previously categorised with MP. For example I have an image with category "flower", subcategory "Drosera" and the XMP:lr:hierarchical subjects had been saved to the file. When I select the image (or probably when the folder was initially scanned), the top level "flower" is created, as is subcategory "Drosera". The category has Drosera ticked (but not the parent). However, MP has also created an additional spurious subcategory named "flower,Drosera" and placed it under "Audio"! This was partly a consequence of a bug listed further below (no 7?).
5. In another image (from the internet), the IPTC keyword is "Macbeth,ColorChecker" and it is duplicated in XMP:dc as Subject[1]=Macbeth and Subject[2]=ColorChecker. These are correctly recognized as two separate categories, but they are created as subcategories of "Audio".
6. Select an image with no existing keywords or captions and then ctrl-I (Edit IPTC/XMP). Select tab "keywords"; from the dropdown source, change "history" to "categories (database)" ; pick "Photographs -> Landscapes" and assign to image; save. The IPTC:keyword is given as "Photographs|Landscapes", and hierarchicalSubject[1]="Photographs|Landscapes" but XMP:dc only has Subject[1]="Landscapes". Also the parent "Photographs" has not been ticked in the database, only the child "Landscape" is enabled.
7. Select another image with no keywords and, in the categories subwindow, click on "Photographs->Pets" to assign that category to the image. The IPTC:keyword becomes "Photographs,Pets" (note the comma separator rather than '|'. XMP:dc provides Subject[1] and Subject[2] and XMP:lr: has hierarchicalSubject[1]="Photographs" as well as hierarchicalSubject[2]="Photographs|Pets"
8. as a secondary bug in this situation, the details listing does not show the updated keywords - they are visible in the Info tab for the file, but are never listed until I browse to a different folder and return.
9. Exit MP; Overwrite the two files changed above with backup copies; restart MP, select the files and select menu "view->update files from catalog". The files are not touched, but .xmp sidecar files are produced. MP does not read these files, possibly because there is already embedded XMP data in the image file. No IPTC data is created, therefore no IPTC:keywords.
10. Exit MP; delete the .XMP sidecar files; restart MP. Note the image files have no XMP or IPTC keyword content. Choose "update catalog from files" and nothing happens. The existing categories remain in the database, assigned to the files. On the other hand, if the process is done with image files that have some embedded keyword/subject content, then the category DB will be resynchronized as expected.
11. At this stage, I deleted the XnView.db file, and copied over a fresh set of images from my backups. Restarted MP and visited the test folder; because it reset the category list, the newly imported entries went under Others, rather than Audio.
12. Select 2 images with no existing keywords or captions; in the categories subwindow, tick "Photographs->Family"; (optional: check IPTC tab of Info window showing new keywords and reselect both images); then edit IPTM/XMP; type a caption; save-all. If I did the optional step then all is as expected. However if I went straight from category selection to Edit-IPTC then the previous keywords (or null) are retained and not updated to reflect categories.
13. clear the DB and refresh the files. Select 1 image with no existing keywords or captions; in the categories subwindow, tick "photographs->Family"; browse out to a different folder and back again; select the file just changed and also select another image with no caption or keywords; ctrl-I, type a new caption; save-all and look at details list. Keywords have been duplicated from first file to second, but categories DB has not been updated.
14. By the way, using '|' as a separator with no space is not a good choice, because other programs do not necessarily recognise it and regard the compound as a single word - making searches fail.

[xnview]

[*]The "Categories" sub-window is completely greyed out unless I have an image selected. I cannot even import or export category lists.
[*]Browse to the starting folder 1 described above and select a file. Under categories management, export the current category list. Then also under "categories management" select "clear list" and then re-import the same list. The ordering seems odd - they are almost alphabetical, except "Audios" is now at the bottom.
[*] Because of this automatic sorting, I can see no way to sort the list to my liking, putting common categories to the top. I found no UI for it, so was hoping to manually sort them via file.

Ok, right...

[*] Browse to another directory, containing images previously categorised with MP. For example I have an image with category "flower", subcategory "Drosera" and the XMP:lr:hierarchical subjects had been saved to the file. When I select the image (or probably when the folder was initially scanned), the top level "flower" is created, as is subcategory "Drosera". The category has Drosera ticked (but not the parent).

The setting "Automatically apply parent keywords..." is when you select a category in the UI
For this file, the IPTC/XMP keyword is "flower,Drosera", not 2 words!! And the XMP:lr:hierarchical subject has only a category (Drosera), if you have 2 words, lr must be "subject[1]=flower,Drosera, subject[2]=flower"

However, MP has also created an additional spurious subcategory named "flower,Drosera" and placed it under "Audio"! This was partly a consequence of a bug listed further below (no 7?).
[*]In another image (from the internet), the IPTC keyword is "Macbeth,ColorChecker" and it is duplicated in XMP:dc as Subject[1]=Macbeth and Subject[2]=ColorChecker. These are correctly recognized as two separate categories, but they are created as subcategories of "Audio".

Right a bug!

[*]Select an image with no existing keywords or captions and then ctrl-I (Edit IPTC/XMP). Select tab "keywords"; from the dropdown source, change "history" to "categories (database)" ; pick "Photographs -> Landscapes" and assign to image; save. The IPTC:keyword is given as "Photographs|Landscapes", and hierarchicalSubject[1]="Photographs|Landscapes" but XMP:dc only has Subject[1]="Landscapes". Also the parent "Photographs" has not been ticked in the database, only the child "Landscape" is enabled.

You think that setting "Automatically apply parent keywords..." must used here??

[*]Select another image with no keywords and, in the categories subwindow, click on "Photographs->Pets" to assign that category to the image. The IPTC:keyword becomes "Photographs,Pets" (note the comma separator rather than '|'. XMP:dc provides Subject[1] and Subject[2] and XMP:lr: has hierarchicalSubject[1]="Photographs" as well as hierarchicalSubject[2]="Photographs|Pets"

I can't reproduce, i have always | not , as separator

[*] as a secondary bug in this situation, the details listing does not show the updated keywords - they are visible in the Info tab for the file, but are never listed until I browse to a different folder and return.

Right

[*]Exit MP; Overwrite the two files changed above with backup copies; restart MP, select the files and select menu "view->update files from catalog". The files are not touched, but .xmp sidecar files are produced. MP does not read these files, possibly because there is already embedded XMP data in the image file. No IPTC data is created, therefore no IPTC:keywords.

Yes, right IPTC/XMP must be updated!!

[*]Exit MP; delete the .XMP sidecar files; restart MP. Note the image files have no XMP or IPTC keyword content. Choose "update catalog from files" and nothing happens. The existing categories remain in the database, assigned to the files. On the other hand, if the process is done with image files that have some embedded keyword/subject content, then the category DB will be resynchronized as expected.

Right, the entry in database is not removed before the update...

[*]Select 2 images with no existing keywords or captions; in the categories subwindow, tick "Photographs->Family"; (optional: check IPTC tab of Info window showing new keywords and reselect both images); then edit IPTM/XMP; type a caption; save-all. If I did the optional step then all is as expected. However if I went straight from category selection to Edit-IPTC then the previous keywords (or null) are retained and not updated to reflect categories.

right, a bug

[*]clear the DB and refresh the files. Select 1 image with no existing keywords or captions; in the categories subwindow, tick "photographs->Family"; browse out to a different folder and back again; select the file just changed and also select another image with no caption or keywords; ctrl-I, type a new caption; save-all and look at details list. Keywords have been duplicated from first file to second, but categories DB has not been updated.

right...

[*]By the way, using '|' as a separator with no space is not a good choice, because other programs do not necessarily recognise it and regard the compound as a single word - making searches fail.[/list]

so a settings to choose between |,/\.: ??

[CameronD]

...
The setting "Automatically apply parent keywords..." is when you select a category in the UI
For this file, the IPTC/XMP keyword is "flower,Drosera", not 2 words!! And the XMP:lr:hierarchical subject has only a category (Drosera), if you have 2 words, lr must be "subject[1]=flower,Drosera, subject[2]=flower"

I am not so much worried about the meaning of "Automatically apply parent keywords..." as to the lack of consistency in behaviour.
If I create image metadata with that setting, then the database gets corrupted and I re-import the image metadata, I would expect that the same categories get restored. But this is not happening in all cases. XnViewMP will write XMP data in one way, but will interpret it differently when it reads it back in.

[*]Select another image with no keywords and, in the categories subwindow, click on "Photographs->Pets" to assign that category to the image. The IPTC:keyword becomes "Photographs,Pets" (note the comma separator rather than '|'. XMP:dc provides Subject[1] and Subject[2] and XMP:lr: has hierarchicalSubject[1]="Photographs" as well as hierarchicalSubject[2]="Photographs|Pets"

I can't reproduce, i have always | not , as separator

My misunderstanding. See below.

[*]By the way, using '|' as a separator with no space is not a good choice, because other programs do not necessarily recognise it and regard the compound as a single word - making searches fail.[/list]

so a settings to choose between |,/\.: ??

Again this might be a misunderstanding on my part. I have only just realised that the comma I am seeing is not stored in the file. Instead, XnView is displaying two separate entries and inserting a comma as the separator. Exiftool uses the '*'. I did not even realise the IPTC keywords were stored as separate words.
I think if the code can reliably save/restore the hierarchy using the XMP:lr:hierarchicalSubject then leaving the categories separate in IPTC and XMP:dc:Subject then that is good enough for me.
Above all, my main aim was to be able to restore the same state on loss of the DB or copying a subset of photos to another system.

I shall try the test version soon.

[xnview]

But this is not happening in all cases. XnViewMP will write XMP data in one way, but will interpret it differently when it reads it back in.

in which case you have not the same thing??
Perhaps i need to add '*' as keywords separator if exiftools use it?

[CameronD]

But this is not happening in all cases. XnViewMP will write XMP data in one way, but will interpret it differently when it reads it back in.

in which case you have not the same thing??
Perhaps i need to add '*' as keywords separator if exiftools use it?

I need to understand the issue better first, before commenting on this.

The use of '*' is a shorthand for display only - it is not used inside the metadata itself.
In the case of two separate keywords, instead of reporting:

Code: Select all

Subject[1] = flower
Subject[2] = Drosera

the exiftool text dump simply shows

Code: Select all

Subject = flower*Drosera

In the same way that XnView is using comma.

However, sometimes my workflow has managed to create

Code: Select all

Subject="flower, Drosera"

where there is a single keyword entry and the comma is embedded on the file's metadata.

Because I was not aware of this distinction until now, I might have not reported it correctly.

I will do some more tests. I am quite confident that it is some process within MP that writes it in this way, but I will need to reproduce it.

I think the difference is visible in the "keywords" column of the details listing. When the keywords are stored separately, XnViewMP separates them with a comma and no space, but when they have been merged in the metadata they seem to be shown with a comma and a space.

So, Pierre, can you confirm the expected behaviour?
Is it that the only place the keyword separator should appear is in the XMP:lr:HierarchicalSubject items?
Should multiple keywords in IPTC and XMP:subject always be stored as multiple entries?

[xnview]

However, sometimes my workflow has managed to create

Code: Select all

Subject="flower, Drosera"

where there is a single keyword entry and the comma is embedded on the file's metadata.
Because I was not aware of this distinction until now, I might have not reported it correctly.

I will do some more tests. I am quite confident that it is some process within MP that writes it in this way, but I will need to reproduce it.

Strange, yes please check and tell me...

So, Pierre, can you confirm the expected behaviour?

in the iptc panel, you can see if it's only one or two keywords...

Is it that the only place the keyword separator should appear is in the XMP:lr:HierarchicalSubject items?
Should multiple keywords in IPTC and XMP:subject always be stored as multiple entries?

yes