v0.72 DB-Category <-> IPTC keyword problems.

[CameronD]

...
in the iptc panel, you can see if it's only one or two keywords...

yes, I see - separate keywords in the IPTC block are shown on separate lines.
If I see keywords on one line then it means they are stored in the image metadata as a single item.

[CameronD]

However, sometimes my workflow has managed to create

Code: Select all

Subject="flower, Drosera"

where there is a single keyword entry and the comma is embedded on the file's metadata.
Because I was not aware of this distinction until now, I might have not reported it correctly.

I will do some more tests. I am quite confident that it is some process within MP that writes it in this way, but I will need to reproduce it.

Strange, yes please check and tell me...

I think the issue of commas in keywords is a problem all of my own making.
I wrote an exiftool script to try to correct the problems of bug 6 above, where XnViewMP was writing keyword entries as "parent|child".
Since the details column and IPTC panels were showing categories separated by comma, I assumed that that was the correct format, and wrote exiftool commands to perform the operation to simply replace the '|' by ','.

And now I discover that exiftool displays multiple entries as separated by a comma+space by default, but does not reverse the process by default.
I mentioned in a post above about the '*' separator, but that is a change made by exiftool-gui.

[CameronD]

Here are my results for checking against the beta QT5 32-bit Windows version from mid-April 2015.
In summary, remaining bugs are mainly to do with:

• display updates - the Browser window details columns do not match updated keywords in files.
• DB-Categories assigned via Edit IPTC window. multiple bugs remain here. For my purposes, I would be happy to never touch the keywords tab, and just use the DB-categories window, however there are still cases where applying the same caption to a batch of files will mess up the link between DB-category and XMP content.

1 The "Categories" sub-window is completely greyed out unless I have an image selected. I cannot even import or export category lists.
2. Browse to the starting folder 1 described above and select a file. Under categories management, export the current category list. Then also under "categories management" select "clear list" and then re-import the same list. The ordering seems odd - they are almost alphabetical, except "Audios" is now at the bottom.

1 and 2 are fixed.

3. Because of this automatic sorting, I can see no way to sort the list to my liking, putting common categories to the top. I found no UI for it, so was hoping to manually sort them via file.

I would still would like some way to change the display order of the list, but I understand it is not as trivial as changing the order in the import file, so it is not high importance.

4. Browse to another directory, containing images previously categorised with MP. For example I have an image with category "flower", subcategory "Drosera" and the XMP:lr:hierarchical subjects had been saved to the file. When I select the image (or probably when the folder was initially scanned), the top level "flower" is created, as is subcategory "Drosera". The category has Drosera ticked (but not the parent). However, MP has also created an additional spurious subcategory named "flower,Drosera" and placed it under "Audio"! This was partly a consequence of a bug listed further below (no 7?).

Not a bug. The problem was with source images, I suspect I may have introduced all these through incomplete understanding of the requirements.

5. In another image (from the internet), the IPTC keyword is "Macbeth,ColorChecker" and it is duplicated in XMP:dc as Subject[1]=Macbeth and Subject[2]=ColorChecker. These are correctly recognized as two separate categories, but they are created as subcategories of "Audio".

Mostly fixed. Except once, but it is not reproducible (see more detail below).

6. Select an image with no existing keywords or captions and then ctrl-I (Edit IPTC/XMP). Select tab "keywords"; from the dropdown source, change "history" to "categories (database)" ; pick "Photographs -> Landscapes" and assign to image; save. The IPTC:keyword is given as "Photographs|Landscapes", and hierarchicalSubject[1]="Photographs|Landscapes" but XMP:dc only has Subject[1]="Landscapes". Also the parent "Photographs" has not been ticked in the database, only the child "Landscape" is enabled.

Not fixed - The IPTC keyword is still stored and displayed as a new single keyword: "Photographs|Landscapes". Also, the parent is not automatically selected separately. I think the automatic parent selection should follow the user preference from Browser->metadata.

7. Select another image with no keywords and, in the categories subwindow, click on "Photographs->Pets" to assign that category to the image. The IPTC:keyword becomes "Photographs,Pets" (note the comma separator rather than '|'. XMP:dc provides Subject[1] and Subject[2] and XMP:lr: has hierarchicalSubject[1]="Photographs" as well as hierarchicalSubject[2]="Photographs|Pets"

originally was probably not a bug, just a misunderstanding on my part (see my suggestion under point 14). However, I notice the "Pets" checkbox is tristate, and when I click once, the "Photographs" box is ticked, but the "Pets" box is blocked in blue and when I click again it becomes ticked. It is the condition for multiple selection where not all selected items have the same value. But I only have one file selected, so it should be automatically ticked. I think this is a bug existing from the earlier versions, but it was not as obvious with a different theme.

8. as a secondary bug in this situation, the details listing does not show the updated keywords - they are visible in the Info tab for the file, but are never listed until I browse to a different folder and return.

Not fixed

9. Exit MP; Overwrite the two files changed above with backup copies; restart MP, select the files and select menu "view->update files from catalog". The files are not touched, but .xmp sidecar files are produced. MP does not read these files, possibly because there is already embedded XMP data in the image file. No IPTC data is created, therefore no IPTC:keywords.
10. Exit MP; delete the .XMP sidecar files; restart MP. Note the image files have no XMP or IPTC keyword content. Choose "update catalog from files" and nothing happens. The existing categories remain in the database, assigned to the files. On the other hand, if the process is done with image files that have some embedded keyword/subject content, then the category DB will be resynchronized as expected.

9, 10. Almost fixed. Select some or all files in folder, "update files from catalogue" and the details listing columns for caption and keywords are blanked out. This is just a display refresh problem - files have been updated properly, and no XMP sidecar is produced.

11. At this stage, I deleted the XnView.db file, and copied over a fresh set of images from my backups. Restarted MP and visited the test folder; because it reset the category list, the newly imported entries went under Others, rather than Audio.

fixed as under points 1, 2.

12. Select 2 images with no existing keywords or captions; in the categories subwindow, tick "Photographs->Family"; (optional: check IPTC tab of Info window showing new keywords and reselect both images); then edit IPTM/XMP; type a caption; save-all. If I did the optional step then all is as expected. However if I went straight from category selection to Edit-IPTC then the previous keywords (or null) are retained and not updated to reflect categories.

The reported bug seems to be fixed, but no XMP:hierarchicalSubject is created. The same "not all items are equal" checkmark happens as for point 7.

13. clear the DB and refresh the files. Select 1 image with no existing keywords or captions; in the categories subwindow, tick "photographs->Family"; browse out to a different folder and back again; select the file just changed and also select another image with no caption or keywords; ctrl-I, type a new caption; save-all and look at details list. Keywords have been duplicated from first file to second, but categories DB has not been updated.

Result is different, but imperfect. Either the conditions to reproduce are more complex than I thought, or one bug has replaced another. This time, the keywords are not copied over to the second file, and so the categories do not need updating. However, after the save-all, the original image loses its XMP:hierarchicalSubject, while the XMP:dc:Subject entries remain untouched. This is a complex scenario, because of the option settings for "IPTC update keywords on multiple selection", and because it might depend on the selection order ...

13b. The same 2 images, both with the same caption, one with keywords, the other with no keywords. Select both and ctrl-I. Change to the "Keywords" tab. Flip through images (left/right arrow buttons under the preview) until the existing keywords are loaded into the box. Then save-all. The IPTC and XMP entries are copied to the second image, but the category DB is not updated for the changed image.

14. By the way, using '|' as a separator with no space is not a good choice, because other programs do not necessarily recognise it and regard the compound as a single word - making searches fail.

not a problem - my misunderstanding. However, to avoid misunderstanding in future it might be good if an option is added to Browser metadata to allow the user to specify a separator character or string for display only between multiple keywords. This could be used in the details columns and also the IPTC tab.

Other bugs noticed during testing:

"export category list" does not automatically append any extension to the new file name. This means that it will not find it when you try to re-import it because the import dialogue requires a .txt extension.

Importing from images into new database - this is the one instance of bug 5 occurring again.
start with Fresh DB (deleted old one).
browse to folder 1, image with multiple XMP:subject, but no XMP:hierarchicalSubject - XnViewMP automatically adds entries under "Other"
browse to folder 2, (contains copy of same image) recreates category entries under Audio instead of matching the ones under "Other".
I then exited folder 2, selected a different image and right-click-deleted the new incorrect categories under audio. Return to image in folder 2 and the "Drawings" category is ticked, as is "Photographs->family". Unfortunately none of this was reproducible - I tried the same thing again from scratch and the steps must have been slightly different.

Display update bug: Select a single image that has previously had category "Photographs->Family" applied. Untick both categories in the Categories window. The Keywords are removed from the file's XMP and IPTC lists, but remains in the details column until you leave the folder and return.

A request to reduce confusion: could the "categories" section within IPTC be relabelled as "IPTC-Categories"? Then it is more obviously different from your DB categories.

[CameronD]

I have some more bugs to report, or perhaps other manifestations of the same bug.
The report in the previous post was done on a clean install for another user. Today I found the same/similar bug using the beta QT5 test code on my working images, retaining my existing db:

The first category on my personal list is "bird" and it has child categories such as "duck", "raptor", "wader" and so on.
When I use the category window to assign bird->raptor to a single jpeg image (I have not tried multiple images) it automatically selects bird in line with my preference to automatically select parent, and both entries are written to the IPTC and XMP locations as expected.
However, the heirarchical Subjects are written as

Code: Select all

heirarchicalSubject[1] = raptor
heirarchicalSubject[2] = bird

instead of

Code: Select all

heirarchicalSubject[1] = bird|raptor
heirarchicalSubject[2] = bird

So it looks as if my "bird" category is being treated as the place to put miscellaneous categories instead of "Other".

The same test image that caused the non-reproducible bug in the previous post also caused the same problem here:
That image has 3 xmp subjects: "Macbeth", "Colorchecker" and "UV", but no hierarchicalSubject entry.
When I ran the beta MP version and browsed to a new test folder with this image, the program created new child categories:

• Macbeth and colorchecker were placed under "bird" and
• "UV" was placed under "zz_pad" (which I had previously created as a placeholder for the end of the imported list, and was somehow automatically being used as the parent category for unallocated categories)

When I search for any of these categories, the search engine returns just the one file.
I can find nothing in the DB tag structure that would indicate why this happened, except that the common factor is that on both systems where the bug occurred, the wrong parent had Tags.TagID value of 1.
That is the birds category in today's example and the Audio category the first time it happened. The "Other" category is not number 1 in either case.

I then renamed the folders with the bird->Macbeth images, optimised the database including removing orphan entries, deleted the spurious categories (right click->delete).
Then when I browsed to the folders again, this time the categories were created under Other, as expected.

Edit: when I used the old X64 version to test the search count bug I noticed that the "birds" entry now has been moved to the bottom of the category window..

[xnview]

12. Select 2 images with no existing keywords or captions; in the categories subwindow, tick "Photographs->Family"; (optional: check IPTC tab of Info window showing new keywords and reselect both images); then edit IPTM/XMP; type a caption; save-all. If I did the optional step then all is as expected. However if I went straight from category selection to Edit-IPTC then the previous keywords (or null) are retained and not updated to reflect categories.

The reported bug seems to be fixed, but no XMP:hierarchicalSubject is created.

When XMP:hierarchicalSubject must be created???

13. clear the DB and refresh the files. Select 1 image with no existing keywords or captions; in the categories subwindow, tick "photographs->Family"; browse out to a different folder and back again; select the file just changed and also select another image with no caption or keywords; ctrl-I, type a new caption; save-all and look at details list. Keywords have been duplicated from first file to second, but categories DB has not been updated.

Result is different, but imperfect. Either the conditions to reproduce are more complex than I thought, or one bug has replaced another. This time, the keywords are not copied over to the second file, and so the categories do not need updating. However, after the save-all, the original image loses its XMP:hierarchicalSubject, while the XMP:dc:Subject entries remain untouched. This is a complex scenario, because of the option settings for "IPTC update keywords on multiple selection", and because it might depend on the selection order ...

You have not the 'Always clear all field' checked??

However, to avoid misunderstanding in future it might be good if an option is added to Browser metadata to allow the user to specify a separator character or string for display only between multiple keywords. This could be used in the details columns and also the IPTC tab.

, is not good as separator?

Importing from images into new database - this is the one instance of bug 5 occurring again.
start with Fresh DB (deleted old one).
browse to folder 1, image with multiple XMP:subject, but no XMP:hierarchicalSubject - XnViewMP automatically adds entries under "Other"
browse to folder 2, (contains copy of same image) recreates category entries under Audio instead of matching the ones under "Other".
I then exited folder 2, selected a different image and right-click-deleted the new incorrect categories under audio. Return to image in folder 2 and the "Drawings" category is ticked, as is "Photographs->family". Unfortunately none of this was reproducible - I tried the same thing again from scratch and the steps must have been slightly different.

If you can reproduce it, please send me picture file, and the step? I have always the good category in Other

[CameronD]

The first category on my personal list is "bird" and it has child categories such as "duck", "raptor", "wader" and so on.
When I use the category window to assign bird->raptor to a single jpeg image (I have not tried multiple images) it automatically selects bird in line with my preference to automatically select parent, and both entries are written to the IPTC and XMP locations as expected.
However, the heirarchical Subjects are written as

Code: Select all

heirarchicalSubject[1] = raptor
heirarchicalSubject[2] = bird

instead of

Code: Select all

heirarchicalSubject[1] = bird|raptor
heirarchicalSubject[2] = bird

So it looks as if my "bird" category is being treated as the place to put miscellaneous categories instead of "Other".

I have tested this problem with v 0.75 beta (May 11 2015).
The bug still exists when I assign the category from the "Categories" sub-window or dockable window, or whatever it is called. (view->data pane->Categories).
However, if I do the category assignment and then "View -> Update files from catalog", then the hierarchical subject is returned...

Code: Select all

heirarchicalSubject[1] = bird
heirarchicalSubject[2] = bird|raptor

I do not know if it is significant, but this time the order is reversed: [1] is the parent and [2] is parent|child.

The next stage: if I use the Categories pane to deselect all categories, browse to another folder and back, then the IPTC keywords and XMP:Subjects have been removed, but one of hierarchicalSubjects has been left in place. Thus it retains:

Code: Select all

heirarchicalSubject[1] = bird|raptor

Again, if I run "View -> Update files from catalog", then the hierarchical subject is cleared.

[CameronD]

When XMP:hierarchicalSubject must be created???

If I assign categories via the "Categories" data pane, then XMP:hierarchicalSubject entries are created.
It seems logical to me that if I have chosen the options to keep categories and keywords matching, and if I am choosing from the category list itself (in the IPTC keyword tab), then the assigned XMP subjects should behave the same way as assinging from the Categories pane..
I understand that there are logically a lot of possible pathways and if you want to say that new hierarchicalSubject entries can only be created through the "Categories" pane then I am happy to accept that.

One thought has just occurred to me - in the default category structure, what would happen if I created a subcategory "Drawings->Family" ? Is the name duplication at the second level allowed? I can see that if the use of XMP:hierarchicalSubject was not strictly followed then you could not guarantee reconstruction of a corrupted database, as the single keyword entries could be ambiguous, especially without automatic parent selection.

You have not the 'Always clear all field' checked??

I do not know where this option is (in the GUI), and I do not know what it does.
In xnview.ini, I have:

Code: Select all

[IPTC]
clearAlways=true

in both my old x64 version and new beta test ones.

However, to avoid misunderstanding in future it might be good if an option is added to Browser metadata to allow the user to specify a separator character or string for display only between multiple keywords. This could be used in the details columns and also the IPTC tab.

, is not good as separator?

Forget about it - it would have been useful for diagnosis, but only after I realised what a mess I had made. So the time when I needed it was when I would not have understood that I needed it. I can get the same information by using exiftool, or windows explorer (which uses ';' as the keyword separator).

Importing from images into new database - this is the one instance of bug 5 occurring again.
...

If you can reproduce it, please send me picture file, and the step? I have always the good category in Other

I think this is one of my mistakes again. I am not certain, but it seems to be associated with when I again run the x64 v0.72 version of MP. I was doing that occasionally to check if behaviour was different between versions. It is possible that the old version updated the DB categories wrongly, but that I only noticed it after I went back to the new beta version.

[xnview]

Do you have always issues with lastest version?

[CameronD]

Do you have always issues with lastest version?

I am still having some issues, but this was reading old images into the new version on my laptop.

At this stage it is possible that the images still had wrong keywords from previous mistakes or bugs, so I need to look into it more closely.

This may take some time.

[tobias2]

Hi,

I am new to xnview and have similar issues described above (using latest version 0.82 now)
Not sure whether it should go to a separate thread or not, but here we go:
if a picture contains iptc keywords "key1" and "key2" and the xmp dc subject tag only has "key1" xnview sometimes (i wish it was consistent ) shows only key 1 as a category.
In the info tab and the keyword editor (ctrl-i) key1 and key2 are both shown.

As soon as I add or remove a category from xnview to the picture (for example "key3") key2 gets removed from the iptc keywords.
I tried to workaround this by removing all xmp tags using "exiftool -XMP:all= test.jpg". this removes the xmp part, but it does not cause xnview to pickup the iptc keywords as a category.

desired behavior:
-xnview should import iptc keywords and xmp dc subjects
-adjusting categories from xnview should adjust iptc keywords and xmp dc subjects (ideally xnview would keep dc subject and iptc keywords in sync ?)

thanks in advance for your answer!

[CameronD]

desired behavior:
-xnview should import iptc keywords and xmp dc subjects
-adjusting categories from xnview should adjust iptc keywords and xmp dc subjects (ideally xnview would keep dc subject and iptc keywords in sync ?)

Hi Tobias,
I can give you an answer only from a user perspective...

1. xnview will import keywords/subject as category - if you have the option enabled (Settings->Browser->metadata)- but I think it assumes iptc-keyword is the same as xmp-subject
2. likewise an option allows you to enable assigning keyword/subject from the category
3. there is another option, I think only in the "Edit IPTC" window; Options tab, that says IPTC, XMP or both.

I know of no setting in xnview that allows you to identify if IPTC and XMP values are no longer in sync. I use an exiftool command for that.

[tobias2]

Hi cameron,

thanks for responding / trying to help out.
Re-reading my own post I should correct the version i use: it's 0.81. not sure if that matters.
regarding the settings i have enabled in the settings->metadata
"import xmp subject or iptc keywords to categories" : enabled
"export db category to xmp subject and iptc keyword" : enabled
"create XMP data": enabled
"create IPTC data":enabled
all other options are disabled.
I enabled "create iptc data" because another app (piwigo) depends on reading the iptc keyword data.

Regarding your tip on exiftool:
I synced the keywords to the xmp dc subject for a few pictures. result in xnview mp: nothing changes
next step: dropping the picture from the xnview mp and re-adding it imports all keywords... (i did so by moving the test-file to a directory that was not watched by xnview mp)
so looks like issue can be worked around!

maybe another (for me easier) solution would be if an end user can select whether he wants to import only iptc keywords ?

I will do some more testing / keyword syncing now and see how this plays out and would appreciate it if the author can share some more on the behavior when the iptc / xmp subjects are not in sync..

thanks!

[CameronD]

I enabled "create iptc data" because another app (piwigo) depends on reading the iptc keyword data.

Strictly speaking, this is the old IPTC-IIM. The current IPTC standard is just to use the XMP equivalents.
I would really like to be able to eliminate IPTC-IIM, but even xnviewMP requires this for the "details" file listing.

Regarding your tip on exiftool:
I synced the keywords to the xmp dc subject for a few pictures. result in xnview mp: nothing changes
next step: dropping the picture from the xnview mp and re-adding it imports all keywords... (i did so by moving the test-file to a directory that was not watched by xnview mp)
so looks like issue can be worked around!

maybe another (for me easier) solution would be if an end user can select whether he wants to import only iptc keywords ?

If another application modifies the file, and it behaves properly and updates the file modification date, then xnview will reread the file contents and update its database. Except that it will not resync embedded metadata with catalogue categories.
To do that, you select the files and use the menu option view->update catalog from files. Yes, it's an editing function on the view menu.

Or, in major cases I just delete the db file and have xnviewMP rescan.

[CameronD]

Do you have always issues with lastest version?

I have done some testing. There are still a few bugs, but, because I have changed the way I process files, I had not noticed.

Display Updates:
There are still situations where the display does not update until I select away and back. Sometimes I just need to select a new file, other times I need to browse to a new folder and back.
To reproduce, put Browser in details view, Keywords column enabled; Categories and Info panes both visible; XMP or iptc info tab showing; select one file:

1. click on a category to activate it - Keywords column does not change; IPTC and XPM tabs in info pane do not change. Deselect image and reselect: IPTC and XMP tabs now show updated keywords/subject and tooltip is updated; Keywords column is still not updated. Select another folder, and reselect original folder - new Keywords are displayed.
2. select an image - select view->update catalogue from file - categories are all unticked in the Categories pane until you deselect and reselect file.
3. select an image with keywords removed outside of xnview - select view->update files from catalogue - same behaviour as in the first case.

Entering Keywords from Edit IPTC window
This is still broken, but I have just stopped using it, so had not noticed.
To reproduce:

• select one image file only
• tools->metadata->edit IPTC. Select Keywords tab; from dropdown box select Categories (database) to give the category list.
• click a parent to list the child entries
• double click the child and get the following keywords added:

  Code: Select all

  parent|child
  parent

• Note, parent and child are merged on the line. This erroneous line remains as an IPTC entry.
• The XMP Subjects are correctly rendered, but the hierarchicalSubject has only "parent|child"

Second test:

• select two image files (empty of keywords - to avoid confusion with multiple image options)
• edit IPTC again, adding keywords the same way.
• click Save All
• No categories are changed for either file. although the XMP:subject is updated.

Re-importing categories from new files
This works well now, with just a few problems partly of my own making. I have identified the situation causing some apparent lack of reproducibility.

• I created a few test images, with parent+child, parent only, child only keywords and also made copies that were missing the hierarchicalSubject entries.
• delete the xnview.db file, restart xnview and browse to the test folder.
• new categories will be created as parent/child, but sometimes there will be a second "child" category at the top level. I think whether this happens or not depends on the order in which the files are processed, which seems to be the file display sort order. If the first file found has parent and child but no hierarchicalSubject then they will be created at the top level. Once a hierarchicalSubject entry is found then subsequent "child" entries will be placed below the parent, even if the image file does not supply the hierarchy. (This is good)
• my problem is having numerous files missing hierarchicalSubject entries. The solution is to make sure I have saved a good copy of the category structure, delete the DB, restore the category structure and then rescan the files. This way it should not depend on the order in which files are found. If I find the files that are missing the hierarchy then I can always "update from category"