Writing Categories to XMP overwrites XMP subjects from other sources [Solved]

[amimaster]

TL;DR;

Just make sure your XnView is updated.

Hello,
I'm facing an issue that might be quite difficult to understand, so I'll explain my general workflow to allow you to understand better.

Some years ago I moved from Lightroom to XnView MP for adding XMP keywords to my image folder containing thousands of images.
The workflow I established allowed me to define a set of keywords I was interested in (100+), then grouping them using category sets
(e.g., group "Place" contained "interior, outdoors, beach, forest, street, ...") allowed me to have a nice interface to assign keywords from each group and to "convert" them into keyword by using the "export categories to XMP" setting. Everything worked fine and I tagged plenty of images.
Then I stopped doing that due to lack of time, but continued taking photos and stacking them.
Recently, I trained an AI classifier to output some keywords (different from those I used in XnView) and wrote a script for saving them in XMP metadata.
Since I needed a nice interface for manually validating my model output (and to actually manage keywords), I opened XnView again.

So, what happens is the following (where A* are XnView categories and B* are my new keywords coming from my script):

- I start with a image with no tags.
- I assign tags A1, A2, A3 using XnView Category Groups. They are correctly written in the Xmp.dc.subject tag within the image file.
- I use my script to add subjects B1, B2, that XnView have never seen.
- I can see the tags correctly applied using Gwenview ExifTool, reporting A1, A2, A3, B1, B2.
- I open XnView, select "Update Catalog from Files", then select the image
- B1, B2 are added to the list in the Categories tab, and I can see them in XMP viewer, but I cannot see them in the Assigned Category Group nor in assigned categories.
- If I try to edit the categories, e.g. adding A4, as soon as I click on another image, my XMP subjects reverts to A1, A2, A3, A4, discarding everything else that has been imported.

I remember during my several tries that sometimes XMP subjects are removed altogether by XnView, leaving only image rating and another tag.. I would edit this post with more specific information as soon as I can access my pc...

Is that some kind of bug, or I'm not recalling an important step? I know that categories group are stored internally and keywords are stored in XMP, but I thought that the update of the catalog would also reflect on categories, and that "Export categories to XMP" would just add new categories to XMP leaving there the ones that are already present... Is there any workaround for this?

[Gerald_Ire]

Hello amimaster, just a short note on this topic...

1st: wow, things you do... thumbs up!
Handle keywords/subject isnt easy...
Our experience with a software developer and the implentation of metadata showed us the pain...

Cant give you a solution, only some things to check in your workflow:

Check XnViewMP Tools/Settings/Metadata/... for some options...

As far as I remember "keywords/subject" are in a container. To add more keywords/subject you have to ad container_A + container_B to get them all together...

Check ExifTool Forum for more infos...

[michel038]

Perhaps "Update catalog from files" needs an import option to be enabled : Settings>Metadata>Import XMP...

I got problems some years ago with import (Ratings, Colours, Face tags, new Iptc data...)
A strong method is to remove/Import folders in Settings>Catalog table

[amimaster]

Hello amimaster, just a short note on this topic...

1st: wow, things you do... thumbs up!

Thanks!

Handle keywords/subject isnt easy...
Check XnViewMP Tools/Settings/Metadata/... for some options...

In the tab Metadata > IPTC & XMP I have the following enabled:

- Import XMP subject or IPTC keyword to Catalog's categories (so subjects are imported as categories)
- Export Catalog Category to XMP Subject and IPTC Keyword (so categories are written to XMP)
- Create XMP data / Create IPTC data (what happens if i have enabled the above, but not these two? Where's going to export the categories?)
- Read hierarchical keywords (|)
- Write hierarchical keywords

I don't see anything else that I might need / could be related to my problem...

As far as I remember "keywords/subject" are in a container. To add more keywords/subject you have to ad container_A + container_B to get them all together...

I'm not sure I understand what you're suggesting... If you are talking about the XPM data format, I am storing the subjects under the same XMP key that XnViews writes to (that should be Xmp.dc.subject). It actually writes also hierarchical keywords to Xmp.lr.hierarchicalSubject, which I wasn't doing in my script, so I updated it but nothing changed.


Upon making some more tests I discovered some interesting behaviors.
Category Sets are not the problem here, I tried repeating the steps above by assigning tags from the Category tab and not from the Category Sets, same behavior.
I got the feeling that the problem is related to one of the following hypotheses:

1. XnView has some inconsistencies as it actually relies on different systems (IPTC-IIM / XMP) to do different things:
- When I add my custom "subjects" to the file (B1, B2, B3...) it actually adds the corresponding categories to the category list in the left pane, however it does NOT flag them (i.e., the checkbox is blank), even if I can read the subjects being assigned in XPM on the "Info" panel on the right. So my intuition is that XnView uses XMP tags to populate the category list but then relies on IPTC-IIM to populate its internal DB that tells if the category is assigned to the image. My script only changes XMP data. I tried disabling "Create IPTC data" but nothing changes, at least for this image that already has IPTC-IIM data associated.
After all, if we are to manage some XnView "categories", how to they translate to IPTC keywords and XMP subjects? Which one contains the "true" data?

2. The internal DB is not updated correctly when visualizing the image or "updating catalog from files", and its internal state is different to what is displayed in the UI, although this is unlikely:
- When I add XMP tags with my script, then assign a new tag with XnView, it removes all the tags added from the script and SOMETIMES it falls back to the newly added keyword + another keyword that has been added in the past but is not present neither in XMP nor in IPTC-IIM data.

3. There is something weird going on with IPTC-IIM:
After repeating many times the loop above, I noticed that in IPTC data there is still the very first keyword I assigned to the image via the interface. It is NOT marked as an assigned category in the category list, and if I open the dialog to edit IPTC data it is actually there, but if I try to remove it and click "write", XnView just crashes (stdout log below):

Code: Select all

IPTC GET Keywords <test.png>
corrupted size vs. prev_size
[1]    120611 IOT instruction (core dumped)  xnviewmp

Well, the plot thickens...

[amimaster]

OK, now I feel VERY stupid.

I'm using Arch Linux and I forgot I installed xnview using AUR, so my system upgrades didn't upgrade XnView too, I was still at a 2021 build.
The latest version seems to have fixed the problem, since It now correctly "ticks" the XMP tags that it finds in the XMP data.

Thank you!

[Gerald_Ire]

OK, now I feel VERY stupid.

Nope! If you talk only with "me,myself & I" about a project you sometimes overseeing little things...
So it work! Thanks for sharing...