Tagging JPEG creates xmp-sidecar

[m.Th.]

These programs (I didn't work too much with RawTherapee and I didn't saw at all SilkyPix) doesn't support XMP standard?

No, they don't, nor does Rawtherapee. Rawtherapee produces a sidecar in the form "abc.orf.pp3" next to abc.orf, Silkypix produces a sidecar "abc.orf.5.spd" in a subfolder "SILKYPIX_DS". Capture One uses a similar strategy.

There is no DAM out there which is able to handle different sidecars (except JPhototagger, but that is another story). After all, one ends up with a lot of orphaned sidecar files while the corresponding raw files are already deleted/moved/renamed etc.

So:
* Sidecar lifetime management - Must Have.
* Support for other sidecar format - Nice to Have.

I think that (un)fortunately the market share for the other sidecar format are quite small.

[hic]

These programs (I didn't work too much with RawTherapee and I didn't saw at all SilkyPix) doesn't support XMP standard?

No, they don't, nor does Rawtherapee. Rawtherapee produces a sidecar in the form "abc.orf.pp3" next to abc.orf, Silkypix produces a sidecar "abc.orf.5.spd" in a subfolder "SILKYPIX_DS". Capture One uses a similar strategy.

There is no DAM out there which is able to handle different sidecars (except JPhototagger, but that is another story). After all, one ends up with a lot of orphaned sidecar files while the corresponding raw files are already deleted/moved/renamed etc.

So:
* Sidecar lifetime management - Must Have.
* Support for other sidecar format - Nice to Have.

I think that (un)fortunately the market share for the other sidecar format are quite small.

The market share for Rawtherapee is quite big, so I don't think that handling other sidecars is "nice to have".

And I guess it would not be too demanding. Here is an example of a batch script which does the job for deleting sidecars:

rem Input of image filename
set IMAGE_OLD=%1

rem extract path
for %%i in (%IMAGE_OLD%) DO set PATH_OLD=%%~dpi

rem Adjust path to hooks (e.g. for Capture One; for Photivo and RawTherapee just leave "%PATH_HOOK%" empty)

set PATH_HOOK="SYLKIPIX_DS"

rem Adjust suitable hook suffix

set HOOK_SUFFIX=".5.spd"
if not exist %PATH_OLD%%PATH_HOOK% GOTO END

for %%i in (%IMAGE_OLD%) DO set NAME_OLD=%%~nxi
set HOOK=%PATH_OLD%%PATH_HOOK%\%NAME_OLD%%HOOK_SUFFIX%

del "%HOOK%"
:end

Unfortunately I have no idea how to code this in a programming language...

[m.Th.]

So:
* Sidecar lifetime management - Must Have.
* Support for other sidecar format - Nice to Have.

I think that (un)fortunately the market share for the other sidecar format are quite small.

The market share for Rawtherapee is quite big, so I don't think that handling other sidecars is "nice to have".

Hmmmm.... let's see:



Source: http://digital-photography-school.com/p ... ll-results

Yes, I know that RawTherapee is a popular solution, but let us not forget the sense of its popularity in the global market. I really would like to support its proprietary sidecars but, sincerely, I cannot say that it is a "must have".

And I guess it would not be too demanding.

"not demanding"... ...what?

1. To support parsing (import/export of) custom sidecars? Well, 1st, it depends on the format, data which is stored and, more, if they release the specifications for the format. But the most dangerous part is that it is proprietary and hence we depend on them - if they will turn around and implement a tag/bit of information which is incompatible with our architecture, then we're in trouble. Of course, we can limit ourselves to RCK (Rating/Coloring/Keywords).

2. To copy/move/delete/rename (CMDR) the sidecars together with the images? No it isn't demanding. Just that Pierre must allow the user to specify which are the sidecars - something like this:

Companion-files-mockup.png (8.03 KiB) Viewed 4429 times

...and after this, to change a little the code for each CMDR action (see above) to look also for the files with the same name and sidecar extension and do the same operation for it.

Here is an example of a batch script which does the job for deleting sidecars:

rem Input of image filename
set IMAGE_OLD=%1

rem extract path
for %%i in (%IMAGE_OLD%) DO set PATH_OLD=%%~dpi

rem Adjust path to hooks (e.g. for Capture One; for Photivo and RawTherapee just leave "%PATH_HOOK%" empty)

set PATH_HOOK="SYLKIPIX_DS"

rem Adjust suitable hook suffix

set HOOK_SUFFIX=".5.spd"
if not exist %PATH_OLD%%PATH_HOOK% GOTO END

for %%i in (%IMAGE_OLD%) DO set NAME_OLD=%%~nxi
set HOOK=%PATH_OLD%%PATH_HOOK%\%NAME_OLD%%HOOK_SUFFIX%

del "%HOOK%"
:end

Unfortunately I have no idea how to code this in a programming language...

Thanks!

...but I'm a little reserved WRT to sidecars in subfolders (also RawShooter was the same) because one must configure yet another parameter (the subdir) which, incidentally, (in the case of SilkyPix) can depend on the version number. This also can create problems in the CMDR (on-disk management) engine because instead of the simple case of same-dir sidecars...

Code: Select all

cSidecar=ReplaceExtension(cImageFullPath, 'XMP);
if FileExists(cSidecar) then
   CopyFile(cSidecar, cDestDir); //the same applies to Move

...the program must have a more complex logic for building cSidecar and (temporary) adjusting the cDestDir.

Certainly doable, tough.

[JohnFredC]

(Didn't know there are so many sidecar formats! )

So, in addition to reading "strange" sidecar formats, it might be useful to "merge/consolidate sidecars" to a particular sidecar file/format chosen by the user.

Perhaps XnView could even allow the user to select which sidecar format is the default.

Not only that, what about a "synchronize all sidecars" tool, too? (...now that's a "blue sky" idea if there ever was one!)

Perhaps a "sidecar plugin" approach would be the only reasonable way to accomplish such things.

[oops66]

Hello,
It's what why I asked for the .mie file format support in xnview (in 11 nov 2007 ) (the "best" benchmark note for embedded metadata) ... used bye exiftool too.
... so probably the best way to store metadata in sidecar ????

What is MIE?
MIE stands for "Meta Information Encapsulation". The MIE format is an extensible, dedicated
meta information format which supports storage of binary as well as textual meta information.
MIE can be used to encapsulate meta information from many sources and bundle it together with
any type of file.
Features
Below is very subjective score card comparing the features of a number of common file and meta
information formats, and comparing them to MIE. The following features are rated for each

Code: Select all

format with a score of 0 to 10:
  1) Extensible (can incorporate user-defined information).
  2) Meaningful tag ID's (hint to meaning of unknown information).
  3) Sequential read/write ability (streamable).
  4) Hierarchical information structure.
  5) Easy to implement reader/writer/editor.
  6) Order of information well defined.
  7) Large data lengths supported: >64kB (+5) and >4GB (+5).
  8) Localized text strings.
  9) Multiple documents in a single file.
 10) Compact format doesn't squander disk space or bandwidth.
 11) Compressed meta information supported.
 12) Relocatable data elements (ie. no fixed offsets).
 13) Binary meta information (+7) with variable byte order (+3).
 14) Mandatory tags not required (an unecessary complication).
 15) Append information to end of file without editing.

format with a score of 0 to 10:

Code: Select all

                          Feature number                   Total
     Format  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15   Score
     ------ ---------------------------------------------  -----
     MIE    10 10 10 10 10 10 10 10 10 10 10 10 10 10 10    150
     PDF    10 10  0 10  0  0 10  0 10 10 10  0  7 10 10     97
     PNG    10 10 10  0  8  0  5 10  0 10 10 10  0 10  0     93
     XMP    10 10 10 10  2  0 10 10 10  0  0 10  0 10  0     92
     AIFF    0  5 10 10 10  0  5  0  0 10  0 10  7 10  0     77
     RIFF    0  5 10 10 10  0  5  0  0 10  0 10  7 10  0     77
     JPEG   10  0 10  0 10  0  0  0  0 10  0 10  7 10  0     67
     EPS    10 10 10  0  0  0 10  0 10  0  0  5  0 10  0     65
     CIFF    0  0  0 10 10  0  5  0  0 10  0 10 10 10  0     65
     TIFF    0  0  0 10  5 10  5  0 10 10  0  0 10  0  0     60
     EXIF    0  0  0 10  5 10  0  0  0 10  0  0 10  0  0     45
     IPTC    0  0 10  0  8  0  0  0  0 10  0 10  7  0  0     45

By design, MIE ranks highest by a significant margin. Other formats with reasonable scores are
PDF, PNG and XMP, but each has significant weak points. What may be surprising is that TIFF,
EXIF and IPTC rank so low. [/color]

See here for .mie specification:
The .pdf file for .mie!

http://newsgroup.xnview.com/viewtopic.p ... e+exiftool
http://newsgroup.xnview.com/viewtopic.p ... 67#p117667

[m.Th.]

Hello,
It's what why I asked for the .mie file format support in xnview (in 11 nov 2007 ) (the "best" benchmark note for embedded metadata) ... used bye exiftool too.
... so probably the best way to store metadata in sidecar ????

If Pierre wants to implement it, I'm not against it.

However, you forgot to mention in your score card (btw, nice work) some factors:

- Feature Coverage / Popularity (XMP is the most popular)
- Standardization (XMP is an ISO standard)
- Tooling/Coding (we have it already - no coding)

...also, looking at http://en.wikipedia.org/wiki/Extensible ... a_Platform I see that RawTherapee supports XMP.

I understand you about some technical advantages of other metadata formats but I'm quite reserved WRT need to support them.

As an aside, about the lack of compactness of XMP, we can do what Microsoft Office does - zip it and change the extension to eg. XMZ. (Perhaps you know, the new Office formats - docx, xlsx, pptx are just zipped xml)

[oops66]

@m.Th ... as usual ... it's only a proposal, of course
For more info. about sidecar files:
http://en.wikipedia.org/wiki/Sidecar_file
...
Meta Information Encapsulation (MIE)
Meta Information Encapsulation sidecar files. The MIE format is an extensible, dedicated meta information format part of ExifTool. MIE files can be used to encapsulate meta information from many sources and bundle it together with any type of file.
A variation of this are copies of the source file which contain largely the same information, but in a different format or from a previous version:...

EDIT: Sorry for my approximative english , but when you say: "WRT" ... is it what meaning ?
Answer: "WRT" is "With Respect To". This is internet shorthand for saying "I want to respond to a specific point made earlier in this conversation". WRT is ideal for steering a conversation in the direction you wish to take it.
http://netforbeginners.about.com/od/w/f/What-Is-WRT.htm

[hic]

...also, looking at http://en.wikipedia.org/wiki/Extensible ... a_Platform I see that RawTherapee supports XMP.

No, it doesn't. it is planned for the future, but it seems to be unlikely that it will be in the near future...

hic

[CameronD]

...
No, I don't think so. I would say just 3 because DB is already updated:

Export to:
* DB only
* Metadata when possible
* Metadata and sidecar
* Sidecar

Yes, the only sensible way is for XnView to treat the DB as the "master copy", which it does at the moment.
This implies that auto-import must happen in order that the DB remains the master version.
It also implies that programs are assumed to modify mtime sensibly, so that XnView knows which files it needs to reimport.

It does pose a bit more of a problem for:

sidecar has a higher priority on embedded metadata, right?

Surely it must be more complex than this. After all, other software we might need to use might not know how to use the sidecar.
If XnView can detect that the image file has been recently changed, but the sidecar has not been modified since the DB entry was made, then the embedded metadata should take priority.

If both image and sidecar file have been modified since the DB date, and XnView detects a certain item has been modified differently in both sources, then this is beyond any automatic assumption of priority and XnView should ideally pop up some window prompting manual choice.
This becomes even more critical if there is the possibility of multiple sidecar files for the one image.

[hic]

If both image and sidecar file have been modified since the DB date, and XnView detects a certain item has been modified differently in both sources, then this is beyond any automatic assumption of priority and XnView should ideally pop up some window prompting manual choice.
This becomes even more critical if there is the possibility of multiple sidecar files for the one image.

I guess I should explain something: the other mentioned sidecar files usually don't contain metadata but development settings of the raw converter. Thus it would only be necessary to keep track of it while copying, deleting, moving or renaming a raw file. Nothing more. No DB entry has to be written to these sidecars.

Hic

[m.Th.]

...
No, I don't think so. I would say just 3 because DB is already updated:

Export to:
* DB only
* Metadata when possible
* Metadata and sidecar
* Sidecar

Yes, the only sensible way is for XnView to treat the DB as the "master copy", which it does at the moment.
This implies that auto-import must happen in order that the DB remains the master version.

Of course. Everything outside of DB is just like that: "outside". There are way too many factors (read: uncontrollable modification sources) involved to have them as primary metadata source.

It also implies that programs are assumed to modify mtime sensibly, so that XnView knows which files it needs to reimport.

Yes. But this (at least on Windows) is (more or less) standard. Anyway we don't have a better choice, isn't it? (Relying on hashes is way to slow+complicated for this purpose - of course, IMHO)

It does pose a bit more of a problem for:

sidecar has a higher priority on embedded metadata, right?

Surely it must be more complex than this. After all, other software we might need to use might not know how to use the sidecar.

I think that you're overanalyzing a bit here. If the "other software doesn't know how to use the sidecar" - don't use it. I have a hard time to see a software fiddling with a separate file without knowing what to do with that file.

If XnView can detect that the image file has been recently changed, but the sidecar has not been modified since the DB entry was made, then the embedded metadata should take priority.

Well, here is a problem. Theoretically you're right. But in practice, we cannot distinguish between image data change and embedded metadata change. One can change the image data (remember there are enough image editors which are NOT non-destructive - incidentally XnView Classic/MP is one of them) and then while the embedded metadata can be older than the one from sidecar, the file's modified date/time is newer than sidecar's.

If both image and sidecar file have been modified since the DB date, and XnView detects a certain item has been modified differently in both sources, then this is beyond any automatic assumption of priority and XnView should ideally pop up some window prompting manual choice.
This becomes even more critical if there is the possibility of multiple sidecar files for the one image.

Sure.

[m.Th.]

If both image and sidecar file have been modified since the DB date, and XnView detects a certain item has been modified differently in both sources, then this is beyond any automatic assumption of priority and XnView should ideally pop up some window prompting manual choice.
This becomes even more critical if there is the possibility of multiple sidecar files for the one image.

I guess I should explain something: the other mentioned sidecar files usually don't contain metadata but development settings of the raw converter. Thus it would only be necessary to keep track of it while copying, deleting, moving or renaming a raw file. Nothing more. No DB entry has to be written to these sidecars.

Hic

Yes, I know - but is good that you stress it.

Basically this thread has two topics now:

1. The problems with metadata management (automatic in/out, storage: embedding, sidecar etc.)
2. A request (yours) - which IMHO fits better in http://newsgroup.xnview.com/viewtopic.php?f=62&t=29532 which deals with a file processing enhancement: "do CMDR (Copy/Move/Delete/Rename) with multiple files together based on some criteria". Btw, I updated the topic - have a look.

The problem with your request about the CMDR engine is not its existence per se (many other programs offer similar features, including 'image stacking') but the fact that the companion files are (in some cases which you need - and I understand you) somewhere else and not in the same directory. And this doesn't apply to all companion files (note that I avoid the word "sidecars" because we're talking now about a file-agnostic engine) so we need then for each extension to specify where is that kind of companion file and the program will need at each CMDR action to look for the existence of that directory(ies) and look inside to see if the companion exists... ...hmmm... I think that the things go up exponentially.

...but looking at your messages got me thinking: perhaps a custom script parametrized with a {folder} ran after each CMDR will help you?

[CameronD]

I guess I should explain something: the other mentioned sidecar files usually don't contain metadata but development settings of the raw converter.

Thank you - yes, I did not understand the contents of those files.

[CameronD]

...
Of course. Everything outside of DB is just like that: "outside". There are way too many factors (read: uncontrollable modification sources) involved to have them as primary metadata source.

And every other software developer regards their DB, or embedded metadata as the "primary" source. And XnView as one of these uncontrollable outside modifiers.

It also implies that programs are assumed to modify mtime sensibly, so that XnView knows which files it needs to reimport.

Yes. But this (at least on Windows) is (more or less) standard. Anyway we don't have a better choice, isn't it? (Relying on hashes is way to slow+complicated for this purpose - of course, IMHO)

mtime semantics are followed in all operating systems I know of, which I think covers all MP platforms.
Unfortunately it seems particularly common for image handling programs to override this, in some cases without option. When I find such a program I tend to stop using it.

But I agree that yes, it really is the only sane way to operate - not foolproof, but at least efficient.

I think that you're overanalyzing a bit here. If the "other software doesn't know how to use the sidecar" - don't use it.

I think you misunderstood my comment. If I use software B to manipulate something about images, and it can only modify embedded metadata, then in that case the embedded metadata is the latest and therefore "correct" value. It is the same argument as my next point, which is that there can be no automatic "priority" assigned to metadata source.

If XnView can detect that the image file has been recently changed, but the sidecar has not been modified since the DB entry was made, then the embedded metadata should take priority.

Well, here is a problem. Theoretically you're right. But in practice, we cannot distinguish between image data change and embedded metadata change...

I don't see a major problem at all - if dates are newer then simply check if any mapped item is different between file and DB. There is not a lot of data to check. If the user has selected to keep embedded metadata in sync then update the DB. If the user has selected to ignore embedded metadata then ignore it.

[CameronD]

... The default behavior of XnView (and others softs) must be (for the integrity of the users data) ... "do not touch the originals users files/pictures by default". ....

One minor point - There is too much ambiguity about what constitutes "originals". For example, XnView is not necessarily going to know that my "originals" are sometimes raw files (stored elsewhere) and the jpegs might be "developed" versions.
The option presumably must apply to all image files, or none, and the wording should be clear about this.