XnF (frame) files – The next level in multimedia management

[m.Th.]

XnFrame (XnF) files – The next level in organizing and accessing multimedia

The problem

There are a lot of multimedia files today (audio and video) and most of them contain different, distinct parts which distinguish themselves from the whole multimedia stream.

This is particularly true for recorded raw/unprocessed data (both audio and video) as well as for digitized multimedia which comes from a multitude of sources – mainly from analog media like videotapes (Beta, VHS etc.), audio cassettes, vinyl but also from other digital media like DV tapes, audio recordings of live events in which the recorder is left “on” during the entire duration of the event etc.

While for images there is already implemented in some programs / platforms a feature called “area tagging”, used mostly for person detection and note taking (see Flickr, Facebook), for multimedia, as far as we know, even if it is a huge demand, there is no such a solution.

What we have

XnView MP has already a very good cataloging engine – and, as always, there are plans to enhance it – however this module works only with files.
We already know how to extract a frame from video files (see Tools | Settings | Thumbnail | Misc – the options “Show 4 thumbnails in video’s thumbnail” and “For video files, extract frame at…”) and we know how to navigate in the multimedia stream (try to play a video or audio file – a player with a slider appears – when the user moves the slider the player goes to the according position)

The solution

Enter XnF Files.

We can gather and use all these things together to provide a top-notch feature which will allow us:
• Visual direct access at any desired part of the sequential multimedia file. Basically are human-meaning visual bookmarks in the big video/audio stream
• Best-in-class cataloging of these parts using the entire Rating-Color-Keywords infrastructure of XnView MP

In few words, an XnF file format is a frame together with its position and a link to the source multimedia, file format which XnView MP knows how to handle accordingly.

XnF file format

The XnF file format is composed from the following parts/fields:
1. File Signature – The string “XNF” – the program will check for this signature in order to identify the format, besides of the file extension. If this check fails, the program will stop the loading process.
2. File Version – Integer: 1 – used to distinguish different future format versions (additions, changes etc.)
3. Source Link – the full UNC path to the multimedia file from which the frame was extracted. The path will be stored exactly in the same way in which XnView MP stores the paths already (see the Folders table in the database). When the user will double-click on an XnF thumbnail in XnView MP the program will try to load the multimedia file specified by this field. If the file is not found, then it will try to find it in the same directory with the XnF file. If the source file is still missing, the multimedia loading process will be aborted.
4. Frame Index – The position in source file from where the frame was extracted. At double-click, the program will navigate in video (or audio) file and start to play from there.
5. The frame – a WebP compressed with high-quality setting image of the extracted frame. For videos the nature of the frame is obvious, for audio files, the frame will be the waveform starting from that position.


In this way we'll have the leading multimedia management engine on the market. Yep, is that simple.

The original document is posted here: http://is.gd/NjhH4A

[m.Th.]

...Perhaps a better approach from coding perspective:

Save the frame in a 'normal' JPG/JPG XR/JPG2000 file and use the embedded XMP to store the other custom metadata fields which we want. And at the end change the extension of the file to Xnf.

Generate (writing) phase:
-------------------------------

So, the steps of the algorithm would be:

The user:

1. User plays the movie c:\bar\foo.mp4
2. User stops the movie at frame 1234567
3. User goes to Tools | Extract Frame... (or whatever command - in right-click popup menu etc.)

And now the program:

1. Get the frame
2. Save it to foo.001.jpg (001 means that is the first frame file extracted for the movie "foo") ...or save it to foo.1234567.jpg??? - no, I don't think so because we can have 24 fps or more so the frame index grows very very fast.
3. Add the following tags to the embedded XMP:

Code: Select all

   xnvmp:signature="XNF"
   xnvmp:version=1
   xnvmp:sourceFile="c:\bar\foo.mp4"
   xnvmp:frameindex=1234567

Save them in a rdf:Description tag or where it it easy for you to read them.
4.Rename the file to foo.001.xnf

You're done!

(minimal coding required here - you just use the already existing XMP code)

Reading phase and generate the thumbnail:
-------------------------------------------------

1. Read the xnf file as a JPG, JPG XR (perhaps better?) or JPG2000.
2. Generate in the normal way the thumbnail from the file. (Perhaps check the xnvmp:signature to see that it is a valid file.)

(No additional coding required for this phase)

When user clicks on the thumbnail
-----------------------------------------
1. Read the xnvmp:sourceFile and try to load the fullpath file from there (c:\bar\foo.mp4 in our example). If the file doesn't exist, try to find 'foo.mp4' in the current directory. If the video file still doesn't exist, just display the frame from foo.001.xnf
2. If the video (or audio) file is successfully loaded at the step 1 then read the xnvmp:frameindex tag and move the player at that frame index in the media file.
3. Play from there, if the XnView MP is configured to automatically play the media.

(minimal coding required here - just move the player at the desired position)


Notes: Perhaps we must do a little testing to see what container we should use: JPG, JPG2000 or JPG XR. Theoretically JPG XR should be the best.

[m.Th.]

1. Bump!

2. Here's the code to play the audio at the specified timecode: http://stackoverflow.com/questions/2473 ... 9#24733919

The code to get the frame/play the video at specified timecode is already inside. See the code around Tools | Settings | Thumbnail | For video files extract frame at ____%