OK, I can (almost) always reproduce it this way - with 3 short MPG files and with some AVI and MKV files, but never with one short MPG file (same source as the others, weird), anyway, I tested with many videos and it looks I found the problem.
XnView: MP 0.87 beta 3, not released yet (probably the same in beta 2 and previous versions) - 64-bit
OS: Windows 10 - 64-bit
0. Play few videos in preview, wait at least one short video to play to the end, switch to viewer and back, close all viewer tabs, do not select any images.
1. Use details view.
2. Set option "Use middle click" to "Browser <> Viewer" (Interface > Switching mode).
3. Select any video file in browser.
4. Middle click the video to go to viewer.
5. No dialog.
6. Close video viewer tab.
7. Select any image in browser.
8. Now middle click the same video directly, to go to viewer.
Actual behaviour (bug): "This picture will be converted to RGB with 8bits per component" dialog is displayed.
9. Close video viewer tab.
10. Now middle click the same or different video directly, to go to viewer.
Actual behaviour (bug): "This picture will be converted to RGB with 8bits per component" dialog is displayed, always, no need to select an image before.
I tried to middle click the same videos 10-20x quickly, and I had this dialog just once, after some 15x clicks, but when I selected image first, I could almost always reproduce it, just I need to play some videos first, it doesn't work without that step 0.
All in all, this dialog is displayed almost always if I select image before video, almost never if not.