How to make it work with huge image libraries

[brazenvoid]

I have folders with 500k to a million images (png, jpg) each on a HDD. The images can be tiny to as large as 30MB each. I have to analyze them, tag some of interest and then move them to another folder. A number of passes will happen on filtered images till only the most unique will actually be considered.

1. As Xnview goes not responding for an unbearably long time for the main folders, what I do is to move 5k images each to a sandbox folder and then tag the files.
2. Once I open the first image, Xnview starts something in background, perhaps analyzing (?) the files in the folder, slowing down my browsing pace.
3. Browser view starts analysis of the folder once it opens and takes a minute to analyze all the images which results, I don't derive any use from. During this time too Xnview stutters.
4. The sole reason I go to browser view is to keep a virtual bookmark of the last image I was on. As once I complete move of the tagged files from the view window, the window abnormally retains the index instead of the image and navigates to the new image now on that index. This behavior makes me lose the way of my progress. Doing the move in the browser view makes the view window retain the image and not the index.
5. At first initiation, the "Move To" function also exhibits a long hang, especially if the sandbox folder coexists in the same folder where all the source folders are. So I have all the source folders in another folder. The filter folders with this sandbox folder exist in another. Still even though the filter folders and this sandbox have less than 20k images. It still hangs for 5-10m.

I keep windows defender disabled while I work with Xnview as otherwise the delays are even worse.

Please help me improve this workflow, perhaps there is a way to disable/reduce/limit the analysis and somehow speedup the Move To using some more trickery

[brazenvoid]

By disabling caching of thumbnails in Catalog setting I was able to eliminate lag on view window at least. Unexpectedly that also removed the lag on "Move To" as well.

[xnview]

in the main folder, do you have a lot of subfolders?

[brazenvoid]

The folder struct is like this:

Root
----Sources
--------Source 1
.
.
--------Source 6
----Filtering
--------Sandbox
--------Pass 1
.
.
--------Pass 5

[brazenvoid]

Oh, I forgot to mention something, though I am skeptical that it matters. The source folders are updated with new imagery continuously. The rate is random, on average it is one per two seconds.

I do not open these folders in XnView though. I manage all image transfers from source to sandbox using CLI.

[brazenvoid]

The move to issue is back somehow. I have cleared the catalogues and the caches but it still goes into a comma for a long time. It again goes into an unresponsive state after I click ok too. That duration fluctuates wildly, sometimes few seconds to many minutes.

[brazenvoid]

Moving the filter folders one more step inside another folder seems to have worked. It seems as though as long as there are folders with thousands of files inside the same folder as the source folder for the move makes xnview start analyzing them too...

New Struct
--Source N
--Filters
----Sandbox
----Passes
------Pass N

An exclusion list would be great so that xnview can't start on something it can never hope to do in a sane way.

[brazenvoid]

It is back again, and after more testing, it is clear that xnview can't handle large folders regardless of where they are. Whatever analysis it starts, it hangs the display. It doesn't seem that UI is on a separate thread.

Such folders, the time they are moved to completely new location, not existent before, for a limited time work instantly but soon they get bogged down regardless. The movement is impossible to work out from within the program consistently. If only I could move the tagged files externally.

[xnview]

if i understand correctly, you have a big delay when you have subfolders? How many subfolders?