How to make it work with huge image libraries

[brazenvoid]

I have folders with 500k to a million images (png, jpg) each on a HDD. The images can be tiny to as large as 30MB each. I have to analyze them, tag some of interest and then move them to another folder. A number of passes will happen on filtered images till only the most unique will actually be considered.

1. As Xnview goes not responding for an unbearably long time for the main folders, what I do is to move 5k images each to a sandbox folder and then tag the files.
2. Once I open the first image, Xnview starts something in background, perhaps analyzing (?) the files in the folder, slowing down my browsing pace.
3. Browser view starts analysis of the folder once it opens and takes a minute to analyze all the images which results, I don't derive any use from. During this time too Xnview stutters.
4. The sole reason I go to browser view is to keep a virtual bookmark of the last image I was on. As once I complete move of the tagged files from the view window, the window abnormally retains the index instead of the image and navigates to the new image now on that index. This behavior makes me lose the way of my progress. Doing the move in the browser view makes the view window retain the image and not the index.
5. At first initiation, the "Move To" function also exhibits a long hang, especially if the sandbox folder coexists in the same folder where all the source folders are. So I have all the source folders in another folder. The filter folders with this sandbox folder exist in another. Still even though the filter folders and this sandbox have less than 20k images. It still hangs for 5-10m.

I keep windows defender disabled while I work with Xnview as otherwise the delays are even worse.

Please help me improve this workflow, perhaps there is a way to disable/reduce/limit the analysis and somehow speedup the Move To using some more trickery

[brazenvoid]

By disabling caching of thumbnails in Catalog setting I was able to eliminate lag on view window at least. Unexpectedly that also removed the lag on "Move To" as well.

[xnview]

in the main folder, do you have a lot of subfolders?

[brazenvoid]

The folder struct is like this:

Root
----Sources
--------Source 1
.
.
--------Source 6
----Filtering
--------Sandbox
--------Pass 1
.
.
--------Pass 5

[brazenvoid]

Oh, I forgot to mention something, though I am skeptical that it matters. The source folders are updated with new imagery continuously. The rate is random, on average it is one per two seconds.

I do not open these folders in XnView though. I manage all image transfers from source to sandbox using CLI.

[brazenvoid]

The move to issue is back somehow. I have cleared the catalogues and the caches but it still goes into a comma for a long time. It again goes into an unresponsive state after I click ok too. That duration fluctuates wildly, sometimes few seconds to many minutes.

[brazenvoid]

Moving the filter folders one more step inside another folder seems to have worked. It seems as though as long as there are folders with thousands of files inside the same folder as the source folder for the move makes xnview start analyzing them too...

New Struct
--Source N
--Filters
----Sandbox
----Passes
------Pass N

An exclusion list would be great so that xnview can't start on something it can never hope to do in a sane way.

[brazenvoid]

It is back again, and after more testing, it is clear that xnview can't handle large folders regardless of where they are. Whatever analysis it starts, it hangs the display. It doesn't seem that UI is on a separate thread.

Such folders, the time they are moved to completely new location, not existent before, for a limited time work instantly but soon they get bogged down regardless. The movement is impossible to work out from within the program consistently. If only I could move the tagged files externally.

[xnview]

if i understand correctly, you have a big delay when you have subfolders? How many subfolders?

[brazenvoid]

Sorry for the delayed response. I had lost hope and have been struggling with it since, unfortunately there are no better alternatives...

• The first delay occurs when I click "move to". It scales with how many files are contained in the source folder. If there are like 500 then the delay is about 5-10 seconds. I think it scales linearly.

• The second delay is subjective and it can happen many times. It manifests when navigating to select the destination. I think its a windows issue. This scales with how many files are contained in each folder you expand and any children it has in the hierarchy. If a large dump (200k+ images) is also in the same parent then it locks up for an extended time, usually more than an hour.
  • Using the path input bypasses this.

• The third delay occurs when I submit the destination and scales with the number of total and tagged files.

I have tried different patterns and hierarchies. It always helps tremendously if large dumps are not around destination and source folders. So I usually keep the destination and source together, and when destination has more than 20k images, I move them to a separate folder.

I have also tried disabling the catalogue and metadata, in a bid to stop it from doing anything extra but it only works for a limited time. After sometime browsing the images, it silently starts doing something in the background, albeit it rarely disturbs the browsing. Though accidentally going into browser mode makes it work intensively at it, closing the browser does not stop it, have to kill it.

Perhaps I can do it through CLI? Like tag the images in the UI and then move them through some CLI command?

[xnview]

• The first delay occurs when I click "move to". It scales with how many files are contained in the source folder. If there are like 500 then the delay is about 5-10 seconds. I think it scales linearly.

So you have the delay before to open the 'move to' dialog?

• The third delay occurs when I submit the destination and scales with the number of total and tagged files.

Could you make a screencast?

[brazenvoid]

Sorry, that will show a lot more than I am comfortable sharing and editing that out is beyond my capabilities.

It just goes not responding the usual way, if there is anything particular you want to know I can write it here. If there is some kind of logging mechanism in the app, I may be able to work with that too.

[xnview]

could you describe when do you have delay/not responding?

[brazenvoid]

This all started because letting Xnview do tagging in a folder with hundreds of thousands or millions of images, locks it up. Especially if you go to the browser.

So I devised a divide and conquer strategy . For example, if the folder structure is like this:

- Dump Folder (400k+ images, 500 KB - 40 MB each)
- Sandbox
- Filtered

What I do is:

1. Use power shell to move first 1000 images from dump folder to sandbox.

2. Open an image in sandbox folder through xnview and start tagging files to move to Filtered folder.

2a. If cataloging and meta data are at default settings, xnview starts some process in the background after some time which does continuous work and generates a lot of disk reads. It goes not responding for each huge image e.g. a 15 MB+ PNG.

2c. I have since disabled all meta, disabled catalog integrity check on startup, and set compression to none for thumbnails.This works as long as Xnview is not opened in the dump folder.

3. Afterwards I click "Move To", it can go to not responding for a while.

4. If the destination selection opens in the same folder where these three folders are, it again goes into a not responding state. This I know is because of the UI as it searches all 3 to find any sub-folders and as the dump folder has so many files, it takes a while. At first there are + actions before them and when it resumes, those go away.

5. Further after selecting destination, it can again go to a not responding state but this is very rare.

6. After the move is complete (after tagged selection), I close xnview, use power shell to empty sandbox and cycle these steps again.

[brazenvoid]

I have now devised another strategy, which works for longer, with the following structure:

- Dump Folder
- Filtered Final
- Workspace
---- Sandbox
---- Filtered

1. I copy 1000 to Sandbox, and move tagged files to Filtered.

2. This way the destination dialog opens always inside workspace with folders with relatively not as many files which avoids that not responding state.

3. When Filtered has about 10k or more images, I move them to Filtered Final. 20k+ images brings back those "after move to" and "after select destination" not responding states.

Perhaps I should try moving filtered to yet another level below.