1.4 beta - Batch conversion with skip existing file setting - extreme slow!

[masterjp]

I often use the skip existing picture feature in batch conversion, because I want to save time during conversion large picture archives.
Today I want to convert new files, but I already converted them a few days before. So XnviewMP starts the conversion.
I used the 4 core speed setting in conversion dialog, but the whole conversion works in single cpu core speed!
It is very slowly. In previous versions 1.3 or older the performance with skip existing files and multi core cpu usage was fantastic and fast.
Since version 1.3.1 and 1.4 betar the speed is very slowly, because it will use one cpu core only

Why did you change this great algorithm? Please make it great again!



XNVIEWMP 1.3.1 and 1.4 beta:
======================

Input files: 72506
Extracted pages: 72506
Overwritten: -72506
Skipped: 72506

Total Duration: 13 minute(s)


XNVIEWMP 1.3:
=============

Input files: 32627
Extracted pages: 32627
Overwritten: -32627
Skipped: 32627

Total Duration: 16 second(s)

Finished at: Do Nov 24 20:48:19 2022

[xnview]

Strange, in the 2 tests you have not the same number of files. Which output format? Which input format?

[masterjp]

The input files are jpeg files with max. 6000 x 4000 Pixels and I convert them to webp.

[xnview]

could you send me a sample file used? are you reproduced the difference even on 100 files for each version? Without multiple core, you have same time in 1.3/1.3.1?
16seconds to convert 32000 files is strange to me, and i see 'Skipped: 32627' so no files are written???

[masterjp]

The old version 1.3 was very fast, because it did not calculate the file size saving informationens for each file and whole file saving for the complete conversion. I convert a big collection of files to another directory, where around 90 percent of the files already exists. This cost much time for checking, if the path and files already exists. Maybe it works in single core mode only.

I optimized some directorys with the pingo optimizer, which convert the jpeg files to arithmetic compression.
But this is no problem for XnViewMP. I can load the files.

[xnview]

The old version 1.3 was very fast, because it did not calculate the file size saving informationens for each file and whole file saving for the complete conversion. I convert a big collection of files to another directory, where around 90 percent of the files already exists. This cost much time for checking, if the path and files already exists. Maybe it works in single core mode only.

It's in 1.4.0 not in 1.3.1, so for me it's not the problem. So if i understand you correctly, it's only if files already exist? i've tried but no difference.
Is it possible to re-test 1.3.1 & 1.3.0?

[masterjp]

Thank you for your help.

It is very difficult to analyse the reason for my problem.
The problem is very special, because I have so many files
and it will only happens, if you convert a huge picture archiv or under special condition.

I will do some further tests under the same conditions and with all versions (1.3 and 1.4 beta 1) to precisly allocate the reason for my problem.

[xnview]

any news?

[user0]

I experience extreme slowdown once as well, no file skips though

brief details as far as I remember:
1.4.0 beta, had 2-5 .png with and without alpha, was doing canvas resize plus something else

will try to dig deeper later

[masterjp]

I have not reported for quite a while because I have been very intensively comparing and testing XnviewMP version 1.3.0 and version 1.3.1 (and now also version 1.4.0).

In my previous report I had written about a speed problem in both versions. Version 1.3.0 is still much faster under the same conditions! Depending on the situation, the time difference can be significant.

The problem occurs only under certain conditions, but it can affect all users equally.

I have one main folder with many subfolders with images.
New folders with new images are regularly added to the main folder.

I regularly convert these to a destination folder using the skip file function, because there are already folders and files from older conversions in the destination folder. This way I save time converting and don't have to recreate all the files and folders. The peculiarity is that I deleted certain files or folders in the destination folder because I don't like them anymore.

When XnViewMP 1.3.0 starts the batch conversion, it seems to first check all folders and files in the destination directory for existence. Only then does the program create the missing folders and files in the destination directory using all the threads set. During the conversion, a message about the converted files appears regularly in the status area. This is how it should be and it is perfect!

---

When XnViewMP 1.3.1 or 1.4.0 starts, the conversion status message screen freezes. It does not update automatically. Only when the conversion is done, all status messages appear!
For smaller conversion tasks, the status messages may appear periodically in the status area. Here XnviewMP 1.3.1 and 1.4.0 reacts quite differently than version 1.3.0.

The fastest version with the same conversion settings is version 1.3.0!
Version 1.3.0 => time for conversion 1 minute and 48 seconds => Status Log file: https://www.dropbox.com/s/i589xcvhu2dpk ... 0.txt?dl=0
Version 1.3.1 => time for conversion 3 minutes and 10 seconds => Status Log file: https://www.dropbox.com/s/b82jmwhn0sm91 ... 0.txt?dl=0

Here is a link to the video file, which shows the performance of XnviewMP 1.3.0 first and than 1.4.0 and my used settings:
https://www.dropbox.com/s/dvacgb5d2j3ht ... n.mkv?dl=0

In my test with 32.000 files the difference is not so big, but I have done tests with > 50.000 files with other directories in the past and the time difference was very big! It depends on the amount of files and folders skipped.

Also, using Windows Task Manager, I could see that versions 1.3.0 and 1.3.1 or later have very different CPU usage during batch conversion.

Version 1.3.0 starts first with only one thread to check the files and directories for existence. Then the conversion starts with the choosed number of threads!

Version 1.3.1 always starts with full CPU load (4 threads set in conversion dialog!). After that, the program uses only 1-2 threads (25% cpu load) over a longer period! This is very strange and could explain, why the newer version 1.3.1 and 1.4.0 need much longer for the conversion.

----

Here you can see a batch conversion of XnviewMP 1.3.0 - It takes 6 seconds to check, if all folders and files are converted from source to destination folder! I did the same test with Version 1.3.1 and 1.4.0 and the newer versions need 47 seconds! The newest version is 5-7 times slower!
This is only a fast test with small pictures and a collection auf 17.000 pictures to demonstrate the time difference. If you use larger output files the time difference will be much larger e.g. 30-60 Minutes for the same batch conversion.

Here are source files with folder structure for testing performance (packed size 6 MB - unpacked 29 GB!) :
https://www.dropbox.com/s/mix2d9r91tzsp ... er.7z?dl=0

I converted them to webp (80% quality) and used resize to longest side (2560 pixel).
For testing you can use a smaller output size (e.g. 128-320 pixels longest side) to save time!

Here is the destination folder with the resized files:
https://www.dropbox.com/s/hgdei1btx0atq ... on.7z?dl=0

I used the following settings:

Xnviewmp-settings2.png

Xnviewmp-settings1.png

[xnview]

Thanks to your detailed description I can reproduce the problem.
This one was well hidden

[masterjp]

Thank you Pierre! ) You are great!

[xnview]

This problem is supposed to be fixed in XnView MP 1.4.1. Please check and confirm the bug fix here.

[masterjp]

Your version 1.4.1 works perfect and fix the bug! Thank you.