1.4 beta - Batch conversion with skip existing file setting - extreme slow!

New issues i.e. issues that have been introduced in the latest test version.

Moderators: XnTriq, helmut, xnview

User avatar
masterjp
Posts: 305
Joined: Fri Feb 13, 2009 4:37 pm
Location: Duesseldorf, Germany

1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by masterjp »

I often use the skip existing picture feature in batch conversion, because I want to save time during conversion large picture archives.
Today I want to convert new files, but I already converted them a few days before. So XnviewMP starts the conversion.
I used the 4 core speed setting in conversion dialog, but the whole conversion works in single cpu core speed!
It is very slowly. In previous versions 1.3 or older the performance with skip existing files and multi core cpu usage was fantastic and fast.
Since version 1.3.1 and 1.4 betar the speed is very slowly, because it will use one cpu core only :!:

Why did you change this great algorithm? Please make it great again! :)



XNVIEWMP 1.3.1 and 1.4 beta:
======================

Input files: 72506
Extracted pages: 72506
Overwritten: -72506
Skipped: 72506

Total Duration: 13 minute(s)


XNVIEWMP 1.3:
=============

Input files: 32627
Extracted pages: 32627
Overwritten: -32627
Skipped: 32627

Total Duration: 16 second(s)

Finished at: Do Nov 24 20:48:19 2022
PC: Intel 8700k + Asus Z370-F + 16 GB RAM G.Skill
OS: Windows 10 Pro x64 (latest updates) | XN-View (latest stable version) |XnViewMP (latest stable version) | XnConvert (latest) | Adobe Photoshop Elements 2022 & Elements XXL 9
User avatar
xnview
Author of XnView
Posts: 38830
Joined: Mon Oct 13, 2003 7:31 am
Location: France

Re: 1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by xnview »

Strange, in the 2 tests you have not the same number of files. Which output format? Which input format?
Pierre.
User avatar
masterjp
Posts: 305
Joined: Fri Feb 13, 2009 4:37 pm
Location: Duesseldorf, Germany

Re: 1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by masterjp »

The input files are jpeg files with max. 6000 x 4000 Pixels and I convert them to webp.


Image
PC: Intel 8700k + Asus Z370-F + 16 GB RAM G.Skill
OS: Windows 10 Pro x64 (latest updates) | XN-View (latest stable version) |XnViewMP (latest stable version) | XnConvert (latest) | Adobe Photoshop Elements 2022 & Elements XXL 9
User avatar
xnview
Author of XnView
Posts: 38830
Joined: Mon Oct 13, 2003 7:31 am
Location: France

Re: 1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by xnview »

could you send me a sample file used? are you reproduced the difference even on 100 files for each version? Without multiple core, you have same time in 1.3/1.3.1?
16seconds to convert 32000 files is strange to me, and i see 'Skipped: 32627' so no files are written???
Pierre.
User avatar
masterjp
Posts: 305
Joined: Fri Feb 13, 2009 4:37 pm
Location: Duesseldorf, Germany

Re: 1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by masterjp »

The old version 1.3 was very fast, because it did not calculate the file size saving informationens for each file and whole file saving for the complete conversion. I convert a big collection of files to another directory, where around 90 percent of the files already exists. This cost much time for checking, if the path and files already exists. Maybe it works in single core mode only.

I optimized some directorys with the pingo optimizer, which convert the jpeg files to arithmetic compression.
But this is no problem for XnViewMP. I can load the files.
Last edited by masterjp on Wed Nov 30, 2022 9:04 am, edited 1 time in total.
PC: Intel 8700k + Asus Z370-F + 16 GB RAM G.Skill
OS: Windows 10 Pro x64 (latest updates) | XN-View (latest stable version) |XnViewMP (latest stable version) | XnConvert (latest) | Adobe Photoshop Elements 2022 & Elements XXL 9
User avatar
xnview
Author of XnView
Posts: 38830
Joined: Mon Oct 13, 2003 7:31 am
Location: France

Re: 1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by xnview »

masterjp wrote: Tue Nov 29, 2022 8:54 pm The old version 1.3 was very fast, because it did not calculate the file size saving informationens for each file and whole file saving for the complete conversion. I convert a big collection of files to another directory, where around 90 percent of the files already exists. This cost much time for checking, if the path and files already exists. Maybe it works in single core mode only.
It's in 1.4.0 not in 1.3.1, so for me it's not the problem. So if i understand you correctly, it's only if files already exist? i've tried but no difference.
Is it possible to re-test 1.3.1 & 1.3.0?
Pierre.
User avatar
masterjp
Posts: 305
Joined: Fri Feb 13, 2009 4:37 pm
Location: Duesseldorf, Germany

Re: 1.4 beta - Batch conversion with skip existing file setting - extreme slow!

Post by masterjp »

Thank you for your help.

It is very difficult to analyse the reason for my problem.
The problem is very special, because I have so many files
and it will only happens, if you convert a huge picture archiv or under special condition.

I will do some further tests under the same conditions and with all versions (1.3 and 1.4 beta 1) to precisly allocate the reason for my problem.
PC: Intel 8700k + Asus Z370-F + 16 GB RAM G.Skill
OS: Windows 10 Pro x64 (latest updates) | XN-View (latest stable version) |XnViewMP (latest stable version) | XnConvert (latest) | Adobe Photoshop Elements 2022 & Elements XXL 9