Batch Processing running on multiple cores

[hermandsen]

Hi!
I just started using XnView after switching from IrfanView - both great products.
I'm missing one thing from both though: Batch processing running over multiple threads.

If you're using a dual-core cpu, or a quad-core cpu, you'll only see 50% / 25% usage when running a batch job.
Using one thread per cpu-core could really speed up this process.

// hermandsen

[xnview]

XnConvert uses multiple cores to process a picture.

[Hacker]

I wholeheartedly support this request. It would be very nice if XnView could eg. resize four images (in case of a quad-core processor) at the same time instead of just one after another.

Roman

[helmut]

Sure enough, I also support this request. According to Pierre multiple processors are supported by XnConvert, already.

A note:
When looking at the processor load one has to take the file accessing into account. Depending on the (file) size of the images and the type of processing, the bottle neck might not be the processor(s) which perform the unpacking, the processing, and packing of image data, but the hard disk which has to supply the image data and make it persistent after processing.

[aureagenus]

(I apologize for any inconveniences that my poor English may cause.)

I also support this request.

In my opinion, processing multiple files simultaneously is probably the easiest and most effective way of utilizing multicore cpus 100%.
Because, it is not easy to parallelize standard algorithms or make any parallelized algorithms be able to use resource of cpu 100%.
If multiple files are to be processed simultaneously, multi-core cpus can be utilized without having any notable bottlenecks since each files are modified and saved independently.

Pixillion Image Converter and Contenta Converter use this method. And during the batch processing, the cpu usage graph in Windows Task Manager shows almost 100%. Also, it is really fast.
In regard to this method, sometimes many Lightroom users do what they call 'manual parallel processing method.'
Even though many of processing algorithms in Lightroom has already been parallelized, many people does not seem to be fully satisfied with Lightroom because it does not use their cpus 100%.
http://feedback.photoshop.com/photoshop ... n_parallel

Thus, Lightroom users do this in order to utilize their cpus 100%, separating bunch of pictures into a few independent groups and processing them simultaneously.

I think if XnView can separate files of the list of the batch window into a few group and process them simultaneously, it can easily utilize any multi-core cpus, regardless of parallel efficiency of processing algorithm.

Best Regards,
Joonsuk Kim

[loopingz]

To my experience a big bottleneck is the storage. I am used to store on a hard drive or a (fast) NAS. Anyway it is not fast enough. Before processing I move all my pic to a local ssd and work on it. Batch rename is then almost instant and so is timestamp change. For timestamp change it is a huge improvment as Xnview would look frozen for a long time if pics were on the NAS.

[maerean]

multiple cores/cpus computers have been available since 2006 and this feature has been asked for in this forum since 2011.
we're almost in 2014 now and my cpu is only being used at 13% (1/8) of its potential which means I have to wait 8 times as much as I should for the convert steps to take place.
please implement this feature.
it's definitely a feature worth buying a licence for. you can quote me on this one.

[cicciobello]

[quote="maerean"]my cpu is only being used at 13% (1/8) of its potential

You are lucky! New Intel's 8 core/16 threads CPU use 6% of CPU potential, I suppose.

[aureagenus]

To my experience a big bottleneck is the storage.

Actually, the storage is not a big bottleneck. To fully utilize my cpu, I decided to use nconvert, which is command line batch image processor, instead of XnView. In addition, thanks to Joey from stackoverflow.com http://stackoverflow.com/questions/6727 ... -processes, I could run multiple instances of nconvert simultaneously to convert image sequences.

Here is the code ...

Code: Select all

cls
@echo off
for /f %%i in ('dir "../2_Modified/" ^| find "파일"') do set numFiles=%%i
setlocal EnableDelayedExpansion
set /a "cntStart = 1"
set /a "cntFinal = 15"
set /a "maxIdx = numFiles/15"
set /a "remIdx = numFiles-maxIdx*15"
set /a "currIdx = maxIdx*15+1"
for /l %%i in (1,1,%maxIdx%) do (
  call :loop !cntStart! !cntFinal!
  set /a "cntStart += 15"
  set /a "cntFinal += 15"
)
if %remIdx% gtr 0 (
  echo Starting processing instance for %currIdx% ~ %numFiles%
  start /b /high nconvert -quiet -out lwfjp2 -n %currIdx% %numFiles% 1 -o ../3_JPEG2000/Page###.jp2 ../2_Modified/Page###.tif
)
goto :eof

:loop
call :checkinstances
if %INSTANCES% leq 12 (
    echo Starting processing instance for %1 ~ %2
    start /b /high nconvert -quiet -out lwfjp2 -n %1 %2 1 -o ../3_JPEG2000/Page###.jp2 ../2_Modified/Page###.tif
    goto :eof
)
echo Waiting for instances to close ...
ping -n 1 ::1 >nul 2>&1
goto loop
goto :eof

:checkinstances
for /f %%t in ('tasklist /fo csv /fi "IMAGENAME eq nconvert.exe"^|find /c "nconvert.exe"') do set INSTANCES=%%t
goto :eof

Using my Intel Core i7-4960X 6-Cores, the conversion process much faster than that of using XnView. I've run the code both in Plextor M5 Pro 512GB and WD Green 2TB, there was not a significant performance loss due to storage speed.