Page 1 of 1

Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 12:55 am
by golemus
Hi. I have a large photo collection and I would like to convert it to more modern format so that it consumes less space.

I have been comparing AVIF and JXL a lot and am probably ending towards AVIF because barely no normal apps (according to my tests on Galaxy S21FE) are able to view JXL files.

So there is large folder tree structure full of pictures, mainly JPG, newer ones are HEIC, then there are some CR2 from my old Canon EOS 400D and RW2 from my Panasonic Lumix LX-7 and NEF raw files also.

As there is 40 000 pics and hundreds of folders I cannot do every folder separately but have to automate some rutine to go through the tree hierarchy.

I am currently doing this for video/mp4 files with ffmpeg and some .bat files in windows and after that comes the picture project. There is .bat file that goes recursively through the whole tree hierarchy. Unfortunately videos are blended with pics partially to same folders.




First I was thinking of using ffmpeg also for this but ffmpeg seems unable to transfer EXIF and other metadata (if this is incorrect let me know).

So now I am thinking about XNView MP. It has batch mode but it seems to be aimed for single folders.

Can XNView MP batch mode or even single conversion be launched from command line with appropriate parameters?

(ps. not directly related but first I tried XNShell but it seems to have bug with AVIF conversion and produced files that many targets were unable to open).


Also I don't necessarily want to process yet the RAW files, JPGs are most important as they consume most space.

Mostly I will keep the resolution same but some very large pics I am considering resizing in the conversion.



Good tips how to approach this?

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 9:05 am
by masterjp
I was in the same situation and converted big folder structures with JPEG/RAW-Files from my digital canon cameras with XnviewMP to more efficiency picture formats (HEIF/HEIC, WEBP, AVIF and JXL) for smaller archiving big amounts of photos for viewing on android smartphone and tablet. XnViewMP is the best choice for this job and can create the same folder structure from your source in the destination path!
I did tests with all modern picture formats, because I want to save much space and a good quality to size relationship.
But all my tests ends in big compatibility problems on android systems, bad performance in viewing, high CPU usage and battery drain compared with old formats (jpeg/webp). In most cases the new file formats (avif, heic, etc.) do not support hardware acceleration on android systems.
Many new formats have no native support in android 12-15. Maybe in the future.
The AVIF format is not totally supported by android 12 or 13. Android support color format 4:2:0 only!

I did some tests with HEIC and AVIF.
Both formats are great, but slow in compression and viewing. They use software for viewing and do not use hardware decoding.
Further they get slowly loaded by android apps, consume more CPU power and it results in faster battery drain!
So I do not recommend this modern and good formats for tablets or smartphones!

The best solution is encoding pictures to JPEG with the new JPEG-LI encoder in XnviewMP!
You will have 100% compatibility in all android apps for viewing, fast loading of pictures, very high compression near the size of webp and avif!
The JPEG-LI Encoder save up to 35% space for the same quality compared to a standard/progressive jpeg file.
It use the latest technics and advantages of JPEG-XL file format, but with high compatibility!
It works perfect on all my android viewing apps!

For picture viewing I can recommend the follwing apps,
which support JPEG, JPEG-LI, AVIF, HEIC, etc:

- FossifyGallery (Freeware) https://play.google.com/store/apps/deta ... fy.gallery
- A+ Gallery

If you need further infos or recommandations for jpeg-li encoding I can write down my favorite settings.

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 2:52 pm
by golemus
masterjp wrote: Tue Feb 25, 2025 9:05 am But all my tests ends in big compatibility problems on android systems, bad performance in viewing, high CPU usage and battery drain compared with old formats (jpeg/webp). In most cases the new file formats (avif, heic, etc.) do not support hardware acceleration on android systems.
I am aware of lack of hardware acceleration. Samsung Galaxy S22 and newer have (AFAIK) AV1 hardware acceleration so I would I assume? that the same applies for avif. So it should be coming.

Anyway currently I just feel that it is most important that people can open pics, even if it is slow.
I did some tests with HEIC and AVIF.
Both formats are great, but slow in compression and viewing. They use software for viewing and do not use hardware decoding.
Further they get slowly loaded by android apps, consume more CPU power and it results in faster battery drain!
So I do not recommend this modern and good formats for tablets or smartphones!
I'd be interested in your heic vs avif results. My smartphone now outputs heic but I don't want to encode jpg:s to heic because it is proprietary format, not all PCs can open it without buying heic extensions and in general I feel that heic/h265 encoding is a bit buggy and not reliable.
The best solution is encoding pictures to JPEG with the new JPEG-LI encoder in XnviewMP!
You will have 100% compatibility in all android apps for viewing, fast loading of pictures, very high compression near the size of webp and avif!
The JPEG-LI Encoder save up to 35% space for the same quality compared to a standard/progressive jpeg file.
It use the latest technics and advantages of JPEG-XL file format, but with high compatibility!
It works perfect on all my android viewing apps!
35% is not enough gains that I would see sense of going through the effort of converting the collection. I will check it out anyway. So does it output standard jpeg?


For picture viewing I can recommend the follwing apps,
which support JPEG, JPEG-LI, AVIF, HEIC, etc:
If you need further infos or recommandations for jpeg-li encoding I can write down my favorite settings.
yeah why not. But you don't have to write, just take screenshot of settings window it will be faster.

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 6:16 pm
by masterjp
Here is an example file, which was encoded by avif (Quality 70% - color 4:2:0), heic (50% Quality) and JPEG-LI (75% Quality - 4:2:0 color format - progressive). I used these settings to get an nearly equal filesize.

There are only a few quality differences. AVIF can better handle flat textures and text edges look sharper.
But the JPEG-LI encode looks great and has a very good speed/compression behaviour.
AVIF compression tooks up to 5 times more.

---
My recommend setting for JPEG-LI is min. 75% quality (better 80% if save space is not important).
Progressive format is a bit slower and has lower filesize than JPEG Standard (=baseline).

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 6:32 pm
by masterjp
I am aware of lack of hardware acceleration. Samsung Galaxy S22 and newer have (AFAIK) AV1 hardware acceleration so I would I assume? that the same applies for avif. So it should be coming.

Anyway currently I just feel that it is most important that people can open pics, even if it is slow.


masterjp: I have Samsung devices, too (Samsung S23 and Samsung Tab S7).
Both have AV1 hardware accelaration, but in most cases only for videos (!).
AVIF picture format is something different!

Most Freeware apps for picture viewing use software decoding for pictures, so you need CPU power only!
The Samsung Gallery app (pre-installed) ist very fast, good optimized and a great program for viewing pictures (JPEG, AVIF, HEIC).
Some good sorting features are missing, but the performance is fantastic.

---

I'd be interested in your heic vs avif results. My smartphone now outputs heic but I don't want to encode jpg:s to heic because it is proprietary format, not all PCs can open it without buying heic extensions and in general I feel that heic/h265 encoding is a bit buggy and not reliable.

masterjp: I agree with you to ignore the HEIC format. Samsung support it for their camera app. Apple support it, too.
In the beginning of HEIC and AVIF I had many problems with both formats. Sometimes picture with odd picture size (e.g. 2491 x 2031 pixel) were saved with green lines in the picture. It is a basic problem of avif format, because it accept unodd only.
Maybe the later avif encoders could fix it. In my further testings sometimes HEIC or AVIF files get different colors (e.g. red is slightly darker or brighter).

---

35% is not enough gains that I would see sense of going through the effort of converting the collection. I will check it out anyway. So does it output standard jpeg?

masterjp: The saved JPEG-LI files use the standard JPEG format, which you can read or view with all windows, android, etc. programs and apps.
Adobe Photoshop Elements works only with installed Camera RAW!

All the other popular picture viewer and converter (IrfanView, XnView, XnViewMP, Gimp, Faststone Viewer, Paint, all WEB-Browsers) can handle the jpeg-li files without problems.

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 6:35 pm
by masterjp
Here are my recommend settings for JPEG-LI encoding:
settings.png

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Tue Feb 25, 2025 8:23 pm
by Kadet
golemus wrote: Tue Feb 25, 2025 2:52 pm 35% is not enough gains that I would see sense of going through the effort of converting the collection. I will check it out anyway. So does it output standard jpeg?

If you want profits greater than 35%, you need to know what you lose. In the picture below you have on the right a test image PNG; in the middle AVIF with 70% quality; and for the same file size, on the left there is an image in JPEGLI with 33% quality. When watching in 100% zoom comparison is not bad, the copies are quite similar to the original. But what do we see after increasing to 400%? AVIF has completely lost noise/granularity. JPEGLI, like JPEG, also blurred the picture, but also clearly you can see squares.
You will receive distortions, wanting saving too much space. However, from this two evil, I will choose AVIF.

If you want to keep photography nuances, buy a larger disk and compress (or not) it to JXL not less than with 95% quality. Soon all software will be known jxl format.
TEST_JPEGLI_AVIF.png

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Wed Feb 26, 2025 9:28 pm
by golemus
In the 1-2 pics I tested AVIF beats JXL clearly.

I must say that majority of my collection is pretty bad pics (mobile phone) and avif works very well as it removes also noise. AVIF algorithm seems particularly good for pics with have a lot of dark/black.

It might be different playground for somebody that has only high quality DSLR material.

I tested also JPEG LI but I got worse quality results than normal JPG. I am not 100% sure if i used the right settings but I tried to respect the ones that were written above.

And it consumes much more space than AVIF or JXL so I don't really like it.

Re: Batch conversion of large collection (40 000 pics) of JPG to AVIF or JXL

Posted: Fri Feb 28, 2025 7:44 pm
by masterjp
AVIF has fantastic compression, but you need to test it on your android device!

On my Samsung S10 with Android 12 I have big problems with the samsung gallery app.
It only supports AVIF files with color format 4:2:0.

I tested this format with 70% quality and with portrait photos I had small but annoying color differnces.
The skin tone was much darker than on the original jpeg photo.

If you use fossify gallery on android, the avif decoder can handle it perfectly.

Maybe on your Samsung S22 the quality is better. Please try it, before you convert all your photos!
Check the colors.

WIth XnViewMP the avif files are viewed correctly, so you can not check it on windows/linux systems.