JPEG Corruption

Discussions on GFL SDK, the graphic library for reading and writing graphic files

Moderators: XnTriq, xnview

Post Reply
User avatar
dominique
Posts: 72
Joined: Thu Nov 08, 2007 9:22 am

JPEG Corruption

Post by dominique » Fri Jun 20, 2008 8:13 am

Dear Pierre and GFLSDK users,
I developed a software for massive and recurssive image processing with GFLSDK in C++. It worked fine since some strange JPEG corruption appears this week :
http://a.dominique.guerin.free.fr/LIBGFL/image1.jpg
File corruption detected using Photoshop
http://a.dominique.guerin.free.fr/LIBGFL/image2.jpg
No file error detected with Photoshop
I don't think it's a library problem. I think it's a network problem while image transfert from processing server to data server.
Is there any way to detect file corruption like this? My google does not help me.
Or any other functions to check write operations (I check all returned codes while calling writing functions)?
Or any error code provided for decoding JPEG operations?
Merci d'avance ;)
Dom

User avatar
xnview
Author of XnView
Posts: 33205
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: JPEG Corruption

Post by xnview » Fri Jun 20, 2008 9:38 am

Ok, i can change GFLSDK to detect image1.jpg, but on image2.jpg it's not easy!
Pierre.

User avatar
dominique
Posts: 72
Joined: Thu Nov 08, 2007 9:22 am

Post by dominique » Fri Jun 20, 2008 10:04 am

It's a really big problem :)
It's a good begining if you can return an error code on image1. I think this image miss pixels. We can see it on the "offseted" borders after the bytes error.
The second image is a mistery... it may be the result of a bit shift... something like that... I have an peace of solution using histogram correlation from different image's parts.
A good idea is to provide the MD5 of the JPEG file without writing it on the disk (only perform it on RAM to avoid disk access). So I can compare the MD5 before (in RAM) and after the transfert (on the distant disk). Perhaps you can add it for advanced users who want to save file through network ;)
Dom

User avatar
xnview
Author of XnView
Posts: 33205
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Post by xnview » Fri Jun 20, 2008 11:31 am

But you have this problem when your write jpeg with GFL??
Pierre.

User avatar
dominique
Posts: 72
Joined: Thu Nov 08, 2007 9:22 am

Post by dominique » Fri Jun 20, 2008 11:59 am

Yes I use GFL to write JPEG and insert IPTC throuth the network.
I'm not really shure that's a GFL problem. Perhaps it's the operating systems or network... It's difficult to test each part of the process.
In one hand I just want you to confirm that's impossible to have this kind of problem with GFL... so I can say my boss that my software and GFL have no bug ;)
In the other hand it should be interesting to had a file verification option in GFL (automated image processing with error log, image tester...).
Dom

User avatar
xnview
Author of XnView
Posts: 33205
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Post by xnview » Fri Jun 20, 2008 12:14 pm

dominique wrote:In one hand I just want you to confirm that's impossible to have this kind of problem with GFL... so I can say my boss that my software and GFL have no bug ;)
I have never heard from users about such problem
Pierre.

User avatar
dominique
Posts: 72
Joined: Thu Nov 08, 2007 9:22 am

Post by dominique » Fri Jun 20, 2008 12:32 pm

Ok
Now I think it could be interesting to add a MD5 verification in my software...
Perhaps I can had a kind of RAMDisk to write the file on the Processing server RAM and copy it to the distant disk and then compare the 2 files' MD5.
Does GFL have a function to write a file to a buffer/stream instead of a disk path?
Dom

User avatar
xnview
Author of XnView
Posts: 33205
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Post by xnview » Fri Jun 20, 2008 12:36 pm

dominique wrote: Does GFL have a function to write a file to a buffer/stream instead of a disk path?
Yes, gflSaveBitmapToMemory :-)
Pierre.

User avatar
dominique
Posts: 72
Joined: Thu Nov 08, 2007 9:22 am

Post by dominique » Fri Jun 20, 2008 12:40 pm

That's why I like GFL ;)
I'll test this solution.
Dom

User avatar
dominique
Posts: 72
Joined: Thu Nov 08, 2007 9:22 am

Post by dominique » Thu Jul 03, 2008 4:18 pm

I try a simple solution and have the first interesting result.
My software compute MD5 on processed images from the buffer provide by gflSaveBitmapToMemory on the local server.
I use a simple command line tool the compute MD5 on transfered processed images directly on the distant server.
And...
I found 1 corrupted image during processing (~4000 images processed).
So this solution does not correct the problem but allow corruption detection and correction.

I see 2 problems that give this corruption...
1 - Local network problems
2 - Operating system problem while calling gflSaveBitmap to save on a distant server.

More tests and results will give more informations.
I hope I'll give you more explainations next time.
Rgds,
Dom

Post Reply