JPEG Corruption

[dominique]

Dear Pierre and GFLSDK users,
I developed a software for massive and recurssive image processing with GFLSDK in C++. It worked fine since some strange JPEG corruption appears this week :
http://a.dominique.guerin.free.fr/LIBGFL/image1.jpg
File corruption detected using Photoshop
http://a.dominique.guerin.free.fr/LIBGFL/image2.jpg
No file error detected with Photoshop
I don't think it's a library problem. I think it's a network problem while image transfert from processing server to data server.
Is there any way to detect file corruption like this? My google does not help me.
Or any other functions to check write operations (I check all returned codes while calling writing functions)?
Or any error code provided for decoding JPEG operations?
Merci d'avance

[xnview]

Ok, i can change GFLSDK to detect image1.jpg, but on image2.jpg it's not easy!

[dominique]

It's a really big problem
It's a good begining if you can return an error code on image1. I think this image miss pixels. We can see it on the "offseted" borders after the bytes error.
The second image is a mistery... it may be the result of a bit shift... something like that... I have an peace of solution using histogram correlation from different image's parts.
A good idea is to provide the MD5 of the JPEG file without writing it on the disk (only perform it on RAM to avoid disk access). So I can compare the MD5 before (in RAM) and after the transfert (on the distant disk). Perhaps you can add it for advanced users who want to save file through network

[xnview]

But you have this problem when your write jpeg with GFL??

[dominique]

Yes I use GFL to write JPEG and insert IPTC throuth the network.
I'm not really shure that's a GFL problem. Perhaps it's the operating systems or network... It's difficult to test each part of the process.
In one hand I just want you to confirm that's impossible to have this kind of problem with GFL... so I can say my boss that my software and GFL have no bug
In the other hand it should be interesting to had a file verification option in GFL (automated image processing with error log, image tester...).

[xnview]

In one hand I just want you to confirm that's impossible to have this kind of problem with GFL... so I can say my boss that my software and GFL have no bug

I have never heard from users about such problem

[dominique]

Ok
Now I think it could be interesting to add a MD5 verification in my software...
Perhaps I can had a kind of RAMDisk to write the file on the Processing server RAM and copy it to the distant disk and then compare the 2 files' MD5.
Does GFL have a function to write a file to a buffer/stream instead of a disk path?

[xnview]

Does GFL have a function to write a file to a buffer/stream instead of a disk path?

Yes, gflSaveBitmapToMemory

[dominique]

That's why I like GFL
I'll test this solution.

[dominique]

I try a simple solution and have the first interesting result.
My software compute MD5 on processed images from the buffer provide by gflSaveBitmapToMemory on the local server.
I use a simple command line tool the compute MD5 on transfered processed images directly on the distant server.
And...
I found 1 corrupted image during processing (~4000 images processed).
So this solution does not correct the problem but allow corruption detection and correction.

I see 2 problems that give this corruption...
1 - Local network problems
2 - Operating system problem while calling gflSaveBitmap to save on a distant server.

More tests and results will give more informations.
I hope I'll give you more explainations next time.
Rgds,