Page 1 of 1
Problem when reading images in multi-thread environment
Posted: Thu Jul 28, 2005 10:43 am
by Ithier
Hi,
I have a multi-threaded application which runs on a computer with a CPU with HyperThreading.
When I load two PNG at the same time, sometimes (perhaps 1 time out of 3), the loaded PNG is corrupted.
I have tried the same application with a computer which does not have HyperThreading and all is fine.
Here is an example of a corrupted image :
and the original :
It seems to be the gflLoadBitmapFromMemory function as I have extracted the buffer justs after the call and the data is already corrupted.
I use GFL 2.41 and Windows XP or Windows Server 2003 and the application is compiled with gcc 3.2.3.
Does someone has already encoutered this problem ?
Is GFL thread safe ?
Ithier
Re: Problem when reading images in multi-thread environement
Posted: Sat Jul 30, 2005 12:15 pm
by xnview
Ithier wrote:Hi,
I have a multi-threaded application which runs on a computer with a CPU with HyperThreading.
When I load two PNG at the same time, sometimes (perhaps 1 time out of 3), the loaded PNG is corrupted.
I have tried the same application with a computer which does not have HyperThreading and all is fine.
Here is an example of a corrupted image :
and the original :
It seems to be the gflLoadBitmapFromMemory function as I have extracted the buffer justs after the call and the data is already corrupted.
I use GFL 2.41 and Windows XP or Windows Server 2003 and the application is compiled with gcc 3.2.3.
Does someone has already encoutered this problem ?
Is GFL thread safe ?
Yes, the problem is only on PNG file??
Posted: Tue Aug 02, 2005 4:14 pm
by Ithier
Yes, at least it does not happen with jpeg files (I have not tested with other file formats).
Posted: Wed Aug 03, 2005 8:13 am
by xnview
Ithier wrote:Yes, at least it does not happen with jpeg files (I have not tested with other file formats).
Could you send me a sample project, please?
Posted: Wed Aug 03, 2005 10:17 am
by Ithier
Here is some code to test the problem. I remember you that it happens only on a computer with hyper-threading or with multiple cpu.
It happens in debug and in release mode.
The code, reads a file in memory and then create two threads which load the file and save it in jpeg.
The problem does not happen all the time, I have run the program 5 times so it has generated 10 jpeg files. On a computer with 2 cpu, 6 files were corrupted. On a computer with one cpu but with hyperthreading 3 files were corrupted. On a single cpu computer, 0 file was corrupted.
Code: Select all
#include "libgfl.h"
#include <fstream>
#include <windows.h>
GFL_UINT8* pData = NULL;
int data_size = 0;
DWORD Thread (LPVOID Param)
{
//Load PNG file from memory
GFL_LOAD_PARAMS load_option;
gflGetDefaultLoadParams( &load_option );
GFL_FILE_INFORMATION GflFI;
GFL_BITMAP* GflBitmap;
gflLoadBitmapFromMemory(pData, data_size, &GflBitmap, &load_option, &GflFI);
//Output filename (Id of the thread)
char filename[32];
sprintf(filename, "%d.jpg", GetCurrentThreadId());
//Save the file
GFL_SAVE_PARAMS save_option;
gflGetDefaultSaveParams(&save_option);
save_option.FormatIndex = gflGetFormatIndexByName("jpeg");
gflSaveBitmap(filename, GflBitmap, &save_option);
return 0;
}
int main(int argc, char* argv[])
{
gflLibraryInit();
//Read PNG file in memory
std::ifstream ficDoc (argv[1], std::ios::in|std::ios::binary|std::ios::ate);
data_size = ficDoc.tellg();
pData = new GFL_UINT8[data_size];
ficDoc.seekg (0, std::ios::beg);
ficDoc.read (reinterpret_cast<char*>(pData), data_size);
ficDoc.close();
for (int i=0; i<2; ++i)
{
DWORD dwThreadId;
CreateThread (NULL, // no security attribute
0, // default stack size
(LPTHREAD_START_ROUTINE) Thread,
NULL, // thread parameter
0, // not suspended
&dwThreadId); // returns thread ID
}
Sleep (3000); //Wait for the threads to terminate
return 0;
}
If you need I can send you the dsp file (for Visual C++ 6.0) but all the options are standard except C++ -> Code Generation -> Use runtime-library which is "Debug Multithread" or "Multithread".
Good luck
Ithier
Posted: Wed Aug 03, 2005 2:42 pm
by xnview
Ithier wrote:Here is some code to test the problem. I remember you that it happens only on a computer with hyper-threading or with multiple cpu.
It happens in debug and in release mode.
Ok, i'll try to find a dual proc computer

hey Ithier
Posted: Sat Aug 06, 2005 10:37 pm
by MaierMan
Can you determine the "stage" when corruption occures.
(eg. by setting CS arround gflLoadBitmap and/or gflSaveBitmap).
I think that would give Pierre a better clue.
(I know about the problems of debugging MT-code on a single processor box :p)
Something like:
Code: Select all
#include <libgfl.h>
#include <fstream>
#include <iostream>
#include <windows.h>
#include <string>
#include <time.h>
class CondCS
{
CRITICAL_SECTION s;
public:
CondCS() { InitializeCriticalSection(&s); }
~CondCS() { DeleteCriticalSection(&s); }
void lock(const BOOL x, const char *i, std::string name)
{
if (x)
{
EnterCriticalSection(&s);
std::cerr << name << " - lock: " << i << std::endl;
}
}
void unlock(const BOOL x, const char *i, std::string name)
{
if (x)
{
std::cerr << name << " - unlock: " << i << std::endl;
LeaveCriticalSection(&s);
}
}
};
CondCS sec;
#define TOPROT 2 // [0-3[
#define PROT(x) (TOPROT & x)
class Thread;
DWORD ThreadFunc(Thread *thread);
static unsigned hrand(unsigned high)
{
return (unsigned)((double)(rand() % RAND_MAX) / (double)RAND_MAX * (double)high);
}
class Thread
{
public:
unsigned id;
unsigned short prio;
HANDLE hThread;
DWORD dwThread;
GFL_UINT8 *pData;
GFL_UINT32 sSize;
std::string name;
Thread(unsigned i, GFL_UINT8 *d, GFL_UINT32 s) : id(i), hThread(NULL), dwThread(0), pData(d), sSize(s)
{
hThread = CreateThread(
NULL,
0,
(LPTHREAD_START_ROUTINE)ThreadFunc,
(LPVOID)this,
CREATE_SUSPENDED,
&dwThread
);
prio = hrand(3);
SetThreadPriority(
hThread,
(prio == 2)
? THREAD_PRIORITY_HIGHEST
: (
!prio
? THREAD_PRIORITY_IDLE
: THREAD_PRIORITY_NORMAL
)
);
if (!hThread)
std::cerr << "Failed to create!" << std::endl;
char n[1024];
sprintf(n, "Thread %.2d (%p) p%d", id, hThread, prio);
name.assign(n);
}
~Thread() { if (hThread) CloseHandle(hThread); };
void Execute() { if (hThread) ResumeThread(hThread); }
};
DWORD ThreadFunc(Thread *thread)
{
DWORD err;
GFL_LOAD_PARAMS load_option;
gflGetDefaultLoadParams(&load_option);
GFL_FILE_INFORMATION GflFI;
GFL_BITMAP *GflBitmap;
sec.lock(PROT(1), "load", thread->name);
err = MAKELONG(gflLoadBitmapFromMemory(
thread->pData,
thread->sSize,
&GflBitmap,
&load_option,
&GflFI
), 0);
sec.unlock(PROT(1), "load", thread->name);
std::string filename(thread->name);
filename.append(".jpg");
GFL_SAVE_PARAMS save_option;
gflGetDefaultSaveParams(&save_option);
save_option.FormatIndex = gflGetFormatIndexByName("jpeg");
sec.lock(PROT(2), "save", thread->name);
err |= MAKELONG(0, gflSaveBitmap(
(char*)filename.c_str(),
GflBitmap,
&save_option
));
sec.unlock(PROT(2), "save", thread->name);
return err;
}
#define THREADS 10
int main(int argc, char* argv[])
{
if (argc != 2)
{
std::cout << "/me takes 1 arg" << std::endl;
return 1;
}
gflLibraryInit();
srand((unsigned)time(NULL));
std::ifstream ficDoc(
argv[1],
std::ios::in | std::ios::binary | std::ios::ate
);
GFL_UINT32 sSize = ficDoc.tellg();
GFL_UINT8 *pData = new GFL_UINT8[sSize];
ficDoc.seekg (0, std::ios::beg);
ficDoc.read (reinterpret_cast<char*>(pData), sSize);
ficDoc.close();
HANDLE *pHandles = new HANDLE[THREADS];
Thread **pThreads = new Thread*[THREADS];
std::cout << "Protected stages: " << TOPROT << std::endl;
for (unsigned i = 0; i < THREADS; ++i)
{
pThreads[i] = new Thread(i, pData, sSize);
pHandles[i] = pThreads[i]->hThread;
}
for (unsigned i = 0; i < THREADS; ++i)
{
pThreads[i]->Execute();
}
WaitForMultipleObjects(
THREADS,
pHandles,
TRUE,
INFINITE
);
for (unsigned i = 0; i < THREADS; ++i)
{
DWORD ec;
if (GetExitCodeThread(pHandles[i], &ec))
{
std::cout << pThreads[i]->name << ") - load: " << LOWORD(ec) << " - save: " << HIWORD(ec) << std::endl;
}
else
{
std::cout << pThreads[i]->name << "): ABANDONED" << std::endl;
}
delete pThreads[i];
}
delete [] pData;
delete [] pHandles;
delete [] pThreads;
return 0;
}
But Im only single-core, so output is fine for all four TOPROTs :p
Posted: Sun Aug 07, 2005 10:33 pm
by Guest
Hi,
Thank you for your code and for your time, in fact I already knew where the problem is, as I explain in my first post, because just after the loading of the PNG file I have written the GFL_BITMAP buffer in a file. I have read the file (like a bitmap file without headers) and the data was already corrupted. So I am quite sure that the problem come from the gflLoadBitmapFromMemory function.
Ithier
Posted: Mon Sep 05, 2005 3:00 pm
by Ithier
Hi Pierre,
Did you have time to look at my problem ?
If you need some help to test special version of SDK (with debugging output for example), I can do that on my dual-proc computer. Do not hesitate to contact me, I would be very glad to help you.
Best Regards
Ithier
Posted: Sun Sep 18, 2005 1:55 pm
by xnview
Ithier wrote:Hi Pierre,
Did you have time to look at my problem ?
If you need some help to test special version of SDK (with debugging output for example), I can do that on my dual-proc computer. Do not hesitate to contact me, I would be very glad to help you.
Sorry currently not the time to check this problem...