Problem when reading images in multi-thread environment

Discussions on GFL SDK, the graphic library for reading and writing graphic files

Moderators: XnTriq, xnview

Post Reply
User avatar
Ithier
Posts: 47
Joined: Fri Nov 19, 2004 10:50 am
Location: Paris, France
Contact:

Problem when reading images in multi-thread environment

Post by Ithier » Thu Jul 28, 2005 10:43 am

Hi,

I have a multi-threaded application which runs on a computer with a CPU with HyperThreading.
When I load two PNG at the same time, sometimes (perhaps 1 time out of 3), the loaded PNG is corrupted.
I have tried the same application with a computer which does not have HyperThreading and all is fine.
Here is an example of a corrupted image :
Image
and the original :
Image

It seems to be the gflLoadBitmapFromMemory function as I have extracted the buffer justs after the call and the data is already corrupted.
I use GFL 2.41 and Windows XP or Windows Server 2003 and the application is compiled with gcc 3.2.3.

Does someone has already encoutered this problem ?
Is GFL thread safe ?

Ithier

User avatar
xnview
Author of XnView
Posts: 32052
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Re: Problem when reading images in multi-thread environement

Post by xnview » Sat Jul 30, 2005 12:15 pm

Ithier wrote:Hi,

I have a multi-threaded application which runs on a computer with a CPU with HyperThreading.
When I load two PNG at the same time, sometimes (perhaps 1 time out of 3), the loaded PNG is corrupted.
I have tried the same application with a computer which does not have HyperThreading and all is fine.
Here is an example of a corrupted image :
Image
and the original :
Image

It seems to be the gflLoadBitmapFromMemory function as I have extracted the buffer justs after the call and the data is already corrupted.
I use GFL 2.41 and Windows XP or Windows Server 2003 and the application is compiled with gcc 3.2.3.

Does someone has already encoutered this problem ?
Is GFL thread safe ?
Yes, the problem is only on PNG file??
Pierre.

User avatar
Ithier
Posts: 47
Joined: Fri Nov 19, 2004 10:50 am
Location: Paris, France
Contact:

Post by Ithier » Tue Aug 02, 2005 4:14 pm

Yes, at least it does not happen with jpeg files (I have not tested with other file formats).

User avatar
xnview
Author of XnView
Posts: 32052
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Post by xnview » Wed Aug 03, 2005 8:13 am

Ithier wrote:Yes, at least it does not happen with jpeg files (I have not tested with other file formats).
Could you send me a sample project, please?
Pierre.

User avatar
Ithier
Posts: 47
Joined: Fri Nov 19, 2004 10:50 am
Location: Paris, France
Contact:

Post by Ithier » Wed Aug 03, 2005 10:17 am

Here is some code to test the problem. I remember you that it happens only on a computer with hyper-threading or with multiple cpu.
It happens in debug and in release mode.

The code, reads a file in memory and then create two threads which load the file and save it in jpeg.
The problem does not happen all the time, I have run the program 5 times so it has generated 10 jpeg files. On a computer with 2 cpu, 6 files were corrupted. On a computer with one cpu but with hyperthreading 3 files were corrupted. On a single cpu computer, 0 file was corrupted.

Code: Select all

#include "libgfl.h"
#include <fstream>
#include <windows.h>

GFL_UINT8* pData = NULL;
int data_size = 0;

DWORD Thread (LPVOID Param)
{
	//Load PNG file from memory
	GFL_LOAD_PARAMS load_option;
	gflGetDefaultLoadParams( &load_option );
    GFL_FILE_INFORMATION    GflFI;
    GFL_BITMAP*     GflBitmap;
    gflLoadBitmapFromMemory(pData, data_size, &GflBitmap, &load_option, &GflFI);

	//Output filename (Id of the thread)
	char filename[32];
	sprintf(filename, "%d.jpg", GetCurrentThreadId());

	//Save the file
   	GFL_SAVE_PARAMS save_option;
    gflGetDefaultSaveParams(&save_option);
    save_option.FormatIndex = gflGetFormatIndexByName("jpeg");
    gflSaveBitmap(filename, GflBitmap, &save_option);

	return 0;
}


int main(int argc, char* argv[])
{
	gflLibraryInit();
	//Read PNG file in memory
    std::ifstream ficDoc (argv[1], std::ios::in|std::ios::binary|std::ios::ate);
	data_size = ficDoc.tellg();
    pData = new GFL_UINT8[data_size];
    ficDoc.seekg (0, std::ios::beg);
    ficDoc.read (reinterpret_cast<char*>(pData), data_size);
	ficDoc.close();
	
	for (int i=0; i<2; ++i)
	{
		DWORD dwThreadId;
		CreateThread (NULL,								// no security attribute 
					  0,								// default stack size 
					  (LPTHREAD_START_ROUTINE) Thread,	
					  NULL,								// thread parameter 
					  0,								// not suspended 
					  &dwThreadId);						// returns thread ID
	}
	Sleep (3000); //Wait for the threads to terminate
	return 0;
}
If you need I can send you the dsp file (for Visual C++ 6.0) but all the options are standard except C++ -> Code Generation -> Use runtime-library which is "Debug Multithread" or "Multithread".

Good luck :wink:

Ithier

User avatar
xnview
Author of XnView
Posts: 32052
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Post by xnview » Wed Aug 03, 2005 2:42 pm

Ithier wrote:Here is some code to test the problem. I remember you that it happens only on a computer with hyper-threading or with multiple cpu.
It happens in debug and in release mode.
Ok, i'll try to find a dual proc computer :-)
Pierre.

User avatar
MaierMan
Posts: 78
Joined: Wed Aug 04, 2004 8:32 pm
Contact:

hey Ithier

Post by MaierMan » Sat Aug 06, 2005 10:37 pm

Can you determine the "stage" when corruption occures.
(eg. by setting CS arround gflLoadBitmap and/or gflSaveBitmap).
I think that would give Pierre a better clue.

(I know about the problems of debugging MT-code on a single processor box :p)

Something like:

Code: Select all

#include <libgfl.h>
#include <fstream>
#include <iostream>
#include <windows.h>
#include <string>
#include <time.h>

class CondCS
{
    CRITICAL_SECTION s;
public:
    CondCS() { InitializeCriticalSection(&s); }
    ~CondCS() { DeleteCriticalSection(&s); }
    void lock(const BOOL x, const char *i, std::string name)
	{
		if (x)
		{
			EnterCriticalSection(&s);
			std::cerr << name << " - lock: " << i << std::endl;
		}
	}
	void unlock(const BOOL x, const char *i, std::string name)
	{
		if (x)
		{
			std::cerr << name << " - unlock: " << i << std::endl;
			LeaveCriticalSection(&s);
		}
	}
};
CondCS sec;

#define TOPROT 2 // [0-3[
#define PROT(x) (TOPROT & x)

class Thread;
DWORD ThreadFunc(Thread *thread);

static unsigned hrand(unsigned high)
{
	return (unsigned)((double)(rand() % RAND_MAX) / (double)RAND_MAX * (double)high);
}

class Thread
{
public:
	unsigned id;
	unsigned short prio;

	HANDLE hThread;
	DWORD dwThread;
	GFL_UINT8 *pData;
	GFL_UINT32 sSize;
	
	std::string name;
	
	Thread(unsigned i, GFL_UINT8 *d, GFL_UINT32 s) : id(i), hThread(NULL), dwThread(0), pData(d), sSize(s)
	{
		hThread = CreateThread(
			NULL,
			0,
			(LPTHREAD_START_ROUTINE)ThreadFunc,
			(LPVOID)this,
			CREATE_SUSPENDED,
			&dwThread
		);
		prio = hrand(3);
		SetThreadPriority(
			hThread,
			(prio == 2)
			? THREAD_PRIORITY_HIGHEST
			: (
				!prio
				? THREAD_PRIORITY_IDLE
				: THREAD_PRIORITY_NORMAL
			)
		);
		if (!hThread)
			std::cerr << "Failed to create!" << std::endl;

		char n[1024];
		sprintf(n, "Thread %.2d (%p) p%d", id, hThread, prio);

		name.assign(n);
	}
	~Thread() { if (hThread) CloseHandle(hThread); };
	void Execute() { if (hThread) ResumeThread(hThread); }
};

DWORD ThreadFunc(Thread *thread)
{
	DWORD err;
    GFL_LOAD_PARAMS load_option;
    gflGetDefaultLoadParams(&load_option);
    GFL_FILE_INFORMATION GflFI;
    GFL_BITMAP *GflBitmap;

    sec.lock(PROT(1), "load", thread->name);
    err = MAKELONG(gflLoadBitmapFromMemory(
		thread->pData,
		thread->sSize,
		&GflBitmap,
		&load_option,
		&GflFI
	), 0);
    sec.unlock(PROT(1), "load", thread->name);

	std::string filename(thread->name);
	filename.append(".jpg");

    GFL_SAVE_PARAMS save_option;
    gflGetDefaultSaveParams(&save_option);
    save_option.FormatIndex = gflGetFormatIndexByName("jpeg");

    sec.lock(PROT(2), "save", thread->name);
    err |= MAKELONG(0, gflSaveBitmap(
		(char*)filename.c_str(),
		GflBitmap,
		&save_option
	));
    sec.unlock(PROT(2), "save", thread->name);
return err;
}

#define THREADS 10
int main(int argc, char* argv[])
{
	if (argc != 2)
	{
		std::cout << "/me takes 1 arg" << std::endl;
		return 1;
	}
	gflLibraryInit();
 	srand((unsigned)time(NULL)); 

    std::ifstream ficDoc(
		argv[1],
		std::ios::in | std::ios::binary | std::ios::ate
	);
    GFL_UINT32 sSize = ficDoc.tellg();
    GFL_UINT8 *pData = new GFL_UINT8[sSize];

    ficDoc.seekg (0, std::ios::beg);
    ficDoc.read (reinterpret_cast<char*>(pData), sSize);
    ficDoc.close();

    HANDLE *pHandles = new HANDLE[THREADS];
	Thread **pThreads = new Thread*[THREADS];
	
	std::cout << "Protected stages: " << TOPROT << std::endl;

    for (unsigned i = 0; i < THREADS; ++i)
    {
        pThreads[i] = new Thread(i, pData, sSize);
		pHandles[i] = pThreads[i]->hThread;
    }
	for (unsigned i = 0; i < THREADS; ++i)
	{
		pThreads[i]->Execute();
	}
    WaitForMultipleObjects(
        THREADS,
        pHandles,
        TRUE,
        INFINITE
    );
    for (unsigned i = 0; i < THREADS; ++i)
    {
		DWORD ec;
        if (GetExitCodeThread(pHandles[i], &ec))
		{
			std::cout << pThreads[i]->name << ") - load: " << LOWORD(ec) << " - save: " << HIWORD(ec) << std::endl;
		}
		else
		{
			std::cout << pThreads[i]->name << "): ABANDONED" << std::endl;
		}
		delete pThreads[i];
    }    
    delete [] pData;
    delete [] pHandles;
	delete [] pThreads;

return 0;
}
But Im only single-core, so output is fine for all four TOPROTs :p

Guest

Post by Guest » Sun Aug 07, 2005 10:33 pm

Hi,

Thank you for your code and for your time, in fact I already knew where the problem is, as I explain in my first post, because just after the loading of the PNG file I have written the GFL_BITMAP buffer in a file. I have read the file (like a bitmap file without headers) and the data was already corrupted. So I am quite sure that the problem come from the gflLoadBitmapFromMemory function.



Ithier

User avatar
Ithier
Posts: 47
Joined: Fri Nov 19, 2004 10:50 am
Location: Paris, France
Contact:

Post by Ithier » Mon Sep 05, 2005 3:00 pm

Hi Pierre,

Did you have time to look at my problem ?

If you need some help to test special version of SDK (with debugging output for example), I can do that on my dual-proc computer. Do not hesitate to contact me, I would be very glad to help you.

Best Regards

Ithier

User avatar
xnview
Author of XnView
Posts: 32052
Joined: Mon Oct 13, 2003 7:31 am
Location: France
Contact:

Post by xnview » Sun Sep 18, 2005 1:55 pm

Ithier wrote:Hi Pierre,

Did you have time to look at my problem ?

If you need some help to test special version of SDK (with debugging output for example), I can do that on my dual-proc computer. Do not hesitate to contact me, I would be very glad to help you.
Sorry currently not the time to check this problem...
Pierre.

Post Reply