LCF-AT Posted October 6, 2018 Share Posted October 6, 2018 Hi guys, I have again a small question about piping a CLI output to my buffer.So I just wanna start a CLI app with paramters and reading the output (content I also see in CMD window) so long till its finished.So the only problem I have so far is how to wait till its finished.Lets say the CLI app needs a while (a minute etc).My question is what function/s I need to use for that?I see I cant use WaitForSingleObject with INFINITE paramter so this dosent work because the CLI app dosent exit itself so long I didnt had reading the bytes.If I just read the bytes using PeekNamedPipe / ReadFile function then I dont get the whole content = it quits to early and the CLI app is still working. My goal is it to read the bytes on fly and checking whether the pipe / process is still working / activ or not. Below some example code: _CreatePipe PROC Local RetVal:DWORD invoke RtlZeroMemory,addr SecuAttr,sizeof SecuAttr invoke RtlZeroMemory,addr StartInfo,sizeof StartInfo invoke RtlZeroMemory,addr PI,sizeof PI mov RetVal,0 mov SecuAttr.nLength,sizeof SECURITY_ATTRIBUTES mov SecuAttr.bInheritHandle,TRUE invoke CreatePipe,addr hRead,addr hWrite,addr SecuAttr,NULL .if eax == FALSE invoke MessageBox,0,chr$("CreatePipe failed!"),chr$("Problem!"),MB_ICONWARNING mov eax,FALSE ret .endif invoke SetHandleInformation, hRead, HANDLE_FLAG_INHERIT, 0 .if eax == FALSE invoke CloseHandle,hRead invoke CloseHandle,hWrite invoke MessageBox,0,chr$("SetHandleInformation failed!"),chr$("Problem!"),MB_ICONWARNING mov eax,FALSE ret .endif mov RetVal,eax mov StartInfo.cb,sizeof StartInfo invoke GetStartupInfo,addr StartInfo xor eax,eax mov StartInfo.lpReserved,eax mov eax,hWrite mov StartInfo.hStdOutput,eax mov StartInfo.hStdError,eax mov StartInfo.dwFlags, STARTF_USESTDHANDLES mov StartInfo.wShowWindow,SW_SHOW mov eax,RetVal ret _CreatePipe endp ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; .elseif ax == IDB_TEST invoke VirtualAlloc,0,50000h,MEM_COMMIT,PAGE_EXECUTE_READWRITE mov esi, eax push esi invoke _CreatePipe .if eax != FALSE lea edi, chr$("cmd.exe /c C:\tool.exe -t -r") invoke CreateProcess, NULL, edi, NULL,NULL,TRUE,CREATE_NO_WINDOW,NULL,NULL, addr StartInfo, addr PI .if eax != FALSE @@: invoke Sleep,300 invoke PeekNamedPipe,hRead,offset PIPEBUFFER,sizeof PIPEBUFFER,offset PIPEREADBYTES,addr AVAL,NULL .if eax != FALSE .if AVAL != 0h invoke ReadFile,hRead,esi,PIPEREADBYTES,addr BYTESREAD,0 .if eax != FALSE add esi, BYTESREAD jmp @B .else invoke MessageBox,0,chr$("ReadFile failed!"),chr$("Problem!"),MB_ICONWARNING jmp @F .endif .else pop esi push esi invoke MessageBox,0,esi,chr$("Info!"),MB_ICONINFORMATION .endif .else invoke MessageBox,0,chr$("PeekNamedPipe failed!"),chr$("Problem!"),MB_ICONWARNING .endif @@: invoke CloseHandle, [PI.hThread] invoke CloseHandle, [PI.hProcess] invoke CloseHandle, hRead invoke CloseHandle, hWrite .else invoke CloseHandle, hRead invoke CloseHandle, hWrite .endif .endif pop esi invoke VirtualFree,esi,0,MEM_RELEASE How to handle the wait operation correctly and to read all bytes / content so long the CLI app is activ / working?Maybe anyone could help a little to fix that problem. Thank you Link to comment
evlncrn8 Posted October 6, 2018 Share Posted October 6, 2018 https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-getexitcodeprocess that at least lets you check 100% if the process has exited, the rest is just polling, to check if there's input and so on also, mov StartInfo.hStdOutput,eax mov StartInfo.hStdError,eax was that intentional ? theres stdout, stdin, and stderr.. also you didnt set hread from what i saw but i didnt go through the code in detail 2 Link to comment
atom0s Posted October 7, 2018 Share Posted October 7, 2018 When you use CreateProcess the PROCESS_INFORMATION class gives back the main process handle and the main thread handle. You can use those to pull the exit codes of either. - GetExitCodeProcess and GetExitCodeThread Keep in mind if the application is multi-threaded, it is not guaranteed that the main thread handle will remain alive for the entire duration of the app. 1 Link to comment
LCF-AT Posted October 7, 2018 Author Share Posted October 7, 2018 Hi guys, thanks for your answers.Ok I am just checking whether the process is still active or not using GetExitCodeProcess function.Ok,so this seems to work so far and I can read while pipe content.Now I found another small problem.So if I start the CLI app and it does run for a longer while then I wanna also stop the process similar like using CRTL+C key press in CMD window.First I just tried to use TermniateProcess function with the hprocess ID of CLI app but this dosent work and CLI app keeps working in background / taskmanager and I need to wait till its finished by itself.Now I checked again the internet and found something about using GenerateConsoleCtrlEvent function to send CTRL_C_EVENT.I also found some example code like this... https://codetitans.pl/blog/post/sending-ctrl-c-signal-to-another-application-on-windows [DllImport("kernel32.dll", SetLastError = true)] static extern bool AttachConsole(uint dwProcessId); [DllImport("kernel32.dll", SetLastError = true, ExactSpelling = true)] static extern bool FreeConsole(); [DllImport("kernel32.dll")] static extern bool SetConsoleCtrlHandler(ConsoleCtrlDelegate handler, bool add); // Delegate type to be used as the Handler Routine for SCCH delegate Boolean ConsoleCtrlDelegate(CtrlTypes type); // Enumerated type for the control messages sent to the handler routine enum CtrlTypes : uint { CTRL_C_EVENT = 0, CTRL_BREAK_EVENT, CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT = 5, CTRL_SHUTDOWN_EVENT } [DllImport("kernel32.dll")] [return: MarshalAs(UnmanagedType.Bool)] private static extern bool GenerateConsoleCtrlEvent(CtrlTypes dwCtrlEvent, uint dwProcessGroupId); public static void StopProgram(uint pid) { // It's impossible to be attached to 2 consoles at the same time, // so release the current one. FreeConsole(); // This does not require the console window to be visible. if (AttachConsole(pid)) { // Disable Ctrl-C handling for our program SetConsoleCtrlHandler(null, true); GenerateConsoleCtrlEvent(CtrlTypes.CTRL_C_EVENT, 0); // Must wait here. If we don't and re-enable Ctrl-C // handling below too fast, we might terminate ourselves. Thread.Sleep(2000); FreeConsole(); // Re-enable Ctrl-C handling or any subsequently started // programs will inherit the disabled state. SetConsoleCtrlHandler(null, false); } } ...I tried this like this for testing... invoke CreateProcess, NULL,addr commandline,NULL,NULL,TRUE,CREATE_NO_WINDOW,NULL,NULL,addr StartInfo,addr PI .if eax != FALSE invoke Sleep,2000 invoke AttachConsole,PI.dwProcessId .if eax != FALSE invoke SetConsoleCtrlHandler,NULL,TRUE .if eax != FALSE invoke GenerateConsoleCtrlEvent,CTRL_C_EVENT,NULL .if eax != FALSE invoke Sleep,2000 invoke FreeConsole invoke SetConsoleCtrlHandler,NULL,FALSE .endif .endif .endif ...so it seems to work and the CLI app gets closed but I am not so sure whether this code example is good idea to use it on that way etc.In Olly it does crash after I did traced over this code after (not in outside of Olly).So is it needed to use FreeConsole & SetConsoleCtrlHandler at the end if the CLI is no more present?I dont think so or?I am using already cmd.exe /c paramter in CreateProcess function. Other issue I see is that I dont use the flag CREATE_NEW_PROCESS_GROUP (in the link above they told to use it but then CTRL_C_EVENT isnt working).Maybe you have some another little clue how I should do it correctly to send CTRL_C_EVENT & wait without to get any trouble later and I also dont wanna exit the CLI process rough you know. greetz Link to comment
evlncrn8 Posted October 7, 2018 Share Posted October 7, 2018 https://docs.microsoft.com/en-us/windows/console/generateconsolectrlevent should do it invoke FreeConsole invoke SetConsoleCtrlHandler,NULL,FALSE it might well crash there, considering you still have the handler registered so maybe flip that code around.. setevent then freeconsole.. see what happens .. 1 Link to comment
LCF-AT Posted October 8, 2018 Author Share Posted October 8, 2018 Hi again, ok I used it so now with freeconsole & Setconsolehandler at the end.So if I dont do it then it dosent work anymore after first try.My thread code look so now... local WASATTACHED:DWORD mov WASATTACHED, 0 @@: .if STOP == 1 ; <--- stop CLI invoke GetExitCodeProcess,[PI.hProcess],addr ExitCode .if ExitCode == STILL_ACTIVE && WASATTACHED == NULL invoke AttachConsole,PI.dwProcessId mov WASATTACHED, 1 invoke SetConsoleCtrlHandler,NULL,TRUE invoke GenerateConsoleCtrlEvent,CTRL_C_EVENT,NULL .endif .endif invoke RtlZeroMemory,addr PIPEBUFFER,sizeof PIPEBUFFER invoke Sleep,30 invoke PeekNamedPipe,hRead,offset PIPEBUFFER,sizeof PIPEBUFFER-10h,offset PIPEREADBYTES,addr AVAL,NULL .if eax != FALSE .if AVAL != 0h invoke RtlZeroMemory,addr PIPEBUFFER,sizeof PIPEBUFFER invoke ReadFile,hRead,addr PIPEBUFFER,PIPEREADBYTES,addr BYTESREAD,0 .if eax != FALSE invoke SendMessage,RICHEDITHANDLE,EM_REPLACESEL,FALSE,addr PIPEBUFFER invoke RichSetPos, RICHEDITHANDLE, -1h jmp @B .else invoke MessageBox,0,chr$("ReadFile failed!"),chr$("Problem!"),MB_ICONWARNING jmp @F .endif .else invoke GetExitCodeProcess,[PI.hProcess],addr ExitCode .if ExitCode == STILL_ACTIVE jmp @B .endif .endif .else invoke MessageBox,0,chr$("PeekNamedPipe failed!"),chr$("Problem!"),MB_ICONWARNING .endif @@: invoke CloseHandle, hRead invoke CloseHandle, hWrite invoke CloseHandle, [PI.hThread] invoke CloseHandle, [PI.hProcess] .if STOP == 1 && WASATTACHED == 1 invoke FreeConsole invoke SetConsoleCtrlHandler,NULL,FALSE .endif mov STOP, 0 xor eax,eax Ret ...seems to work fine so far. Thanks again for the help. greetz Link to comment
LCF-AT Posted March 30, 2021 Author Share Posted March 30, 2021 Hi again, so I have another addon question about starting console windows,So how to manage to start more than one console instances?Lets say I wanna run a CLI app few times via CreateProcess function and I wanna also check whether they are still running and I wanna also force to exit them if I want by sending CTRL_C_EVENT (same code as above).Is that doable?In example above I have to use that functions... CreateProcess ---------------- AttachConsole SetConsoleCtrlHandler GenerateConsoleCtrlEvent GetExitCodeProcess CloseHandle FreeConsole SetConsoleCtrlHandler ...which are working for one instance without problems but now I wanna start more than one instances at same time or one by one you know.So above in code example I saw the info... // It's impossible to be attached to 2 consoles at the same time, // so release the current one. ...which tells me that there is any limit and I only can attach only one CLI.Is that right? CreateProcess ---------------- invoke AttachConsole,PI.dwProcessId <-- PID invoke SetConsoleCtrlHandler,NULL,TRUE <-- No PID invoke GenerateConsoleCtrlEvent,CTRL_C_EVENT,NULL <-- No PID invoke GetExitCodeProcess,[PI.hProcess],addr ExitCode <-- PID invoke CloseHandle, [PI.hThread] <-- PID invoke CloseHandle, [PI.hProcess] <-- PID invoke FreeConsole <-- No PID invoke SetConsoleCtrlHandler,NULL,FALSE <-- No PID Above you can see the functions I can use a PID with and some of them I can not use a specific PID.When I start more than one CLI apps at same time then I can not control all at the samt time?How to manage that problem to get access to all running CLI instances I did started to send Events into the right CLI window? Or have I just to set a lock on that attach / FreeConsole code to handle each PID seperated one by one?Some kind of "right of way" like in traffic etc?Maybe you can tell me or show some code lines how to manage that better to run more than one instances at same time.Thank you. greetz Link to comment
fearless Posted April 1, 2021 Share Posted April 1, 2021 https://www.codeproject.com/articles/13368/multiple-consoles-for-a-single-application https://codereview.stackexchange.com/questions/195157/using-multiple-console-windows-for-output 1 Link to comment
LCF-AT Posted April 1, 2021 Author Share Posted April 1, 2021 Hi fearless, thanks for this info but I'am not sure about that whether I could use it (as I think) and its also some C stuff (no clue about that) you know.So if I read it right then it wants to create some child processes for another CLI window/s but in my case I want to run diffrent instances seperated from each other.Otherwise lets say I do start 4 CLI windows (anyhow with that C code) then it will build child processes from first started CLI right?But what is when I have to exit the first CLI process?Then child processes get closed too or not? greetz Link to comment
LCF-AT Posted May 4, 2021 Author Share Posted May 4, 2021 Hi guys, so I'am still fighting to find a method to run/start multible CMD apps (starting works) and to send a CTRL_C_EVENT into every single started CMD app if I want to do that. Maybe I start with a question about that CTRL_C_EVENT command.Normaly I am using it to tell the CMD window to stop the running CMD app.My question in this case is whether there is any diffrent between using the CTRL_C_EVENT and/or using a EXIT call (maybe taskmanager / kill process of CMD/app)?So I think to send a CTRL_C_EVENT is a more better solution than using a hard process kill but is there a diffrent? 1,) Normal case: I'am setup CreateProcess function paramters with the commandline paramters for the cmd and the app/tool (ffmpeg etc) and the path. cmd.exe /c ffmpeg -hide_banner etc Using cmd.exe with /c command... /C Carries out the command specified by string and then terminates Lets say I am doing that a few times to run diffrent ffmpeg instances.So far no problem yet but now I wanna close diffrent still running ffmpeg processes in a soft and elegant method by sending a CTRL_C_EVENT command but for this I have to attach the console windows of each running process.I was trying to do that like I told before using that functions below in a own thread for every running ffmpeg instance.... invoke AttachConsole,PI.dwProcessId <-- PID invoke SetConsoleCtrlHandler,NULL,TRUE <-- No PID invoke GenerateConsoleCtrlEvent,CTRL_C_EVENT,NULL <-- No PID invoke GetExitCodeProcess,[PI.hProcess],addr ExitCode <-- PID invoke CloseHandle, [PI.hThread] <-- PID invoke CloseHandle, [PI.hProcess] <-- PID invoke FreeConsole <-- No PID invoke SetConsoleCtrlHandler,NULL,FALSE <-- No PID ...as you can see there are no PIDs I can set for diffrent functions what is a problem.The functions can be overlap by the threads are running and using them to etc you know.My question are now... - What should I note if I use that functions above to exit a single console window if I have running more than one console window? - Are there other methods?Maybe starting console apps without CMD.exe tool and just calling exit process (hard kill etc)? Byte the way, I did noticed when I am using the code functions above that something is changed is no more in original state.I see that the "l" key on keyboard dosent pint the "l" anymore in any tools like notepad etc so instead of that the "l" key does pop up a CMD window!=?Not sure yet what it is to trigger that issue. Anyway, so if you got some more ideas how to handle that problem to exit diffrent running console window processes in a smart / soft way then just tell me. greetz Link to comment
deepzero Posted May 4, 2021 Share Posted May 4, 2021 There is a difference, as programs can handle cntrl+c as they wish (including performing cleanup before terminating, or simply ignoring it). But I doubt your cmd-ffmpeg combo is doing that. If you just want to kick off a conversion and kill it if it takes too long, then the simplest solution imo is to ignore consoles and cmd and whatnot, and just use CreateProcess to launch ffmpeg with arguments directly and TerminateProcess it if it takes too long. if you really need to write to its stdin or log its stdout, you can do that (check CreateProcess documentation). As so often, though, I am not sure i 100% understand your motives and what you try to accomplish. Is there any reason you are launching ffmpeg with cmd /C? What is your goal? -- after reading the two year old thread above i am not really any wiser. Launching a subprocess and accessing its stdout is a common task, and msdn provides best practice for it: https://docs.microsoft.com/de-de/windows/win32/procthread/creating-a-child-process-with-redirected-input-and-output 1 Link to comment
LCF-AT Posted May 4, 2021 Author Share Posted May 4, 2021 Tach deep, why do you doubt that deep?Above you can see the functions I use to send CTRL+C command.So it takes a little while til the console process get closed (few seconds) but I think it does it with that cleanup.Ok, so you mean I shouldnt use CMD.exe itself anymore and just using TerminateProcess.I also thought about that but could be maybe to rough specially if something gets written on disk.Or what do you think? So I have these goals. - starting multible commandline tools instances from one main process (GUI) = Working using CreateProcess function - A) Getting control of all console processes I did started to read informations / pipes (of all created console instances at same time) and to send infos like CTRL+C etc commands into any of them etc. - B.) Not need to read pipes to get informations but need to send CTRL+C command into to exit the running console softly / clean.Otherwise I have to use the woodhammer method (kill process) At the moment I'am just trying out method B more or less so I dont need to read infos from pipe yet.Just need to find a good method to exit running console instances nice and clean for 100%.I dont wanna get some running left overs in background or other issues. greetz Link to comment
deepzero Posted May 4, 2021 Share Posted May 4, 2021 So it occurred to me you are probably using it to dump some stream. And in that case the cntrl+c thing is indeed legitimate. A process can only be attached to one console at a time, if you have several threads try to cntrl+c different windows, they will collide and weird things happen. So you need a global lock / mutex so that only one thread at a time does the AttachConsole dance. If you pm me your streamdumping command or whatever it is you do with ffmpeg, i can try to make it work next weekend. 1 Link to comment
LCF-AT Posted May 4, 2021 Author Share Posted May 4, 2021 Hi again, so you mean if I wanna use that attach - freeconsole functions then I should execute that function loop ONLY ONCE at this time to prevent that overlapping I told beofre yes? Example: .data THREDEXERUNS dd FALSE Thread Proc createprocess stuff ...... .while THREDEXERUNS == TRUE invoke Sleep,500 <--- others have to wait a half second in loop .endw mov THREDEXERUNS, TRUE <--- Set Lock invoke AttachConsole,PI.dwProcessId <-- PID invoke SetConsoleCtrlHandler,NULL,TRUE <-- No PID invoke GenerateConsoleCtrlEvent,CTRL_C_EVENT,NULL <-- No PID invoke GetExitCodeProcess,[PI.hProcess],addr ExitCode <-- PID invoke CloseHandle, [PI.hThread] <-- PID invoke CloseHandle, [PI.hProcess] <-- PID invoke FreeConsole <-- No PID invoke SetConsoleCtrlHandler,NULL,FALSE <-- No PID mov THREDEXERUNS, FALSE <--- Set Unlock xor eax,eax ret ; <--- Thread exit Like this?Just need to secure that last finish functions for the console itself to attch / send CTRL event / Setconsole as last function and then I give it free access again for the next waiting one right?I do not use CRIT_SECTION stuff here just wanna keep it simple here.But you mean it so right or similar right? Question: So if I attach the console with PID X the I think it also means that all next execution functions like SetConsoleCrtlEvent is used for that attached X console app right?Will see whether it works and whether the "l" key dosent pop up a running visible CMD window to front anymore instead of printing the "l" in notepad and other apps. PS: Dont have written much code to dump streams yet because just wanna run ffmpeg with paramters and create a controled exit you know.So at the moment it does work so far during my tests to run X instnaces at same time and to exit them as I wanted.Maybe just a random success because the attach functions didnt overlap by random of course.I will try it now with the lock. greetz Link to comment
deepzero Posted May 5, 2021 Share Posted May 5, 2021 Quote But you mean it so right or similar right? yes. however, you should use a critical-section or something similar, because otherwise there is always a small chance you get unlucky and end up with a collision again. But the code you posted should as-is make this very very rare, yes. Test it and see if it works. 1 Link to comment
LCF-AT Posted May 13, 2021 Author Share Posted May 13, 2021 Hi again, I have some another important questions to prevent error / exceptions by using THREADs and would like to ask how to do it correctly. 1.) How to work with threads I want to access "one by one"? So normaly I was always using some simple thread lock commands to prevent accessing same thread at same time as you can see in my pre post using THREDEXERUNS into a while / endw loop.In the last days I had to fight with diffrent exceptions "C0000374" and "C0000005" and my app just did exit.Its really hard to find out the reason of the C0000374 (heap corruption issue) exception because it dont get logged where it happens = PITA.Somehow I thought it should be happens inside of my running threads.As I said I was just using my own simple lock command trying to lock my thread if any did already access it but I get some randomly exceptions earlier or sooner = Bad.Now I did remember what "deep" said that I should better use critical-section commands for that and I did it now like MSDN said in that example... https://docs.microsoft.com/en-us/windows/win32/sync/using-critical-section-objects // Global variable CRITICAL_SECTION CriticalSection; int main( void ) { ... // Initialize the critical section one time only. if (!InitializeCriticalSectionAndSpinCount(&CriticalSection, 0x00000400) ) return; ... // Release resources used by the critical section object. DeleteCriticalSection(&CriticalSection); } DWORD WINAPI ThreadProc( LPVOID lpParameter ) { ... // Request ownership of the critical section. EnterCriticalSection(&CriticalSection); // Access the shared resource. // Release ownership of the critical section. LeaveCriticalSection(&CriticalSection); ... return 1; } ...and of wonder it seems to work and I don't get any exceptions anymore after testing it for few hours.Thats the good news. SUB Question about those "Critical Section" functions: How to use a RTL_CRITICAL_SECTION object?In the example of MSDN above you can see the code does Init/Spin one object and using that object in one thread.My question here is what is if I run diffrent and also other threads I wanna lock too, so can I then use the same object I did init OR have I to init another objects?Lets say I wanna run 3 diffrently threads should I then init/spin 3 times a object section OR can I just init ONE section object and using them for EVERY X (100+) threads?How to make it right in that case?Just important to know that for me.One Init/Spin section object for everything or one Init/Spin section object for one thread?Another small addone question I have is whether I can use that secction object functions "ONLY INSIDE THREADS" or whether I can use them also for "NORMAL PROCS"?Before I call any threads I have to prepair some allocated code memory blocks I have to send with the thread.Somehow I must also LOCK that prepair PROC and here I am using again my simple LOCK method that why I am asking for whether I can use the object section function also for normal procs or not?If NOT....how to lock such PROCs then? Summury: Use crit functions inside threads = OK.What to do use on normal PROCs if my simple LOCK is too unsafe? So now I was using that section object in my thread and didnt got any trouble so far = good.But with that method I can run one and same thread one by one what is good for thread which can be bypassed quickly (no wait operations inside etc) what brings me to my next question.... 2.) How to work with same thread I want to access MULTIBLE TIMES? How to manage that situation? Example: I made a downloader routine which is very large and works fine for one by one access but when I wanna download a large file X MegeBytes then the actually process who did access this routine keeps running inside of this routine till its finished and all other processes who wanna access the same routine have to wait so long to prevent possible errors/exceptions etc.That pretty tricky.I also was trying to use just LOCAL variables but somehow I also get exceptions +/- if more than one process is accessing that routine at same time. Master question: What can I do and what can I NOT DO inside of the thread which get accessed multible times to prevent exceptions?Are there any functions I should better NOT USE?In my thread I'am using some heap functions GetProcessHeap / HeapAlloc / HeapFree and other VirtualAlloc / GlobalAlloc & Free functions which could maybe by the reason for exceptions.I do remember that fearless told something about that a longer while ago when I had problems with that C0000374 exceptions.So what to do in that case?Alloc Space / Free Space....maybe they can be overlapping by running MThreads at same time you know.How to prevent that? All in all my goal is it to find out (or somebody does tell me) how to deal with thread (single/multible) correctly for 100% just to prevent error exceptions which brings my apps calling the exit routine you know.Maybe just only a newbie question for you but I still found no good working 100% solution for that.Any kind of new infos and hints how to manage that better are very welcome.Thank you. greetz Link to comment
deepzero Posted May 13, 2021 Share Posted May 13, 2021 All the heap-functions you mentioned should be safe to use from different threads. Generally the Win32 api is pretty threadsafe. GDI handles often cannot be shared across threads, but it doesnt look like you need that. Generally, your threads shouldnt do anything that changes a shared resource. For example, freely modify global variables. However, this very much depends on your specific code. I am guessing your heap corruption comes from either a) double free - two threads free the same pointer b) use after free - thread A uses a pointer that was already freed by another thread c) classic heap-overflow where you write past the allocated memory. Quote My question here is what is if I run diffrent and also other threads I wanna lock too, so can I then use the same object I did init OR have I to init another objects? All your threads have to use the same CRITICAL_SECTION variable! It is this variable that synchronizes them! Imagine it like a ball in a AA meeting - only the one who has the ball (=critical section) can talk. As for the init function, only ONE thread must call it. Ideally you do this before you start your M subthreads. Quote Example: I made a downloader routine which is very large and works fine for one by one access but when I wanna download a large file X MegeBytes then the actually process who did access this routine keeps running inside of this routine till its finished and all other processes who wanna access the same routine have to wait so long to prevent possible errors/exceptions etc.That pretty tricky.I also was trying to use just LOCAL variables but somehow I also get exceptions +/- if more than one process is accessing that routine at same time. As you said, the function must be threadsafe, which can be a little tricky. Having it only access local variables is a good idea. You can send me the function to look at if you want. https://en.wikipedia.org/wiki/Thread_safety Quote Before I call any threads I have to prepair some allocated code memory blocks I have to send with the thread.Somehow I must also LOCK that prepair PROC and here I am using again my simple LOCK method that why I am asking for whether I can use the object section function also for normal procs or not?If NOT....how to lock such PROCs then? I dont understand... Critical sections can only be used to synchronize threads in one process. 1 Link to comment
LCF-AT Posted May 14, 2021 Author Share Posted May 14, 2021 Hi deep, thanks for your answer so far.So just to make it clear.....you said I have to use only ONE C-Section for ALL of my threads?Anyway what kind of threads?All using same C-section = ONE global C-Section-Object for everything?At the moment I am using 2 C-sections for 2 diffrent threads but this is then Wrong right?But I am using 2 diffrent threads who doing diffrent things so if I use one same C-Section for both threads then they have to wait all right?Hhhmm.Thats not what I want.So they should be work seperated from each other.Lets say one thread does hang because of any reason then the other threads dosent also work anymore for itself.Not sure whether I do understand it wrong or not. I also got an exception in app inside of Enter or Leave C-section function and I dont know why.My exception log tells me that... Exception: C0000005 (STATUS_ACCESS_VIOLATION) regEax 00000000 regEcx FFFFFFFF regEdx FFFFFFFA regEbx FFFFFFFA regEsp 010BFC18 regEbp 010BFCA8 regEsi 0094F230 regEdi 0094F220 regEip 7716F533 Exception Address $ ==> 7716F533 FF40 14 INC DWORD PTR DS:[EAX+14] <--- exception eax = 0 $+3 7716F536 8B1E MOV EBX,DWORD PTR DS:[ESI] $+5 7716F538 895C24 50 MOV DWORD PTR SS:[ESP+50],EBX ; ntdll.77197D50 $+9 7716F53C 8D6424 00 LEA ESP,DWORD PTR SS:[ESP] $+D 7716F540 E8 AB46FEFF CALL 77153BF0 ; ntdll.RtlGetCurrentServiceSessionId $+12 7716F545 85C0 TEST EAX,EAX API address and name at or before Exception: 7716F490 RtlDllShutdownInProgress Stack ----------------- 00000000 | 010BFC18 | 0094F220 bones.exe <--- pointer to Crit section 00000004 | 010BFC1C | 0094F224 bones.exe 00000008 | 010BFC20 | FFFFFFFA 0000000C | 010BFC24 | 00000000 00000010 | 010BFC28 | 00000000 00000014 | 010BFC2C | 00000000 00000018 | 010BFC30 | FFFFFFFF ...not sure why it does reach this code and pops a exception.Somehow it didnt work today = AVio.I will try just using one C-Section for everything.One more question about the TryEnterCriticalSection function.Should I use this before calling EnterC function to make it more safe?Just asking etc.Do you have any short example WITH using tryC function? What about not thread routines which get called from threads.....how to lock them to get called one by one to prevent overlapping issues etc?I mean before I call CreateThread I have to prepair a own struct to fill datas and allocated memory blocks into to send that struct with the CreateThread function so that the thread has all infos to work with.This prepair routine I call just normaly not as thread and just LOCK then via while / endw at the top (simple lock).My threads are running in loop mode and if they finished they send a message to WM_COMMAND to prepair the next thread you know. I test again now with one C-section to see whether it works. greetz Link to comment
deepzero Posted May 14, 2021 Share Posted May 14, 2021 Quote Should I use this before calling EnterC function to make it more safe?Just asking etc.Do you have any short example WITH using tryC function? No. EnterCriticalSection takes the lock, or blocks/freezes until it can take the lock. Try.. takes the lock, or returns false if it cannot take it. You have to call only one function (otherwise you might take the lock twice). You probably want EnterCriticalSection. One critical section protects all the resources used in that "critical sections" of your code. If you have two different, unrelated "critical sections" (e.g. two different functions that only one thread my call at a time), then you can and should use two critical sections. Quote What about not thread routines which get called from threads... What do you mean by "not thread routines"? As I understand the win32api, you should be able to call GetProcessHeap+HeapAlloc in one thread, and then GetProcessHeap+HeapFree in another, yes. No locking is required from your side. Hard to tell what might cause your exception, it could be that your EnterCritSection/LeaveCritSection calls go out of sync? Ie you call LeaveCrit.. too often? It could be related to how you terminate threads. Do you use TerminateThread at all? (you should not). 1 Link to comment
LCF-AT Posted May 14, 2021 Author Share Posted May 14, 2021 Hi deep, ok, just using Enter & Leave.Yes I have 2 threads running which dont have to do with each other = I was using 2 C-Objects one for every thread.Not sure whether it makes sense to use more than one C-Sections.....so if I see it right then the EnterC function does read the thread ID and does checking it whether its already locked or not.......same happens in my thread 2 too but this thread has a other ID......so I think in this case its really just ok to use one C-Section or?At the moment I am using just one now as you said before. As I said, I have 2 threads running and they just restart itself if they finished by sending a sendmessage with ID to Wnd procedere and there gets the new thread prepaired and started again (loop mode).Somehow I have also to LOCK the not thread procs to prevent some overlap issues you know.But here I can not use EnterC functions which only working with threads.So how to LOCK protect normal procs? On the other hand its still tricky to handle all those thread stuff for Multible running thread using NO EnterC / LeaveC functions.Somehow like a download accelerator who is working at the same time and not one by one you know.In this case you have to run the same thread X times 4 / 8 / 16 and higher etc.In this case you have to write a thread code which is really safe or 100% = HOW?It starts already with the alloc / free stuff.Do you know some advanced thread code examples?Just wanna lern a little bit to really just prevent errors / exceptions in my stuff you know. greetz Link to comment
kao Posted May 14, 2021 Share Posted May 14, 2021 Hi LCF-AT, I apologize but you're mixing up terms and until you get the basics sorted out, both your questions and the answers will be confusing. I'll do my best to clarify some things, but multithreading is a huge subject and would probably take 5-6 lectures in Computer Science just to cover the basics. If you want more details, there's always google and programming books. Or just ask... 1) What is a thread. One explanation that I found is this: "Thread is an execution unit that consists of its own program counter, a stack, and a set of registers where the program counter mainly keeps track of which instruction to execute next, a set of registers mainly hold its current working variables, and a stack mainly contains the history of execution". Think about it for a bit. Each thread has its own context. It means each thread has its own set of register values (EAX, ESP, EBP, EIP, etc..). When Windows kernel switches between threads, it magically sets all registers to correct values, so you don't have to worry about them. But all process memory (think "global variables") is shared between all threads. All file/GDI/network handles are shared between all threads. These things are called "shared resources". And it's up to you to make sure that your threads play nice with those resources. 2) There is no difference between "code inside thread" or "normal proc". When you start a process in Windows, there is always one thread, called main thread. All the code is executed in the context of that one thread. When you create another thread, it will run some code. It can be some procedure. But if you passed address of PE entrypoint to CreateThread, it would run the exact same code as the main thread, only in a different thread. So - code is a code, no matter which thread executes it. 3) You can't run the same thread X times. You can run the same code X times - but in different threads. Similar to that, a thread cannot restart itself. You can have a jump to the beginning of code creating a big loop, or you can create a new thread running the same code. But no restarts.. 4) What's the big deal with multi-threading. When there's one thread, everything is simple. Global variables, local variables, it all works the same. But when there are more threads, some conflicts begin: When one thread modifies a global variable, and other thread reads the same variable, it gets the new (probably unexpected) value. After one thread calls CreateFile with write access, other thread will not be able to open the same file for write - it will get access denied. When one thread calls SetFilePointer on a file handle, and other thread calls WriteFile on the same file handle, unexpected data will be written to an unexpected place. When one thread calls FreeMemory and other thread is still writing to that memory block, you'll get access violation. The best way to make sure that there are no issues, is to avoid global variables, and use local variables instead. Local variables go into stack, and - as mentioned in #1 - each thread has it's own ESP/EBP. But sometimes it's hard to avoid using global variables. Or, in case of files, you just can't open one file multiple times. In those cases, you need make sure only one thread is accessing the resource at any given time. 4) What a critical section does. Critical section guards access to one shared resource (see #1). Maybe it's a file, into which X threads need to write some data. Maybe it's some global variable containing counter that all threads need to update. Whatever it is, the general rule is - one resource = one critical section object. Another resource = another critical section object. It is NOT one thread = one critical section object. Think of a critical section as a reservations book for a shared car. For each car, you have a separate book. If you want to take the car today, you write your name in the book. Once your name is in the book, everyone else can see it and know you are the one taking the car. Later you come, take the car keys and drive somewhere. You come back, you put the keys back and you erase your reservation from the book. Then next person can come, reserve a car, take the keys and go somewhere. Simple system that works fine. Only sometimes, some assh*le comes and takes the car keys without looking at the book. That's what happens when you don't call EnterCriticalSection from your thread. Also sometimes someone forgets to give the car keys back. Then nobody can take a car. That's what happens when you don't call LeaveCriticalSection. 5) How to use critical section. You have some shared resource to protect? You call InitializeCriticalSection. Usually this is done from a main thread during initialization. You want to access shared resource from your thread, you call EnterCriticalSection. All other threads that call EnterCriticalSection with that critical section object will wait until you're done - Windows kernel takes care of that. You're done with the shared resource, you call LeaveCriticalSection. When cleaning up, you call DeleteCriticalSection. Again, usually it's done from main thread. As mentioned above, as soon as you call EnterCriticalSection for some resource, all other threads will not be able to access that resource. They will be stuck in the "waiting" state. Therefore, you should make sure only minimum necessary code is between EnterCriticalSection and LeaveCriticalSection. Some pseudo-example: // you have 1 global variable: // myFileSection - critical section that guards access to the file void ThisIsADownloadThread() { WriteConsole("writing downloaded block to file",...); EnterCriticalSection(myFileSection, ....); hFile = CreateFile("somefile.data", ....); SetFilePointer(hFile, myOffset, FILE_BEGIN); WriteFile(hFile, myData, myDataSize, ...); CloseHandle(hFile); LeaveCriticalSection(myFileSection, ...); WriteConsole("finished writing data to file",...); } If you want, you can call other methods after entering critical section - all that code is protected and will not be interrupted. So, this one is perfectly fine too: void WriteMyData(char *fileName, byte* data, int dataSize) { hFile = CreateFile(fileName, ....); SetFilePointer(hFile, myOffset, FILE_BEGIN); WriteFile(hFile, data, dataSize, ...); CloseHandle(hFile); } void ThisIsADownloadThread() { DownloadMyData(&buffer); WriteConsole("writing downloaded block to file",...); EnterCriticalSection(myFileSection, ....); WriteMyData("somefile.data", &buffer, bufferSize); LeaveCriticalSection(myFileSection, ...); WriteConsole("finished writing data to file",...); } However, this code is not OK: void WriteMyData(char *fileName, byte* data, int dataSize) { hFile = CreateFile(fileName, ....); SetFilePointer(hFile, myOffset, FILE_BEGIN); WriteFile(hFile, data, dataSize, ...); CloseHandle(hFile); } void ThisIsADownloadThread() { EnterCriticalSection(myFileSection, ....); DownloadMyData(&buffer); WriteConsole("writing downloaded block to file",...); WriteMyData("somefile.data", &buffer, bufferSize); WriteConsole("finished writing data to file",...); LeaveCriticalSection(myFileSection, ...); } Sure, it will work, but in a very bad way. First thread will grab the critical section and then start downloading data. Other thread will try to enter critical section but will not be able to. So, it will just wait without doing anything. After first thread finishes download, it will leave critical section, and only then second thread will wake up and start downloading. See the problem? So, remember to put only minimum necessary code between Enter/LeaveCriticalSection. As mentioned earlier, it's also important to call LeaveCriticalSection. Even if your code in one thread got some exception, your SEH handler should clean up and call LeaveCriticalSection - otherwise all other threads will be stuck. Final thing about critical section objects, you can use the same critical section object in more than one place. So this one is also perfectly fine: void Write1(char *fileName) { byte buffer[1]; buffer[0] = '1'; EnterCriticalSection(myFileSection, ....); hFile = CreateFile(fileName, ....); SetFilePointer(hFile, 0, FILE_END); WriteFile(hFile, &buffer, 1, ...); CloseHandle(hFile); LeaveCriticalSection(myFileSection, ...); } void Write2(char *fileName) { byte buffer[1]; buffer[0] = '2'; EnterCriticalSection(myFileSection, ....); hFile = CreateFile(fileName, ....); SetFilePointer(hFile, 0, FILE_END); WriteFile(hFile, &buffer, 1, ...); CloseHandle(hFile); LeaveCriticalSection(myFileSection, ...); } void ThisIsADownloadThread() { WriteConsole("writing 1",...); Write1("dummy.file"); WriteConsole("writing 2",...); Write2("dummy.file"); WriteConsole("finished",...); } It was a long post but I hope it was at least a bit useful. Cheers, kao. P.S. My final suggestion is - learn multi-threading by making a simple program first. For example, make a multi-threaded program that writes something to a console or into a file For example, one thread should write "X", other one "Y", third one "Z". Play around with it. Try adding Sleep(30000) in different places and see what happens. 1 1 Link to comment
LCF-AT Posted May 14, 2021 Author Share Posted May 14, 2021 Hello kao, just wanna say thank you for trying to help me with your explaination about threads.Its really not so easy to understand it completely correctly but its really a important theme. About point 1.So I did understand that a thread works for itself by creating some kind of another code page (2 threads like 2 paper pages) and that I should use local variables +/- stack sizes to store or when using global variables = using LOCKs.But there is still a open question... - All code in normal state without using thread are just into the MAIN thread once. So if I now create a thread does it mean that everything get created another time in the new thread OR ONLY the thread procedere itself which I have used with CreateThread function?So that was still unclear for me. Example: MainThread proc ... WM_COMMAND IDC_RUNTHREAD invoke CreateThread,0,5000,addr TestThread,edi,NULL,NULL MainThread endp TestThread proc struct:DWORD invoke EnterCriticalSection,addr CriticalSection .... invoke SomeRoutine invoke SendMessage,hWnd,WM_COMMAND,IDC_RUNTHREAD,NULL ; hWnd is from MainThread .... invoke LeaveCriticalSection,addr CriticalSection ret TestThread endp SomeRoutine proc local temp:DWORD invoke something,addr temp SomeRoutine endp So above you can see my main thread proc / Wnd proc.Now if I press a button etc to trigger WM_COMMAND with my ID to start a thread then it get started and runs.Inside of that thread it calls a SomeRoutine proc = Does this code created another time for every thread?Like my question before... MainThread = all code of my app.Start another thread or more = all created again & again like 5 threads created = 5 entire code of all?Like I would start 5 Ollys with same app? So in that thread above you can see that it calls SendMessage with ID to start it again.My question here is who does get the sendmessage?Does it get the same thread ID which is running and where it was calling from OR does the normal MAINThread get the sendmessage?Do you know what I mean?So if the normal main thread does get the sendmessage then it could be a problem if I have running MANY threads who calling the same ID = overlapping or so or? About exceptions between Enter and leave functions.So I did not notice it yet.I am using my old exception log code (few years ago maybe you do remember) but I didnt add something like that.Maybe I should do that anyhow else.But mostly if any exception happens in my app then it dosent work go on = exits. One more question about alloc and free spaces.Can it be possible that this could make trouble?So lets say I run 100 threads and in all of them I am using alloc & free functions then it could be possible that something wrong happens or?Or are those functions like global alloc & free also secure and using some LOCK/CHECK to prevent errors?So what happens when the function gloabl alloc / free gets executed multible times at the same moment? Maybe I'am a little bit too unsure but in most cases I get any error & exceptions (maybe after few minutes or after few days etc like a random) and this sucks you know and to prevent this I have to know how to make it correctly. Thanks again for your help kao and to make it more clear.If you got some more infos to tell then just tell me. greetz Link to comment
kao Posted May 15, 2021 Share Posted May 15, 2021 Hi again, I can only repeat my suggestion - try to learn on a small sample app. Don't try to learn on your very complex downloader (or whatever) app, there are too many hidden things. Start simple and then add more stuff. As for your example code, it is exactly like the bad example in my previous post: 2 hours ago, kao said: Sure, it will work, but in a very bad way. First thread will grab the critical section {and start working}. Other thread will try to enter critical section but will not be able to. So, it will just wait without doing anything. After first thread finishes {work}, it will leave critical section, and only then second thread will wake up and start {work}. See the problem? So, remember to put only minimum necessary code between Enter/LeaveCriticalSection. So, you essentially created a multi-threaded application where only one thread is working at any given time. Not very useful, is it? 1 hour ago, LCF-AT said: MainThread = all code of my app.Start another thread or more = all created again & again like 5 threads created = 5 entire code of all?Like I would start 5 Ollys with same app? I tried to explain it in #1, but here it is again. When a new thread is created, only a small (kernel) structure containing all registers is created. This is the only unique thing for each thread. All threads share the same memory - which also means they all share the same code. 2 hours ago, LCF-AT said: thread above you can see that it calls SendMessage with ID to start it again.My question here is who does get the sendmessage? Message is processed by a thread that is running message loop (GetMessage/TranslateMessage). Usually that's the main thread of the application. https://stackoverflow.com/a/29185012 2 hours ago, LCF-AT said: So lets say I run 100 threads and in all of them I am using alloc & free functions then it could be possible that something wrong happens VirtualAlloc should be thread-safe. HeapAlloc is thread safe (unless you use HEAP_NO_SERIALIZE flag, which you have probably never heard of). More, it depends on where you store your pointers that VirtualAlloc/HeapAlloc returns. If you store them as local variables, it should be OK. And finally, it depends on whether you pass these pointers from one thread to another. For example, one thread allocates memory buffer, other thread downloads some data into that buffer and third thread writes from this buffer to file. That's extremely hard to get right, so try to avoid such situations. If your app keeps getting issues with memory and heap corruptions, GFlags is your best friend: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/gflags-and-pageheap 1 Link to comment
LCF-AT Posted May 15, 2021 Author Share Posted May 15, 2021 Hi again, do you have some example code link from github etc I could see of any sources where you think thats good etc? About C-Section functions again.Ok understand, just using them for important codes to protect them.Otherwise I can place them also at top / bottom for quick working routines. Question: Can I also use Enter/leave functions multible time inside of a routine?Yes or? I forgot last time to remember that also the main thread IS a thread (stupid me) and that I also can use those C-Crit functions IN them as well. That means I can also protect my "normal" routines with them if I do access them?But what is if I send a mesage from my extern thread to main thread to execute the routine then it should not work because the thread ID is same right!? Still unclear for me now.I tried to debug a app called VirtualDub (x86) in Olly and this app does use a lot of Init/Enter/Leave function in the main thread and also more than one time (Procedere & sub Procedere with same object).Why this?Just a double protect if the first Enter didnt work or something? Example: Test proc / Main thread only invoke Enter.....,addr Sec0 .... invoke Enter.....,addr Sec0 ..... ..... invoke Leave....,addr Sec0 .... invoke Leave....,addr Sec0 Does it make any sense?As I said, somehow I'am getting more confused now than before. greetz Link to comment
aIjundi Posted May 15, 2021 Share Posted May 15, 2021 24 minutes ago, LCF-AT said: Otherwise I can place them also at top / bottom for quick working routines. Locking everything, even when it is quick, would hurt performance. You only need to lock when modifying or accessing some resource that is shared between threads. If a routine does not access/modify any shared resource, you don't need to lock at all 28 minutes ago, LCF-AT said: Can I also use Enter/leave functions multible time inside of a routine?Yes or? Yes, provided you don't enter twice in a row and each enter is followed by a leave. 1 Link to comment
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now