I appreciate how humble he is: This I understand, this I don't, here I have experience, but not here. Example: "It took months to understand the bug, but then a fix was created fairly promptly (not by me – I still don’t understand that part of the code)"
Also nice how the issues are handled by team members, that know more about stuff in their field. Like, maybe other people wouldn't be skilled enough to find the issue, but he wasn't skilled enough to fix it.
Learning along the way. When tackling issues, there are always some knowns and unknowns and the experience is gained. Thanks to the way randomascii is written, I'v had some success using WPA.
First thing I do on a new Windows setup is turn on "handles", "threads", "Commit size", "NP pool" columns in task manager details...
If you want to see some real offenders... have a quick look at Asus's "LightingService.exe" (the daemon that controls their rgb LED coloring suite). Gets up to 2m+ handles after a day or two of running on my system.
The next biggest offender on my system? Their ez updater "EzUpdt.exe" at ~60,000 handles.
Haven't bothered looking at it with process explorer/procmon yet, but I'm sure it's a quick fix.
I didn't even check the handles on LightingService. That shit got uninstalled fast after I saw it eating an entire CPU core and then some.
I honestly didn't even want any lighting control software but all the RAM that had the speed and latency that I wanted were decorated with them so I needed software to set them to a static color. Thankfully I learned about OpenRGB.
It's a shit-show also, with every vendor having their own solution and 0 interop. RGB addressing isn't rocket science I'm not sure why every app needs its own proprietary daemon to handle this - mercy on your soul if you have corsair + asus + gigabyte in your system as I do.
I think it's because no one standardized on protocols for addressing RGB lighting on RAM, GPUs, and motherboards. We at least have those ARGB headers fairly standardized now, but wiring up even things that are slotted into the motherboard would suck (having just a few fans with those is already a mess).
Also the manufacturers probably love it because it pushes people towards not mixing products from different companies.
That's an awesome set of tools! I've written much jankier versions of these myself, and I'll definitely look into using the official ones.
That said, I don't know that they would have helped in the article's case. It looks like there was a higher-level resource getting leaked, which would also most likely leak these.
Heh, if Teams was a linux application the first line of the documentation would be about editing sysctl.config to allow far more open files by default.
Just enabling the Handles column made me realize that the "Asus Update Service" SysTray Icon/application had a handle leak: as it had 64K handles open.
There's also an Asus Lighting Service that still has 10K handles open.
It seemed like in the old days you had to read the Petzold book to learn Windows programming. It actually explained what was going on underneath the hood. And if you wanted more, you'd read something by Jeffrey Richter, David Solomon, or Mark Russinovich.
Now? You watch a couple YouTube videos, or perhaps a terrible Apress book. You get something to appear on the page, and viola!, you're a programmer.
There’s a real problem here: those are still the books to read if you really want to understand Windows. They’re great books but they’re old, often out of print, and the onus is on the reader to distinguish outdated info from still-relevant info. Yet as far as I can tell, nobody’s publishing better books about Windows programming today.
It’s remarkably hard in 2021 for a newbie to start learning Windows programming “properly”.
It's because MS treats win32 (or how ever you want to call the "legacy" api) like the plague despite it being the past,present and arguably the future.
I still can't name a single UWP app that isnt a crude mockery of some win32 app that runs circles around it in both performance and UI design.
I think that’s an unfair characterization of the Terminal issue you linked, and the Terminal developers in general. If you follow the Terminal repo it is clear that the devs are not fans of the “modern” Windows technologies they are forced to work with.
This doesn't surprise me at all. Vendor tools, like the ASUS utilities, or Dells update application, and many others are commonly outsourced from different companies and seem to import a string of security flaws. Providing the program, not providing a secure program tends to be the metric for delivery on these applications.
Just for others who may not see where the handles column is: "Open task manager, go to details tab, right click on one of the already listed columns, click select columns, then scroll down to check handles."
After sorting by handles, I see similar results. Logitech Hub Agent has 13,000 handles. It seems these gaming companies hire the worst coders without care for actual performance. Yet...they advertise their products as cutting edge performance. Quite the irony.
> It seems these gaming companies hire the worst coders without care for actual performance. Yet...they advertise their products as cutting edge performance. Quite the irony.
Unfortunately gaming companies routinely hire inexperienced coders because they're cheap. By the time those coders have skilled up, they've been burned by the gaming industry in one of the many ways that the gaming industry burns coders, and so they no longer work in gamedev.
Another two good utilities to use is performance monitor, and process explorer. Perf mon is good for showing usage over time with its built in graphing and recording.
"Process Hacker" can sometimes display more information as well; for example, I see that igfxEM.exe on my machine (Intel graphics driver) is leaking around 5000 handles to the registry key HLKM\system\ControlSet001\Control\Class\{5c4c3332-344d-483c-8739-259e934c9cc8}\0001 over a week of uptime…
Oh I used to love leaky HANDLE hunts. These things could bring windows 3.n (and maybe NT 4.v too?) to its knees pretty quickly so I was always really careful about diligently freeing them.
And, like many commenters below constantly surprised that so many apps still ship with these problems, since it's so easy to spot them. Windows perfmon also gives you a nice graph, so you can correlate e.g. GUI behaviour with a jump in handles being created that then never get freed.
Whilst it's easy to spot the leak, the fact that you can leak handles by entirely forgetting they exist can make it a little harder for the programmers to find where their code leaked the handle and so fix it.
If you leak 6GB of RAM, there's 6GB of evidence about what was leaked exactly. If it's full of terrible love poetry you can rule out "FootBallScores" and focus on the "TeenagePoems" data structure and related code. But if you leak 50 000 HANDLEs then er... oops?
The post shows you can narrow it down to say, Event HANDLES but after that it gets increasingly sticky. Hopefully somewhere there's a C++ object that owns the handle and that has leaked which you can trace, but as I understand it, the handles themselves might be all that leaked, leaving you to instrument software so that you can find out which code made all the handles and then trace back those that seem leaked.
One good rule I used to use and make everyone in my team use was when you get a handle write the 'anti' call at the same time. Good portions of the older windows API is allocate functions and destroy functions. If you write both at the same time your mental overhead is less.
many start using 'auto managed' style languages (c++, java, etc) where the life cycle is not as clear. The life cycle is the same though but you own it in an indirect way. Who makes it. Who uses it. Who destroys it. In some languages that is easier to do, others you own it front to back. When doing this I try to start with create/destroy then usage. It is a style that helps remove leaks before they happen. They can still slip in there...
I have used the common string thing a few times to help narrow leaks (does not work in all cases :( ). You also can use tools like valgrind, boundchecker, purify, etc. Think MS has a couple that I can not remember off the top of my head.
Historically there were two kinds of leak. Your program might grow more than intended, but eventually give everything back when it finished - or it might seize some resources permanently by mistake, so that you need to restart the computer to "fix" it.
Modern operating systems mostly rule out the latter type of leak, you could leak files I guess, and of course Cloud users could leak things like S3 objects, even whole instances, but many resources are now automatically cleaned up when you exit.
As a result of that though, for long-lived processes, such as Chrome but also most background tasks and server software, just "I definitely clean up the mess eventually" doesn't get the job done, the OS was going to do that too. The user doesn't care whether the resources would have been returned half a second after they closed your program when "clean_up_everything()" is called by the main thread, or, a second after that when the OS cleans everything left behind.
So this is a real problem, about the actual meaning of our programs, and (though they are still a good idea) can't be helped by good programming techniques, garbage collection, Rust's Drop trait, the C++ RAII way of thinking, deferred clean-up in languages like Zig or Python, or anything else I'm aware of.
We need to actually express in our programs the intent to hang on to only what's actually needed and clean everything else up as we go. And it can be sorely tempting to consider that "It gets cleaned up eventually" is good enough, that's where leaks get in.
Spot on. But as for 'right now' I have to be a bit more practical and work with what I got.
Mostly these days most machines have a decent amount memory so leaks are not as noticeable, unless you look. If in the early days if I leaked 50MB of memory and my machine had 16MB. I had a real issue and the machine would be borked. If I do the same today you would not notice it.
It is why I stressed watching life cycle of an object. You made this thing, who is cleaning this mess up, and when. 'When' could be anywhere from never 'I need this all the time' to bunch it up when idle/reuse (garbage collection style), or 'right now' I need this memory back right now. There are trade offs and you need to watch for that too. The write it down while you are thinking of it has served me very well over the years. I personally got bit by not following my own rule a few weeks ago. I allocated something and I had not cleaned up correctly. I got 'lucky' and that was actually the right thing to do. But in the code review I rightfully got dinged on it.
One thing I wish many more docs would do is 'this makes object xyz use remove_xyz to clean it up'. Or 'this looks like it is creating an object it is not, this is returning some global'. Right there in the doc. It would help so much.
That temptation of 'eventually' is one that some languages push hard. But I find many leaks that I have chased over the years were just a poor understanding of the calls being used (bad docs, not reading them, or a combo). You may have had a different experience.
> can't be helped by good programming techniques, garbage collection, Rust's Drop trait, the C++ RAII way of thinking, deferred clean-up in languages like Zig or Python, or anything else I'm aware of.
Arena allocation. Every allocation must be attributed to some arena (eg current tab, current network request, current frame being rendered, etc); when the arena goes away, so do all its allocations.
The managed languages create additional hurdles for handle leaks. My prior experience with the .net GC is that it responds to memory pressure, and not necessarily to open handles. So when people write code that relies on GC for cleanup, it has a bit of a blind spot for the handles -- they don't take much memory in your process so it won't know to invoke GC.
I wonder if these problems would happen less on Linux.
On the one hand, most distros default to 1024 for the (soft) limit on open fds, which would be easier to hit if you were leaking fds and thus easier to detect.
On the other hand, a lot of things that HANDLEs get used for might end up being userspace pointers from userspace libraries instead, so the leak would be of memory instead of fds and hard to detect again.
You can track handles within your process using GetProcessHandleCount. We have discovered some leaks in Windows APIs this way (which were fixed pretty fast).
On the topic of leaking HANDLEs, Synergy/Barrier's service leaks zombie processes and causes systems to malfunction over time, and this hasn't been fixed in years.
Ouch. Zombie handles are so much worse. Now we're talking serious memory, and since it isn't attributed to the process that leaks the handles it is easy to not realize what is happening.
More details: as I explain in the first link in the blog post, if a process leaks process handles then roughly 64 KB of memory will be used for each leaked handle (each zombie process) but that memory will not be attributed to any process. If you have the Handles column open you will see a large count in one process, but that process will not have a large memory footprint. If you kill that process you will reclaim the memory from all of the zombies.
Also nice how the issues are handled by team members, that know more about stuff in their field. Like, maybe other people wouldn't be skilled enough to find the issue, but he wasn't skilled enough to fix it.
Learning along the way. When tackling issues, there are always some knowns and unknowns and the experience is gained. Thanks to the way randomascii is written, I'v had some success using WPA.