Microsoft Antispyware: Torn Apart
You’ve all heard about Microsoft Antispyware Beta 1 by now. Scripting News gave it a minor mention, Neil’s World has a list of pros and cons, Asymptomatic just links it, and Peter Provost has some nice screenshots. Numerous others also cover it. But, what nobody’s doing is actually testing how much spyware the new Microsoft anti-spyware program can remove.
Luckily for me, a quick VMware Workstation installation of Windows XP Professional with 3 Gb of HD and 128MB of RAM will allow me to fill up a virtual system full of spyware–and then test it all. VMware is also offering a public beta of version 5.0, for those wanting a controlled test environment. The operating system is a raw version of XP–no service packs preinstalled, and no windows update for me. I want to maximize exposure to spyware. Since it’s running on a LAN inside a firewall, it will be safe from most viruses. I only want spyware, not miscellanious vagrant malware!
Virtual OS installed, the next step is to infect it with spyware in a systematic way. The first nasty thing I can think of to install is Kazaa, our spyware-laden p2p buddy. This gets us the GAIN network. Now I’ll install the Gamespot Download Manager, reportedly bundled with spyware. Unfortunately, I could download from HTTP–it seems as if the download manager were unavailable. Real Player, Weatherbug, MorpheusUltra, and tried Suprnova.com, which didn’t work. To increase my surfing and spyware acquiring speed, I used this list of spyware, Firefox’s tabs, and many, many google windows to try to install *each* of the the products.
Here’s some of what the system looks like after a complete infection. IE is covered in new “tools,” and the desktop is totally trashed:
Microsoft Antispyware comes to the rescue. It identified 53 spyware threats over 12 infected processes, 5031 files, and 6330 registry entries. It also gave default recommended actions for each of them, and thread-level ratings. Descriptive text on the right panel made it clear exactly what each piece of rogue software did.
Next up is Lavasoft’s Adaware Personal, with the latest definitions, of course. It labelled 10 processes, 685 registry keys, and 259 files as spyware, for a total of 1594 objects. Compared to Microsoft’s default scan, this seems sparse. At first glance, it seems as if Lavasoft either has an incompetent product, or a crippled “free” version of their real product. I don’t see enough results! Scientific comparison pending, Microsoft’s new product looks like a winner.
Our final contendor is the tough, free Spybot Search and Destroy. It offers a one-click “find spyware” button, automatic updates, and system background protection. It checks your system against a database of over 13,000 spyware with signatures, so you can bet if you’ve got spyware, it knows about it. Of course, it’s only as good as its database, which may or may not be frequently updated. One bad thing about its austere interface is the lack of spyware explanation that the Microsoft product has. When I find spyware, I want to know what it is. The final report? 20 nasties, not many views into the data. I don’t know where they came from–just that they’re there.
Here’s an Excel graph of the final comparison between the new Microsoft Antispyware Beta 1, Lavasoft Adaware Personal Edition, and Spybot S & D, the three most popular free antispyware programs:
As you can see, Microsoft’s Antispyware product dominates the bunch. For a more specific graph, see: specific-spyware-removed.png. Hands down, go Microsoft. It caught more spyware than any of the others–nearly perfect coverage. Still, keep another product on hand for the occasional messup.
| This entry was posted on Monday, January 17th, 2005 at 4:23 am and is tagged with microsoft antispyware beta, antispyware beta 1, microsoft antispyware beta 1, vmware workstation, s tabs, workstation installation, virtual os, raw version, thread level, virtual system, gain network, gamespot, public beta, google, descriptive text, morpheusultra, preinstalled, registry entries, test environment, systematic way. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback. |
26 Responses to “Microsoft Antispyware: Torn Apart”
Leave a Reply
Great idea testing in vmware.
Thanks for the ping
. Nice to see an actual test like that. Did any of the tools find any false positives, or did they miss anything important? A lot of the time I find that Ad-Aware detects loads of tracking cookies which aren’t really a major threat.
new systematic review.
Thanks Neil! None of the tools found false positives. Their problem is missing important things. For example, even the Microsoft/Giant product missed a couple things picked up by Ad-Aware. For example, the Alexa stuff–which is patched in later editions of windows–is considered spyware by Lavasoft and obviously not by Microsoft.
Tracking cookies are pretty much ignored by the Microsoft product. They’re not spyware threats anyway…
Excellent review! I remember several articles about those Anti-Spyware products and most of them seem to fail miserably. So much about cool tools. You did a great job. You should apply as a freelance writer for some magazines …
Some more: why do you consider real player and game spy spyware?
Real Player and Gamespy both are (or were) known to also install some spyware, so I thought I’d give them a try!
Thanks for the link!
I recently installed Anti-Spyware on a buddy’s computer, you know the kind – blindly clicks on any OK button that pops up. Anti-Spyware found 8 spyware threats and removed all of them effectively without hurting the system. Nice job for a live computer in the wild, which is a bit more sensitive than a fresh VM with fresh spyware installed.
I get a couple of false positives on my own machine, since I run some remote control software (VNC). It also seemed to think that libexpat.dll was the PeopleOnPage Browser Hijacker, just like on this guy’s system. In subsequent scans, it has changed its mind.
Excellent article! Good writing; creative testing; useful info!
Just wondering, did you run adaware and spybot on the same os after you ran microsoft antispyware? Wouldn’t that mean that most of everything would’ve already been cleaned up before adaware and spybot could run?
I ran them, but I didn’t remove any spyware–I just ran the scans. So, that problem you mention didn’t actually happen! ^_^
Just because you didn’t select “remove” doesn’t mean it still didn’t remove. Microsoft may have removed some of the really bad ones without asking, or maybe microsoft does it in the background while the scan is running. To do a fair test what you need to do is create one virutal HD (not sure how VM works, but this would work in Qemu), install XP and all the mallicious stuff on there, shut the virtual PC down, then make 3 copies of the virtual HD. On each copy run each product, and then compare results. I might try that myself when I have some time, and how do we know you don’t work for microsoft
Don’t take my comments wrong, that is a great idea to compare the scanners! Keep up the good work!
Also, how do you know the numbers really are what they say they are? Maybe Microsoft counts duplicates of the same spyware to make it look like it found more. Another suggestion is after running the scans, run the Microsoft scan and see how many it finds on the scanned and repaired VM.
Good Luck!
To reassure Jeff, who doubts the vericity of my procedure:
Repeated scans with the microsoft product always show the same spyware, therefore it doesn’t remove spyware unless either you have set it to, or you ask it to. Neither were done, therefore the spyware wasn’t removed by Microsoft antispyware.
Also, I counted spyware by hand, using spyware names. I made sure there were no duplicated, and generalized as much as possible. For example, all Claria products were generalized to “Claria” because some antispyware products just group them like that–although MSFT does not.
Oh really? I’m suprised, didn’t expect that much from an MS product. Well at least they’re actually trying to imrpove their OS (as long as they’re not greedy about it and make this thing freeware)
@Jeff, it’s actually worse if they release this as either a free product or integrate it with the OS. Either way, they will completely destroy an entire segment of the anti-malware industry. It’s better if Microsoft sells this and competes with the industry without totally undercutting them. Remember what happened to netscape?
Poor “scientific” approach on a number of items but the most glaring and easies to point out: Your graph showing the bars for the number of hits by Microsoft, Ad-aware, and Spybot S&D is misleading at best. How is it that you base this on a 100% scale (vertical scale)? You trully dont know how much was actually installed in your victim system. You only know what was detected by each. Therefore, you make it look like Microsoft found 90% of spyware…not quite true.
John, john, I’m not so stupid. It’s a relative measure of performance based on all of the spyware found.
Are you sure you instructed each apllication to actually perform the same task?
What is Spy or Adware is relative, some people view certian cookies as spyware some don’t. If you set the Microsoft one to scan every single drive and inside archieves and to detect minor threats such as cookies then ofcourse it would find a dam site more than running SpyBot on just the critical threats and ignoring the cookie threats, also its hard to really compare unless you actually read each report to confirm that SpyBot and Lavasoft don’t count something as 1 threat when Microsoft counts it multipule times. Do you even know what was on the infected system? Did you only find thoose nice common spyware things like kazaa or some of the more obscure ones? Did you repeat the test, if so how many times and are the stats averages? This is not scientific in the least bit, making the results completely unreliable. More information is needed on exactly how the tests where carried out. And FYI spybot does list detials about threats. It sometimes hides it so not to confuse users.
Andy falls into John’s line of thought. Really guys, there’s no major design flaw in the study:
1. Spyware counts are manual and aggregate. The total amount of spyware is the manual union of the spyware reported by all of the tools.
2. Spyware was placed on the system by myself, so there were no surprises there. All the common ones like Kazaa and Real were installed that I could think of, and then random threats were installed where they could be aquired.
3. The results are completely deterministic in theory–and in practice, because I checked. MS Antispyware will always find xx items if xx items are findable. Running another antispyware product to look for spyware first without removing any in the program leaves all spyware intact.
“Microsoft’s new product looks like a winner.”
Sorry to bring this back up on you, but this is not a new product.
All it is, is Giant Antispyware (of which they bought out) with Microsoft Beta written all over it. It is the best in spyware/adware removal, and I trust it with all my heart and soul (kicks adaware’s pay version and spybot’s butt 3 times over), but Microsoft deserves no credit to this at all…other than possibly making it free.
Elliott,
Thanks for the review.
Personally I don’t care who wrote the Microsoft product – Microsoft bought Giant because their product was highly regarded, and now they have to work out how to market it without getting yet another law suit from someone who cries foul with their overpriced or inadequate product (e.g. the Real Networks piece of spyware that I have to install to listen to BBC broadcasts online).
That aside, I really think the crux of this is what you said in your summary: “another product on hand for the occasional messup”. I’ve been blindly trusting the Microsoft product for months and it seems to have done a good job on the whole but ignored a whole load of tracking cookies on each machine I ran Ad-Aware SE 1.06 (personal) against today.
I love the
GiantMicrosoft product, but think it needs to improve its handling of tracking cookies. Until then, I’ll have to run two scans on my PCs.BTW, I think it’s interesting to see that you get flamed for suggesting that a Microsoft product could be good. I regularly blog on Microsoft stuff and get caned for being “sponsored by Microsoft” (I wish I was!).
Mark
Watch out for cookies when using the Microsoft AntiSpyware beta
Last year I blogged about Microsoft’s acquisition of Giant Software and I’ve been using their AntiSpyware…
Jeff: “Oh really? I’m suprised, didn’t expect that much from an MS product.” … that’s probably because MS didn’t make MS Anti-spyware, they bought it, and relatively recently at that!
MS Anti-Spyware is basically Giant Anti-Spyware rebranded. They bought one of the best. We’ll see if they can keep up the same quality level.
[...] This is based on my tests of antispyware products, but since FlexBeta had the same results, I don’t think I’m crazy here. [...]
[...] Further reading Adware/Spyware thread (pcreview.co.uk) Cookie demonstration (privacy.net) Microsoft AntiSpyware: Torn Apart [...]