A NIMDA Experience

September 18th

I do know that after 9:34 am, this virus began propogating itself to directories on the local server in the form of .EML files.

9:53 am
Still undetected, the virus has now spread via the WAN link to our Corporate server. Meanwhile, the virus is still spreading like wildfire on the IIS/home server directories.

10:13 am
I receive an e-mail alert that the storage space on our internal IIS/Home server was below 250 mb. Unforunately, our users have a tendancy to dump very large database files (2gb+) onto this server and considering the server usually only has 3-4 gb of free space, it gets filled every 6 months or so by someone. The server still had about 240mb free on it and I assumed that was the case and began running a Perl script to recurse the user home directories and find out who was consuming the space. The script takes about 20 minutes to run completly so I went to grab a cup of coffee.

10:17 am
I received an e-mail from a user who had noticed some very strange files on the IIS/Home server, all ending in .EML extensions. (An EML file is an Email file. It's essentially an e-mail saved to a file and can be opened in Outlook)

10:18 am
Before I could look into the EML files, we received support calls from 4 different users stating that they couldn't save any files to their home directories. I pulled up Windows Explorer real quick and the 240mb of free space was completely gone. Still thinking it was a user's large database file, I let my Perl script continue to run and moved a 500mb file of archived logs off of the server and onto my local PC, just to give users some free space to work with.

10:22 am
I received another e-mail from a second user who also noticed the .EML files in her home directory on the IIS/Home server. Unfortunately, she actually attached the .EML files to the e-mail. I hesitated for a moment before opening the attachment, so instead of opening the EML file, I did a Quick View with Windows 2000. I saw that the .EML files seemed strange. The subject was extremely long and seemed like garbage and the body didn't contain anything. The files did have a README.EXE file attached though. Needless to say, I was more than a little suspiscous. I immediately closed the Quick View and did a permanent delete on the e-mail.

At this point, I connected to the D$ share of the IIS/Home server and began a File Find for *.EML. I also started a full virus scan of the server using InoculateIT (running the latest signature's of course)

10:42 am
The full virus scan of the server turned up empty, but my File Find had found over 10,000 .EML files! I sent a quick page off to our CIO and we kept looking into the .EML files and starting checking some of the major antivirus sites. I had a bad feeling it was something new.

10:54 am
After checking Computer Associates, McAfee, Norton and Trend Micro's sites, we didn't find any new advisories. We were thinking it might be a new strain of the APOST.A virus, simply because of the README.EXE file that was attached. I sent a quick page again off to our CIO keeping him updated.

11:27 am
Russ Cooper of NTBugtraq sent an e-mail adivsory out warning of a new IIS worm propogating named W32.nimda.amm. He only described the IIS side of the effect and didn't mention anything about any .EML files. It seemed odd to us because, as far as we knew, our IIS side of the server was perfectly fine. We began checking our IIS server logs to see if we could find anything else.

11:58 am
We sent an e-mail out to all of our users informing them of the .EML files on the network and to NOT launch them under any circumstances. We still hadn't found anything unusual in the IIS server logs, but that was because we weren't looking for the NIMDA/Code Red type patterns.

12:06 pm
I ran a File Find for *.EML on the Corporate server and to my suprise discovered 468 .EML files on that server. I had initially thought that it was isolated at least to our location, but it had somehow spread over the WAN. I deleted all 468 .EML files, re-ran File Find and didn't find any additional .EML files on the Corporate server.

12:21 pm
I deleted the 11,483 .EML files on the IIS/Home server that I had discovered and started a new File Find again.

12:24 pm
File Find found 6,000 new .EML files on the IIS/Home server! All the files had a time stamp of 12:22 am. We were starting to feel like we were slowly losing control of the situation.

12:29 pm
With the .EML extension of the files, we felt at the time that the point of entry for the virus was via an e-mail message to a user who ran the README.EXE attachment.As a precaution, we decided to shut down our Exchange Internet Mail Service to prevent any possible viruses from coming in/out via e-mail.

12:41 pm
Re-running the File Find on the IIS/Home Server found 16,000 .EML files now. Somehow we got an additional 10,000 .EML files since the last File Find at 12:24!

12:52 pm
Symantec issues their first notification. (Thank goodness for web based e-mail like Yahoo and Hotmail!) Same general information as Russ Coopers, but still no mention of the .EML files! Management still isn't convinced that we're suffering from the Nimda virus since we're seeing .EML files and apparantly no one else is.

We spend the next hour and a half deleting over 35,000 .EML files on five different servers.

2:30 pm
Trend Micro to the rescue! They release their first notification on it, complete with sketchy technical details. They do however, mention the prolification of .EML files. (whew)

We're still combating the re-appearance of thousands of .EML files on several servers. As fast as we can delete the files, something is creating more. We're convinced that we have users desktop PC's that are infected (at least one, that's all it needs)

3:15 pm
It took some convincing, but we got Management's approval and we're shutting down all PC's at our location. It's never a good position to be in, but we need to ensure that we can stop the spread of these .EML files.

We're continuing to delete thousands of .EML files on the servers, which thankfully aren't re-appearing anymore now, while the desktop support group begins the arduous process of manually checking over a hundred desktop PC's for any .EML files.

5:00 pm
Thanks to several people on the NTBugtraq list, we've got some pretty detailed preliminary instructions on how to manually find and clean the NIMDA virus from desktop PC's. Over the next five hours, this is done to all of the PC's. We've got twelve PC's that turn out "positive" for the NIMDA virus and we quarantine those PC's from the network.

8:00 pm
We check Yahoo web mail and get an e-mail that Russ Cooper of NTBugtraq sent out at 6:19 outlining more technical information about the NIMDA virus. Full detailed instructions on the IE versions that are effected as well as the three major attack vectors(IE, E-mail and IIS) are included. Unfortunately, over 90% of our desktops are running Windows 9x with IE 4.x. The decision is made to completely shut down Internet access until a patch is released for InoculateIT for NIMDA and we get all of those PC's upgraded to IE 5.01 SP2.

11:18 pm
Finally done, and a bit tired. Amazingly, our CIO is still here with us. Thirty minutes later, I leave for the forty-five minutes drive home.

September 19th

While the signatures we're updating, I did a quick File Find on the IIS/Home server to see if any .EML files popped up overnight. NOOO! There's over 5,000 .EML files back on this server!

We get approval from management to keep users at our location off of their computers again until we can get this contained. It's not fun, but the majority of them understand. At least users at our other three locations can still use their computers, but the Internet and Internet e-mail is still down. We spend the next 5 hours cleaning servers and continuing to clean user's desktop PC's. We also begin applying IE 5.01 SP2 patches to desktops

12:25 pm
The IIS/Home server that was initially infected is still heavily infected. For every .EML file we delete, more take it's place. Scanning the server shows active executables running in memory are infected with the NIMDA virus. Over 500 executables in the server are infected including key executables for the NT operating system itself, PowerChute, WinZip, Computer Associates Arcserve and ironically, even InoculateIT itself.

We decide that this server is beyond recovery. It's going to need a complete system rebuild and data needs to be recovered back from Monday's back-up (pre-NIMDA) tape. We isolate the computer from the network and spend the next three hours documenting every single piece of information we can from the server, to make the rebuild easier.

1:45 pm
Since we're going to need to pull down critical patches, etc from the Internet, we get approval from management to open the Net back up as well as e-mail. Considering our users at this location are still powered down, it's not too bad. An e-mail is sent out to all the other users having them stay off of the Net.

4:15 pm
Fresh install of NT on the IIS/Home server begins. At least this will give us a chance to get all the Compaq ROMPAQ's updated to the latest versions. :)

September 20th

7:45 am
Restore went fine to the rebuilt IIS/Home server. Begin a full scan on all of the other servers to see if NIMDA popped up overnight.

8:30 am
All of our servers are showing 100% clean now. We spend the next three hours re-assigning permissions and finalizing last minute details on the IIS/Home Server.

12:15 pm
Notification is given to users at our location that they can power their PC's back on and log into the network to resume normal work operations. We all hold our breath while full scans are running on the IIS/Home server.

2:10 pm
Full scans show that everything is clean. Looks like we got it completely. Everything is back 100% operational and all users are working as normal.

Summary

Roughly two and a half-days later and a completely rebuilt server, we're back online. So what did we learn from this? Well, our systems are certainly kept up to date with hotfixes and security patches now. We always tried to stay a few months behind the curve, to let the other guys test things out first. No longer the case, we're not going to be the early-adopters. To the extent that we're gambling the stability of our systems in an effort to prevent the next big NIMDA virus from hitting us. However, for us that's a gamble that we've decided we're willing to take to ensure that we don't spend another two and a half-days dealing with this problem next time. But then again, no matter how many times you think that's going to happen, it never seems to work does it.