Operating systems changes are almost always driven by hardware changes. As CPUs got better, faster, and smarter, the O/S needed to catch up to their capabilities, especially in the area of memory management - the lifeblood of any O/S.
Windows 98 was the ultimate DOS O/S. It also did a very good job running multiple Windows 16-bit apps, and 32-bit Windows apps as long as you had enough memory, and a video card with enough RAM. It would do amazing things in 128MB of RAM, and killed with 1GB.
As processors went to 64-bit and 128-bit, Windows NT/2000/XP evolved. I still have a system running Windows 2000 desktop, but it's a single-purpose machine sitting on a platform that the O/S was designed to run on. I still prefer some of the nuts-and-bolts things I can do on Win2K that require a "wizard" on newer versions of Windows.
XP was the ultimate desktop for Windows 32-bit programs. With a gig of RAM, it did well. With 2-4 GB of RAM, it screamed. But when 64-bit programs came along, and multiple processors on the same die, XP became obsolete. XP-64 was an attempt to deal, but it simply couldn't handle more than two CPUs.
There were a lot of other technologies - like multi-media support - that were antiquated on XP. Many application programmers provided their own solutions. Many others asked Microsoft to beef up the O/S to handle multi-media better so they could keep their apps simpler. Microsoft decided - as it usually does - that having the O/S do the work keeps it simpler for application programmers, and keeps the look-and-feel of apps from different programmers more consistent. This required some reprogramming of older multi-media apps once you got beyond XP.
Vista was the "bridge" product that incorporated everything Microsoft had to throw at the new hardware and software. It was overengineered, overcomplicated, and tried to communicate with all of the existing hardware from manufacturers who were still in business. It can address and manage 4 or more processors, and a LOT more RAM. If you had hardware from a manufacturer that had gone bust, or decided not to write drivers for Vista, you were screwed. If you were an early adopter of the 64-bit version of Vista, you were part of the learning curve for everybody - Microsoft, the application programmers, and the hardware manufacturers. Vista 32-bit runs pretty well, although it's slow to boot.
Windows 7 is the "fixed" version of Vista. The underlying subsystems are substantially the same. Everybody - Microsoft, hardware manufacturers, and application programmers are substantially more comfortable by now.
As far as viruses go, when you have over 90% of the desktops in the world, you're a target. It's got more to do with the will of the bad guys than it does with vulnerability of the O/S.
Even Vista was way way better.