And then factor in attrition---how many people in the upper demos have changed their habits, their listening preferences, and (sorry to bring it up, but always a factor in upper demos)...died.
76 million Americans were born during the Baby Boom (1946-1964). By 2012 (the latest numbers available), 11 million of them had died. It's been offset by immigrants born in the same years who've come here, but as you've pointed out, David, that doesn't equate to the same musical tastes.
Another good point. There are many boomers whose tastes have migrated over the years, and to whom the 60's and 70's pop / CHR oldies are just no longer relevant.
I went to the Pitbull concert this Saturday and was amazed at how many younger boomers were there. There is a stereotype that if I once liked "At the Hop" and "Peggy Sue" I should like to hear them today. I'll take "Messin' Around" and "International Love" many times over the older stuff.