The word "fan" comes from the word "fanatic", and most people who watch, listen or even attend games are not fanatics. The fanatics are a relatively small minority.
To add on to your point, Mikey: My wife belongs to a volunteer organization that used to do fundraising in December by taking over some vacant shopping mall storefront and offering giftwrapping services to Christmas shoppers. She would show up on certain days and do her bit. One day, a tall, well-attired Black fellow comes in with an armful of purchases, and she assists him to select wrapping papers and bows and ribbon, and then she wraps his purchases. While she's doing this, she noticed a buzz in the store but just kept doing her thing. When she finished and the fellow had paid and left, one of her co-workers asks if she knew who that was. Turns out she was assisting Jerry Rice, the hall-of-fame running back from the San Francisco 49ers. She had absolutely no clue. (And had I been there that day, I probably wouldn't have recognized Mr. Rice either.)