In some cases, the cities lose population or the teams lose fans.
Example: the RCA Dome in Indianapolis seats about 56k for football (memory here) and the new Lucas Oil Field which is under construction will seat over 70,000. The RCA Dome has had one game that wasn't a sellout that I can think of since 2000 - something like a 32 game streak before not having a sellout.
It make sense to have a larger stadium then? Well, yes, as long as the team does well and the city and surrounding area is doing well economically. If Eli Lilly goes out of business, Indianapolis will have issues filling their new stadium. If the Colts find themselves 3-13 back-to-back years, they might have issues filling the new stadium.