My best guess, is that for many viewers, the novelty simply wore off. I dismissed it and never watched it the first season because it seemed lame to me. I started watching in the middle of season 2 because I saw a marathon on another channel, and coupled with the popularity of the songs on the iTunes charts, I figured it was worth a sample. I went back and caught up on episodes and I was hooked. By the second episode of the third season, I stopped recording it and have only seen a few moments here and there (mostly while waiting for "New Girl" to come on)
Personally, the storylines and slightly uneven writing never bothered me. Those are a trademark of all Ryan Murphy projects- and "Glee" at its best moments, just like "Nip/Tuck"- was worth sitting through the crap.
I think what brings down the show is the singing. There are many talented musicians in the cast, but I don't want to hear showtunes. I know it may be unrealistic to exclude them from the show, but I think "Glee" will continue to decline in popularity as long as showtunes are a mainstay. I'd rather hear a million covers of "Red Solo Cup" because it means the show is staying young and current- a beacon of what's in in pop culture. Plus, every song seems to be performed by Kurt or Rachel, characters that I like when they're just acting, but who I immediately reach for the remote when they start to sing.