Not necessarily. Delaying shows is commonplace, especially in the western US, where putting on someone like Imus live would make no sense since it would be 2 or 3 AM in the Mountain or Pacific time zones.
Rush found the perfect niche time when he started in 1988 by going from Noon-3PM ET. That way, he hits either early or late midday in the whole country and avoids drive time. But many hosts don't have that luxury, so local programmers will delay their shows and air them when it fits their station's local needs best.
The biggest drawback is major national breaking news. If you're listening to a four hour old show and a major news event occured an hour ago, it sounds weird for the host to not be speaking about it, which of course, would be impossible. The exception is a truly seismic event such as 9-11, in which case, your station, hopefully, is airing something else live anyway.
The other drawback is callers can't call a delayed show, but since only a miniscule number of callers get on anyway, most programmers don't care about that; nor should they. A good host is what people listen to.