I think it's possible ESPN may lose say 25 stations over the next couple of years. Of which, let's say 15 go to FSR and 10 to YSR. Of the 15 to FSR though, YahooSports will likely pick up the rimshot am FSR affiliate that was clearing already in that market. So any loss for ESPN, will probably result in a gain for Yahoo Sports Radio.
However, the main key to survival (and possibly prosperity) has to do more with the acceptance and continued clearance of 3 sports stations in a market.
Or I suppose they could also somehow bump FSR out of the ClearChannel deal which I don't even know if that's even negotiable.
My biggest concern for YSR, and this was mentioned above, would be another company like CBS or DirectV launching a 4th network with big bucks behind their affiliate push. Although, if you look at One on One, SNR, Sportsfan Radio network, Prime Sports, and others maybe they just don't see it as profitable even if it would bury a competitor along the way.
This may answer the question of why SNR/YSR has had 5 or 6 owners over the last 15 years? It comes down to how much money are you willing to lose for the slim hope of somehow battling the giants. But underdogs do come through sometimes. Take a look at the origins of Cumulus and where they are now.