There is no hard-and-fast limit. It depends on how far you're willing to compress the video. The more you compress, the poorer the picture quality. SD subchannels require less compression than HD subs.
Trinity Broadcasting Network runs five SD subs on their stations - IMHO they look pretty good but that's about the limit. Here in Nashville two stations run one HD sub and two SDs - in each case the HD and one of the SDs look pretty good, the second SD not so much so.
Yup, it all depends on the equipment and how it's used. In Milwaukee for instance, WDJT/CBS runs 4 subs; two in 4:3 SD with full-screen video, one in 16:9 SD, all with a 1080i main channel, and somehow the HD channel still looks great because they vary the bitrate on the other three depending on the programming. Admittedly though their sister independent WMLW carries a 720p main signal, Bounce on the 2nd, widescreen Movies! on the third, and SD Telemundo on the fourth; that struggles a little more for some reason.
Compare that to WTMJ/NBC though. Three subchannels, one Local AccuWeather (data/small video) and one Live Well (widescreen full video), and then the main channel in 1080i. In that case though, it's universally agreed by many DTV followers in Milwaukee that their video quality is pitiful because they have consistent bitrates across all three channels, so that Live Well is pushing bits it doesn't deserve during its inexplicable hours of infomercials, one of which is in primetime. Somehow, NBC there looks worse than 720p in my eyes.
Then the public TV stations; the main one runs 720p on their main signal, two full-screen 4:3 video feeds, and then a data-only weather station. Looks OK. Same with the other sister station, which has a 720p main signal, a 480i widescreen rebroadcast of the first station, Create on the third, two audio channels with a still card of song/artist, and the last and sixth channel airing a traffic camera channel which is sent in the lowest video bandwidth permissible. It just seems to depend on how you push and shape the bandwidth in every case, so six subchannels can be done, but carefully.
Also don't forget a few years back, some of the ION stations were pushing the failed
USDTV service out under contract along with several other failed 'pay over the air' systems, so they know already how to push out multiple signals.