There is a theoretical limit: 99 (!) subchannels, but unless all of them were audio-only nobody would be stupid enough to try it. KWDA in the DFW area has 9 subchannels, and the picture quality on several suffers noticeably.
A rule of thumb I use is, no more than seven SD subchannels, or one HD plus three SD, unless some of them are very low-bandwidth. Even then you'd need a good statmux to keep the picture quality reasonable.