Size wouldn't be a problem, but power consumption would be. Current digital TV decoding chips are fairly power hungry, and will go through a battery pretty fast. And the problem is that you aren't going to save a lot of power by eliminating that portion of the circuitry dedicated to decoding the MPEG-2 video and making an audio only device.
The other issue right now would be cost -- the digital TV decoding chip would be a signficant expense considering the low price point that most radios sell for today. How many people would be willing to pay $99 for a TV-band radio?