There are 243 plus million people who have been within range of hearing a radio, usually as background noise. It a gross exaggeration to call them "listeners". When my furnace or air conditioner kicks in, I can hear the sound of the fan that's blowing air. That doesn't mean I'm an HVAC "listener".
First, only 48 markets are measured by PPM which will detect "accidental" listening. The other 200 markets and all the counties in the national survey that are not part of a measured metro are diary measured. Accidental listening does not get reported in the diary. And in the PPM, the amount of "only listening by accident" is minimal as a granular examination of the data will show.