The FM band had lots of unique content from the 40's well into the 50's. But several things impeded any audience growth of FM.
First, before the development of AFC circuits, FM reception drifted. Second, the Armstrong-Sarnoff battle kept FM from being more actively promoted at the national level. Third, as TV developed, radio had enough trouble adapting and sustaining AM stations for anyone to spend much time on FM.
The move from 42-50 MHz to 88-108 MHz after the war didn't help FM initially (thousands of receivers became obsolete after the transition -- IIRC, the last "old-band" stations went off the air in 1948), but it was the right decision over the long term. There wasn't enough room in the old band for growth -- it couldn't expand up or down. Same problem as today, ironically.
As a surprise to some, the FM band had begun a slow but impressive growth again in the 60's, with station counts increasing plenty of pre-simulcast-ban innovations like WOR-FM's early FM rock format and the development of Beautiful Music at stations like KFOG, WDVR and WJIB to ethnic FMs such as WOOK-FM and WZEN.
I'm guessing that the start of Stereo FM in 1961 had a hand in the start of FM's success as well.
And unlike color TV, Sarnoff did not want FM to succeed as the money consumers would spend on FM might decrease the money spent on a TV set. At that time, such opposition was very significant to the state of FM.
RCA didn't invent FM, and they couldn't reap the patent rewards. That had to have something to do with Sarnoff's opposition, besides his personal feud with Armstrong.