Whether the actors previously mentioned were racist or not is almost immaterial. Unfortunately, racism and Hollywood are not strangers to each other. Cases in point:
*African Americans became the victims of one of the most vicious stereotypical labellings in 1915 with the release of D. W. Griffith's The Birth of a Nation, in which they were portrayed as dim-witted, lazy, drunk, sleazy, and two-faced. Griffith, the son of a Confederate general, added insult to injury by casting mostly white actors in blackface to portray the black "characters." The effect on African Americans in both society and the movie business was devastating, and the worst of it lasted for decades.
*Hattie McDaniel, who won the Best Supporting Actress Oscar for her portrayal of Mammy in 1939's Gone With The Wind, was not allowed to attend the world premiere of GWTW in Atlanta due to Georgia's strict segregationist laws in effect at the time. Clark Gable and others connected with the film protested and wanted the premiere moved out of Atlanta, but McDaniel, not wanting to rock any boats, urged them to go on ahead to Atlanta.
*Staying with McDaniel, she suffered two more slights because of her race. On what should have been the greatest night of her life, she and her husband, the only blacks in attendance at the 1940 Oscar presentations, were placed in the very back of the room at the Ambassador Hotel, where the ceremonies were being held. They were not allowed to sit with other people from GWTW.
Upon her death in 1952, McDaniel's final wishes included to be interred at Hollywood Memorial Cemetery. The then-owner of the cemetery denied that request, because he had a policy of whites-only being buried there. Hattie McDaniel was subsequently interred at a cemetery in South Los Angeles. The Hollywood graveyard, now known as Hollywood Forever Cemetery, and under more modern ownership, now has a cenotaph for her as a tribute.
*Hal Roach Studios, which produced The Little Rascals, aka Our Gang, gave its child actors catered box lunches in its dining hall. However, black child actors such as Buckwheat and others had to take a sack lunch outside and sit on the sidewalk on Washington Blvd. outside the studio.
*Louis B. Mayer, one of the founding fathers of MGM, lost his 1951 power struggle with Dore Schary in part to a creative difference in the casting of a black actor. Mayer, who always referred to black people as "Schwarzas", a Yiddish term for black people, not necessarily derogatory, wanted the black character to be Uncle Tom in nature, while Schary wanted the character to be proud and defiant. Schary, much to Mayer's dismay, went over his head to the head of MGM's parent company Loew's Inc., and won that argument. Mayer was ousted a short time later.
Since Walter Brennan isn't around anymore to answer the question at hand, I guess we'll never really know for sure, but whether he was racist or not, he certainly would not have been alone.