It's probably worth acknowledging that as a white male, I don't walk in the shoes of people who have been maligned by Rush. On the other hand, he would only be an investor in a business without the power to hire or fire people. I guess it's all a matter of where you draw the line. I personally would never work with him or be part of a business team that includes him, but that's a little different from broader social groups driving him out of a business deal where he wasn't in a position to hurt employees with his biased attitude.
I'm not sure if I completely agree with Conor Friedersdorf's article on the whole issue - while I would only accuse someone of "being a racist" where their behavior is far worse than the societal norm, I think doing or saying something racist is far more common. We've come a long way but have a long way to go on bias in our society. Rather than viewing the statement "what you just said was racist" as equivalent to an accusation of pedophilia, it would be better to examine the situation calmly, decide if it's true, rectify the situation if needed and move on.