This used to be a strong racist argument in the past against interracial marriage/friendship. However, times have changed, and in a great deal of the country, even the areas with a low percentage of minorities, it is simply not that hateful. In fact, the people saying these racist things are either presently living in an area with a lot of Blacks or they are stuck in the past like the old White man who remarked to me along these lines once.
I knew two young guys on this forum who lived in Chicago and New York City respectively, and it was no wonder they were so racist. They’re simply fish out of water, and Blacks etc. tend to be non-benevolent when they are in the majority.