Forgive the language that's considered politically incorrect as a racial slur now –it had a different, less evil meaning in the era of the story, simply meaning what 'black' or African American means today (except coming from the mouth of Butch Lovemaiden, the stereotypical Southern lawman, whose "boy," is no less demeaning)–but this explanation of Ned's as answer to the Colonel's question of behavior is priceless:
There's pride in that, and each man treats the other with respect though a difference in social status and culture is certainly accepted as a natural state of being. These families have crossed racial barriers in their bloodlines, and without hesitation acknowledge it without shame or concern, and yet the wall stands more because of social status than color, though color certainly marks the division as clearly as football jerseys at a game.
The Lost Children: A Charity Anthology