Frederick Douglass in his Life and Times of Frederick Douglass (1881) (Chapter 20) describes an incident when he was attacked while working at a Baltimore, Maryland shipyard while he was still a slave, that sheds a lot of light on the dynamics of white racism among non-slaveholders under the slavery system. His current master had allowed him to take a job, though a large portion of his wages had to be remitted to the master.
The facts which led to this brutal outrage upon me, illustrate a phase of slavery which was destined to become an important element in the overthrow of the slave system, and I may therefore state them with some minuteness. That phase was this--the conflict of slavery with the interests of white mechanics and laborers. In the country this conflict was not so apparent; but in cities, such as Baltimore, Richmond, New Orleans, Mobile, etc., it was seen pretty clearly. The slaveholders, with a craftiness peculiar to themselves, by encouraging the enmity of the poor laboring white man against the blacks, succeeded in making the said white man almost as much a slave as the black slave himself. The difference between the white slave and the black slave was this: the latter belonged to one slaveholder, and the former belonged to the slaveholders collectively. The white slave had taken from him by indirection what the black slave had taken from him directly and without ceremony. Both were plundered, and by the same plunderers. The slave was robbed by his master of all his earnings, above what was required for his bare physical necessities, and the white laboring man was robbed by the slave system, of the just results of his labor, because he was flung into competition with a class of laborers who worked without wages. The slaveholders blinded them to this competition by keeping alive their prejudice against the slaves as men--not against them as slaves. They appealed to their pride, often denouncing emancipation as tending to place the white working man on an equality with negroes, and by this means they succeeded in drawing off the minds of the poor whites from the real fact, that by the rich slave-master, they were already regarded as but a single remove from equality with the slave. The impression was cunningly made that slavery was the only power that could prevent the laboring white man from falling to the level of the slave's poverty and degradation. To make this enmity deep and broad between the slave and the poor white man, the latter was allowed to abuseand whip the former without hindrance. But, as I have said, this state of affairs prevailed mostly in the country. In the city of Baltimore there were not unfrequent murmurs that educating slaves to be mechanics might, in the end, give slave-masters power to dispose altogether with the services of the poor white man. But with characteristic dread of offending the slaveholders, these poor white mechanics in Mr. Gardiner's ship-yard, instead of applying the natural, honest remedy for the apprehended evil, and objecting at once to work there by the side of slaves, made a cowardly attack upon the free colored mechanics, saying they were eating the bread which should be eaten by American freemen, and swearing that they would not work with them. The feeling was really against having their labor brought into competition with that of the colored freeman, and aimed to prevent him from serving himself, in the evening of life, with the trade with which he had served his master, during the more vigorous portion of his days. Had they succeeded in driving the black freemen out of the ship-yard, they would have determined also upon the removal of the black slaves. The feeling was very bitter, toward all colored people in Baltimore about this time (1836), and they-- free and slave--suffered all manner of insult and wrong. (my emphasis)
It's also important to keep in mind that racism as a doctrine of white superiority, as distinct from prejudices and hatreds, mostly began developing in the early 19th century. And changes in technology and organization in the slave states also encouraged a shift in the arguments and beliefs used to justify slavery. As Bruce Franklin wrote in The Victim as Criminal and Artist (1978):
Slavery, as we now recognize, went through a fundamental change around 1830, completing its evolution from a predominantly small-scale, quasi-domestic institution appended to hand-tool farming and manufacture into the productive base of an expanding agricultural economy, utilizing machinery to process the harvested crops and pouring vast quantities of agricultural raw materials, principally cotton, into developing capitalist industry in the northern states and England. Prior to the 1830s, open assertions of the "permanent inferiority" of Blacks "were exceedingly rare." [The quotation is from The Black Image in the White Mind (1971) by George Fredrickson.] In fact, many eighteenth-century and early nineteenth-century apologists for slavery defended it as a means of "raising"and "civilizing" the poor, benighted, childlike Negro. But in the 1830s there emerged in America a world-view based on the belief that Blacks were inherently a race inferior to whites, and as part of this world-view there developed a scientific theory of Blacks as beings halfway, or even less than halfway, between animals and white people. This was part of the shift of Blacks from their role as children, appropriate to a professedly patriarchal society which offered them the opportunity of eventual development into adulthood, into their role as subhuman beasts of burden, the permanent mainstay of the labor force of expanding agribusiness. (my emphasis)
This is important in understanding the general context in which the Civil War occurred, especially since the Lost Cause dogma actually has clouded the nature of the source of the sectional conflict in popular understanding. The older, patriarchal justifications for slavery as a long-term program to uplift the blacks, as cynical and hyprocritical as it was for many, was still common as a defense of the Peculiar Institution in Upper South states like Virginia and Maryland.
But in the other slave states, slavery in the decades just preceding the Civil War was justified as a necessary and permanent institution required by the alleged biological inferiority of blacks. The fact that some slaves notoriously had half or more white heritage in their bloodlines didn't seem to affect this justification at all.
(See the Index to Confederate "Heritage" Month post 2005 for links to all this year's posts.)