So, in my book is farm run by a widow and her daughter, who happens to marry my "hero" (starts in 1858). They have black farm workers and two who take care of the house, but none are slaves. They have aspirations to go "somewhere", but where? And they like it where they are, pretty much. They get paid, some, and share in the proceeds generated by farming. The whites still live in the Big House and the blacks live in small houses which they have repaired and made to be neat.
One, the house sort of maitre d' is fairly sophisticated, but where could he go, rationally? It's a hundred miles to a slave free state, at least. A black female does the cooking and washing and ends up married to the other hero of the book, a slave that our white hero frees. So, he's a freedman now, great, and gets paid, but where can he go?
He ends up following our hero into the Confederate Army and has a remarkable event that alters history, but that's in Book Two.
Our two heroes spent on summer at his uncles in Chambersburg, PA, which has some relevance later in the book (2). The farm workers are all white and do not take kindly to the black person's presence. The uncle is a progressive sort, a "Republican".