Despite all the rebel flags – which are some “bad boy cool thing” here – actually, the majority fought for the North, with the minority fighting for the South because there simply wasn’t anything for locals, generally speaking to gain from a Southern victory. In fact, I suspect even in the Deep South (Georgia, Alabama etc.) they didn’t really want to fight for the plantation owners, but probably they were scared of a slave uprising if they didn’t!
For some history – the state of West Virginia was formed because the majority didn’t want to leave the Union. Also, Andrew Johnson was from East Tennessee (my home) and he opposed the planters because, again, despite the fact he owned slaves, he just saw the planters as oppressive to the small farmers living in his section of the state.