Chaney was a native of Meridian and the eldest son in a family of five children. His mother, a domestic servant, was protective; his father, a plasterer, left his mother when James was in his mid-teens. He was slightly built, but athletic. He was described as shy in public, but a cutup in his home.
Chaney first encountered problems at the Catholic school for Negroes he attended in 1959, when he was sixteen. Chaney was suspended for a week when he refused to remove a yellow paper NAACP "button." The next year he was expelled from school for fighting. Chaney tried to join the army, but his asthma resulted in a 4-F disqualification. Unemployed and restless, Chaney joined the Negro plasterer's union, where he apprenticed with his father. His work as a plasterer ended in 1963 after a fight with his father.
According to William Huie, Chaney "outside of the Movement was a Nobody facing a lifetime of being a "boy" helper to a white carpenter or painter or plumber. Inside the Movement he was Somebody; people would listen to him and he had something to contribute."