Summers was born in Bremerton, Washington, served in the United States Army in Vietnam and did not play his first Major League Baseball game until he was 28 years old. He was signed by the Oakland Athletics as an amateur free agent in 1971, after being discovered in a men's softball league following his service in Vietnam.
Summers came from a family of athletes, with a father who was a prizefighter in the United States Navy and a mother who was a pro bowler. Summers received his nickname "Champ" from his father: "Dad took one look at me when I was born and said, 'He looks like he's just gone 10 rounds with Joe Louis.'"