Walters was born in New York City. His father was a British immigrant and insurance salesman. From age 6, Walters lived in Britain and France with his family. At 16, he returned to the United States and worked for his father as an insurance claims adjuster and investigator.
His formal education beyond elementary school consisted entirely of boarding school instruction at Stonyhurst College, a 400-year-old Jesuit school in Lancashire, England. He did not attend a university. In later years, he seemed to enjoy reflecting on the fact that he had risen fairly high and accomplished much despite a near-total lack of formal academic training.