Cashen was born in Baltimore, Maryland in 1925 and grew up admiring and playing baseball. He played second base at Loyola College in Maryland but was unable to hit well and did not pursue a playing career. Instead, he joined the Baltimore News-American becoming an award-winning sportswriter over 17 years at the newspaper. He spent nights at University of Maryland School of Law, graduating with a JD in 1958. He assumed his career would either be in journalism or law. Instead, he was hired by Jerry Hoffberger to be a publicity director for two Baltimore race tracks and later the advertising head at Hoffberger's National Brewing Company. In 1965, Hoffberger's brewing company purchased the Baltimore Orioles and Cashen eagerly accepted the position of executive vice president of the team.