Buff is the son of the architect Conrad Buff III and the grandson of children's book creators Mary and Conrad Buff (painter Conrad Buff II). He was born in Los Angeles. Buff attended Pasadena City College for two years before joining the U. S. Navy. Buff learned film editing while working for the Navy's Motion Picture Office in Hollywood. In the first phase of his civilian career Buff was the "visual effects editor" on several successful films, ranging from The Empire Strikes Back (1980) through Ghostbusters (1984). Buff was an assistant editor on Return of the Jedi (1983); he worked with editor Sean Barton and director Richard Marquand. His first editing credit was as the co-editor with Barton for Jagged Edge (1985), which was also directed by Marquand.