The refractive index (RI) of a solid depends on the illumination wavelength, temperature and material properties, such as the chemical composition, crystal structure, and isotropy. RI measurements, however, also depend on environmental conditions, such as the temperature, pressure, CO2 concentration and humidity of the surrounding air. As these environmental conditions are not always reported, reported values of the RI are often irreproducible. Here we describe a new optical set-up based on the angle of minimum deviation to traceably measure the RI at controlled temperature, pressure, humidity, and CO2 concentration of the surrounding air. Advantages of the reported method are that (I) we perform RI measurements without the need for an independent measurement of the prism angles, and (II) correlations in the uncertainty propagation reduce the sensitivity coefficients greatly. The absolute RI of fused silica at 20.00 °C is 1.470 091 at 404.66 nm, 1.467 169 at 435.83 nm, 1.460 459 at 546.07 nm, and 1.459 237 at 579.07 nm. The expanded uncertainty (k = 2) of the set-up and procedure is 1.4 × 10−6 for 404.66 nm, 435.83 nm, and 546.07 nm and 1.7 × 10−6 for 579.07 nm. The main factors affecting the expanded uncertainty are the calibration uncertainty of the rotary stage, and the repeatability of the measurement.