Many neural control systems are at least roughly optimized, but how is optimal control learned? There are algorithms for this purpose, but in their current forms, they are not suited for biological neural networks because they rely on a type of communication that is not available in the brain, namely, weight transport—transmitting the strengths, or “weights,” of individual synapses to other synapses and neurons. Here we show how optimal control can be learned without weight transport. Our method involves a set of simple mechanisms that can compensate for the absence of weight transport in the brain and so may be useful for neural computation generally.