The extreme learning machine (ELM) neural network has got noticeable momentum in the computational intelligence and machine learning communities. However, the random initialization of the input weights and biases of classical ELM increases its sensibility to input perturbations and results in poor network stability. In this work, we propose a novel approach, named confidence random bias ELM (CRB-ELM), that inherits the randomness of the ELM for bias tuning based on confidence interval and confidence level. The experimental comparison of CRB-ELM to the classical ELM and the base projection vector machine reports that CRB-ELM achieves higher performance in classification and regression problems, being more stable over several benchmark datasets.