Dimensionality reduction is an important problem class in machine learning and data mining, as the dimensionality of data sets is steadily increasing. This work is a contribution in the line of research on iterative unsupervised kernel regression (UKR), a class of methods for dimensionality reduction that employ regression methods to find low-dimensional representations of high-dimensional patterns. We introduce a hybrid optimization approach of iteratively constructing a solution and performing gradient descent in the data space reconstruction error (DSRE). Further, we introduce a variable kernel function that increases the flexibility of UKR learning. The variable kernel function increases the model capacity, but introduces new parameters that have to be tuned.