-
Generalised envelope spectrum-based signal-to-noise objectives: Formulation, optimisation and application for gear fault detection under time-varying speed conditions
Authors:
Stephan Schmidt,
Daniel N. Wilke,
Konstantinos C. Gryllias
Abstract:
In vibration-based condition monitoring, optimal filter design improves fault detection by enhancing weak fault signatures within vibration signals. This process involves optimising a derived objective function from a defined objective. The objectives are often based on proxy health indicators to determine the filter's parameters. However, these indicators can be compromised by irrelevant extraneo…
▽ More
In vibration-based condition monitoring, optimal filter design improves fault detection by enhancing weak fault signatures within vibration signals. This process involves optimising a derived objective function from a defined objective. The objectives are often based on proxy health indicators to determine the filter's parameters. However, these indicators can be compromised by irrelevant extraneous signal components and fluctuating operational conditions, affecting the filter's efficacy. Fault detection primarily uses the fault component's prominence in the squared envelope spectrum, quantified by a squared envelope spectrum-based signal-to-noise ratio. New optimal filter objective functions are derived from the proposed generalised envelope spectrum-based signal-to-noise objective for machines operating under variable speed conditions. Instead of optimising proxy health indicators, the optimal filter coefficients of the formulation directly maximise the squared envelope spectrum-based signal-to-noise ratio over targeted frequency bands using standard gradient-based optimisers. Four derived objective functions from the proposed objective effectively outperform five prominent methods in tests on three experimental datasets.
△ Less
Submitted 6 May, 2024; v1 submitted 26 April, 2024;
originally announced May 2024.
-
Multifidelity Surrogate Models: A New Data Fusion Perspective
Authors:
Daniel N Wilke
Abstract:
Multifidelity surrogate modelling combines data of varying accuracy and cost from different sources. It strategically uses low-fidelity models for rapid evaluations, saving computational resources, and high-fidelity models for detailed refinement. It improves decision-making by addressing uncertainties and surpassing the limits of single-fidelity models, which either oversimplify or are computatio…
▽ More
Multifidelity surrogate modelling combines data of varying accuracy and cost from different sources. It strategically uses low-fidelity models for rapid evaluations, saving computational resources, and high-fidelity models for detailed refinement. It improves decision-making by addressing uncertainties and surpassing the limits of single-fidelity models, which either oversimplify or are computationally intensive. Blending high-fidelity data for detailed responses with frequent low-fidelity data for quick approximations facilitates design optimisation in various domains.
Despite progress in interpolation, regression, enhanced sampling, error estimation, variable fidelity, and data fusion techniques, challenges persist in selecting fidelity levels and developing efficient data fusion methods. This study proposes a new fusion approach to construct multi-fidelity surrogate models by constructing gradient-only surrogates that use only gradients to construct regression surfaces. Results are demonstrated on foundational example problems that isolate and illustrate the fusion approach's efficacy, avoiding the need for complex examples that obfuscate the main concept.
△ Less
Submitted 21 April, 2024;
originally announced April 2024.
-
A spectral regularisation framework for latent variable models designed for single channel applications
Authors:
Ryan Balshaw,
P. Stephan Heyns,
Daniel N. Wilke,
Stephan Schmidt
Abstract:
Latent variable models (LVMs) are commonly used to capture the underlying dependencies, patterns, and hidden structure in observed data. Source duplication is a by-product of the data hankelisation pre-processing step common to single channel LVM applications, which hinders practical LVM utilisation. In this article, a Python package titled spectrally-regularised-LVMs is presented. The proposed pa…
▽ More
Latent variable models (LVMs) are commonly used to capture the underlying dependencies, patterns, and hidden structure in observed data. Source duplication is a by-product of the data hankelisation pre-processing step common to single channel LVM applications, which hinders practical LVM utilisation. In this article, a Python package titled spectrally-regularised-LVMs is presented. The proposed package addresses the source duplication issue via the addition of a novel spectral regularisation term. This package provides a framework for spectral regularisation in single channel LVM applications, thereby making it easier to investigate and utilise LVMs with spectral regularisation. This is achieved via the use of symbolic or explicit representations of potential LVM objective functions which are incorporated into a framework that uses spectral regularisation during the LVM parameter estimation process. The objective of this package is to provide a consistent linear LVM optimisation framework which incorporates spectral regularisation and caters to single channel time-series applications.
△ Less
Submitted 29 October, 2023;
originally announced October 2023.
-
Latent Space Perspicacity and Interpretation Enhancement (LS-PIE) Framework
Authors:
Jesse Stevens,
Daniel N. Wilke,
Itumeleng Setshedi
Abstract:
Linear latent variable models such as principal component analysis (PCA), independent component analysis (ICA), canonical correlation analysis (CCA), and factor analysis (FA) identify latent directions (or loadings) either ordered or unordered. The data is then projected onto the latent directions to obtain their projected representations (or scores). For example, PCA solvers usually rank the prin…
▽ More
Linear latent variable models such as principal component analysis (PCA), independent component analysis (ICA), canonical correlation analysis (CCA), and factor analysis (FA) identify latent directions (or loadings) either ordered or unordered. The data is then projected onto the latent directions to obtain their projected representations (or scores). For example, PCA solvers usually rank the principal directions by explaining the most to least variance, while ICA solvers usually return independent directions unordered and often with single sources spread across multiple directions as multiple sub-sources, which is of severe detriment to their usability and interpretability.
This paper proposes a general framework to enhance latent space representations for improving the interpretability of linear latent spaces. Although the concepts in this paper are language agnostic, the framework is written in Python. This framework automates the clustering and ranking of latent vectors to enhance the latent information per latent vector, as well as, the interpretation of latent vectors. Several innovative enhancements are incorporated including latent ranking (LR), latent scaling (LS), latent clustering (LC), and latent condensing (LCON).
For a specified linear latent variable model, LR ranks latent directions according to a specified metric, LS scales latent directions according to a specified metric, LC automatically clusters latent directions into a specified number of clusters, while, LCON automatically determines an appropriate number of clusters into which to condense the latent directions for a given metric. Additional functionality of the framework includes single-channel and multi-channel data sources, data preprocessing strategies such as Hankelisation to seamlessly expand the applicability of linear latent variable models (LLVMs) to a wider variety of data.
The effectiveness of LR, LS, and LCON are showcased on two crafted foundational problems with two applied latent variable models, namely, PCA and ICA.
△ Less
Submitted 10 July, 2023;
originally announced July 2023.
-
A Novel and Fully Automated Domain Transformation Scheme for Near Optimal Surrogate Construction
Authors:
Johann Bouwer,
Daniel N. Wilke,
Schalk Kok
Abstract:
Recent developments in surrogate construction predominantly focused on two strategies to improve surrogate accuracy. Firstly, component-wise domain scaling informed by cross-validation. Secondly, regression to construct response surfaces using additional information in the form of additional function-values sampled from multi-fidelity models and gradients. Component-wise domain scaling reliably im…
▽ More
Recent developments in surrogate construction predominantly focused on two strategies to improve surrogate accuracy. Firstly, component-wise domain scaling informed by cross-validation. Secondly, regression to construct response surfaces using additional information in the form of additional function-values sampled from multi-fidelity models and gradients. Component-wise domain scaling reliably improves the surrogate quality at low dimensions but has been shown to suffer from high computational costs for higher dimensional problems. The second strategy, adding gradients to train surrogates, typically results in regression surrogates. Counter-intuitively, these gradient-enhanced regression-based surrogates do not exhibit improved accuracy compared to surrogates only interpolating function values.
This study empirically establishes three main findings. Firstly, constructing the surrogate in poorly scaled domains is the predominant cause of deteriorating response surfaces when regressing with additional gradient information. Secondly, surrogate accuracy improves if the surrogates are constructed in a fully transformed domain, by scaling and rotating the original domain, not just simply scaling the domain. The domain transformation scheme should be based on the local curvature of the approximation surface and not its global curvature. Thirdly, the main benefit of gradient information is to efficiently determine the (near) optimal domain in which to construct the surrogate. This study proposes a foundational transformation algorithm that performs near-optimal transformations for lower dimensional problems. The algorithm consistently outperforms cross-validation-based component-wise domain scaling for higher dimensional problems. A carefully selected test problem set that varies between 2 and 16-dimensional problems is used to clearly demonstrate the three main findings of this study.
△ Less
Submitted 30 March, 2023;
originally announced March 2023.
-
On the momentum diffusion over multiphase surfaces with meshless methods
Authors:
Johannes C. Joubert,
Daniel N. Wilke,
Patrick Pizette
Abstract:
This work investigates the effects of the choice of momentum diffusion operator on the evolution of multiphase fluid systems resolved with Meshless Lagrangian Methods (MLM). Specifically, the effects of a non-zero viscosity gradient at multiphase interfaces are explored. This work shows that both the typical Smoothed Particle Hydrodynamics (SPH) and Generalized Finite Difference (GFD) diffusion op…
▽ More
This work investigates the effects of the choice of momentum diffusion operator on the evolution of multiphase fluid systems resolved with Meshless Lagrangian Methods (MLM). Specifically, the effects of a non-zero viscosity gradient at multiphase interfaces are explored. This work shows that both the typical Smoothed Particle Hydrodynamics (SPH) and Generalized Finite Difference (GFD) diffusion operators under-predict the shear divergence at multiphase interfaces. Furthermore, it was shown that larger viscosity ratios increase the significance of this behavior. A multiphase GFD scheme is proposed that makes use of a computationally efficient diffusion operator that accounts for the effects arising from the jump discontinuity in viscosity. This scheme is used to simulate a 3D bubble submerged in a heavier fluid with a density ratio of 2:1 and a dynamic viscosity ratio of 100:1. When comparing the effects of momentum diffusion operators, a discrepancy of 57.2% was observed in the bubble's ascent velocity.
△ Less
Submitted 17 March, 2023;
originally announced March 2023.
-
Traction chain networks: Insights beyond force chain networks for non-spherical particle systems
Authors:
Daniel N. Wilke
Abstract:
Force chain networks are generally applied in granular materials to gain insight into inter-particle granular contact. For conservative spherical particle systems, i.e. frictionless and undamped, force chains are information complete due to symmetries resulting from isotropy and constant curvature of a sphere. In fact, for conservative spherical particle systems, given the geometry and material, t…
▽ More
Force chain networks are generally applied in granular materials to gain insight into inter-particle granular contact. For conservative spherical particle systems, i.e. frictionless and undamped, force chains are information complete due to symmetries resulting from isotropy and constant curvature of a sphere. In fact, for conservative spherical particle systems, given the geometry and material, the force chain network uniquely defines the contact state that includes elastic forces, penetration distance, overlap volume, contact areas and contact pressures in a particle system. This is, however, not the case for conservative non-spherical particle systems. The reason is that a force chain network is not sufficient to uniquely define the contact state in a conservative non-spherical particle system. Additional information is required to define the contact state of non-spherical granular systems. Traction chain networks are proposed to complement force chain networks for the improved quantification of the state of contact of a granular system.
△ Less
Submitted 7 June, 2021;
originally announced June 2021.
-
GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
Authors:
Younghwan Chae,
Daniel N. Wilke,
Dominic Kafka
Abstract:
Mini-batch sub-sampling (MBSS) is favored in deep neural network training to reduce the computational cost. Still, it introduces an inherent sampling error, making the selection of appropriate learning rates challenging. The sampling errors can manifest either as a bias or variances in a line search. Dynamic MBSS re-samples a mini-batch at every function evaluation. Hence, dynamic MBSS results in…
▽ More
Mini-batch sub-sampling (MBSS) is favored in deep neural network training to reduce the computational cost. Still, it introduces an inherent sampling error, making the selection of appropriate learning rates challenging. The sampling errors can manifest either as a bias or variances in a line search. Dynamic MBSS re-samples a mini-batch at every function evaluation. Hence, dynamic MBSS results in point-wise discontinuous loss functions with smaller bias but larger variance than static sampled loss functions. However, dynamic MBSS has the advantage of having larger data throughput during training but requires the complexity regarding discontinuities to be resolved. This study extends the gradient-only surrogate (GOS), a line search method using quadratic approximation models built with only directional derivative information, for dynamic MBSS loss functions. We propose a gradient-only approximation line search (GOALS) with strong convergence characteristics with defined optimality criterion. We investigate GOALS's performance by applying it on various optimizers that include SGD, RMSprop and Adam on ResNet-18 and EfficientNetB0. We also compare GOALS's against the other existing learning rate methods. We quantify both the best performing and most robust algorithms. For the latter, we introduce a relative robust criterion that allows us to quantify the difference between an algorithm and the best performing algorithm for a given problem. The results show that training a model with the recommended learning rate for a class of search directions helps to reduce the model errors in multimodal cases.
△ Less
Submitted 23 May, 2021;
originally announced May 2021.
-
Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
Authors:
Dominic Kafka,
Daniel Nicolas Wilke
Abstract:
Gradient-only and probabilistic line searches have recently reintroduced the ability to adaptively determine learning rates in dynamic mini-batch sub-sampled neural network training. However, stochastic line searches are still in their infancy and thus call for an ongoing investigation. We study the application of the Gradient-Only Line Search that is Inexact (GOLS-I) to automatically determine th…
▽ More
Gradient-only and probabilistic line searches have recently reintroduced the ability to adaptively determine learning rates in dynamic mini-batch sub-sampled neural network training. However, stochastic line searches are still in their infancy and thus call for an ongoing investigation. We study the application of the Gradient-Only Line Search that is Inexact (GOLS-I) to automatically determine the learning rate schedule for a selection of popular neural network training algorithms, including NAG, Adagrad, Adadelta, Adam and LBFGS, with numerous shallow, deep and convolutional neural network architectures trained on different datasets with various loss functions. We find that GOLS-I's learning rate schedules are competitive with manually tuned learning rates, over seven optimization algorithms, three types of neural network architecture, 23 datasets and two loss functions. We demonstrate that algorithms, which include dominant momentum characteristics, are not well suited to be used with GOLS-I. However, we find GOLS-I to be effective in automatically determining learning rate schedules over 15 orders of magnitude, for most popular neural network training algorithms, effectively removing the need to tune the sensitive hyperparameters of learning rate schedules in neural network training.
△ Less
Submitted 29 June, 2020;
originally announced July 2020.
-
Investigating the interaction between gradient-only line searches and different activation functions
Authors:
D. Kafka,
Daniel. N. Wilke
Abstract:
Gradient-only line searches (GOLS) adaptively determine step sizes along search directions for discontinuous loss functions resulting from dynamic mini-batch sub-sampling in neural network training. Step sizes in GOLS are determined by localizing Stochastic Non-Negative Associated Gradient Projection Points (SNN-GPPs) along descent directions. These are identified by a sign change in the direction…
▽ More
Gradient-only line searches (GOLS) adaptively determine step sizes along search directions for discontinuous loss functions resulting from dynamic mini-batch sub-sampling in neural network training. Step sizes in GOLS are determined by localizing Stochastic Non-Negative Associated Gradient Projection Points (SNN-GPPs) along descent directions. These are identified by a sign change in the directional derivative from negative to positive along a descent direction. Activation functions are a significant component of neural network architectures as they introduce non-linearities essential for complex function approximations. The smoothness and continuity characteristics of the activation functions directly affect the gradient characteristics of the loss function to be optimized. Therefore, it is of interest to investigate the relationship between activation functions and different neural network architectures in the context of GOLS. We find that GOLS are robust for a range of activation functions, but sensitive to the Rectified Linear Unit (ReLU) activation function in standard feedforward architectures. The zero-derivative in ReLU's negative input domain can lead to the gradient-vector becoming sparse, which severely affects training. We show that implementing architectural features such as batch normalization and skip connections can alleviate these difficulties and benefit training with GOLS for all activation functions considered.
△ Less
Submitted 23 February, 2020;
originally announced February 2020.
-
Resolving learning rates adaptively by locating Stochastic Non-Negative Associated Gradient Projection Points using line searches
Authors:
Dominic Kafka,
Daniel N. Wilke
Abstract:
Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for neural network training algorithms. Stochastic sub-sampling during training decreases computational cost and allows the optimization algorithms to progress over loca…
▽ More
Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for neural network training algorithms. Stochastic sub-sampling during training decreases computational cost and allows the optimization algorithms to progress over local minima. However, it also results in discontinuous cost functions. Minimization line searches are not effective in this context, as they use a vanishing derivative (first order optimality condition), which often do not exist in a discontinuous cost function and therefore converge to discontinuities as opposed to minima from the data trends. Instead, we base candidate solutions along a search direction purely on gradient information, in particular by a directional derivative sign change from negative to positive (a Non-negative Associative Gradient Projection Point (NN- GPP)). Only considering a sign change from negative to positive always indicates a minimum, thus NN-GPPs contain second order information. Conversely, a vanishing gradient is purely a first order condition, which may indicate a minimum, maximum or saddle point. This insight allows the learning rate of an algorithm to be reliably resolved as the step size along a search direction, increasing convergence performance and eliminating an otherwise expensive hyperparameter.
△ Less
Submitted 14 January, 2020;
originally announced January 2020.
-
Empirical study towards understanding line search approximations for training neural networks
Authors:
Younghwan Chae,
Daniel N. Wilke
Abstract:
Choosing appropriate step sizes is critical for reducing the computational cost of training large-scale neural network models. Mini-batch sub-sampling (MBSS) is often employed for computational tractability. However, MBSS introduces a sampling error, that can manifest as a bias or variance in a line search. This is because MBSS can be performed statically, where the mini-batch is updated only when…
▽ More
Choosing appropriate step sizes is critical for reducing the computational cost of training large-scale neural network models. Mini-batch sub-sampling (MBSS) is often employed for computational tractability. However, MBSS introduces a sampling error, that can manifest as a bias or variance in a line search. This is because MBSS can be performed statically, where the mini-batch is updated only when the search direction changes, or dynamically, where the mini-batch is updated every-time the function is evaluated. Static MBSS results in a smooth loss function along a search direction, reflecting low variance but large bias in the estimated "true" (or full batch) minimum. Conversely, dynamic MBSS results in a point-wise discontinuous function, with computable gradients using backpropagation, along a search direction, reflecting high variance but lower bias in the estimated "true" (or full batch) minimum. In this study, quadratic line search approximations are considered to study the quality of function and derivative information to construct approximations for dynamic MBSS loss functions. An empirical study is conducted where function and derivative information are enforced in various ways for the quadratic approximations. The results for various neural network problems show that being selective on what information is enforced helps to reduce the variance of predicted step sizes.
△ Less
Submitted 15 September, 2019;
originally announced September 2019.
