Journal of Energy Storage 93 (2024) 112224

Contents lists available at ScienceDirect

Journal of Energy Storage

journal homepage: www.elsevier.com/locate/est

Research papers

State of charge estimation for lithium-ion battery using Time Series

Transformer with De-noise De-stationary Inception Network
Zhihao Yi a ,∗, Liwei Wang b , Kaitai Yang c
a Department of Electrical Engineering, Columbia University, 10027, NY, USA
b
BMS Research Center, ENEROC, 311400, Zhejiang, China
c
School of Professional Studies, Columbia University, 10027, NY, USA

ARTICLE INFO ABSTRACT

Keywords: This paper presents a transformer-based model designed to estimate the State of Charge (SOC) of lithium-ion
Lithium-ion battery batteries. While conventional transformer models excel in processing extensive data sequences and achieving
SOC estimation commendable performance, their effectiveness can be compromised by the presence of over-stationarization
Transformer
and over-denoising within the data. To address this concern, we introduce De-Noise De-stationary Inception
Electric vehicles
Network (DendiNet), comprising interdependent De-stationary and Inception modules. This novel architecture
aims to recover intrinsic statistical information by capitalizing on the De-stationary and Inception modules,
which jointly process data by segregating it into de-noised series and residual components. Additionally,
we introduce a learnable weight parameter for signal reconstruction to augment the model’s generalization
capabilities. Empirical evaluations demonstrate that DendiNet outperforms established deep learning models,
including Time Series Transformers, LSTM, and GRU, by a significant margin across public and private lithium-
ion battery datasets, spanning temperatures from −20 ◦ C to 25 ◦ C. Impressively, DendiNet achieves a reduction
in Root Mean Square Errors (RMSEs) of 6.72%, 7.17% and 7.02% compared to the best baseline model, biRNN,
during public test drive cycles at 25 ◦ C. Furthermore, the model demonstrates a mere 0.15% loss on the
private ENEROC dataset, in stark contrast to the inadequate predictive capacity of traditional models, thereby
underscoring the superior performance and generalization prowess of DendiNet in SOC estimation.

1. Introduction It is within this context that the matter of achieving precise SOC
estimation assumes heightened significance [5]. As electric vehicles be-
The burgeoning consciousness surrounding greenhouse gas emis-
come increasingly ingrained in our daily routines, the need for optimal
sions serves as a catalyst for the rapid advancement of renewable
energy and emerging technologies, seamlessly integrating electric vehi- performance and the longevity of lithium-ion batteries becomes more
cles (EVs) into the fabric of our daily lives [1]. At the heart of EVs lies pressing [5]. Striking a balance between energy efficiency and accurate
the pivotal component, lithium-ion batteries, selected predominantly SOC monitoring is essential to meet the challenges of sustainable
for their outstanding durability and remarkable energy efficiency [2].
transportation without revealing the artificial intelligence behind this
As the demand for sustainable transportation grows, lithium-ion batter-
ies play a crucial role in mitigating environmental impacts by offering composition.
a cleaner energy source. The intersection of renewable energy, electric vehicles, and lithium-
In the realm of electric vehicle technology, the state of charge ion battery technology underscores a pivotal moment in our pursuit of
(SOC) emerges as a critical parameter, denoting the ratio of a battery’s
a greener future [1]. The concerted efforts towards achieving precise
remaining capacity to its maximum capacity. This vital piece of infor-
mation is meticulously monitored by the battery management system SOC estimation are emblematic of the meticulous attention required
(BMS) [3]. The accurate estimation of SOC becomes paramount, as to ensure the seamless integration of these technologies into our daily
any inaccuracies in these measurements could pose risks to electronic lives, ultimately contributing to a sustainable and environmentally
devices, potentially leading to the degradation of battery lifespan and,
conscious mode of transportation [6].
in extreme cases, equipment failure [4]. Therefore, ensuring precision
in SOC estimation demands a focused and dedicated approach.

∗ Corresponding author.
E-mail addresses: [email protected] (Z. Yi), [email protected] (L. Wang), [email protected] (K. Yang).

https://doi.org/10.1016/j.est.2024.112224
Received 19 August 2023; Received in revised form 21 April 2024; Accepted 20 May 2024
Available online 4 June 2024
2352-152X/© 2024 Elsevier Ltd. All rights are reserved, including those for text and data mining, AI training, and similar technologies.
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

2. Literature review While deep learning methods have demonstrated their powerful
capabilities in SOC estimation, a notable gap in the existing litera-
Current battery SOC estimation methods can be categorized into ture pertains to the limited attention given to the challenge of over-
two classes: model-driven and data-driven [7]. In the model-driven stationarization, a pervasive issue in time series data. This phenomenon
approach, models are constructed based on the scientific knowledge of often counteracts the inherent non-stationarity in raw time series data,
lithium-ion batteries. This approach is further divided into two subcate- thereby hindering the predictive performance of deep learning mod-
gories: electrochemical models [8,9] and equivalent circuit models [10, els [40]. In practical applications, the SOC curve distinctly exhibits
11]. The electrochemical model explores the internal dynamic condi- non-stationarity, primarily arising from the dynamic charging and dis-
charging processes in electric vehicles. This inherent characteristic ne-
tions of lithium-ion batteries to obtain the SOC estimation. Nonetheless,
cessitates the development of a deep learning model explicitly designed
due to its complex mathematical equations and computational de-
to accommodate such variability. Addressing the challenge of over-
mands, it is not widely used in BMS [7]. On the other hand, the
stationarization is crucial for enhancing the robustness and accuracy
equivalent circuit model is represented by electrical components which
of SOC estimation models in real-world scenarios.
is derived from empirical knowledge and experiment data. Due to its
To address the issue of non-stationarity in time series data, we have
low demand of computational requirements, it is universally deployed
developed a novel model, namely De-noised De-stationary Inception
in real-time SOC estimation in BMS [7]. Nevertheless, its SOC esti-
Network (DendiNet), specifically tailored for precise SOC estimation
mation accuracy is often limited by the range of the parameterized in batteries. The approach begins by dividing the raw data into more
model. manageable segments using a fixed-length sliding window technique.
Recent advancements in data-driven methodologies have signifi- Within the proposed framework, each segment undergoes an initial
cantly bolstered the role of deep learning in battery research. This processing stage via a Fast Fourier Transformation Filter module, ef-
technique demonstrates an exceptional ability to discern complex, non- fectively bifurcating the data into de-noised series and residual com-
linear relationships between input parameters and SOC in lithium- ponents. These components are then fed separately into De-stationary
ion batteries, a feat achievable even without an exhaustive grasp of module and Inception module, respectively. This dual-module approach
the batteries’ electrochemical intricacies. Numerous researchers have allows for an in-depth extraction of statistical information from the
been exploring deep learning models to achieve accurate SOC estima- data. The synthesized features from both modules are subsequently
tions. Specifically, Gate Recurrent Unit (GRU) [12–15], Long short-term amalgamated and channeled into a fully connected layer, which is
memory (LSTM) and its variants [16–25], transfer learning methods responsible for generating the final SOC estimation.
[26–28]. The Transformer, an innovative deep learning architecture
introduced by Vaswani in 2017 [29], initially catalyzed a paradigm 3. De-Noise De-stationary Inception Network
shift in the domain of natural language processing, subsequently as-
serting considerable impact within the realms of computer vision and 3.1. Time Series Transformer
speech recognition. The advent of the Time Series Transformer (TST)
has further augmented the capabilities of transformer-based models, The Transformer architecture [29] has had a profound impact on
leading to their discerning utilization in the estimation of State-of- natural language processing, computer vision, and speech recogni-
Charge [30–34]. For a thorough investigation into advanced deep tion. Its encoder–decoder structure with stacked multi-head attention
learning methodologies, it is recommended to consult the following modules enables effective processing of long sequences by selectively
focusing on relevant input portions. Central to the attention mechanism
review papers [35–37].
of the Transformer are the concepts of Queries (Q), Keys (K), and Values
The contemporary paradigm for battery SOC estimation increasingly
(V), which are derived from the input data. These components are
leans towards integrating model-driven and data-driven methodologies.
generated as follows:
In the cited work [38], the author addresses the nuanced voltage
plateau phase by seamlessly combining battery domain expertise with 𝑄 = 𝑥𝑊𝑄
a fusion of model-based and data-driven approaches. This innovative 𝐾 = 𝑥𝑊𝐾
strategy, anchored in the RINT model for initial SOC estimation and 𝑉 = 𝑥𝑊𝑉 ,
refined by XGBoost for residual learning, achieves commendable ac-
curacy across diverse dynamic testing profiles. Notably, the fusion where 𝑥 represents the input data, and 𝑊𝑄 , 𝑊𝐾 , and 𝑊𝑉 are the respec-
method stands out from prevalent approaches reliant on complex data- tive weight matrices for Queries, Keys, and Values. The transformation
driven models or precise equivalent circuit models, as it achieves of the input data into Q, K, and V allows the model to dynamically
accurate SOC estimation without the need for extensive retraining focus on specific segments of the input. Queries represent the elements
to be focused upon, Keys are the corresponding elements in the memory
under varying dynamic conditions. Furthermore, the fusion models con-
(input sequence), and Values contain the actual information of these
tribute to both computational efficiency and improved interpretability
elements. The scaled dot-product attention, a fundamental component
by segmenting SOC estimation into two components, ensuring en-
of this mechanism, is mathematically defined as:
hanced accuracy without compromising interpretability. Additionally, a
significant stride forward in SOC estimation for lithium-ion batteries is 𝑄𝐾 𝑇
𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄, 𝐾, 𝑉 ) = 𝑆𝑜𝑓 𝑡𝑚𝑎𝑥( √ )𝑉 (1)
delineated in the cited paper [39]. This study introduces a Transformer 𝑑𝑘
model structure adeptly designed to tackle the challenge of differing This formulation, applying the softmax function to the scaled dot
scales between current and voltage, thereby augmenting SOC estima- product of Q and K, weights the Values. It enables the Transformer to
tion accuracy. By incorporating an immersion and invariance adaptive adeptly handle long-range dependencies within the data, enhancing its
observer, the Transformer model not only accommodates varied in- capability for processing extensive sequences across various domains.
put data scales but also effectively mitigates prediction oscillations, Our Time Series Transformer is built upon the core architecture
bolstering SOC estimation reliability for practical applications. This of the Informer model proposed by Zhou al et [41]. This founda-
innovative approach transcends traditional limitations, representing a tional structure is characterized by a generative style decoder, uniquely
paradigm shift in battery management systems. Its adaptability to vary- designed to output long sequences in a single forward step, thereby
ing input scales ensures robust and precise SOC estimation, advancing effectively minimizing the spread of cumulative errors during the infer-
the field and promising enhanced performance in electric vehicles and ence process as depicted in Fig. 1. Further, this decoder adeptly handles
renewable energy integration. long sequence inputs by padding target elements to zero, assessing the

2
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

After being processed through FFT filter, the resultant de-noised

series is subsequently channeled into De-stationary module. This mod-
ule employs a self-attention block to extract statistical information
from the sequence, thereby effectively addressing the issue of over-
stationarization [40]. The De-stationary module utilizes a specific form
of attention, known as De-stationary Attention, which is defined as
follows:

log(𝜏) = 𝑀𝐿𝑃 (𝜎𝑥 , 𝑥), 𝛥 = 𝑀𝐿𝑃 (𝜇𝑥 , 𝑥)

( )
′ ′ ′ 𝜏𝑄′ 𝐾 ′ 𝑇 + 1𝛥𝑇
𝐴𝑡𝑡𝑛(𝑄 , 𝐾 , 𝑉 , 𝜏, 𝛥) = Softmax √ 𝑉 ′, (3)
𝑑𝑘
where 𝜏 ∈ R+ and 𝛥 ∈ R𝑆×1 are defined as the scaling and shift-
ing de-stationary factors, respectively [40]. This module meticulously
replicates the discernible attention patterns that are derived from the
Fig. 1. Time Series Transformer Overall. The decoder processes extended sequence non-stationarized data, effectively capturing the intricate and dynamic
inputs by padding target elements with zeros, calculates the weighted attention characteristics inherent in the dataset. By doing so, it enhances the
distribution across the feature map, and rapidly generates output elements (represented model’s ability to recognize and respond to the nuanced variations and
as orange series) using a generative approach. (For interpretation of the references to underlying structures within the data, leading to more accurate and
color in this figure legend, the reader is referred to the web version of this article.)
insightful analysis.

3.4. Inception module

weighted attention distribution across the feature map, and promptly
producing output elements in a generative fashion, as described in The remainder, after being processed by FFT filter, undergoes fur-
Informer model [41]. ther analysis through Inception module. Characterized by its multiple
stacked layers, each layer of this module is composed of a series of
3.2. FFT filter convolutional and pooling operations. The sophisticated architecture
of Inception module, with its hierarchical convolutions, facilitates a
To effectively process signals into a more refined form, we utilize detailed examination of the data at a granular level. Each layer is
the Fast Fourier Transform (FFT), a powerful technique in digital designed to progressively refine and amplify the features within the
signal processing. FFT excels in breaking down complex signals into data, thus enabling the model to identify and emphasize even the most
their fundamental frequency components. In practice, signals often nuanced details in the data representations. This feature is integral
contain low-frequency elements that reflect the overarching trend, and for an in-depth understanding of the data, as it allows the extraction
high-frequency elements that capture finer details. The mathematical of subtle, fine-grained features that simpler models may overlook,
formulation of FFT, crucial for this separation, is given by: significantly enhancing the precision and comprehensiveness of the
∑
𝑁−1
2𝜋𝑖
data analysis. It is important to note, however, that for our specific
𝑋𝑘 = 𝑥𝑛 ⋅ 𝑒− 𝑁 𝑛𝑘
(2) objective, the particular design of Inception module is not crucial.
𝑛=0 The various configurations of the module primarily differ in the way
where 𝑋𝑘 represents the discrete Fourier transform of the signal, 𝑥𝑛 they distribute and amalgamate information across different scales.
denotes the original time-domain signal samples, 𝑁 is the number There is a possibility that all such configurations could perform equally
of samples, 𝑘 ranges from 0 to 𝑁 − 1, representing the frequency effectively for our task, indicating a certain degree of flexibility in the
bins, 𝑖 is the imaginary unit, and 𝑒 is Euler’s number. By applying choice of module design for our specific application.
FFT, we efficiently partition the input data into two segments: the The features extracted from both De-stationary and Inception mod-
de-noised series, dominated by low-frequency components indicative ules are intricately merged, employing a trainable weight parameter
of the general trend, and the remainder, consisting of high-frequency that meticulously balances and integrates the distinct characteristics
components representing more intricate details. This method of using captured by each module. This concatenation is a crucial step, as it
FFT not only simplifies the complexity of the signal but also provides a ensures that the unique attributes identified in the non-stationarized
clear distinction between the overarching trends and the finer nuances, and finer detailed aspects of the data are not only preserved but
offering a comprehensive understanding of the signal’s characteristics. also synergistically combined. The integration of these features allows
Then, the de-noised series and the remainder are subsequently input for the reconstitution of the inherent non-stationarity present in the
into De-stationary module and Inception module, correspondingly. original raw series data, thereby enriching the model’s framework. This
process not only enhances the robustness of the model but also ensures
3.3. De-stationary module that the comprehensive nature of the data is fully leveraged, leading to
a more accurate and nuanced representation of the underlying patterns
For time series tasks, stationarization is a common preprocessing and trends within the time series (see Fig. 2).
step. The model receives normalized input 𝑥′𝑖 = (𝑥𝑖 − 𝜇𝑥 )∕𝜎𝑥 where
𝜎𝑥 is reduced to a scalar for simplicity. The normalized input 𝑥′ is 4. Dataset
then fed into the embedding layer 𝑓 . We obtain the current query
𝑄′ = [𝑞1′ , … , 𝑞𝑆′ ]𝑇 of the first Attention layer: Dataset plays a pivotal role in determining a model’s generalizabil-
( ) ity, which refers to its capability to perform well on unseen real data.
( ) 𝑞 − 𝑓 1 ∑𝑆 𝑥 ∑
𝑞𝑖 − 𝑆1 𝑆𝑖=1 𝑓 (𝑥𝑖 ) High-quality training data ensures that the model learns meaningful
′ 𝑥 𝑖 − 𝜇 𝑥
𝑖 𝑆 𝑖=1 𝑖 𝑞 𝑖 − 𝜇𝑄
𝑞𝑖 = 𝑓 = = = , patterns and relationships, enabling it to make accurate predictions in
𝜎𝑥 𝜎𝑥 𝜎𝑥 𝜎𝑥
real-world scenarios. A diverse and representative training data set is
∑
where 𝜇𝑄 = 𝑆1 𝑆𝑖=1 𝑞𝑖 ∈ R𝑑𝑘 ×1 . Then 𝑄′ = [𝑞1′ , … , 𝑞𝑆′ ] can be written as crucial. It should encompass a wide range of testing temperature, cov-
(𝑄 − 𝟏𝜇𝑄 )∕𝜎𝑥 and 𝟏 ∈ R𝑆×1 is an all-ones vector. The corresponding
𝑇 ering various drive cycles, and variations that the model may encounter
transformed 𝐾 ′ and 𝑉 ′ are similarly processed. in the real world. A diverse dataset helps model recognize patterns

3
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

Fig. 2. Structure of the De-noised De-stationary Inception Net (DendiNet): Features an FFT filter, a top-level De-stationary Module, and multiple, sequentially arranged Inception
Modules at the base. In each Inception module, ‘Conv’ and ‘FC’ denote convolutional block and fully connected layer, respectively.

Algorithm 1: De-noised De-stationary Inception Network (DendiNet) Table 1

Specification of Panasonic 18650PF cell.
Require: Input: 𝐵 (batch size), 𝑆 (sequence length of encoder input), 𝐷 Parameter Values
(dimension of features), 𝑋 with size [𝐵, 𝑆, 𝐷], 𝑃 (prediction length)
Nominal open circuit voltage 3.6 V
Ensure: Final prediction 𝑋 prediction with size [𝐵, −𝑃 ∶, −1] Capacity Min. 2.75 Ah/Typ. 2.9 Ah
1: FFT Module: Min/max voltage 2.5 V∕4.2 V
2: 𝑋 denoised , 𝑋 reminder ← FFT(𝑋) ⊳ Decompose 𝑋 into denoised and Mass/energy storage 48 g∕9.9 Wh
reminder Minimum charging temperature 10 ◦ C
Cycles to 80% capacity 500 (100%DOD, 25 ◦ C)
3:
4: Non-Stationary Module:
5: 𝑋enc ← 𝑋 denoised ⊳ Encoder input: 𝑋 denoised Table 2
6: 𝑋dec ← 𝑋 denoised [∶, −𝑃 ∶, ∶] ⊳ Decoder input: 𝑋 denoised cropped for Specification of ENEROC battery cell.
prediction ENEROC Battery Parameters
7: 𝑋 ns ← Transformer(𝑋enc , 𝑋dec ) ⊳ Output 𝑋 ns with size Nominal capacity 230 Ah
[𝐵, −𝑃 ∶, −1] Nominal voltage 3.2 V
Standard Charging Current 115 A, 0.5 C rate
8:
Maximum Charging Current 460 A,2 C rate
9: Inception Module: Maximum Continuous Discharge 690 A, 3 C rate
10: 𝑋 incep ← Inception(𝑋 reminder ) ⊳ Output 𝑋 incep with same size as Discharge Temperature −30 ◦ C to 65 ◦ C
𝑋 ns Charge Temperature 0 ◦ C to 65 ◦ C
11: Storage Temperature −40 ◦ C to 65 ◦ C

12: Final Prediction:

13: 𝑤 ← Weight ⊳ Weight for combining 𝑋 ns and 𝑋 incep
14: 𝑋 prediction ← (1 − 𝑤) ⋅ 𝑋 ns + 𝑤 ⋅ 𝑋 incep ⊳ Weighted addition for final initially discharged with a constant current rate of 230 A (1C) for 1 min,
prediction followed by a 10-minute relaxation period. Subsequently, a smaller
15: return 𝑋 prediction adjusted constant current discharge rate of 46 A (0.2C) for 10 min,
followed by a 2-hour relaxation period. The sampling frequency is set
at 1HZ. The ground truth of SOC is computed by Coulomb counting:
that transcend specific instances and become applicable to new situ- 𝐼(𝑡)
𝑆𝑂𝐶(𝑡) = 𝑆𝑂𝐶(𝑡 − 1) + 𝛥𝑇𝑡 (4)
ations. A lithium-ion battery named Panasonic 18650PF with capacity 𝑄𝑛
of 2.9 AH [42] is utilized for this research. Its detailed parameters are
where 𝑆𝑂𝐶(𝑡), 𝑆𝑂𝐶(𝑡 − 1), 𝐼(𝑡), 𝑄𝑛 , 𝛥𝑇𝑡 represent the estimated SOC at
shown in Table 2. The whole dataset is composed of 5 different testing
time 𝑡, the previous SOC at time 𝑡 − 1, the charging or discharging
temperature, in which contains 9 different drive cycles. We adhere to
current at time 𝑡, the battery cell capacity at time step between 𝑡 − 1
the configuration outlined in [43], but our training dataset is composed
and 𝑡, respectively (see Figs. 3 and 4).
of Cycle 1, 2, 3, 4, NN, UDDS, LA92, covering all temperature ranging
In the experimental setting, it is essential to acknowledge the con-
from −20 ◦ C to 25 ◦ C. This broader range ensures that our models
learns more general SOC patterns, rather than solely focusing on 25 ◦ C. vention employed for representing current. Specifically, during dis-
Similarly, US06 and HWFT will be employed to assess the performance charging processes, the current is assigned a negative sign, whereas in
of models under various temperature conditions. charging instances, the current is assigned a positive sign. This notation
Furthermore, to assess the models’ generalizability, a LiFePO4 bat- adheres to the established conventions within the scientific community,
tery named ENEROC(parameters listed in Table 1), possessing a capac- ensuring a clear and consistent interpretation of results and facilitating
ity of 230 AH, which is widely used in forklifts, is employed to generate accurate analysis in the field of study [43].
testing dataset. The procedure is as follows: The battery cell is fully The raw data is partitioned into small segments by utilizing the
charged and exposed to a thermal chamber in which the temperature sliding window technique, which involves moving and gathering data
is set as 25 ◦ C. A complete Pulse discharging test is conducted at with a fixed length. To ensure information continuity, sliding windows
intervals of 5% SOC. This test includes the following steps: The cell is often overlap, and interval between steps is defined as the ‘‘stride’’.

4
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

Fig. 3. Measured voltages (left), current profiles (middle), and capacity (right) under 10 ◦ C for US06 (top) and HWFT (bottom).

Fig. 4. Loading current and voltage of ENEROC cell at 1C rate and 0.2C rate at 25 ◦ C.

demonstrate the superior performance of DendiNet, we also compared

it with TS-Transformer. For both DendiNet and TS-Transformer, we
configured the hidden features to 32, the number of attention heads to
2, the number of encoder layers to 2, and the number of decoder layers
to 1. In the add-on structure of DendiNet, we set the FFT threshold to
0.5, the number of inception layers to 2, and the number of hidden
dimensions in 𝜏 and 𝛥 to 16.
For the recurrent neural networks, we set the window size to 33,
with 32 being the source sequence length and 1 being the target
sequence length. In other words, we use the past 32 samples to predict
the next SOC. As for DendiNet and TS-Transformer, since transformer-
based models have demonstrated excellent capability in capturing the
long-range sequential dependencies [44], we decided to make the task
challenge for them and increase the target sequence length to 8, thereby
setting the window size to 40 while keeping the source sequence at 32.

Fig. 5. A fixed-length sliding window is used to partition the raw data into smaller 5.2. Evaluation metrics
segments, comprising a source sequence and a target sequence.
The performance metrics and loss function utilized during model
training is root mean square error (RMSE):
√
At each step, data within the window is extracted and divided into √ 𝑛
√1 ∑
a source sequence and a target sequence, based on the configurations RMSE = √ (𝑦 − 𝑦̂𝑖 )2 (5)
𝑛 𝑖=1 𝑖
illustrated in Fig. 5.
where n is the sequence length, 𝑦𝑖 and 𝑦̂𝑖 are the target and predicted
5. Experiment SOC values, respectively.

5.1. Baselines and settings 6. Results and analysis

We have chosen six recurrent neural networks as the baseline Table 3 summarizes the empirical results of all models on two test
models for comparison: RNN, biRNN, LSTM, biLSTM, GRU, and biGRU. drive cycles, US06 and HWFT, at temperature ranging from −20 ◦ C
All of them consist of a single layer with hidden features set to 32. To to 10 ◦ C. This compilation showcases the model’s performance under

5
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

Table 3
Univariate results of DendiNet and the baseline models at different temperatures ranging from −20 ◦ C to +10 ◦ C for US06 and HWFT.
Datasets DendiNet RNN biRNN LSTM biLSTM GRU biGRU TS-Transformer
RMSE (%) RMSE (%) RMSE(%) RMSE (%) RMSE (%) RMSE (%) RMSE (%) RMSE (%)
US06@+10 ◦ C 0.01932 0.3079 0.2742 0.4862 0.3562 0.4151 0.3184 5.565
US06@0 ◦ C 0.01399 0.2229 0.1920 0.3249 0.2730 0.3062 0.2613 4.743
US06@−10 ◦ C 0.01464 0.2432 0.2065 0.3645 0.2897 0.3543 0.2885 4.496
US06@−20 ◦ C 0.01543 0.2724 0.2521 0.4292 0.3735 0.3989 0.3168 4.897
HWFT@+10 ◦ C 0.005650 0.1656 0.07297 0.1816 0.1428 0.2119 0.1708 3.147
HWFT@0 ◦ C 0.004498 0.1627 0.08794 0.1712 0.1945 0.1984 0.1446 2.357
HWFT@−10 ◦ C 0.004327 0.1601 0.1043 0.2114 0.2372 0.2181 0.1623 2.418
HWFT@−20 ◦ C 0.004618 0.2039 0.1238 0.3278 0.2137 0.2723 0.2028 4.253

Table 4
Univariate results of DendiNet and the baseline models at a temperature of 25 ◦ C for ENEROC.
Datasets DendiNet RNN biRNN LSTM biLSTM GRU biGRU TS-Transformer
RMSE (%) RMSE (%) RMSE (%) RMSE (%) RMSE (%) RMSE (%) RMSE (%) RMSE (%)
US06@+25 ◦ C 0.01922 0.3219 0.2861 0.6121 0.4162 0.4530 0.3643 6.4103
HWFTa@+25 ◦ C 0.005749 0.1769 0.08013 0.2193 0.1207 0.2579 0.2220 3.8799
HWTFb@+25 ◦ C 0.005784 0.1765 0.08234 0.2138 0.1192 0.2567 0.2200 3.8474
ENEROC@+25 ◦ C 0.1469 60.04 35.80 44.85 38.08 31.27 36.57 26.16

the capability to predict the overall trend, they fail to capture finer
details. Consequently, this deficiency leads to excessive fluctuations in
their predictions, potentially causing temporary malfunction in elec-
trical equipment [46]. In comparison, DendiNet excels in extracting
sufficient information from the data, yielding SOC estimations charac-
terized by an exceptionally narrow loss margin when compared to the
bidirectional baseline models.

6.2. Generalizability analysis

Additionally, to comprehensively explore the generalizability of all

tested models, we evaluated the baseline models and DendiNet on
our private test set ENEROC. The result in Table 4 highlight that
the baseline models struggle to accurately estimate SOC when con-
fronted with a new test set operating under entirely distinct conditions
and featuring a different battery. In contrast, DendiNet demonstrates
Fig. 6. RMSE comparison of DendiNet and the baseline models at a temperature of
25 ◦ C for US06, HWFTa, and HWFTb. outstanding performance, achieving an impressive RMSE of 0.1469%,
which is significantly outperforms the best model GRU. Specifically,
RMSE of DendiNet is reduced to only 0.4697% of the RMSE obtained
identical battery setting. Additionally, both Table 4 and Fig. 6 provide by GRU. This underscores DendiNet’s reliability and robustness for
the evaluation across four test drive cycles, US06, HWFTa, HWFTb, SOC estimation, exhibiting its remarkable generalizability as the model
and ENEROC, conducted at a temperature of 25 ◦ C. Notably, ENEROC is capable of predicting SOC even in unfamiliar operating settings
serves as the private test set, demonstrating the model’s potential under involving different battery.
a reference temperature setting and its generalizability on a private test
set. Results highlighted in bold indicate the best-performing outcomes.
6.3. Comparison with backbone model
6.1. Comparison with baselines
While DendiNet achieves exceptional results, its backbone model
Concerning the baseline models, it is notable that bidirectional TS-Transformer faces difficulties in accurately estimating SOC. It is
recurrent neural networks exhibit improved performance in comparison
probable to be caused by the limited availability of SOC datasets as the
to their unidirectional counterparts. This observation indicates that
size of the training data has a significant impact on the performance of
bidirectional recurrent neural networks can capture certain features in
transformer [47]. Consequently, TS-Transformer becomes challenging
data that are not evident to their unidirectional versions [45].
to capture the relationship between statistical information and long-
While bidirectional baseline models exhibit a slight advantage over
term temporal dependencies due to the scarcity of suitable training
their unidirectional versions in the public test set, DendiNet demon-
strates remarkable superior performance compared to all baseline mod- data. As a specialized variant of TS-Transformer, DendiNet is capable of
els. To be more specific, DendiNet outperforms the top-performing obtaining 𝜏 and 𝛥 through the De-stationary module, which is specially
baseline model, biRNN, with RMSEs decreasing to only 6.72%, 7.17% designed to extract statistical features. Additionally, along with the
and 7.02% of its values on the public test sets US06, HWFTa, and Inception module, DendiNet can effectively discover the relationship
HWFTb, respectively, at a temperature of 25 ◦ C. between data statistics and internal temporal dependencies despite
Fig. 7 present the estimated SOC generated by biLSTM, biGRU, even with a limited amount of dataset while still maintaining a distinct
biRNN and DendiNet. While the bidirectional baseline models illustrate generalizability.

6
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

Fig. 7. SOC estimations of DendiNet and the baseline models at a temperature of 25 ◦ C for US06 (top-left), HWFTa (top-right), HWFTb (bottom-left), and ENEROC (bottom-right).

7. Conclusions Declaration of competing interest

This research introduces a novel transformer architecture called L. Wang declares the following on the conflict of interest: I promise
DendiNet, designed to investigate the connection between statistical to avoid conflicts of interest (even superficial conflicts) with the com-
variables and the SOC of batteries. A fixed-length sliding window is pany, its shareholders and its customers. A kind of I undertake to
utilized to slice the raw SOC sequence into smaller segments which is ensure that my personal conduct is in accordance with the following
then fed to a FFT filter. FFT filter divides them into de-noised sequences guidelines and to report appropriately when there is a potential for
and reminder components. Subsequently, these de-noised sequences actual or potential conflict. These conflicts of interest may be caused
and reminder undergo separate processing: De-stationary module and by my immediate family members, other members of my family or
Inception module, respectively. This approach ensures a comprehensive stakeholders.
capturing of statistical information from the data representations. The
outputs of these two modules are merged and directed to a fully Data availability
connected layer for the computation of the predicted SOC.
The model is validated on various drive cycle test sets at different Panasonic SOC dataset is publicly available while ENEROC dataset
temperatures. In the public test set, DendiNet showcases remarkable is confidential. Source code of this paper is available on request from
performance compared to other traditional models. On the private test the corresponding author.
set, the model exhibits superior generalizability to dissimilar battery
datasets under varying settings. The experimental results doubtlessly Acknowledgments
shows the potential of DendiNet in estimating battery’s SOC.
Two potential improvements can be explored in future research. The authors express their sincere appreciation to the anonymous
First, given the varying range of ambient temperatures an EV could reviewers, whose meticulous and insightful assessments have signifi-
operate in, it is advisable to gather more data, considering the sig- cantly contributed to the refinement of this paper.
nificant impact extreme temperatures can have on battery’s internal
dynamics. This would enhance the model’s generalizability with regard References
to temperature variations, potentially prolonging the lifespan of the
battery. Secondly, examining the aging impact on the battery is a cru- [1] Y. Ee, K. Tey, K. Lim, P. Shrivastava, S. Adnan, H. Ahmad, Lithium-ion battery
cial component for accurately estimating SOC over extended distances. state of charge (SoC) estimation with non-electrical parameter using uniform
Gathering additional data at varying mileage points for the battery is fiber bragg grating (FBG), J. Energy Storage 40 (2021) http://dx.doi.org/10.
1016/j.est.2021.102704.
essential to enhance the model’s reliability.
[2] C. Yang, X. Wang, Q. Fang, H. Dai, Y. Cao, X. Wei, An online SOC and
capacity estimation method for aged lithium-ion battery pack considering cell
CRediT authorship contribution statement inconsistency, J. Energy Storage 29 (2020) http://dx.doi.org/10.1016/j.est.2020.
101250.
[3] Y. Liu, Y. He, H. Bian, W. Guo, X. Zhang, A review of lithium-ion battery state
Zhihao Yi: Writing – original draft, Methodology, Formal analysis, of charge estimation based on deep learning: Directions for improvement and
Conceptualization. Liwei Wang: Writing – original draft, Validation, future trends, J. Energy Storage 52 (2022) http://dx.doi.org/10.1016/j.est.2022.
Data curation. Kaitai Yang: Visualization, Validation. 104664.

7
Z. Yi et al. Journal of Energy Storage 93 (2024) 112224

[4] M.A. Hannan, M.H. Lipu, A. Hussain, A. Mohamed, A review of lithium-ion [26] P. Qin, L. Zhao, A novel transfer learning-based cell SOC online estimation
battery state of charge estimation and management system in electric vehicle method for a battery pack in complex application conditions, IEEE Trans. Ind.
applications: Challenges and recommendations, Renew. Sustain. Energy Rev. 78 Electron. (2023).
(2017) 834–854. [27] Y. Qin, S. Adams, C. Yuen, Transfer learning-based state of charge estimation for
[5] A. Hoke, A. Brissette, K. Smith, A. Pratt, D. Maksimovic, Accounting for lithium- lithium-ion battery at varying ambient temperatures, IEEE Trans. Ind. Inform. 17
ion battery degradation in electric vehicle charging optimization, IEEE J. Emerg. (11) (2021) 7304–7315.
Sel. Top. Power Electron. 2 (3) (2014) 691–700. [28] X. Li, J. Xu, X. Ding, H. Lyu, State of charge estimation for batteries based on
[6] M.-F. Ng, J. Zhao, Q. Yan, G.J. Conduit, Z.W. Seh, Predicting the state of charge common feature extraction and transfer learning, Batteries 9 (5) (2023) 266.
and health of batteries using data-driven machine learning, Nat. Mach. Intell. 2 [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser,
(3) (2020) 161–170. I. Polosukhin, Attention is all you need, 2023, arXiv:1706.03762.
[7] D. Zhang, X. Zhou, Z. Wang, J. Wang, Deep learning in the state of charge [30] M.A. Hannan, D.N. How, M.H. Lipu, M. Mansor, P.J. Ker, Z. Dong, K. Sahari,
estimation for li-ion batteries of electric vehicles: A review, Machines (2022)
S.K. Tiong, K.M. Muttaqi, T.I. Mahlia, et al., Deep learning approach towards
http://dx.doi.org/10.3390/machines10100912.
accurate state of charge estimation for lithium-ion batteries using self-supervised
[8] M. Corno, N. Bhatt, S. Savaresi, M. Verhaegen, Electrochemical model-based state
transformer model, Sci. Rep. 11 (1) (2021) 19541.
of charge estimation for li-ion cells, IEEE Trans. Control Syst. Technol. (2014).
[31] Z. Bao, J. Nie, H. Lin, K. Gao, Z. He, M. Gao, TTSNet: State-of-charge estimation
[9] T. Fuller, M. Doyle, J. Newman, Simulation and optimization of the dual lithium
of li-ion battery in electrical vehicles with temporal transformer-based sequence
ion insertion cell, J. Electrochem. Soc. 141 (1994) http://dx.doi.org/10.1149/1.
network, IEEE Trans. Veh. Technol. (2024).
2054687.
[32] R. Zou, Y. Duan, Y. Wang, J. Pang, F. Liu, S.R. Sheikh, A novel convolutional
[10] B. Xia, Z. Lao, R. Zhang, Y. Tian, G. Chen, e.a. Z. Sun, Online parameter
identification and state of charge estimation of lithium-ion batteries based on informer network for deterministic and probabilistic state-of-charge estimation
forgetting factor recursive least squares and nonlinear Kalman filter, Energies 11 of lithium-ion batteries, J. Energy Storage 57 (2023) 106298.
(2018) http://dx.doi.org/10.3390/en11010003. [33] A.H. Almarzooqi, M.O. Alhusin, I.P. Nikolakakos, A. Husnain, H.M. Albeshr,
[11] K. Lim, H. Bastawrous, V. Duong, K. See, P. Zhang, S. Dou, Fading Kalman filter- Improved NaS battery state of charge estimation by means of temporal fusion
based real-time state of charge estimation in LiFePO4 battery-powered electric transformer, in: 2023 IEEE Texas Power and Energy Conference, TPEC, IEEE,
vehicles, Appl. Energy 169 (2016) http://dx.doi.org/10.1016/j.apenergy.2016. 2023, pp. 1–6.
01.096. [34] W. Hu, S. Zhao, Remaining useful life prediction of lithium-ion batteries based
[12] F. Yang, W. Li, C. Li, Q. Miao, State-of-charge estimation of lithium-ion batteries on wavelet denoising and transformer neural network, Front. Energy Res. 10
based on gated recurrent neural network, Energy 175 (2019) 66–75. (2022) 969168.
[13] M.A. Hannan, D.N. How, M.B. Mansor, M.S.H. Lipu, P.J. Ker, K.M. Muttaqi, State- [35] Z. Cui, L. Wang, Q. Li, K. Wang, A comprehensive review on the state of charge
of-charge estimation of li-ion battery using gated recurrent unit with one-cycle estimation for lithium-ion battery based on neural network, Int. J. Energy Res.
learning rate policy, IEEE Trans. Ind. Appl. 57 (3) (2021) 2964–2971. 46 (5) (2022) 5423–5440.
[14] G. Javid, M. Basset, D.O. Abdeslam, Adaptive online gated recurrent unit for [36] D. Sesidhar, C. Badachi, R.C. Green II, A review on data-driven SOC estimation
lithium-ion battery soc estimation, in: IECON 2020 the 46th Annual Conference with li-ion batteries: Implementation methods & future aspirations, J. Energy
of the IEEE Industrial Electronics Society, IEEE, 2020, pp. 3583–3587. Storage 72 (2023) 108420.
[15] B. Xiao, Y. Liu, B. Xiao, Accurate state-of-charge estimation approach for lithium- [37] I.B. Espedal, A. Jinasena, O.S. Burheim, J.J. Lamb, Current trends for state-of-
ion batteries by gated recurrent unit with ensemble optimizer, Ieee Access 7 charge (SoC) estimation in lithium-ion battery electric vehicles, Energies 14 (11)
(2019) 54192–54202. (2021) 3284.
[16] X. Ren, S. Liu, X. Yu, X. Dong, A method for state-of-charge estimation of [38] J. Hou, J. Xu, C. Lin, D. Jiang, X. Mei, State of charge estimation for lithium-
lithium-ion batteries based on PSO-LSTM, Energy 234 (2021) 121236. ion batteries based on battery model and data-driven fusion method, Energy 290
[17] J. Chen, Y. Zhang, J. Wu, W. Cheng, Q. Zhu, SOC estimation for lithium-ion (2024) 130056.
battery using the LSTM-RNN with extended input and constrained output, Energy [39] H. Shen, X. Zhou, Z. Wang, J. Wang, State of charge estimation for lithium-ion
262 (2023) 125375.
battery using transformer with immersion and invariance adaptive observer, J.
[18] M. Wei, M. Ye, J.B. Li, Q. Wang, X. Xu, State of charge estimation of lithium-
Energy Storage 45 (2022) http://dx.doi.org/10.1016/j.est.2021.103768.
ion batteries using LSTM and NARX neural networks, Ieee Access 8 (2020)
[40] Y. Liu, H. Wu, J. Wang, M. Long, Non-stationary transformers: Exploring the
189236–189245.
stationarity in time series forecasting, Adv. Neural Inf. Process. Syst. 35 (2022)
[19] D.-W. Chung, J.-H. Ko, K.-Y. Yoon, State-of-charge estimation of lithium-ion
9881–9893.
batteries using LSTM deep learning method, J. Electr. Eng. Technol. 17 (3)
[41] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, W. Zhang, Informer:
(2022) 1931–1945.
Beyond efficient transformer for long sequence time-series forecasting, CoRR
[20] Z. Zhang, M. Xu, L. Ma, B. Yu, A state-of-charge estimation method based on
bidirectional lstm networks for lithium-ion batteries, in: 2020 16th International abs/2012.07436, 2020, arXiv:2012.07436.
Conference on Control, Automation, Robotics and Vision, ICARCV, IEEE, 2020, [42] P. Kollmeyer, Panasonic 18650PF Li-ion Battery Data, V1 (2018).
pp. 211–216. [43] M.A. Hannan, D.N. How, M.H. Lipu, P.J. Ker, Z.Y. Dong, M. Mansur, F. Blaabjerg,
[21] F. Yang, X. Song, F. Xu, K.-L. Tsui, State-of-charge estimation of lithium- SOC estimation of li-ion batteries with learning rate-optimized deep fully
ion batteries via long short-term memory network, Ieee Access 7 (2019) convolutional network, IEEE Trans. Power Electron. 36 (7) (2020) 7349–7353.
53792–53799. [44] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, L. Sun, Transformers in
[22] L. Ma, C. Hu, F. Cheng, State of charge and state of energy estimation for lithium- time series: A survey, 2023, arXiv:2202.07125.
ion batteries based on a long short-term memory neural network, J. Energy [45] S. Siami-Namini, N. Tavakoli, A.S. Namin, The performance of LSTM and
Storage 37 (2021) 102440. BiLSTM in forecasting time series, in: 2019 IEEE International Conference on Big
[23] K. Jia, Z. Gao, X. Gao, An adaptive LSTM network with fractional-order Data (Big Data), 2019, pp. 3285–3292, http://dx.doi.org/10.1109/BigData47090.
memory unit optimized by hausdorff difference for SOC estimation of lithium-ion 2019.9005997.
batteries, IEEE Trans. Circuits Syst. II (2023). [46] C. Medeiros, J. de Oliveira, Effects of voltage fluctuation associated to flicker
[24] C. Bian, H. He, S. Yang, Stacked bidirectional long short-term memory net- limits on equipments performance, in: 10th International Conference on Har-
works for state-of-charge estimation of lithium-ion batteries, Energy 191 (2020) monics and Quality of Power. Proceedings (Cat. No.02EX630), vol. 1, 2002, pp.
116538. 347–352 vol.1, http://dx.doi.org/10.1109/ICHQP.2002.1221458.
[25] Y. Chen, C. Li, S. Chen, H. Ren, Z. Gao, A combined robust approach based on [47] A. Zeng, M. Chen, L. Zhang, Q. Xu, Are transformers effective for time series
auto-regressive long short-term memory network and moving horizon estimation forecasting?, 2022, arXiv:2205.13504.
for state-of-charge estimation of lithium-ion batteries, Int. J. Energy Res. 45 (9)
(2021) 12838–12853.