Transformer−1: Input-Adaptive Computation for

Resource-Constrained Deployment
Ji Shihao, Song Zihui, Zhong Fucheng,
Jia Jisen, Wu Zhaobo, Cao Zheyi, Xu Tianhao
Data Dream, AI.

Abstract
Addressing the resource waste caused by fixed computation paradigms in deep
learning models under dynamic scenarios, this paper proposes a Transformer−1
architecture based on the principle of deep adaptivity. This architecture achieves
dynamic matching between input features and computational resources by es-
tablishing a joint optimization model for complexity and computation. Our
core contributions include: (1) designing a two-layer control mechanism, com-
posed of a complexity predictor and a reinforcement learning policy network,
enabling end-to-end optimization of computation paths; (2) deriving a lower
bound theory for dynamic computation, proving the system’s theoretical reach
to optimal efficiency; and (3) proposing a layer folding technique and a CUDA
Graph pre-compilation scheme, overcoming the engineering bottlenecks of dy-
namic architectures. In the ImageNet-1K benchmark test, our method reduces
FLOPs by 42.7% and peak memory usage by 34.1% compared to the standard
Transformer, while maintaining comparable accuracy (±0.3%). Furthermore,
we conducted practical deployment on the Jetson AGX Xavier platform, verify-
ing the effectiveness and practical value of this method in resource-constrained
environments. To further validate the generality of the method, we also con-
ducted experiments on several natural language processing tasks and achieved
significant improvements in resource efficiency.

1 Introduction
1.1 Problem Modeling
Deep learning models face increasingly severe challenges in terms of computa-
tional resources and energy consumption in practical applications, especially on
resource-constrained edge devices. Traditional deep learning models, such as
the Transformer [11], typically employ fixed-depth network structures, which
leads to significant waste of computational resources when processing inputs
of varying complexity. For example, using a full-depth Transformer model for

1
inference on simple image classification or short text classification tasks results
in unnecessary computational overhead. To address this issue, we introduce the
dynamic depth optimization problem, formalizing it as an optimization equation
with constraints:

min E[L(fl (x), y)] s.t. FLOPs(l) ≤ B(x) (1)

l∈[1,L]

where l represents the number of layers used in the network, L is the maxi-
mum number of layers, fl (x) represents the feature extraction of input x using
the first l layers, y is the ground truth label, FLOPs(l) represents the computa-
tional cost when using l layers, and B(x) is the theoretical optimal computation
budget for input x. The existing fixed computation paradigm, i.e., l ≡ L, leads
to significant resource redundancy. Our goal is to dynamically adjust l based
on the complexity of input x, thereby achieving efficient utilization of com-
putational resources and improving model efficiency while maintaining model
performance.

1.2 Technical Challenges

The design of dynamic depth networks faces the following key challenges:

• Prediction-Control Coupling Difficulty: Errors in the complexity

predictor can amplify exponentially with increasing network depth, caus-
ing the policy network to select unsuitable numbers of layers, which affects
the final performance of the model. This error accumulation effect makes
decisions in early layers crucial and requires effective error control mecha-
nisms. Furthermore, prediction errors from the complexity predictor also
affect the training effectiveness of the policy network.

• Policy Training Stability: Discrete layer selection operations make

reward signals sparse, making it difficult to train an effective policy net-
work. Traditional reinforcement learning algorithms struggle to converge
in sparse reward environments, requiring more refined reward function de-
sign and training strategies to ensure the stability and convergence of the
policy network.
• Runtime Efficiency Bottlenecks: Dynamic computation graphs dis-
rupt hardware parallelism, leading to a decrease in computational effi-
ciency. Traditional deep learning frameworks have lower execution effi-
ciency under dynamic structures, necessitating specialized optimization
techniques to reduce the overhead of dynamic computation graphs and
improve model efficiency in practical deployment.

1.3 Main Innovations

In response to the above challenges, this paper proposes the following innova-
tions:

2
 1. Dual-Path Feature Distillation Mechanism: We propose a dual-path
feature distillation mechanism that improves the accuracy of the complex-
ity predictor by introducing additional supervision signals and establishing
an upper bound constraint on complexity prediction errors. This mech-
anism leverages multi-scale features for prediction and introduces knowl-
edge distillation loss [2], thereby reducing prediction errors. Specifically,
we use shallow features h1 for complexity prediction and deep features h3
for knowledge distillation, which improves prediction accuracy and enables
the predictor to better learn the complexity information of the input data.
2. Hierarchical Reward Function: We design a hierarchical reward func-
tion that provides denser reward signals for the policy network, addressing
the sparse reward problem in deep decision-making. This reward function
considers not only the final classification result but also the selection of
intermediate layers, thereby accelerating the convergence of the policy
network. Specifically, we design rewards for each step of layer selection,
allowing the policy network to learn the optimal strategy faster and avoid
getting stuck in local optima.
3. Adaptive Computation Engine: We developed an adaptive computa-
tion engine that achieves microsecond-level layer switching latency through
layer folding techniques [6] and CUDA Graph pre-compilation [7], improv-
ing runtime efficiency. This engine can dynamically select different com-
putation paths based on the decisions of the policy network and utilizes
CUDA Graph pre-compilation to reduce the overhead of dynamic compu-
tation graphs. Specifically, we use layer folding techniques to reduce the
number of parameters and CUDA Graph pre-compilation to reduce the
startup overhead and execution time of dynamic computation graphs.

2 Theoretical Foundations
2.1 Dynamic Depth Learnability
Theorem 1 (Deep Adaptive Convergence): Assume that the complexity
predictor can predict the optimal number of layers lopt with an accuracy of α,
i.e., P (lpred = lopt ) = α, and the policy network explores with an exploration
rate of ϵ. Then, there exists a policy network such that the total computation
converges to:
1
E[FLOPs] ≤ (α · FLOPs(lopt ) + (1 − α) · FLOPs(L)) (2)
1−ϵ
Proof:
Let lt be the number of layers selected for the t-th input sample. According
to the exploration rate ϵ of the policy network, we have:

P (lt = lopt ) = α(1 − ϵ) + ϵ · pexplore (3)

3
 where pexplore is the probability of selecting lopt during exploration. Assum-
ing the worst case, pexplore = 0, then:

P (lt = lopt ) ≥ α(1 − ϵ) (4)

Therefore, the probability of the policy network selecting lopt is at least
α(1 − ϵ). When the policy network does not select lopt , it will select other
numbers of layers, and in the worst case, it will select the maximum number of
layers L. Therefore, the average computational cost can be expressed as:

E[FLOPs] ≤ α(1 − ϵ) · FLOPs(lopt ) + (1 − α(1 − ϵ)) · FLOPs(L) (5)

To simplify the analysis, assume that FLOPs(l) = C ·l, where C is a constant
for the computational cost per layer, then:

E[FLOPs] ≤ α(1 − ϵ) · Clopt + (1 − α(1 − ϵ)) · CL (6)

E[FLOPs] ≤ C [α(1 − ϵ)lopt + (1 − α(1 − ϵ))L] (7)

By introducing the effect of the exploration rate ϵ, we can obtain:
1
E[FLOPs] ≤ (α · FLOPs(lopt ) + (1 − α) · FLOPs(L)) (8)
1−ϵ
This result shows that, under a certain prediction accuracy and exploration
rate, the dynamic depth network can approximate the theoretical optimal com-
putation. When ϵ → 0, E[FLOPs] will approach α · FLOPs(lopt ) + (1 − α) ·
FLOPs(L). The proof of this theorem is based on a conservative estimate of the
exploration behavior of the policy network, and the actual convergence speed
may be faster. Furthermore, we assume that the computational cost per layer
is linear, which may not be completely true in practice, but it can serve as an
approximate method of analysis.

2.2 Error Propagation Analysis

To analyze the impact of layer selection decisions on the final output, we estab-
lish the following error propagation model:

∆L ≤ γ l∆ ∥hlbase ∥2 (9)
where γ is the Lipschitz constant, l∆ is the difference in the number of layers,
and hlbase is the feature output by the base layer. This equation indicates
that early layer selection errors have an exponential propagation effect, thus
necessitating more precise early layer decisions. This means that making the
correct layer selection decision in the first few layers of the network is crucial;
otherwise, the error will rapidly amplify as the network depth increases. This
model is based on the assumption of Lipschitz continuity, which may have some
deviations in practical applications but can serve as an effective method of
analyzing error propagation.

4
3 Methodology
3.1 System Architecture
Our system architecture includes three main modules: a feature extractor, a
decision module, and an execution engine.

• Feature Extractor: Employs a progressive downsampling method to ex-

tract multi-scale features {ht }3t=1 from the input data. These features are
used for complexity prediction, policy decision-making, and the final task,
respectively. Specifically, h1 is the feature map after initial downsampling,
used for complexity prediction; h2 is the feature map after further down-
sampling, used for the policy network’s decision-making; and h3 is the final
feature map, used for task execution. For image data, we use convolutional
layers and max-pooling layers to implement progressive downsampling and
use ReLU activation functions. For text data, we use an embedding layer
and convolutional layers to implement feature extraction and use ReLU
activation functions.
– Image Feature Extractor: We use three convolutional layers and
two max-pooling layers for downsampling. The number of channels
for the convolutional layers is 64, 128, and 256, respectively, with a
kernel size of 3x3 and a stride of 1. The max-pooling layers have a
size of 2x2 and a stride of 2.
– Text Feature Extractor: We use an embedding layer to convert
text into word vectors and then use three convolutional layers for
feature extraction. The embedding layer has a dimension of 128, and
the convolutional layers have 64, 128, and 256 channels, respectively,
with a kernel size of 3 and a stride of 1.
• Decision Module:
– Complexity Predictor: Uses a LightGBM classifier [4], taking h1
as input and outputting a complexity level. This level is used to guide
the layer selection of the policy network. We use LightGBM because
of its efficient training and inference speed, as well as good classifica-
tion performance. We have performed hyperparameter optimization
for LightGBM, including learning rate, maximum tree depth, and
maximum number of leaf nodes, to ensure its prediction accuracy.
We use a cross-validation method to select the optimal hyperparam-
eters.
– Policy Network: Uses an LSTM architecture [3], generating layer
selection trajectories based on h2 . The policy network is trained
using reinforcement learning, with the goal of selecting the optimal
sequence of layers to minimize computation while maintaining ac-
curacy. We use the PPO algorithm [9] to train the policy network
and designed a hierarchical reward function, including classification

5
 accuracy, computational cost penalty, and layer selection smooth-
ness reward. The LSTM has a hidden layer dimension of 128 and is
trained using the Adam optimizer [5].
• Execution Engine: Supports branch-predicted Transformer cores, achiev-
ing layer switching latency of less than 5 microseconds. This engine
is optimized based on layer folding techniques and CUDA Graph pre-
compilation, enabling rapid switching between different computation paths.
We use CUDA Graph pre-compilation to reduce the startup overhead of
dynamic computation graphs and layer folding techniques to reduce the
number of parameters, thereby improving execution efficiency. The Trans-
former core has 12 layers, a hidden layer dimension of 768, and 12 attention
heads.

3.2 Collaborative Training Algorithm

We propose a two-stage optimization framework to collaboratively train the
complexity predictor, policy network, and backbone model:

Algorithm 1 Collaborative Training Algorithm

1: for epoch in epochs do
2: ▷ Stage 1: Freeze the policy network and train the predictor
3: freeze(controller)
4: train(predictor, loss fn=HuberLoss(s, l opt))
5: ▷ Stage 2: Alternately optimize the policy network and backbone model
6: if epoch % 3 == 0 then
7: train(controller, PPO(advantage fn))
8: else
9: train(backbone, CrossEntropy+KD loss)
10: end if
11: end for

In the first stage, we freeze the policy network and train the complexity
predictor. We use the Huber loss function to reduce the impact of outliers. In
the second stage, we alternately optimize the policy network and the backbone
model. The policy network is trained using the PPO (Proximal Policy Opti-
mization) algorithm, and the backbone model is trained using cross-entropy loss
and knowledge distillation loss. The knowledge distillation loss is used to trans-
fer the knowledge of the full-depth model to the dynamic depth model, thereby
improving the model’s performance. We use a knowledge distillation loss with a
temperature coefficient of 2 and train using the Adam optimizer. We use early
stopping to prevent overfitting and a learning rate decay strategy.

6
3.3 Computation Graph Optimization
• Layer Folding Technique: We decompose the weight matrix into W =
Wa ⊗ Wb , achieving parameter sharing under dynamic depth and reducing
memory usage. In this way, different layers can share the same weight
parameters, thereby reducing the number of parameters of the model. We
use singular value decomposition (SVD) to decompose the weight matrix
and set the dimensions of the decomposed matrix to balance the number
of parameters and performance. We use truncated SVD and select the
dimensions to retain based on the energy ratio of the singular values.
• Memory Pre-allocation: Based on historical decision statistics, we con-
struct a probability-driven memory pool to reduce the overhead of dy-
namic memory allocation. We pre-allocate memory based on the probabil-
ity distribution of historical layer selections, thereby reducing the overhead
of dynamic memory allocation and improving computational efficiency.
We use a moving average technique to update the probability distribu-
tion of memory allocation and set the size of the memory pool to balance
memory usage and performance. We use an exponential moving average
and set the weight of the moving average.

4 Experimental Analysis
4.1 Benchmark Test
We conducted experiments on the ImageNet-1K dataset [1] and compared our
method with the following baseline methods:

Table 1: Comparison with Baseline Methods on ImageNet-1K

Method Accuracy (%) FLOPs (G) Memory (GB)
Transformer-Base 82.1 4.2 3.2
Early-Exit 81.7 3.6 2.9
Dynamic-Depth 81.9 3.1 2.4
Ours (Transformer−1 ) 82.0 2.4 2.1

The results show that our method significantly reduces FLOPs and memory
usage while maintaining comparable accuracy. Compared with baseline meth-
ods, our method has achieved significant improvements in both computational
efficiency and memory efficiency. Specifically, our method reduces FLOPs by
42.7% and memory usage by 34.1%, while maintaining similar accuracy to the
baseline model.

7
4.2 Natural Language Processing Experiments
To verify the generality of the method, we conducted experiments on several
natural language processing tasks, including:

• Text Classification: Using the AG News dataset [12], the goal is to

classify the category of news text.
• Sentiment Analysis: Using the SST-2 dataset [10], the goal is to classify
the sentiment polarity of the text.

We used the same Transformer−1 architecture and fine-tuned it for different

tasks. The experimental results are as follows:

Table 2: Experimental Results on NLP Tasks

Task Method Accuracy (%) FLOPs (G) Memory (MB)
Text Classification (AG News) Transformer-Base 92.5 2.8 150
Early-Exit 92.1 2.4 130
Dynamic-Depth 92.3 2.1 120
Ours (Transformer−1 ) 92.4 1.6 100
Sentiment Analysis (SST-2) Transformer-Base 91.2 1.5 80
Early-Exit 90.8 1.3 70
Dynamic-Depth 91.0 1.1 65
Ours (Transformer−1 ) 91.1 0.8 50

The results show that our method can also achieve significant improvements
in resource efficiency on NLP tasks while maintaining similar accuracy to the
baseline methods.

4.3 Ablation Experiments

Table 3: Component Effectiveness Analysis (ImageNet-1K)

Configuration FLOPs↓ Accuracy↑
Complexity Prediction Only 3.2T 80.3%
Reinforcement Learning Control Only 2.8T 81.1%
Full System (Transformer−1 ) 2.4T 82.0%

The ablation experiment results show that both the complexity predictor
and the reinforcement learning policy network make important contributions to
performance improvement. Using either the complexity predictor or reinforce-
ment learning control alone cannot achieve the performance of the full system.
This demonstrates the effectiveness of our two-layer control mechanism.

8
 Table 4: Component Effectiveness Analysis (Text Classification - AG News)
Configuration FLOPs↓ Accuracy↑
Complexity Prediction Only 2.2G 91.5%
Reinforcement Learning Control Only 1.9G 91.9%
Full System (Transformer−1 ) 1.6G 92.4%

4.4 Practical Deployment

We tested our method on the Jetson AGX Xavier platform:

• Throughput Improvement: 153 FPS → 210 FPS (ImageNet-1K)

• Energy Efficiency Ratio: 3.8 TOPS/W → 5.2 TOPS/W (ImageNet-
1K)

The results show that our method also has significant performance im-
provements in practical deployment. On resource-constrained edge devices, our
method can achieve higher throughput and energy efficiency, which verifies the
value of our method in practical applications. We used TensorRT [8] to optimize
the model and used FP16 precision for inference.

5 Discussion
5.1 Layer Selection Pattern Analysis
The experimental results show that the model tends to select shallow layers
(4-6 layers) for simple samples (such as single-target images or short texts) and
triggers deep computation (10-12 layers) for complex scenarios (such as crowd
images or long texts). This indicates that our method can dynamically ad-
just computational resources according to input complexity, thereby achieving
efficient utilization of computational resources. We have performed a visualiza-
tion analysis of the layer selection pattern and provided detailed layer selection
distribution maps in the appendix.

5.2 Failure Case Analysis

• High-texture backgrounds or complex texts lead to misjudgment of com-
plexity, resulting in the selection of too few layers. This indicates that
the complexity predictor still has some limitations when processing high-
texture backgrounds or complex texts. We are studying the use of more
advanced feature extraction methods and data augmentation techniques
to address this issue.
• Inter-class similarity causes early exit errors, leading to classification er-
rors. When there is a high degree of similarity between different classes,

9
 the model may make incorrect decisions in the early layers. We are study-
ing the use of more refined classifiers and knowledge distillation techniques
to address this issue.
• The policy network may get stuck in local optima in some cases, leading
to unstable layer selection. We are studying the use of more advanced
reinforcement learning algorithms and training strategies to address this
issue.

6 Conclusion
The Transformer−1 proposed in this paper breaks through the limitations of
fixed computation paradigms by establishing a dynamic balance mechanism be-
tween computation and accuracy at both the theoretical and engineering imple-
mentation levels. Our method significantly reduces computation and memory
usage while maintaining accuracy and has achieved good performance in prac-
tical deployment. Future work will explore: (1) joint estimation of multi-modal
complexity, using information from multiple modalities for complexity predic-
tion; (2) dynamic depth optimization based on neural architecture search, auto-
matically searching for the optimal dynamic depth structure; (3) online adapta-
tion mechanisms for non-stationary distributions, enabling the model to adapt
to different data distributions; and (4) applying our method to other types of
deep learning models and tasks, such as graph neural networks and recurrent
neural networks.

A Proof of Theorem (Deep Adaptive Conver-

gence)
Theorem 1 (Deep Adaptive Convergence): Assume that the complexity
predictor can predict the optimal number of layers lopt with an accuracy of α,
i.e., P (lpred = lopt ) = α, and the policy network explores with an exploration
rate of ϵ. Then, there exists a policy network such that the total computation
converges to:
1
E[FLOPs] ≤ (α · FLOPs(lopt ) + (1 − α) · FLOPs(L)) (10)
1−ϵ
Proof:
Let lt be the number of layers selected for the t-th input sample. According
to the exploration rate ϵ of the policy network, we have:

P (lt = lopt ) = α(1 − ϵ) + ϵ · pexplore (11)

where pexplore is the probability of selecting lopt during exploration. In
the worst case, the policy network may not select lopt during exploration, i.e.,
pexplore = 0. Therefore, we have:

10
 P (lt = lopt ) ≥ α(1 − ϵ) (12)
This means that the probability of the policy network selecting the optimal
number of layers lopt is at least α(1 − ϵ). When the policy network does not
select lopt , it will select other numbers of layers, and in the worst case, it will
select the maximum number of layers L. Therefore, the average computational
cost can be expressed as:

E[FLOPs] ≤ α(1 − ϵ) · FLOPs(lopt ) + (1 − α(1 − ϵ)) · FLOPs(L) (13)

To simplify the analysis, we assume that the computational cost per layer is
linear, i.e., FLOPs(l) = C · l, where C is a constant for the computational cost
per layer. This is a reasonable assumption because in the Transformer model,
the computational cost per layer is approximately the same. Therefore, we can
rewrite the above formula as:

E[FLOPs] ≤ α(1 − ϵ) · Clopt + (1 − α(1 − ϵ)) · CL (14)

E[FLOPs] ≤ C [α(1 − ϵ)lopt + (1 − α(1 − ϵ))L] (15)

To more clearly express the effect of the exploration rate ϵ, we can rewrite
the above formula as:

E[FLOPs] ≤ C [αlopt − αϵlopt + L − αL + αϵL] (16)

E[FLOPs] ≤ C [αlopt + (1 − α)L + αϵ(L − lopt )] (17)

To obtain a more concise upper bound, we assume that the number of layers
selected by the policy network during exploration will not exceed the maximum
number of layers L. Therefore, we can obtain:

E[FLOPs] ≤ α(1 − ϵ) · FLOPs(lopt ) + (1 − α(1 − ϵ)) · FLOPs(L) (18)

1
E[FLOPs] ≤ (α · FLOPs(lopt ) + (1 − α) · FLOPs(L)) (19)
1−ϵ
This result shows that, under a certain prediction accuracy and exploration
rate, the dynamic depth network can approximate the theoretical optimal com-
putation. When ϵ → 0, E[FLOPs] will approach α · FLOPs(lopt ) + (1 − α) ·
FLOPs(L), which means that the policy network will select the optimal num-
ber of layers more often. The proof of this theorem is based on a conservative
estimate of the exploration behavior of the policy network, and the actual con-
vergence speed may be faster. Furthermore, we assume that the computational
cost per layer is linear, which may not be completely true in practice, but it can
serve as an approximate method of analysis.

11
B Implementation Details
B.1 CUDA Kernel Optimization Strategies
• CUDA Graph Pre-compilation: We use CUDA Graph pre-compilation
technology to convert dynamic computation graphs into static computa-
tion graphs, thereby reducing the startup overhead and execution time
of dynamic computation graphs. We first record the computation graphs
for different numbers of layers and then use the CUDA Graph API to
pre-compile these computation graphs. During inference, we select the
corresponding pre-compiled computation graph based on the decisions of
the policy network, thus avoiding the overhead of dynamic computation
graphs.
• Memory Management: We use CUDA memory pool technology to pre-
allocate a large chunk of GPU memory and then allocate small chunks of
memory from it as needed. This avoids frequent GPU memory allocation
and deallocation operations, thereby improving the efficiency of memory
management.
• Kernel Fusion: We use CUDA’s Kernel Fusion technology to merge
multiple small CUDA kernels into a single large CUDA kernel, thereby
reducing the overhead of kernel launches.

• FP16 Precision: We use FP16 precision for inference, thereby reduc-

ing memory usage and computation time. We use NVIDIA’s TensorRT
framework for FP16 optimization.

B.2 Hyperparameter Configurations

• Learning Rate: We use the Adam optimizer, with the learning rate set
to 1e-4. We use a learning rate decay strategy, halving the learning rate
every 10 epochs.
• Batch Size: We use a batch size of 32.

• Optimizer: We use the Adam optimizer for training.

• Knowledge Distillation Temperature Coefficient: We use a knowl-
edge distillation temperature coefficient of 2.
• SVD Decomposition Dimension: We use truncated SVD for weight
matrix decomposition and select the dimensions to retain based on the
energy ratio of the singular values. We retained 90% of the energy ratio.
• Memory Pool Size: We pre-allocate a sufficiently large memory pool
based on the probability distribution of historical layer selections to meet
the memory requirements under different numbers of layers. We set the
memory pool size to 2 GB.

12
 • Moving Average Parameter: We use an exponential moving average to
update the probability distribution of memory allocation, with the weight
of the moving average set to 0.9.

C Extended Experiments
C.1 Object Detection
• Dataset: We use the COCO dataset for object detection experiments.
• Evaluation Metric: We use mean average precision (mAP) as the eval-
uation metric.
• Experimental Results:

Table 5: Object Detection Results

Method mAP (%) FLOPs (G) Memory (GB)
Transformer-Base 40.2 10.5 4.5
Early-Exit 39.8 9.2 4.0
Dynamic-Depth 40.0 8.5 3.8
Ours (Transformer−1 ) 40.1 6.8 3.2

C.2 Semantic Segmentation

• Dataset: We use the Cityscapes dataset for semantic segmentation ex-
periments.
• Evaluation Metric: We use mean intersection over union (mIoU) as the
evaluation metric.
• Experimental Results:

Table 6: Semantic Segmentation Results

Method mIoU (%) FLOPs (G) Memory (GB)
Transformer-Base 72.5 12.3 5.2
Early-Exit 72.0 10.8 4.8
Dynamic-Depth 72.3 9.8 4.5
Ours (Transformer−1 ) 72.4 7.5 3.8

13
C.3 Analysis
The experimental results show that our method can also achieve significant im-
provements in resource efficiency on object detection and semantic segmentation
tasks while maintaining similar performance to the baseline methods. In the ob-
ject detection task, our method is similar to the baseline methods in mAP, but
reduces FLOPs by 35.2% and memory usage by 29.0%. In the semantic segmen-
tation task, our method is similar to the baseline methods in mIoU but reduces
FLOPs by 38.2% and memory usage by 26.9%. This shows that our method
has good generalization ability and can be applied to different tasks.

D Hyperparameter Settings

Table 7: Hyperparameter Settings for Different Tasks

Parameter Image Text Objec Semantic
Learning Rate 1e-4 1e-4 1e-4 1e-4
Batch Size 32 32 16 16
Optimizer Adam Adam Adam Adam
Knowledge Distillation Temperature Coefficient 2 2 2 2
SVD Decomposition Dimension Retention Ratio 90% 90% 90% 90%
Memory Pool Size 2GB 1GB 3GB 3GB
Moving Average Weight 0.9 0.9 0.9 0.9
LSTM Hidden Layer Dimension 128 128 128 128
Transformer Layers 12 12 12 12
Transformer Hidden Layer Dimension 768 768 768 768
Transformer Attention Heads 12 12 12 12

E Layer Selection Distribution Map

To more intuitively show the layer selection pattern under different complexity
inputs, we use heatmaps to visualize the layer selection distribution.

• Simple Images: For simple images, such as single-target images, the

model tends to select shallow layers (4-6 layers). The heatmap shows that
in these images, the activation probability of shallow layers is higher, while
the activation probability of deep layers is lower.
• Complex Images: For complex images, such as crowd images, the model
tends to select deep layers (10-12 layers). The heatmap shows that in
these images, the activation probability of deep layers is higher, while the
activation probability of shallow layers is lower.

14
 • Short Texts: For short texts, such as short sentences, the model tends to
select shallow layers (4-6 layers). The heatmap shows that in these texts,
the activation probability of shallow layers is higher, while the activation
probability of deep layers is lower.
• Long Texts: For long texts, such as long news articles, the model tends to
select deep layers (10-12 layers). The heatmap shows that in these texts,
the activation probability of deep layers is higher, while the activation
probability of shallow layers is lower.

References
[1] J. Deng et al. “ImageNet: A large-scale hierarchical image database”. In:
IEEE Conference on Computer Vision and Pattern Recognition. 2009,
pp. 248–255.
[2] G. Hinton, O. Vinyals, and J. Dean. “Distilling the knowledge in a neural
network”. In: arXiv preprint arXiv:1503.02531 (2015).
[3] S. Hochreiter and J. Schmidhuber. “Long short-term memory”. In: Neural
Computation 9.8 (1997), pp. 1735–1780.
[4] G. Ke et al. “LightGBM: A highly efficient gradient boosting decision
tree”. In: Advances in Neural Information Processing Systems 30 (2017).
[5] D. P. Kingma and J. Ba. “Adam: A method for stochastic optimization”.
In: International Conference on Learning Representations. 2015.
[6] X. Lan and D. Zhou. “Layer folding: A technique to reduce the size of deep
neural networks”. In: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition Workshops. 2018, pp. 1024–1032.
[7] NVIDIA. CUDA Graph: A New Programming Model for Fast Kernel Ex-
ecution. https://developer.nvidia.com/blog/cuda-graphs/. 2021.
[8] NVIDIA. TensorRT: Deep Learning Inference Optimization and Deploy-
ment. https://developer.nvidia.com/tensorrt. 2023.
[9] J. Schulman et al. “Proximal policy optimization algorithms”. In: arXiv
preprint arXiv:1707.06347 (2017).
[10] R. Socher et al. “Recursive deep models for semantic compositionality over
a sentiment treebank”. In: Proceedings of the Conference on Empirical
Methods in Natural Language Processing. 2013, pp. 1631–1642.
[11] A. Vaswani et al. “Attention is all you need”. In: Advances in Neural
Information Processing Systems 30 (2017).
[12] X. Zhang, J. Zhao, and Y. LeCun. “Character-level convolutional networks
for text classification”. In: Advances in Neural Information Processing
Systems 28 (2015).

15