Enhancing Tactile Material Classification with Self-

Supervised TacNet Architectures

Mohamad Marwan El Sidani Linus Eck Felix Kroll (Supervisor)
Technische Universität München Technische Universität München Technische Universität München
Munich, Germany Munich, Germany Munich, Germany
[email protected] [email protected] [email protected]

I. OBJECTIVE: III. TECHNICAL OUTLINE:.

The primary goal of this project is to augment the TacNet II A. Approach:
architecture, developed by Tulbure and Bäuml [1], which was
initially crafted for tactile material classification, by • Architecture Initialization and Baseline Training:
integrating self-supervised learning techniques. This project Initialize the TacNet II model to establish a
performance baseline, focusing on understanding its
integrates TacNet II within a Joint-Embedding Predictive
architecture and optimizing its performance with
Architecture (JEPA) to assess Assran et al.'s [2] claim on labeled tactile data. This stage lays the groundwork for
JEPA's proficiency in learning robust and abstract further adaptations and comparisons.
representations from unlabeled data, potentially
outperforming other self-supervised learning methods in • Adaptation to Self-Supervised Learning with JEPA:
tactile classification tasks. This initiative addresses the Modify TacNet II to predict tactile embeddings using
significant challenge in robotics related to the costly and the JEPA framework, refining data sampling and
time-consuming acquisition of labeled data [3]. masking techniques to enhance learning from
contextual data. Detail the theoretical adjustments and
The innovation lies in using self-supervised learning practical adaptations necessary for integrating TacNet
strategies to enhance TacNet II by integrating it as an encoder II into a self-supervised learning framework.
within the JEPA framework. We hypothesize this approach Incorporate the JEPA Framework’s encoder into an
classification framework and fine tune it on labeled
will predict tactile embeddings instead of direct sensory
data.
outputs, thus improving the system's ability to extract more
generalizable features from materials and enhancing • Adaptation to Self-Supervised Learning with an
performance in tactile material classification. Additionally, Autoencoder: Employ TacNet II as a base for
this project will validate JEPA’s superiority over traditional constructing an autoencoder, training it on unlabeled
autoencoders and test whether TacNet II, adapted to a self- data, then modifying it by removing the decoder and
supervised learning framework, produces credible results. integrating a classification head on the encoder, which
is then fine-tuned on labeled data.
II. RELATED WORK:
• Experimental Evaluation of Encoder Efficacy:
"Baishya and Bäuml [4] along with Tulbure and Bäuml [1] Conduct experiments to compare the adapted TacNet
have significantly contributed to tactile sensing for material II (as a JEPA encoder) with the autoencoder. Focus on
classification with TacNet and its enhanced version, TacNet evaluating the quality and utility of the features each
II, demonstrating high accuracy in interpreting complex model extracts through tactile data classification tasks,
spatio-temporal signals from tactile sensors. Despite their providing insights into their practical applications and
success, the reliance on extensively labeled datasets is a efficiency in classifying materials and directly testing
major impediment in dynamic real-world settings [3]. each method’s adaptation to real-world applicability.
B. Novel Contributions:
To address the challenge of extensively labeled datasets, self-
supervised learning using autoencoders is often employed • Adaptation of Self-Supervised methods to Tactile
[5], allowing the network to extract features from unlabeled Data: We are developing two distinct models: a joint-
data. Although autoencoders are effective for dimensionality embedding architecture (JEPA) and an autoencoder.
Each model is tailored to predict tactile data features,
reduction and image reconstruction, they frequently struggle
contributing to the exploration of self-supervised
to learn abstract, meaningful features essential for
methodologies in the tactile domain.
classification tasks [6].
• Benchmarking and Evaluation: Both models will
Recent advances in self-supervised learning, such as Assran undergo rigorous evaluation on tactile material
et al.'s I-JEPA [2], provide innovative methods for abstract classification tasks. Additionally, an investigation into
representation learning from images without traditional the generalization of the learned embeddings will be
labels or augmentations. While initially tailored for visual conducted. We will compare their performance against
data, applying I-JEPA's principles could greatly enhance each other and the baseline TacNet II model, aiming to
tactile sensing. This approach holds promise for improving highlight advancements in feature extraction,
material classification using unlabeled data, potentially generalization, and classification accuracy.
reducing the need for extensive labeled datasets.
 IV. REFERENCES:
[1] A. Tulbure and B. Bäuml, "Superhuman Performance in Tactile
Material Classification and Differentiation with a Flexible Pressure-
Sensitive Skin," 2018 IEEE-RAS 18th International Conference on
Humanoid Robots (Humanoids), Beijing, China, 2018, pp. 1-9
[2] M. Assran et al., "Self-Supervised Learning from Images with a
Joint-Embedding Predictive Architecture," 2023 IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC,
Canada, 2023, pp. 15619-15629
[3] T. Fredriksson, D. I. Mattos, J. Bosch, and H. H. Olsson, "Data
Labeling: An Empirical Investigation into Industrial Challenges and
Mitigation Strategies," in Product-Focused Software Process
Improvement, PROFES 2020, Lecture Notes in Computer Science, vol.
12562, M. Morisio, M. Torchiano, and A. Jedlitschka, Eds. Cham:
Springer, 2020
[4] S. S. Baishya and B. Bäuml, "Robust material classification with a
tactile skin using deep learning," 2016 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea
(South), 2016, pp. 8-15
[5] K. Berahmand, F. Daneshfar, E.S. Salehi et al. "Autoencoders and
their applications in machine learning: a survey," Artif Intell Rev 57,
2024
[6] W. Guo, J. Cai and S. Wang, "Unsupervised discriminative feature
representation via adversarial auto-encoder, " Appl Intell 50, 2020, pp.
1155–1171