Showing 1–50 of 1,471 results for author: Yang, Q

Hadronic cross section measurements with the DAMPE space mission using 20GeV-10TeV cosmic-ray protons and $^4$He

Authors: F. Alemanno, Q. An, P. Azzarello, F. C. T. Barbato, P. Bernardini, X. J. Bi, I. Cagnoli, M. S. Cai, E. Casilli, E. Catanzani, J. Chang, D. Y. Chen, J. L. Chen, Z. F. Chen, P. Coppin, M. Y. Cui, T. S. Cui, Y. X. Cui, H. T. Dai, A. De Benedittis, I. De Mitri, F. de Palma, A. Di Giovanni, Q. Ding, T. K. Dong , et al. (126 additional authors not shown)

Abstract: Precise direct cosmic-ray (CR) measurements provide an important probe to study the energetic particle sources in our Galaxy, and the interstellar environment through which these particles propagate. Uncertainties on hadronic models, ion-nucleon cross sections in particular, are currently the limiting factor towards obtaining more accurate CR ion flux measurements with calorimetric space-based exp… ▽ More Precise direct cosmic-ray (CR) measurements provide an important probe to study the energetic particle sources in our Galaxy, and the interstellar environment through which these particles propagate. Uncertainties on hadronic models, ion-nucleon cross sections in particular, are currently the limiting factor towards obtaining more accurate CR ion flux measurements with calorimetric space-based experiments. We present an energy-dependent measurement of the inelastic cross section of protons and helium-4 nuclei (alpha particles) on a Bi$_4$Ge$_3$O$_{12}$ target, using 88 months of data collected by the DAMPE space mission. The kinetic energy range per nucleon of the measurement points ranges from 18 GeV to 9 TeV for protons, and from 5 GeV/n to 3 TeV/n for helium-4 nuclei. Our results lead to a significant improvement of the CR flux normalisation. In the case of helium-4, these results correspond to the first cross section measurements on a heavy target material at energies above 10 GeV/n. △ Less

Submitted 30 August, 2024; originally announced August 2024.

Comments: 17 pages, submitted to PRD

WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

Authors: Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, Wen Wang, Zhou Zhao

Abstract: Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domai… ▽ More Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer. △ Less

Submitted 29 August, 2024; originally announced August 2024.

Comments: Working in progress. arXiv admin note: text overlap with arXiv:2402.12208

Galaxies Lighting Up: Discovery of Seventy New Turn-on Changing-look Quasars

Authors: Qian Yang, Paul J. Green, Xue-Bing Wu, Michael Eracleous, Linhua Jiang, Yuming Fu

Abstract: "Changing-look quasars" (CLQs), discovered less than a decade ago, show dramatic, rapid changes in optical/UV continuum and broad line emission. The majority of CLQs have been found dimming as "turn-off" CLQs because most selection methods start from samples of spectroscopically-confirmed quasars. We present here a sample of 82 spectroscopically confirmed "turn-on" CLQs, 70 of which are newly iden… ▽ More "Changing-look quasars" (CLQs), discovered less than a decade ago, show dramatic, rapid changes in optical/UV continuum and broad line emission. The majority of CLQs have been found dimming as "turn-off" CLQs because most selection methods start from samples of spectroscopically-confirmed quasars. We present here a sample of 82 spectroscopically confirmed "turn-on" CLQs, 70 of which are newly identified. The turn-on CLQs are selected from spectroscopically classified galaxies with subsequent significant and dramatic variability in both the optical and mid-infrared bands, indicating a mechanism of changing accretion rate of the supermassive black holes rather than variable obscuration. Based on their bright state Eddington ratios, turn-on CLQs are associated with lower accretion rates compared to turn-off CLQs or typical SDSS quasars with similar redshift and magnitude distributions, even though turn-on CLQs have lower black hole masses. Most turn-on CLQs reside in host galaxies that follow local relations between the central black hole mass and host galaxy properties, such as stellar mass and velocity dispersion. However, their host galaxies have higher mass than normal inactive galaxies, with star formation rates more similar to hosts of Type 2 AGN than to the overall galaxy population. △ Less

Submitted 28 August, 2024; originally announced August 2024.

Comments: Submitted to ApJ; the spectral fitting code QGfit will be available on GitHub; 36 pages, 20 figures

HEAD: A Bandwidth-Efficient Cooperative Perception Approach for Heterogeneous Connected and Autonomous Vehicles

Authors: Deyuan Qu, Qi Chen, Yongqi Zhu, Yihao Zhu, Sergei S. Avedisov, Song Fu, Qing Yang

Abstract: In cooperative perception studies, there is often a trade-off between communication bandwidth and perception performance. While current feature fusion solutions are known for their excellent object detection performance, transmitting the entire sets of intermediate feature maps requires substantial bandwidth. Furthermore, these fusion approaches are typically limited to vehicles that use identical… ▽ More In cooperative perception studies, there is often a trade-off between communication bandwidth and perception performance. While current feature fusion solutions are known for their excellent object detection performance, transmitting the entire sets of intermediate feature maps requires substantial bandwidth. Furthermore, these fusion approaches are typically limited to vehicles that use identical detection models. Our goal is to develop a solution that supports cooperative perception across vehicles equipped with different modalities of sensors. This method aims to deliver improved perception performance compared to late fusion techniques, while achieving precision similar to the state-of-art intermediate fusion, but requires an order of magnitude less bandwidth. We propose HEAD, a method that fuses features from the classification and regression heads in 3D object detection networks. Our method is compatible with heterogeneous detection networks such as LiDAR PointPillars, SECOND, VoxelNet, and camera Bird's-eye View (BEV) Encoder. Given the naturally smaller feature size in the detection heads, we design a self-attention mechanism to fuse the classification head and a complementary feature fusion layer to fuse the regression head. Our experiments, comprehensively evaluated on the V2V4Real and OPV2V datasets, demonstrate that HEAD is a fusion method that effectively balances communication bandwidth and perception performance. △ Less

Submitted 27 August, 2024; originally announced August 2024.

Comments: Accepted by ECCV 2024 Workshop

A Survey on Reinforcement Learning Applications in SLAM

Authors: Mohammad Dehghani Tezerjani, Mohammad Khoshnazar, Mohammadhamed Tangestanizadeh, Qing Yang

Abstract: The emergence of mobile robotics, particularly in the automotive industry, introduces a promising era of enriched user experiences and adept handling of complex navigation challenges. The realization of these advancements necessitates a focused technological effort and the successful execution of numerous intricate tasks, particularly in the critical domain of Simultaneous Localization and Mapping… ▽ More The emergence of mobile robotics, particularly in the automotive industry, introduces a promising era of enriched user experiences and adept handling of complex navigation challenges. The realization of these advancements necessitates a focused technological effort and the successful execution of numerous intricate tasks, particularly in the critical domain of Simultaneous Localization and Mapping (SLAM). Various artificial intelligence (AI) methodologies, such as deep learning and reinforcement learning, present viable solutions to address the challenges in SLAM. This study specifically explores the application of reinforcement learning in the context of SLAM. By enabling the agent (the robot) to iteratively interact with and receive feedback from its environment, reinforcement learning facilitates the acquisition of navigation and mapping skills, thereby enhancing the robot's decision-making capabilities. This approach offers several advantages, including improved navigation proficiency, increased resilience, reduced dependence on sensor precision, and refinement of the decision-making process. The findings of this study, which provide an overview of reinforcement learning's utilization in SLAM, reveal significant advancements in the field. The investigation also highlights the evolution and innovative integration of these techniques. △ Less

Submitted 25 August, 2024; originally announced August 2024.

NEXUS: the North ecliptic pole EXtragalactic Unified Survey

Authors: Yue Shen, Ming-Yang Zhuang, Junyao Li, Adam J. Burgasser, Xiaohui Fan, Jenny E. Greene, Gautham Narayan, Alice E. Shapley, Fengwu Sun, Feige Wang, Qian Yang

Abstract: NEXUS is a JWST Multi-Cycle (Cycles 3-5; 368 primary hrs) GO Treasury imaging and spectroscopic survey around the North Ecliptic Pole. It contains two overlapping tiers. The Wide tier ($\sim 400~{\rm arcmin}^2$) performs NIRCam/WFSS 2.4-5 micron grism spectroscopy with three epochs over 3 years (final continuum ${\rm S/N/pixel>3}$ at F444W$<22.2$). The Deep tier ($\sim 50~{\rm arcmin}^2$) performs… ▽ More NEXUS is a JWST Multi-Cycle (Cycles 3-5; 368 primary hrs) GO Treasury imaging and spectroscopic survey around the North Ecliptic Pole. It contains two overlapping tiers. The Wide tier ($\sim 400~{\rm arcmin}^2$) performs NIRCam/WFSS 2.4-5 micron grism spectroscopy with three epochs over 3 years (final continuum ${\rm S/N/pixel>3}$ at F444W$<22.2$). The Deep tier ($\sim 50~{\rm arcmin}^2$) performs high-multiplexing NIRSpec 0.6-5.3 micron MOS/PRISM spectroscopy for $\sim 10,000$ targets, over 18 epochs with a 2-month cadence (epoch/final continuum ${\rm S/N/pixel>3}$ at F200W$\lesssim 27/29$). All epochs have simultaneous multi-band NIRCam and MIRI imaging ($5σ$ final depths of $\sim 28-29$ in NIRCam and $\sim 25$ in MIRI). The field is within the continuous viewing zone of JWST, and is fully covered by the Euclid Ultra-Deep Field, with 0.9-2 micron deep Euclid spectroscopy and cadenced photometry. NEXUS has three science pillars. First, with its massive and nearly complete (flux-limited) spectroscopic samples and deep photometry, it will perform efficient classification and physical characterization of galaxies and AGNs from $z\sim 1$ to Cosmic Dawn. With the large contiguous area coverage, it will measure the spatial clustering and demography of the first galaxies and SMBHs at $z>6$. Second, multi-epoch observations enable systematic time-domain investigations, focusing on $z>3$ transients and low-mass AGN reverberation mapping. Third, the comprehensive data set will enable knowledge transfer to other legacy fields, create data challenges, and initiate benchmark work for future space missions. With rapid public releases of processed data and an open invitation for collaboration, NEXUS aims for broad and swift community engagement, to become a powerhouse to drive transformative advancements in multiple key science areas of astronomy. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Comments: 19 pages

Research on Improved U-net Based Remote Sensing Image Segmentation Algorithm

Authors: Qiming Yang, Zixin Wang, Shinan Liu, Zizheng Li

Abstract: In recent years, although U-Net network has made significant progress in the field of image segmentation, it still faces performance bottlenecks in remote sensing image segmentation. In this paper, we innovatively propose to introduce SimAM and CBAM attention mechanism in U-Net, and the experimental results show that after adding SimAM and CBAM modules alone, the model improves 17.41% and 12.23% i… ▽ More In recent years, although U-Net network has made significant progress in the field of image segmentation, it still faces performance bottlenecks in remote sensing image segmentation. In this paper, we innovatively propose to introduce SimAM and CBAM attention mechanism in U-Net, and the experimental results show that after adding SimAM and CBAM modules alone, the model improves 17.41% and 12.23% in MIoU, and the Mpa and Accuracy are also significantly improved. And after fusing the two,the model performance jumps up to 19.11% in MIoU, and the Mpa and Accuracy are also improved by 16.38% and 14.8% respectively, showing excellent segmentation accuracy and visual effect with strong generalization ability and robustness. This study opens up a new path for remote sensing image segmentation technology and has important reference value for algorithm selection and improvement. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Green Probabilistic Semantic Communication over Wireless Networks

Authors: Ruopeng Xu, Zhaohui Yang, Yijie Mao, Chongwen Huang, Qianqian Yang, Lexi Xu, Wei Xu, Zhaoyang Zhang

Abstract: In this paper, we propose a multi-user green semantic communication system facilitated by a probabilistic knowledge graph (PKG). By integrating probability into the knowledge graph, we enable probabilistic semantic communication (PSC) and represent semantic information accordingly. On this basis, a semantic compression model designed for multi-user downlink task-oriented communication is introduce… ▽ More In this paper, we propose a multi-user green semantic communication system facilitated by a probabilistic knowledge graph (PKG). By integrating probability into the knowledge graph, we enable probabilistic semantic communication (PSC) and represent semantic information accordingly. On this basis, a semantic compression model designed for multi-user downlink task-oriented communication is introduced, utilizing the semantic compression ratio (SCR) as a parameter to connect the computation and communication processes of information transmission. Based on the rate-splitting multiple access (RSMA) technology, we derive mathematical expressions for system transmission energy consumption and related formulations. Subsequently, the multi-user green semantic communication system is modeled and the optimal problem with the goal of minimizing system energy consumption comprehensively considering the computation and communication process under given constrains is formulated. In order to address the optimal problem, we propose an alternating optimization algorithm that tackles sub-problems of power allocation and beamforming design, semantic compression ratio, and computation capacity allocation. Simulation results validate the effectiveness of our approach, demonstrating the superiority of our system over methods using Space Division Multiple Access (SDMA) and non-orthogonal multiple access (NOMA) instead of RSMA, and highlighting the benefits of our PSC compression model. △ Less

Submitted 21 August, 2024; originally announced August 2024.

Physics-Driven AI Correction in Laser Absorption Sensing Quantification

Authors: Ruiyuan Kang, Panos Liatsis, Meixia Geng, Qingjie Yang

Abstract: Laser absorption spectroscopy (LAS) quantification is a popular tool used in measuring temperature and concentration of gases. It has low error tolerance, whereas current ML-based solutions cannot guarantee their measure reliability. In this work, we propose a new framework, SPEC, to address this issue. In addition to the conventional ML estimator-based estimation mode, SPEC also includes a Physic… ▽ More Laser absorption spectroscopy (LAS) quantification is a popular tool used in measuring temperature and concentration of gases. It has low error tolerance, whereas current ML-based solutions cannot guarantee their measure reliability. In this work, we propose a new framework, SPEC, to address this issue. In addition to the conventional ML estimator-based estimation mode, SPEC also includes a Physics-driven Anomaly Detection module (PAD) to assess the error of the estimation. And a Correction mode is designed to correct the unreliable estimation. The correction mode is a network-based optimization algorithm, which uses the guidance of error to iteratively correct the estimation. A hybrid surrogate error model is proposed to estimate the error distribution, which contains an ensemble of networks to simulate reconstruction error, and true feasible error computation. A greedy ensemble search is proposed to find the optimal correction robustly and efficiently from the gradient guidance of surrogate model. The proposed SPEC is validated on the test scenarios which are outside the training distribution. The results show that SPEC can significantly improve the estimation quality, and the correction mode outperforms current network-based optimization algorithms. In addition, SPEC has the reconfigurability, which can be easily adapted to different quantification tasks via changing PAD without retraining the ML estimator. △ Less

Submitted 20 August, 2024; originally announced August 2024.

Comments: 13 pages

MSC Class: 68T05 ACM Class: I.2.1

Exploiting Fine-Grained Prototype Distribution for Boosting Unsupervised Class Incremental Learning

Authors: Jiaming Liu, Hongyuan Liu, Zhili Qin, Wei Han, Yulu Fan, Qinli Yang, Junming Shao

Abstract: The dynamic nature of open-world scenarios has attracted more attention to class incremental learning (CIL). However, existing CIL methods typically presume the availability of complete ground-truth labels throughout the training process, an assumption rarely met in practical applications. Consequently, this paper explores a more challenging problem of unsupervised class incremental learning (UCIL… ▽ More The dynamic nature of open-world scenarios has attracted more attention to class incremental learning (CIL). However, existing CIL methods typically presume the availability of complete ground-truth labels throughout the training process, an assumption rarely met in practical applications. Consequently, this paper explores a more challenging problem of unsupervised class incremental learning (UCIL). The essence of addressing this problem lies in effectively capturing comprehensive feature representations and discovering unknown novel classes. To achieve this, we first model the knowledge of class distribution by exploiting fine-grained prototypes. Subsequently, a granularity alignment technique is introduced to enhance the unsupervised class discovery. Additionally, we proposed a strategy to minimize overlap between novel and existing classes, thereby preserving historical knowledge and mitigating the phenomenon of catastrophic forgetting. Extensive experiments on the five datasets demonstrate that our approach significantly outperforms current state-of-the-art methods, indicating the effectiveness of the proposed method. △ Less

Submitted 19 August, 2024; originally announced August 2024.

MalLight: Influence-Aware Coordinated Traffic Signal Control for Traffic Signal Malfunctions

Authors: Qinchen Yang, Zejun Xie, Hua Wei, Desheng Zhang, Yu Yang

Abstract: Urban traffic is subject to disruptions that cause extended waiting time and safety issues at signalized intersections. While numerous studies have addressed the issue of intelligent traffic systems in the context of various disturbances, traffic signal malfunction, a common real-world occurrence with significant repercussions, has received comparatively limited attention. The primary objective of… ▽ More Urban traffic is subject to disruptions that cause extended waiting time and safety issues at signalized intersections. While numerous studies have addressed the issue of intelligent traffic systems in the context of various disturbances, traffic signal malfunction, a common real-world occurrence with significant repercussions, has received comparatively limited attention. The primary objective of this research is to mitigate the adverse effects of traffic signal malfunction, such as traffic congestion and collision, by optimizing the control of neighboring functioning signals. To achieve this goal, this paper presents a novel traffic signal control framework (MalLight), which leverages an Influence-aware State Aggregation Module (ISAM) and an Influence-aware Reward Aggregation Module (IRAM) to achieve coordinated control of surrounding traffic signals. To the best of our knowledge, this study pioneers the application of a Reinforcement Learning(RL)-based approach to address the challenges posed by traffic signal malfunction. Empirical investigations conducted on real-world datasets substantiate the superior performance of our proposed methodology over conventional and deep learning-based alternatives in the presence of signal malfunction, with reduction of throughput alleviated by as much as 48.6$\%$. △ Less

Submitted 20 August, 2024; v1 submitted 19 August, 2024; originally announced August 2024.

Comments: Paper accepted to CIKM24 Full Research track

PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding

Authors: Dawei Dai, Yuanhui Zhang, Long Xu, Qianlan Yang, Xiaojing Shen, Shuyin Xia, Guoyin Wang

Abstract: The previous advancements in pathology image understanding primarily involved developing models tailored to specific tasks. Recent studies has demonstrated that the large vision-language model can enhance the performance of various downstream tasks in medical image understanding. In this study, we developed a domain-specific large language-vision assistant (PA-LLaVA) for pathology image understand… ▽ More The previous advancements in pathology image understanding primarily involved developing models tailored to specific tasks. Recent studies has demonstrated that the large vision-language model can enhance the performance of various downstream tasks in medical image understanding. In this study, we developed a domain-specific large language-vision assistant (PA-LLaVA) for pathology image understanding. Specifically, (1) we first construct a human pathology image-text dataset by cleaning the public medical image-text data for domain-specific alignment; (2) Using the proposed image-text data, we first train a pathology language-image pretraining (PLIP) model as the specialized visual encoder for pathology image, and then we developed scale-invariant connector to avoid the information loss caused by image scaling; (3) We adopt two-stage learning to train PA-LLaVA, first stage for domain alignment, and second stage for end to end visual question \& answering (VQA) task. In experiments, we evaluate our PA-LLaVA on both supervised and zero-shot VQA datasets, our model achieved the best overall performance among multimodal models of similar scale. The ablation experiments also confirmed the effectiveness of our design. We posit that our PA-LLaVA model and the datasets presented in this work can promote research in field of computational pathology. All codes are available at: https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA}{https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA △ Less

Submitted 18 August, 2024; originally announced August 2024.

Comments: 8 pages, 4 figs

Tailoring light holes in $β$-$Ga_{2}O_{3}$ via Anion-Anion Antibonding Coupling

Authors: Ke Xu, Qiaolin Yang, Wenhao Liu, Rong Zhang, Zhi Wang, Jiandong Ye

Abstract: A significant limitation of wide-bandgap materials is their low hole mobility related to localized holes with heavy effective masses ($m_h^*$). We identify in low-symmetric wide-bandgap compounds an anion-anion antibonding coupling (AAAC) effect as the intrinsic factor behind hole localization, which explains the extremely heavy $m_h^*$ and self-trapped hole (STH) formation observed in gallium oxi… ▽ More A significant limitation of wide-bandgap materials is their low hole mobility related to localized holes with heavy effective masses ($m_h^*$). We identify in low-symmetric wide-bandgap compounds an anion-anion antibonding coupling (AAAC) effect as the intrinsic factor behind hole localization, which explains the extremely heavy $m_h^*$ and self-trapped hole (STH) formation observed in gallium oxide ($β$-$Ga_{2}O_{3}$). We propose a design principle for achieving light holes by manipulating AAAC, demonstrating that specific strain conditions can reduce $m_h^*$ in $β$-$Ga_{2}O_{3}$ from 4.77 $m_0$ to 0.38 $m_0$, making it comparable to the electron mass (0.28 $m_0$), while also suppressing STH. The light holes show significant anisotropy, potentially enabling two-dimensional transport in bulk material. This study provides a fundamental understanding of hole mass enhancement and STH formation in novel wide-bandgap materials and suggest new pathways for engineering hole mobilities. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Comments: 22 pages, 1 table, 5 figures

Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

Authors: Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

Abstract: The rapid growth in the parameters of large language models (LLMs) has made inference latency a fundamental bottleneck, limiting broader application of LLMs. Speculative decoding represents a lossless approach to accelerate inference through a guess-and-verify paradigm, leveraging the parallel capabilities of modern hardware. Some speculative decoding methods rely on additional structures to guess… ▽ More The rapid growth in the parameters of large language models (LLMs) has made inference latency a fundamental bottleneck, limiting broader application of LLMs. Speculative decoding represents a lossless approach to accelerate inference through a guess-and-verify paradigm, leveraging the parallel capabilities of modern hardware. Some speculative decoding methods rely on additional structures to guess draft tokens, such as small models or parameter-efficient architectures, which need extra training before use. Alternatively, retrieval-based train-free techniques build libraries from pre-existing corpora or by n-gram generation. However, they face challenges like large storage requirements, time-consuming retrieval, and limited adaptability. Observing that candidate tokens generated during the decoding process are likely to reoccur in future sequences, we propose Token Recycling. This approach stores candidate tokens in an adjacency matrix and employs a breadth-first search (BFS)-like algorithm on the matrix to construct a draft tree. The tree is then validated through tree attention. New candidate tokens from the decoding process are then used to update the matrix. Token Recycling requires \textless2MB of additional storage and achieves approximately 2x speedup across all sizes of LLMs. It significantly outperforms existing train-free methods by 30\% and even a training method by 25\%. It can be directly applied to any existing LLMs and tasks without the need for adaptation. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Comments: under review

Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading

Authors: Li Pan, Yupei Zhang, Qiushi Yang, Tan Li, Xiaohan Xing, Maximus C. F. Yeung, Zhen Chen

Abstract: Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two is… ▽ More Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two issues hinder existing methods to precisely interpret diagnostic molecular-pathology features, thereby limiting their grading performance. Moreover, the real-world applicability of existing multimodal approaches is significantly restricted as molecular biomarkers are not always available during clinical deployment. To address these problems, we introduce a novel Focus on Focus (FoF) framework with paired pathology-genomic training and applicable pathology-only inference, enhancing molecular-pathology representation effectively. Specifically, we propose a Focus-oriented Representation Learning (FRL) module to encourage the model to identify regions positively or negatively related to glioma grading and guide it to focus on the diagnostic areas with a consistency constraint. To effectively link the molecular biomarkers to morphological features, we propose a Multi-view Cross-modal Alignment (MCA) module that projects histopathology representations into molecular subspaces, aligning morphological features with corresponding molecular biomarker status by supervised contrastive learning. Experiments on the TCGA GBM-LGG dataset demonstrate that our FoF framework significantly improves the glioma grading. Remarkably, our FoF achieves superior performance using only histopathology slides compared to existing multimodal methods. The source code is available at https://github.com/peterlipan/FoF. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approach

Authors: Shizhou Zhang, Wenlong Luo, De Cheng, Qingchun Yang, Lingyan Ran, Yinghui Xing, Yanning Zhang

Abstract: In this paper, we construct a large-scale benchmark dataset for Ground-to-Aerial Video-based person Re-Identification, named G2A-VReID, which comprises 185,907 images and 5,576 tracklets, featuring 2,788 distinct identities. To our knowledge, this is the first dataset for video ReID under Ground-to-Aerial scenarios. G2A-VReID dataset has the following characteristics: 1) Drastic view changes; 2) L… ▽ More In this paper, we construct a large-scale benchmark dataset for Ground-to-Aerial Video-based person Re-Identification, named G2A-VReID, which comprises 185,907 images and 5,576 tracklets, featuring 2,788 distinct identities. To our knowledge, this is the first dataset for video ReID under Ground-to-Aerial scenarios. G2A-VReID dataset has the following characteristics: 1) Drastic view changes; 2) Large number of annotated identities; 3) Rich outdoor scenarios; 4) Huge difference in resolution. Additionally, we propose a new benchmark approach for cross-platform ReID by transforming the cross-platform visual alignment problem into visual-semantic alignment through vision-language model (i.e., CLIP) and applying a parameter-efficient Video Set-Level-Adapter module to adapt image-based foundation model to video ReID tasks, termed VSLA-CLIP. Besides, to further reduce the great discrepancy across the platforms, we also devise the platform-bridge prompts for efficient visual feature alignment. Extensive experiments demonstrate the superiority of the proposed method on all existing video ReID datasets and our proposed G2A-VReID dataset. △ Less

Submitted 14 August, 2024; originally announced August 2024.

In-line fiber optic optofluidic sensor based on a fully open Fabry-Perot interferometer

Authors: Dewen Duan, Qian Kang, Qianhui Yang, Zihao Zhao, Na Li, Guan-Xiang Du, Yi-Yuan Xie

Abstract: We present an all-fiber, fully open Fabry-Perot interferometer (FPI) cavity that is suitable for fluidic measurement applications. Fabrication of the FPI involves the alignment and bonding of three optical fiber sections using either ceramic glue or low-temperature melting glass. The fabrication procedure allows the protection of the cleaved optical fiber end faces, which serve as the two mirrors… ▽ More We present an all-fiber, fully open Fabry-Perot interferometer (FPI) cavity that is suitable for fluidic measurement applications. Fabrication of the FPI involves the alignment and bonding of three optical fiber sections using either ceramic glue or low-temperature melting glass. The fabrication procedure allows the protection of the cleaved optical fiber end faces, which serve as the two mirrors of the FPI, from damage, thus ensuring the high visibility of the FPI sensor. The FPI's complete openness permits the analyte of interest fluids to flow smoothly into the cavity and interact directly with the light, obviating the need for additional assistance. The fabrication experiment demonstrates that the fabrication procedure can readily achieve a visibility of over 20 dB. Refractive index testing indicates that the sensor exhibits a sensitivity of over 1116 nm/RIU within the range of 1.334-1.375. A comparison of temperature investigations indicates that the fully open cavity FPI fabricated by bonding with low-temperature melting glass exhibits relatively lower temperature immunity than that bonded with ceramic glue. Both have a relatively low temperature fluctuation within the temperature range of 40°C-100°C, with less than 3 nm and 4.5 nm in the over 60°C changes, respectively. Our proposed fully open FPI is an economical, robust, and simple-to-fabricate structure with the potential for mass production. This renders it an appealing option for practical optofluidics applications. △ Less

Submitted 14 August, 2024; originally announced August 2024.

Comments: 6 pages, 10 figures

MSC Class: 7805

Search for QCD Axions in light of String Theory

Authors: Qiaoli Yang, Runchao Huang

Abstract: The QCD axion stands as one of the most promising candidates for resolving the strong CP problem. However, the value of the axion's decay constant $f_a$ and, by extension, its mass $m_a$, remain uncertain within the framework of effective field theory, posing a challenge for experimental detection. Fortunately, fields such as cosmology and astrophysics can offer crucial clues about potential mass… ▽ More The QCD axion stands as one of the most promising candidates for resolving the strong CP problem. However, the value of the axion's decay constant $f_a$ and, by extension, its mass $m_a$, remain uncertain within the framework of effective field theory, posing a challenge for experimental detection. Fortunately, fields such as cosmology and astrophysics can offer crucial clues about potential mass ranges. Additionally, string theory and the more recent swampland principles might shed some light on the subject. The most straightforward string theory compactifications suggest that $f_a$ is around the GUT scale, leading to an anthropic abundance of dark matter. We found that this range can be detected through hydrogen atomic transitions. The recent concept of the dark dimension scenario introduces an alternative possibility. If axions are confined to the four-dimensional Standard Model brane, their decay constant $f_a$ would be on the order of $10^{10}$ GeV. In this scenario, where axions constitute only a minor portion of dark matter, we show that a laser-interferometry setup would be an effective detection method. △ Less

Submitted 13 August, 2024; originally announced August 2024.

Comments: 7 pages, 3 figures

Anisotropy of Nanohertz Gravitational Wave Background and Individual Sources from Supermassive Binary Black Holes: Probe of Cosmic Large Scale Structure

Authors: Qing Yang, Xiao Guo, Zhoujian Cao, Xiaoyun Shao, Xi Yuan

Abstract: Several pulsar timing array (PTA) groups have recently claimed the detection of nanohertz gravitational wave (GW) background, but the origin of this GW signal remains unclear. Nanohertz GWs generated by supermassive binary black holes (SMBBHs) are one of the most important GW sources in the PTA band. Utilizing data from numerical cosmology simulation, we generate mock SMBBHs within the observable… ▽ More Several pulsar timing array (PTA) groups have recently claimed the detection of nanohertz gravitational wave (GW) background, but the origin of this GW signal remains unclear. Nanohertz GWs generated by supermassive binary black holes (SMBBHs) are one of the most important GW sources in the PTA band. Utilizing data from numerical cosmology simulation, we generate mock SMBBHs within the observable universe and treat them as PTA band GW sources. We present their statistical properties, and analyze the isotropic and anisotropic characteristics of the gravitational wave background (GWB) signal they produce. Specifically, we derive the characteristic amplitude and spectrum of the GWB signal, and calculate the angular power spectrum for both GW strains/energy density and the position distribution of GW sources. We predict that the angular power spectrum of GWB energy density has $C_1/C_0\approx0.40\pm0.32$, and $C_l/C_0\simeq \frac{1}{2(2l+1)}$ (for $l>1$). Furthermore, for the upcoming Chinese Pulsar Timing Array (CPTA) and Square Kilometre Array (SKA) PTA, we predict the spatial distribution, numbers and signal-to-noise ratio (SNR) distribution of individual GW sources that may be detected with SNR>8, and study the anisotropy property in the spatial distribution of these individual GW sources. △ Less

Submitted 9 August, 2024; originally announced August 2024.

Comments: 26 pages,18 figures,1 table

Deep Learning-based Unsupervised Domain Adaptation via a Unified Model for Prostate Lesion Detection Using Multisite Bi-parametric MRI Datasets

Authors: Hao Li, Han Liu, Heinrich von Busch, Robert Grimm, Henkjan Huisman, Angela Tong, David Winkel, Tobias Penzkofer, Ivan Shabunin, Moon Hyung Choi, Qingsong Yang, Dieter Szolar, Steven Shea, Fergus Coakley, Mukesh Harisinghani, Ipek Oguz, Dorin Comaniciu, Ali Kamen, Bin Lou

Abstract: Our hypothesis is that UDA using diffusion-weighted images, generated with a unified model, offers a promising and reliable strategy for enhancing the performance of supervised learning models in multi-site prostate lesion detection, especially when various b-values are present. This retrospective study included data from 5,150 patients (14,191 samples) collected across nine different imaging cent… ▽ More Our hypothesis is that UDA using diffusion-weighted images, generated with a unified model, offers a promising and reliable strategy for enhancing the performance of supervised learning models in multi-site prostate lesion detection, especially when various b-values are present. This retrospective study included data from 5,150 patients (14,191 samples) collected across nine different imaging centers. A novel UDA method using a unified generative model was developed for multi-site PCa detection. This method translates diffusion-weighted imaging (DWI) acquisitions, including apparent diffusion coefficient (ADC) and individual DW images acquired using various b-values, to align with the style of images acquired using b-values recommended by Prostate Imaging Reporting and Data System (PI-RADS) guidelines. The generated ADC and DW images replace the original images for PCa detection. An independent set of 1,692 test cases (2,393 samples) was used for evaluation. The area under the receiver operating characteristic curve (AUC) was used as the primary metric, and statistical analysis was performed via bootstrapping. For all test cases, the AUC values for baseline SL and UDA methods were 0.73 and 0.79 (p<.001), respectively, for PI-RADS>=3, and 0.77 and 0.80 (p<.001) for PI-RADS>=4 PCa lesions. In the 361 test cases under the most unfavorable image acquisition setting, the AUC values for baseline SL and UDA were 0.49 and 0.76 (p<.001) for PI-RADS>=3, and 0.50 and 0.77 (p<.001) for PI-RADS>=4 PCa lesions. The results indicate the proposed UDA with generated images improved the performance of SL methods in multi-site PCa lesion detection across datasets with various b values, especially for images acquired with significant deviations from the PI-RADS recommended DWI protocol (e.g. with an extremely high b-value). △ Less

Submitted 8 August, 2024; originally announced August 2024.

Comments: Accept at Radiology: Artificial Intelligence. Journal reference and external DOI will be added once published

Knowledge-Aided Semantic Communication Leveraging Probabilistic Graphical Modeling

Authors: Haowen Wan, Qianqian Yang, Jiancheng Tang, Zhiguo shi

Abstract: In this paper, we propose a semantic communication approach based on probabilistic graphical model (PGM). The proposed approach involves constructing a PGM from a training dataset, which is then shared as common knowledge between the transmitter and receiver. We evaluate the importance of various semantic features and present a PGM-based compression algorithm designed to eliminate predictable port… ▽ More In this paper, we propose a semantic communication approach based on probabilistic graphical model (PGM). The proposed approach involves constructing a PGM from a training dataset, which is then shared as common knowledge between the transmitter and receiver. We evaluate the importance of various semantic features and present a PGM-based compression algorithm designed to eliminate predictable portions of semantic information. Furthermore, we introduce a technique to reconstruct the discarded semantic information at the receiver end, generating approximate results based on the PGM. Simulation results indicate a significant improvement in transmission efficiency over existing methods, while maintaining the quality of the transmitted images. △ Less

Submitted 8 August, 2024; originally announced August 2024.

From squared amplitudes to energy correlators

Authors: Song He, Xuhang Jiang, Qinglin Yang, Yao-Qi Zhang

Abstract: The leading order $N$-point energy correlators of maximally supersymmetric Yang-Mills theory in the limit where the $N$ detectors are collinear can be expressed as an integral of the $1\to N$ splitting function, which is given by the $(N{+}3)$-point squared super-amplitudes at tree level. This provides yet another example that the integrand of certain physical observable -- $N$-point energy correl… ▽ More The leading order $N$-point energy correlators of maximally supersymmetric Yang-Mills theory in the limit where the $N$ detectors are collinear can be expressed as an integral of the $1\to N$ splitting function, which is given by the $(N{+}3)$-point squared super-amplitudes at tree level. This provides yet another example that the integrand of certain physical observable -- $N$-point energy correlator -- is computed by the canonical form of a positive geometry -- the (tree-level) "squared amplituhedron". By extracting such squared amplitudes from the $f$-graph construction, we compute the integrand of energy correlators up to $N=11$ and reveal new structures to all $N$; we also show important properties of the integrand such as soft and multi-collinear limits. Finally, we take a first look at integrations by studying possible residues of the integrand: our analysis shows that while this gives prefactors in front of multiple polylogarithm functions of $N=3,4$, the first unknown case of $N=5$ already involves elliptic polylogarithmic functions with many distinct elliptic curves, and more complicated curves and higher-dimensional varieties appear for $N>5$. △ Less

Submitted 8 August, 2024; originally announced August 2024.

Comments: 10 pages, several figures and tables, and an ancillary file with squared amplitudes up to 12 points, and explicit EC integrands up to 7 points

Resonant Beam Enabled DoA Estimation in Passive Positioning System

Authors: Yixuan Guo, Qingwei Jiang, Mengyuan Xu, Wen Fang, Qingwen Liu, Gang Yan, Qunhui Yang, Hai Lu

Abstract: The rapid advancement of the next generation of communications and internet of things (IoT) technologies has made the provision of location-based services for diverse devices an increasingly pressing necessity. Localizing devices with/without intelligent computing abilities, including both active and passive devices is essential, especially in indoor scenarios. For traditional RF positioning syste… ▽ More The rapid advancement of the next generation of communications and internet of things (IoT) technologies has made the provision of location-based services for diverse devices an increasingly pressing necessity. Localizing devices with/without intelligent computing abilities, including both active and passive devices is essential, especially in indoor scenarios. For traditional RF positioning systems, aligning transmission signals and dealing with signal interference in complex environments are inevitable challenges. Therefore, this paper proposed a new passive positioning system, the RF-band resonant beam positioning system (RF-RBPS), which achieves energy concentration and beam alignment by amplifying echoes between the base station (BS) and the passive target (PT), without the need for complex channel estimation and time-consuming beamforming and provides high-precision direction of arrival (DoA) estimation for battery-free targets using the resonant mechanism. The direction information of the PT is estimated using the multiple signal classification (MUSIC) algorithm at the end of BS. The feasibility of the proposed system is validated through theoretical analysis and simulations. Results indicate that the proposed RF-RBPS surpasses RF-band active positioning system (RF-APS) in precision, achieving millimeter-level precision at 2m within an elevation angle of 35$^\circ$, and an error of less than 3cm at 2.5m within an elevation angle of 35$^\circ$. △ Less

Submitted 7 August, 2024; originally announced August 2024.

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Authors: Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, Yong Li

Abstract: This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it req… ▽ More This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it requires agent to establish self-position and acquire spatial representation of complex urban environment, where landmarks are often invisible. In the absence of navigation instructions, such abilities are vital for the agent to make high-quality decisions in long-range city navigation. With the emergent reasoning ability of large language models (LLMs), a tempting baseline is to prompt LLMs to "react" on each observation and make decisions accordingly. However, this baseline has very poor performance that the agent often repeatedly visits same locations and make short-sighted, inconsistent decisions. To address these issues, this paper introduces a novel agentic workflow featured by its abilities to perceive, reflect and plan. Specifically, we find LLaVA-7B can be fine-tuned to perceive the direction and distance of landmarks with sufficient accuracy for city navigation. Moreover, reflection is achieved through a memory mechanism, where past experiences are stored and can be retrieved with current perception for effective decision argumentation. Planning uses reflection results to produce long-term plans, which can avoid short-sighted decisions in long-range navigation. We show the designed workflow significantly improves navigation ability of the LLM agent compared with the state-of-the-art baselines. △ Less

Submitted 7 August, 2024; originally announced August 2024.

AdapMTL: Adaptive Pruning Framework for Multitask Learning Model

Authors: Mingcan Xiang, Steven Jiaxun Tang, Qizheng Yang, Hui Guan, Tongping Liu

Abstract: In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams such as images, video, and sensor data is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a… ▽ More In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams such as images, video, and sensor data is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a multitask model presents significant challenges due to the complexities of balancing sparsity allocation and accuracy performance across multiple tasks. To tackle these challenges, we propose AdapMTL, an adaptive pruning framework for MTL models. AdapMTL leverages multiple learnable soft thresholds independently assigned to the shared backbone and the task-specific heads to capture the nuances in different components' sensitivity to pruning. During training, it co-optimizes the soft thresholds and MTL model weights to automatically determine the suitable sparsity level at each component to achieve both high task accuracy and high overall sparsity. It further incorporates an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. We demonstrate the effectiveness of AdapMTL through comprehensive experiments on popular multitask datasets, namely NYU-v2 and Tiny-Taskonomy, with different architectures, showcasing superior performance compared to state-of-the-art pruning methods. △ Less

Submitted 7 August, 2024; originally announced August 2024.

Comments: 13 pages, 9 figures, Published at ACM Multimedia (ACM MM) 2024

Leveraging Inter-Chunk Interactions for Enhanced Retrieval in Large Language Model-Based Question Answering

Authors: Tiezheng Guo, Chen Wang, Yanyi Liu, Jiawei Tang, Pan Li, Sai Xu, Qingwen Yang, Xianlin Gao, Zhi Li, Yingyou Wen

Abstract: Retrieving external knowledge and prompting large language models with relevant information is an effective paradigm to enhance the performance of question-answering tasks. Previous research typically handles paragraphs from external documents in isolation, resulting in a lack of context and ambiguous references, particularly in multi-document and complex tasks. To overcome these challenges, we pr… ▽ More Retrieving external knowledge and prompting large language models with relevant information is an effective paradigm to enhance the performance of question-answering tasks. Previous research typically handles paragraphs from external documents in isolation, resulting in a lack of context and ambiguous references, particularly in multi-document and complex tasks. To overcome these challenges, we propose a new retrieval framework IIER, that leverages Inter-chunk Interactions to Enhance Retrieval. This framework captures the internal connections between document chunks by considering three types of interactions: structural, keyword, and semantic. We then construct a unified Chunk-Interaction Graph to represent all external documents comprehensively. Additionally, we design a graph-based evidence chain retriever that utilizes previous paths and chunk interactions to guide the retrieval process. It identifies multiple seed nodes based on the target question and iteratively searches for relevant chunks to gather supporting evidence. This retrieval process refines the context and reasoning chain, aiding the large language model in reasoning and answer generation. Extensive experiments demonstrate that IIER outperforms strong baselines across four datasets, highlighting its effectiveness in improving retrieval and reasoning capabilities. △ Less

Submitted 5 August, 2024; originally announced August 2024.

Implementing NAT Hole Punching with QUIC

Authors: Jinyu Liang, Wei Xu, Taotao Wang, Qing Yang, Shengli Zhang

Abstract: The widespread adoption of Network Address Translation (NAT) technology has led to a significant number of network end nodes being located in private networks behind NAT devices, impeding direct communication between these nodes. To solve this problem, a technique known as "hole punching" has been devised for NAT traversal to facilitate peer-to-peer communication among end nodes located in distinc… ▽ More The widespread adoption of Network Address Translation (NAT) technology has led to a significant number of network end nodes being located in private networks behind NAT devices, impeding direct communication between these nodes. To solve this problem, a technique known as "hole punching" has been devised for NAT traversal to facilitate peer-to-peer communication among end nodes located in distinct private networks. However, as the increasing demands for speed and security in networks, TCP-based hole punching schemes gradually show performance drawbacks. Therefore, we present a QUIC-based hole punching scheme for NAT traversal. Through a comparative analysis of the hole punching time between QUIC-based and TCP based protocols, we find that the QUIC-based scheme effectively reduces the hole punching time, exhibiting a pronounced advantage in weak network environments. Furthermore, in scenarios where the hole punched connection is disrupted due to factors such as network transitions or NAT timeouts, this paper evaluates two schemes for restoring the connection: QUIC connection migration and re-punching. Our results show that QUIC connection migration for connection restoration saves 2 RTTs compared to QUIC re-punching, and 3 RTTs compared to TCP re-punching, effectively reducing the computational resources consumption for re-punching. △ Less

Submitted 3 August, 2024; originally announced August 2024.

Comments: The paper has been accepted for oral presentation at the VTC2024-Fall Conference

AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation

Authors: Zili Wang, Qi Yang, Linsu Shi, Jiazhong Yu, Qinghua Liang, Fei Li, Shiming Xiang

Abstract: Recently, transformer-based models have demonstrated remarkable performance on audio-visual segmentation (AVS) tasks. However, their expensive computational cost makes real-time inference impractical. By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within rest… ▽ More Recently, transformer-based models have demonstrated remarkable performance on audio-visual segmentation (AVS) tasks. However, their expensive computational cost makes real-time inference impractical. By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within restricted frames, and 2) inefficient, burdensome transformer decoder, caused by narrow focus patterns in early stages. In this paper, we introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation transformer that achieves fast, efficient and light-weight simultaneously. Our model leverages an efficient prompt query generator to correct the behaviour of cross-attention. Additionally, we propose ELF decoder to bring greater efficiency by facilitating convolutions suitable for local features to reduce computational burdens. Extensive experiments demonstrate that our AVESFormer significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3 and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an excellent trade-off between performance and speed. Code can be found at https://github.com/MarkXCloud/AVESFormer.git. △ Less

Submitted 3 August, 2024; originally announced August 2024.

Statistical AoI Guarantee Optimization for Supporting xURLLC in ISAC-enabled V2I Networks

Authors: Yanxi Zhang, Mingwu Yao, Qinghai Yang, Dongqi Yan, Xu Zhang, Xu Bao, Muyu Mei

Abstract: This paper addresses the critical challenge of supporting next-generation ultra-reliable and low-latency communication (xURLLC) within integrated sensing and communication (ISAC)-enabled vehicle-to-infrastructure (V2I) networks. We incorporate channel evaluation and retransmission mechanisms for real-time reliability enhancement. Using stochastic network calculus (SNC), we establish a theoretical… ▽ More This paper addresses the critical challenge of supporting next-generation ultra-reliable and low-latency communication (xURLLC) within integrated sensing and communication (ISAC)-enabled vehicle-to-infrastructure (V2I) networks. We incorporate channel evaluation and retransmission mechanisms for real-time reliability enhancement. Using stochastic network calculus (SNC), we establish a theoretical framework to derive upper bounds for the peak age of information violation probability (PAVP) via characterized sensing and communication moment generation functions (MGFs). By optimizing these bounds, we develop power allocation schemes that significantly reduce the statistical PAVP of sensory packets in such networks. Simulations validate our theoretical derivations and demonstrate the effectiveness of our proposed schemes. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Illumination Design for Joint Imaging and Wireless Power Transfer Systems

Authors: Qianyu Yang, Haiyang Zhang, Chunguo Li, Ruiqi Liu, Baoyun Wang

Abstract: This paper presents a novel concept termed Integrated Imaging and Wireless Power Transfer (IWPT), wherein the integration of imaging and wireless power transfer functionalities is achieved on a unified hardware platform. IWPT leverages a transmitting array to efficiently illuminate a specific Region of Interest (ROI), enabling the extraction of ROI's scattering coefficients while concurrently prov… ▽ More This paper presents a novel concept termed Integrated Imaging and Wireless Power Transfer (IWPT), wherein the integration of imaging and wireless power transfer functionalities is achieved on a unified hardware platform. IWPT leverages a transmitting array to efficiently illuminate a specific Region of Interest (ROI), enabling the extraction of ROI's scattering coefficients while concurrently providing wireless power to nearby users. The integration of IWPT offers compelling advantages, including notable reductions in power consumption and spectrum utilization, pivotal for the optimization of future 6G wireless networks. As an initial investigation, we explore two antenna architectures: a fully digital array and a digital/analog hybrid array. Our goal is to characterize the fundamental trade-off between imaging and wireless power transfer by optimizing the illumination signal. With imaging operating in the near-field, we formulate the illumination signal design as an optimization problem that minimizes the condition number of the equivalent channel. To address this optimization problem, we propose an semi-definite relaxation-based approach for the fully digital array and an alternating optimization algorithm for the hybrid array. Finally, numerical results verify the effectiveness of our proposed solutions and demonstrate the trade-off between imaging and wireless power transfer. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Comments: 10 pages, 5 figures

In-plane dielectric constant and conductivity of confined water

Authors: R. Wang, M. Souilamas, A. Esfandiar, R. Fabregas, S. Benaglia, H. Nevison-Andrews, Q. Yang, J. Normansell, P. Ares, G. Ferrari, A. Principi, A. K. Geim, L. Fumagalli

Abstract: Water is essential for almost every aspect of life on our planet and, unsurprisingly, its properties have been studied in great detail. However, disproportionately little remains known about the electrical properties of interfacial and strongly confined water where its structure deviates from that of bulk water, becoming distinctly layered. The structural change is expected to affect water's condu… ▽ More Water is essential for almost every aspect of life on our planet and, unsurprisingly, its properties have been studied in great detail. However, disproportionately little remains known about the electrical properties of interfacial and strongly confined water where its structure deviates from that of bulk water, becoming distinctly layered. The structural change is expected to affect water's conductivity and particularly its polarizability, which in turn modifies intermolecular forces that play a crucial role in many physical and chemical processes. Here we use scanning dielectric microscopy to probe the in-plane electrical properties of water confined between atomically flat surfaces separated by distances down to 1 nm. For confinement exceeding a few nm, water exhibits an in-plane dielectric constant close to that of bulk water and its proton conductivity is notably enhanced, gradually increasing with decreasing water thickness. This trend abruptly changes when the confined water becomes only a few molecules thick. Its in-plane dielectric constant reaches giant, ferroelectric-like values of about 1,000 whereas the conductivity peaks at a few S/m, close to values characteristic of superionic liquids. We attribute the enhancement to strongly disordered hydrogen bonding induced by the few-layer confinement, which facilitates both easier in-plane polarization of molecular dipoles and faster proton exchange. This insight into the electrical properties of nanoconfined water is important for understanding many phenomena that occur at aqueous interfaces and in nanoscale pores. △ Less

Submitted 31 July, 2024; originally announced July 2024.

Beam Focusing for Near-Field Multi-User Localization

Authors: Qianyu Yang, Anna Guerra, Francesco Guidi, Nir Shlezinger, Haiyang Zhang, Davide Dardari, Baoyun Wang, Yonina C. Eldar

Abstract: Extremely large-scale antenna arrays are poised to play a pivotal role in sixth-generation (6G) networks. Utilizing such arrays often results in a near-field spherical wave transmission environment, enabling the generation of focused beams, which introduces new degrees of freedom for wireless localization. In this paper, we consider a beam-focusing design for localizing multiple sources in the rad… ▽ More Extremely large-scale antenna arrays are poised to play a pivotal role in sixth-generation (6G) networks. Utilizing such arrays often results in a near-field spherical wave transmission environment, enabling the generation of focused beams, which introduces new degrees of freedom for wireless localization. In this paper, we consider a beam-focusing design for localizing multiple sources in the radiating near-field. Our formulation accommodates various expected types of implementations of large antenna arrays, including hybrid analog/digital architectures and dynamic metasurface antennas (DMAs). We consider a direct localization estimation method exploiting curvature-of-arrival of impinging spherical wavefront to obtain user positions. In this regard, we adopt a two-stage approach configuring the array to optimize near-field positioning. In the first step, we focus only on adjusting the array coefficients to minimize the estimation error. We obtain a closed-form approximate solution based on projection and the better one based on the Riemann gradient algorithm. We then extend this approach to simultaneously localize and focus the beams via a sub-optimal iterative approach that does not rely on such knowledge. The simulation results show that near-field localization accuracy based on a hybrid array or DMA can achieve performance close to that of fully digital arrays at a lower cost, and DMAs can attain better performance than hybrid solutions with the same aperture. △ Less

Submitted 24 July, 2024; originally announced July 2024.

Comments: 13 pages, 11 figures

A general thermodynamically consistent phase-field-micromechanics model of sintering with coupled diffusion and grain motion

Authors: Qingcheng Yang, Arkadz Kirshtein

Abstract: Sintering is a pivotal technology for processing ceramic and metallic powders into solid objects. A profound understanding of microstructure evolution during sintering is essential for manufacturing products with tailored properties. While various phase-field models have been proposed to simulate microstructure evolution in solid-state sintering, correctly incorporating the crucial densification m… ▽ More Sintering is a pivotal technology for processing ceramic and metallic powders into solid objects. A profound understanding of microstructure evolution during sintering is essential for manufacturing products with tailored properties. While various phase-field models have been proposed to simulate microstructure evolution in solid-state sintering, correctly incorporating the crucial densification mechanism, particularly grain motion, remains a challenge. The fundamental obstacle lies in the ad hoc treatment of the micromechanics of grain motion, where the thermodynamical driving force cannot be derived from the system's free energy. This work presents a novel phase-field-micromechanics model for sintering (PFMMS) that addresses this challenge. We define a unified energy law, under which the governing equations for microstructure evolution in sintering are derived using variational principles. Our approach ensures thermodynamic consistency, with the driving force for grain motion derived from the system's free energy. Consequently, the proposed PFMMS guarantees the evolution of microstructure in a direction that reduces the system's energy and eliminates non-densifying phenomena. We rigorously validate PFMMS against recent benchmarks of theoretical and numerical analysis. It is found that PFMMS captures intrinsic stress distribution along and beyond grain boundaries, exhibits system-size-independent shrinkage strain, and maintains thermodynamic equilibrium states. These features are fundamental requirements for a physically consistent sintering model. △ Less

Submitted 25 July, 2024; originally announced July 2024.

Differential equations and recursive solutions for cosmological amplitudes

Authors: Song He, Xuhang Jiang, Jiahao Liu, Qinglin Yang, Yao-Qi Zhang

Abstract: Recently considerable efforts have been devoted to computing cosmological correlators and the corresponding wavefunction coefficients, as well as understanding their analytical structures. In this note, we revisit the computation of these ``cosmological amplitudes" associated with any tree or loop graph for conformal scalars with time-dependent interactions in the power-law FRW universe, directly… ▽ More Recently considerable efforts have been devoted to computing cosmological correlators and the corresponding wavefunction coefficients, as well as understanding their analytical structures. In this note, we revisit the computation of these ``cosmological amplitudes" associated with any tree or loop graph for conformal scalars with time-dependent interactions in the power-law FRW universe, directly in terms of iterated time integrals. We start by decomposing any such cosmological amplitude (for loop graph, the ``integrand" prior to loop integrations) as a linear combination of {\it basic time integrals}, one for each {\it directed graph}. We derive remarkably simple first-order differential equations involving such time integrals with edges ``contracted" one at a time, which can be solved recursively and the solution takes the form of Euler-Mellin integrals/generalized hypergeometric functions. By combining such equations, we then derive a complete system of differential equations for all time integrals needed for a given graph. Our method works for any graph: for a tree graph with $n$ nodes, this system can be transformed into the {\it canonical differential equations} of size $4^{n{-}1}$ quivalent to the graphic rules derived recently%so-called ``kinematic flow", and we also derive the system of differential equations for loop integrands {\it e.g.} of all-loop two-site graphs and one-loop $n$-gon graphs. Finally, we show how the differential equations truncate for the de Sitter (dS) case (in a way similar to differential equations for Feynman integrals truncate for integer dimensions), which immediately yields the complete symbol for the dS amplitude with interesting structures {\it e.g.} for $n$-site chains and $n$-gon cases. △ Less

Submitted 24 July, 2024; originally announced July 2024.

Comments: 43 pages; many figures

Motion Capture from Inertial and Vision Sensors

Authors: Xiaodong Chen, Wu Liu, Qian Bao, Xinchen Liu, Quanwei Yang, Ruoli Dai, Tao Mei

Abstract: Human motion capture is the foundation for many computer vision and graphics tasks. While industrial motion capture systems with complex camera arrays or expensive wearable sensors have been widely adopted in movie and game production, consumer-affordable and easy-to-use solutions for personal applications are still far from mature. To utilize a mixture of a monocular camera and very few inertial… ▽ More Human motion capture is the foundation for many computer vision and graphics tasks. While industrial motion capture systems with complex camera arrays or expensive wearable sensors have been widely adopted in movie and game production, consumer-affordable and easy-to-use solutions for personal applications are still far from mature. To utilize a mixture of a monocular camera and very few inertial measurement units (IMUs) for accurate multi-modal human motion capture in daily life, we contribute MINIONS in this paper, a large-scale Motion capture dataset collected from INertial and visION Sensors. MINIONS has several featured properties: 1) large scale of over five million frames and 400 minutes duration; 2) multi-modality data of IMUs signals and RGB videos labeled with joint positions, joint rotations, SMPL parameters, etc.; 3) a diverse set of 146 fine-grained single and interactive actions with textual descriptions. With the proposed MINIONS, we conduct experiments on multi-modal motion capture and explore the possibilities of consumer-affordable motion capture using a monocular camera and very few IMUs. The experiment results emphasize the unique advantages of inertial and vision sensors, showcasing the promise of consumer-affordable multi-modal motion capture and providing a valuable resource for further research and development. △ Less

Submitted 23 July, 2024; originally announced July 2024.

Comments: 17 pages,9 figures

DiffX: Guide Your Layout to Cross-Modal Generative Modeling

Authors: Zeyu Wang, Jingyu Lin, Yifei Qian, Yi Huang, Shicen Tian, Bosong Chai, Juncan Deng, Qu Yang, Lan Du, Cunjian Chen, Yufei Guo, Kejie Huang

Abstract: Diffusion models have made significant strides in language-driven and layout-driven image generation. However, most diffusion models are limited to visible RGB image generation. In fact, human perception of the world is enriched by diverse viewpoints, such as chromatic contrast, thermal illumination, and depth information. In this paper, we introduce a novel diffusion model for general layout-guid… ▽ More Diffusion models have made significant strides in language-driven and layout-driven image generation. However, most diffusion models are limited to visible RGB image generation. In fact, human perception of the world is enriched by diverse viewpoints, such as chromatic contrast, thermal illumination, and depth information. In this paper, we introduce a novel diffusion model for general layout-guided cross-modal generation, called DiffX. Notably, our DiffX presents a simple yet effective cross-modal generative modeling pipeline, which conducts diffusion and denoising processes in the modality-shared latent space. Moreover, we introduce the Joint-Modality Embedder (JME) to enhance the interaction between layout and text conditions by incorporating a gated attention mechanism. To facilitate the user-instructed training, we construct the cross-modal image datasets with detailed text captions by the Large-Multimodal Model (LMM) and our human-in-the-loop refinement. Through extensive experiments, our DiffX demonstrates robustness in cross-modal ''RGB+X'' image generation on FLIR, MFNet, and COME15K datasets, guided by various layout conditions. It also shows the potential for the adaptive generation of ''RGB+X+Y(+Z)'' images or more diverse modalities on COME15K and MCXFace datasets. Our code and constructed cross-modal image datasets are available at https://github.com/zeyuwang-zju/DiffX. △ Less

Submitted 24 August, 2024; v1 submitted 22 July, 2024; originally announced July 2024.

Enhancement of 3D Gaussian Splatting using Raw Mesh for Photorealistic Recreation of Architectures

Authors: Ruizhe Wang, Chunliang Hua, Tomakayev Shingys, Mengyuan Niu, Qingxin Yang, Lizhong Gao, Yi Zheng, Junyan Yang, Qiao Wang

Abstract: The photorealistic reconstruction and rendering of architectural scenes have extensive applications in industries such as film, games, and transportation. It also plays an important role in urban planning, architectural design, and the city's promotion, especially in protecting historical and cultural relics. The 3D Gaussian Splatting, due to better performance over NeRF, has become a mainstream t… ▽ More The photorealistic reconstruction and rendering of architectural scenes have extensive applications in industries such as film, games, and transportation. It also plays an important role in urban planning, architectural design, and the city's promotion, especially in protecting historical and cultural relics. The 3D Gaussian Splatting, due to better performance over NeRF, has become a mainstream technology in 3D reconstruction. Its only input is a set of images but it relies heavily on geometric parameters computed by the SfM process. At the same time, there is an existing abundance of raw 3D models, that could inform the structural perception of certain buildings but cannot be applied. In this paper, we propose a straightforward method to harness these raw 3D models to guide 3D Gaussians in capturing the basic shape of the building and improve the visual quality of textures and details when photos are captured non-systematically. This exploration opens up new possibilities for improving the effectiveness of 3D reconstruction techniques in the field of architectural design. △ Less

Submitted 22 July, 2024; originally announced July 2024.

A Benchmark for Gaussian Splatting Compression and Quality Assessment Study

Authors: Qi Yang, Kaifa Yang, Yuke Xing, Yiling Xu, Zhu Li

Abstract: To fill the gap of traditional GS compression method, in this paper, we first propose a simple and effective GS data compression anchor called Graph-based GS Compression (GGSC). GGSC is inspired by graph signal processing theory and uses two branches to compress the primitive center and attributes. We split the whole GS sample via KDTree and clip the high-frequency components after the graph Fouri… ▽ More To fill the gap of traditional GS compression method, in this paper, we first propose a simple and effective GS data compression anchor called Graph-based GS Compression (GGSC). GGSC is inspired by graph signal processing theory and uses two branches to compress the primitive center and attributes. We split the whole GS sample via KDTree and clip the high-frequency components after the graph Fourier transform. Followed by quantization, G-PCC and adaptive arithmetic coding are used to compress the primitive center and attribute residual matrix to generate the bitrate file. GGSS is the first work to explore traditional GS compression, with advantages that can reveal the GS distortion characteristics corresponding to typical compression operation, such as high-frequency clipping and quantization. Second, based on GGSC, we create a GS Quality Assessment dataset (GSQA) with 120 samples. A subjective experiment is conducted in a laboratory environment to collect subjective scores after rendering GS into Processed Video Sequences (PVS). We analyze the characteristics of different GS distortions based on Mean Opinion Scores (MOS), demonstrating the sensitivity of different attributes distortion to visual quality. The GGSC code and the dataset, including GS samples, MOS, and PVS, are made publicly available at https://github.com/Qi-Yangsjtu/GGSC. △ Less

Submitted 19 July, 2024; originally announced July 2024.

MSceneSpeech: A Multi-Scene Speech Dataset For Expressive Speech Synthesis

Authors: Qian Yang, Jialong Zuo, Zhe Su, Ziyue Jiang, Mingze Li, Zhou Zhao, Feiyang Chen, Zhefeng Wang, Baoxing Huai

Abstract: We introduce an open source high-quality Mandarin TTS dataset MSceneSpeech (Multiple Scene Speech Dataset), which is intended to provide resources for expressive speech synthesis. MSceneSpeech comprises numerous audio recordings and texts performed and recorded according to daily life scenarios. Each scenario includes multiple speakers and a diverse range of prosodic styles, making it suitable for… ▽ More We introduce an open source high-quality Mandarin TTS dataset MSceneSpeech (Multiple Scene Speech Dataset), which is intended to provide resources for expressive speech synthesis. MSceneSpeech comprises numerous audio recordings and texts performed and recorded according to daily life scenarios. Each scenario includes multiple speakers and a diverse range of prosodic styles, making it suitable for speech synthesis that entails multi-speaker style and prosody modeling. We have established a robust baseline, through the prompting mechanism, that can effectively synthesize speech characterized by both user-specific timbre and scene-specific prosody with arbitrary text input. The open source MSceneSpeech Dataset and audio samples of our baseline are available at https://speechai-demo.github.io/MSceneSpeech/. △ Less

Submitted 18 July, 2024; originally announced July 2024.

Comments: Accepted by INTERSPEECH 2024

SOMONITOR: Explainable Marketing Data Processing and Analysis with Large Language Models

Authors: Qi Yang, Sergey Nikolenko, Marlo Ongpin, Ilia Gossoudarev, Yu-Yi Chu-Farseeva, Aleksandr Farseev

Abstract: Online marketing faces formidable challenges in managing and interpreting immense volumes of data necessary for competitor analysis, content research, and strategic branding. It is impossible to review hundreds to thousands of transient online content items by hand, and partial analysis often leads to suboptimal outcomes and poorly performing campaigns. We introduce an explainable AI framework SoM… ▽ More Online marketing faces formidable challenges in managing and interpreting immense volumes of data necessary for competitor analysis, content research, and strategic branding. It is impossible to review hundreds to thousands of transient online content items by hand, and partial analysis often leads to suboptimal outcomes and poorly performing campaigns. We introduce an explainable AI framework SoMonitor that aims to synergize human intuition with AI-based efficiency, helping marketers across all stages of the marketing funnel, from strategic planning to content creation and campaign execution. SoMonitor incorporates a CTR prediction and ranking model for advertising content and uses large language models (LLMs) to process high-performing competitor content, identifying core content pillars such as target audiences, customer needs, and product features. These pillars are then organized into broader categories, including communication themes and targeted customer personas. By integrating these insights with data from the brand's own advertising campaigns, SoMonitor constructs a narrative for addressing new customer personas and simultaneously generates detailed content briefs in the form of user stories that can be directly applied by marketing teams to streamline content production and campaign execution. The adoption of SoMonitor in daily operations allows digital marketers to quickly parse through extensive datasets, offering actionable insights that significantly enhance campaign effectiveness and overall job satisfaction △ Less

Submitted 17 July, 2024; originally announced July 2024.

Evaluating the transferability potential of deep learning models for climate downscaling

Authors: Ayush Prasad, Paula Harder, Qidong Yang, Prasanna Sattegeri, Daniela Szwarcman, Campbell Watson, David Rolnick

Abstract: Climate downscaling, the process of generating high-resolution climate data from low-resolution simulations, is essential for understanding and adapting to climate change at regional and local scales. Deep learning approaches have proven useful in tackling this problem. However, existing studies usually focus on training models for one specific task, location and variable, which are therefore limi… ▽ More Climate downscaling, the process of generating high-resolution climate data from low-resolution simulations, is essential for understanding and adapting to climate change at regional and local scales. Deep learning approaches have proven useful in tackling this problem. However, existing studies usually focus on training models for one specific task, location and variable, which are therefore limited in their generalizability and transferability. In this paper, we evaluate the efficacy of training deep learning downscaling models on multiple diverse climate datasets to learn more robust and transferable representations. We evaluate the effectiveness of architectures zero-shot transferability using CNNs, Fourier Neural Operators (FNOs), and vision Transformers (ViTs). We assess the spatial, variable, and product transferability of downscaling models experimentally, to understand the generalizability of these different architecture types. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Surprising Performances of Students with Autism in Classroom with NAO Robot

Authors: Qin Yang, Huan Lu, Dandan Liang, Shengrong Gong, Huanghao Feng

Abstract: Autism is a developmental disorder that manifests in early childhood and persists throughout life, profoundly affecting social behavior and hindering the acquisition of learning and social skills in those diagnosed. As technological advancements progress, an increasing array of technologies is being utilized to support the education of students with Autism Spectrum Disorder (ASD), aiming to improv… ▽ More Autism is a developmental disorder that manifests in early childhood and persists throughout life, profoundly affecting social behavior and hindering the acquisition of learning and social skills in those diagnosed. As technological advancements progress, an increasing array of technologies is being utilized to support the education of students with Autism Spectrum Disorder (ASD), aiming to improve their educational outcomes and social capabilities. Numerous studies on autism intervention have highlighted the effectiveness of social robots in behavioral treatments. However, research on the integration of social robots into classroom settings for children with autism remains sparse. This paper describes the design and implementation of a group experiment in a collective classroom setting mediated by the NAO robot. The experiment involved special education teachers and the NAO robot collaboratively conducting classroom activities, aiming to foster a dynamic learning environment through interactions among teachers, the robot, and students. Conducted in a special education school, this experiment served as a foundational study in anticipation of extended robot-assisted classroom sessions. Data from the experiment suggest that ASD students in classrooms equipped with the NAO robot exhibited notably better performance compared to those in regular classrooms. The humanoid features and body language of the NAO robot captivated the students' attention, particularly during talent shows and command tasks, where students demonstrated heightened engagement and a decrease in stereotypical repetitive behaviors and irrelevant minor movements commonly observed in regular settings. Our preliminary findings indicate that the NAO robot significantly enhances focus and classroom engagement among students with ASD, potentially improving educational performance and fostering better social behaviors. △ Less

Submitted 26 June, 2024; originally announced July 2024.

Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

Authors: Qimin Yang, Rongsheng Wang, Jiexin Chen, Runqi Su, Tao Tan

Abstract: Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q\&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q\&A data exhibit extraordinary disease diagnostic abilities… ▽ More Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q\&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q\&A data exhibit extraordinary disease diagnostic abilities. However, we observed that despite improvements in specific domain knowledge, the performance of medical LLM in long-context understanding has significantly declined, especially compared to general language models with similar parameters. The purpose of this study is to investigate the phenomenon of reduced performance in understanding long-context in medical LLM. We designed a series of experiments to conduct open-book professional knowledge exams on all models to evaluate their ability to read long-context. By adjusting the proportion and quantity of general data and medical data in the process of fine-tuning, we can determine the best data composition to optimize the professional model and achieve a balance between long-context performance and specific domain knowledge. △ Less

Submitted 16 July, 2024; originally announced July 2024.

Comments: 5 pages, 1 figure. Accepted by the Workshop on Long-Context Foundation Models (LCFM) at ICML 2024

Qwen2-Audio Technical Report

Authors: Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, Jingren Zhou

Abstract: We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data an… ▽ More We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community. △ Less

Submitted 15 July, 2024; originally announced July 2024.

Comments: https://github.com/QwenLM/Qwen2-Audio. Checkpoints, codes and scripts will be opensoursed soon

SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling

Authors: Huizheng Wang, Jiahao Fang, Xinru Tang, Zhiheng Yue, Jinxi Li, Yubin Qin, Sihan Guan, Qize Yang, Yang Wang, Chao Li, Yang Hu, Shouyi Yin

Abstract: Benefiting from the self-attention mechanism, Transformer models have attained impressive contextual comprehension capabilities for lengthy texts. The requirements of high-throughput inference arise as the large language models (LLMs) become increasingly prevalent, which calls for large-scale token parallel processing (LTPP). However, existing dynamic sparse accelerators struggle to effectively ha… ▽ More Benefiting from the self-attention mechanism, Transformer models have attained impressive contextual comprehension capabilities for lengthy texts. The requirements of high-throughput inference arise as the large language models (LLMs) become increasingly prevalent, which calls for large-scale token parallel processing (LTPP). However, existing dynamic sparse accelerators struggle to effectively handle LTPP, as they solely focus on separate stage optimization, and with most efforts confined to computational enhancements. By re-examining the end-to-end flow of dynamic sparse acceleration, we pinpoint an ever-overlooked opportunity that the LTPP can exploit the intrinsic coordination among stages to avoid excessive memory access and redundant computation. Motivated by our observation, we present SOFA, a cross-stage compute-memory efficient algorithm-hardware co-design, which is tailored to tackle the challenges posed by LTPP of Transformer inference effectively. We first propose a novel leading zero computing paradigm, which predicts attention sparsity by using log-based add-only operations to avoid the significant overhead of prediction. Then, a distributed sorting and a sorted updating FlashAttention mechanism are proposed with a cross-stage coordinated tiling principle, which enables fine-grained and lightweight coordination among stages, helping optimize memory access and latency. Further, we propose a SOFA accelerator to support these optimizations efficiently. Extensive experiments on 20 benchmarks show that SOFA achieves $9.5\times$ speed up and $71.5\times$ higher energy efficiency than Nvidia A100 GPU. Compared to 8 SOTA accelerators, SOFA achieves an average $15.8\times$ energy efficiency, $10.3\times$ area efficiency and $9.3\times$ speed up, respectively. △ Less

Submitted 14 July, 2024; originally announced July 2024.

Noise Calibration: Plug-and-play Content-Preserving Video Enhancement using Pre-trained Video Diffusion Models

Authors: Qinyu Yang, Haoxin Chen, Yong Zhang, Menghan Xia, Xiaodong Cun, Zhixun Su, Ying Shan

Abstract: In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation… ▽ More In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation that considers both visual quality and consistency of content. Consistency of content is ensured by a proposed loss function that maintains the structure of the input, while visual quality is improved by utilizing the denoising process of pretrained diffusion models. To address the formulated optimization problem, we have developed a plug-and-play noise optimization strategy, referred to as Noise Calibration. By refining the initial random noise through a few iterations, the content of original video can be largely preserved, and the enhancement effect demonstrates a notable improvement. Extensive experiments have demonstrated the effectiveness of the proposed method. △ Less

Submitted 14 July, 2024; originally announced July 2024.

Comments: ECCV 2024, Project Page: https://yangqy1110.github.io/NC-SDEdit/, Code Repo: https://github.com/yangqy1110/NC-SDEdit/

ACM Class: I.2; I.4.3

Asynchronous Feedback Network for Perceptual Point Cloud Quality Assessment

Authors: Yujie Zhang, Qi Yang, Ziyu Shan, Yiling Xu

Abstract: Recent years have witnessed the success of the deep learning-based technique in research of no-reference point cloud quality assessment (NR-PCQA). For a more accurate quality prediction, many previous studies have attempted to capture global and local feature in a bottom-up manner, but ignored the interaction and promotion between them. To solve this problem, we propose a novel asynchronous feedba… ▽ More Recent years have witnessed the success of the deep learning-based technique in research of no-reference point cloud quality assessment (NR-PCQA). For a more accurate quality prediction, many previous studies have attempted to capture global and local feature in a bottom-up manner, but ignored the interaction and promotion between them. To solve this problem, we propose a novel asynchronous feedback network (AFNet). Motivated by human visual perception mechanisms, AFNet employs a dual-branch structure to deal with global and local feature, simulating the left and right hemispheres of the human brain, and constructs a feedback module between them. Specifically, the input point clouds are first fed into a transformer-based global encoder to generate the attention maps that highlight these semantically rich regions, followed by being merged into the global feature. Then, we utilize the generated attention maps to perform dynamic convolution for different semantic regions and obtain the local feature. Finally, a coarse-to-fine strategy is adopted to merge the two features into the final quality score. We conduct comprehensive experiments on three datasets and achieve superior performance over the state-of-the-art approaches on all of these datasets. The code will be available at https://github.com/zhangyujie-1998/AFNet. △ Less

Submitted 13 July, 2024; originally announced July 2024.

Study of a Novel Capacitive Pressure Sensor Using Spiral Comb Electrodes

Authors: Wenjie Chen, Qi Yang, Qi Liu, Yiqun Zhang, Liang He, Yuanlin Xia, Zhuqing Wang, Yubo Huang, Jianfeng Chen, Cao Xia

Abstract: For traditional capacitive pressure sensors, high nonlinearity and poor sensitivity greatly limited their sensing applications. Hence, an innovative design of capacitors based on spiral comb electrodes is proposed for high-sensitivity pressure detection in this work. Compared to traditional capacitive pressure sensors with straight plate electrodes, the proposed sensor with the spiral electrodes i… ▽ More For traditional capacitive pressure sensors, high nonlinearity and poor sensitivity greatly limited their sensing applications. Hence, an innovative design of capacitors based on spiral comb electrodes is proposed for high-sensitivity pressure detection in this work. Compared to traditional capacitive pressure sensors with straight plate electrodes, the proposed sensor with the spiral electrodes increases the overlap areas of electrodes sufficiently, the pressure sensitivity can thus be greatly improved. Moreover, the capacitance variation of the proposed sensor is dominated by the change of the overlap area of the electrodes rather than the electrode's distance, the linearity can also thus be improved to higher than 0.99. Theoretical analysis and COMSOL-based finite element simulation have been implemented for principle verification and performance optimization. Simulation results show that the proposed design has a mechanical sensitivity of 1.5x10-4 m/Pa, capacitive sensitivity of 1.10 aF/Pa, and nonlinear error of 3.63%, respectively, at the pressure range from 0 to 30 kPa. An equivalent experiment has been further carried out for verification. Experimental results also show that both the sensitivity and linearity of capacitive pressure sensors with spiral electrodes are higher than those with straight electrodes. This work not only provides a new avenue for capacitor design, but also can be applied to high-sensitivity pressure detection. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: 20 pages, 14 figures

MSC Class: -

Explicit-NeRF-QA: A Quality Assessment Database for Explicit NeRF Model Compression

Authors: Yuke Xing, Qi Yang, Kaifa Yang, Yilin Xu, Zhu Li

Abstract: In recent years, Neural Radiance Fields (NeRF) have demonstrated significant advantages in representing and synthesizing 3D scenes. Explicit NeRF models facilitate the practical NeRF applications with faster rendering speed, and also attract considerable attention in NeRF compression due to its huge storage cost. To address the challenge of the NeRF compression study, in this paper, we construct a… ▽ More In recent years, Neural Radiance Fields (NeRF) have demonstrated significant advantages in representing and synthesizing 3D scenes. Explicit NeRF models facilitate the practical NeRF applications with faster rendering speed, and also attract considerable attention in NeRF compression due to its huge storage cost. To address the challenge of the NeRF compression study, in this paper, we construct a new dataset, called Explicit-NeRF-QA. We use 22 3D objects with diverse geometries, textures, and material complexities to train four typical explicit NeRF models across five parameter levels. Lossy compression is introduced during the model generation, pivoting the selection of key parameters such as hash table size for InstantNGP and voxel grid resolution for Plenoxels. By rendering NeRF samples to processed video sequences (PVS), a large scale subjective experiment with lab environment is conducted to collect subjective scores from 21 viewers. The diversity of content, accuracy of mean opinion scores (MOS), and characteristics of NeRF distortion are comprehensively presented, establishing the heterogeneity of the proposed dataset. The state-of-the-art objective metrics are tested in the new dataset. Best Person correlation, which is around 0.85, is collected from the full-reference objective metric. All tested no-reference metrics report very poor results with 0.4 to 0.6 correlations, demonstrating the need for further development of more robust no-reference metrics. The dataset, including NeRF samples, source 3D objects, multiview images for NeRF generation, PVSs, MOS, is made publicly available at the following location: https://github.com/LittlericeChloe/Explicit_NeRF_QA. △ Less

Submitted 18 July, 2024; v1 submitted 11 July, 2024; originally announced July 2024.

Comments: 5 pages, 4 figures, 2 tables, conference

Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison

Authors: Qian Yang, Weixiang Yan, Aishwarya Agrawal

Abstract: Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence gener… ▽ More Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence generation, often suffer from overconfidence. Other methods use self-consistency comparison but are affected by confirmation biases. To alleviate these, we propose \textbf{De}compose and \textbf{C}ompare \textbf{C}onsistency (\texttt{DeCC}) for reliability measurement. By comparing the consistency between the direct answer generated using the VLM's internal reasoning process, and the indirect answers obtained by decomposing the question into sub-questions and reasoning over the sub-answers produced by the VLM, \texttt{DeCC} measures the reliability of VLM's direct answer. Experiments across six vision-language tasks with three VLMs show \texttt{DeCC}'s reliability estimation achieves better correlation with task accuracy compared to the existing methods. △ Less

Submitted 11 July, 2024; v1 submitted 10 July, 2024; originally announced July 2024.

Comments: Preprint