-
Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
Authors:
Wei An,
Xiao Bi,
Guanting Chen,
Shanhuang Chen,
Chengqi Deng,
Honghui Ding,
Kai Dong,
Qiushi Du,
Wenjun Gao,
Kang Guan,
Jianzhong Guo,
Yongqiang Guo,
Zhe Fu,
Ying He,
Panpan Huang,
Jiashi Li,
Wenfeng Liang,
Xiaodong Liu,
Xin Liu,
Yiyuan Liu,
Yuxuan Liu,
Shanghao Lu,
Xuan Lu,
Xiaotao Nie,
Tian Pei
, et al. (27 additional authors not shown)
Abstract:
The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic…
▽ More
The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC.
△ Less
Submitted 26 August, 2024;
originally announced August 2024.
-
AmBC-NOMA-Aided Short-Packet Communication for High Mobility V2X Transmissions
Authors:
Xinyue Pei,
Xingwei Wang,
Yingyang Chen,
Tingrui Pei,
Miaowen Wen
Abstract:
In this paper, we investigate the performance of ambient backscatter communication non-orthogonal multiple access (AmBC-NOMA)-assisted short packet communication for high-mobility vehicle-to-everything transmissions. In the proposed system, a roadside unit (RSU) transmits a superimposed signal to a typical NOMA user pair. Simultaneously, the backscatter device (BD) transmits its own signal towards…
▽ More
In this paper, we investigate the performance of ambient backscatter communication non-orthogonal multiple access (AmBC-NOMA)-assisted short packet communication for high-mobility vehicle-to-everything transmissions. In the proposed system, a roadside unit (RSU) transmits a superimposed signal to a typical NOMA user pair. Simultaneously, the backscatter device (BD) transmits its own signal towards the user pair by reflecting and modulating the RSU's superimposed signals. Due to vehicles' mobility, we consider realistic assumptions of time-selective fading and channel estimation errors. Theoretical expressions for the average block error rates (BLERs) of both users are derived. Furthermore, analysis and insights on transmit signal-to-noise ratio, vehicles' mobility, imperfect channel estimation, the reflection efficiency at the BD, and blocklength are provided. Numerical results validate the theoretical findings and reveal that the AmBC-NOMA system outperforms its orthogonal multiple access counterpart in terms of BLER performance.
△ Less
Submitted 26 May, 2024;
originally announced May 2024.
-
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Authors:
DeepSeek-AI,
Aixin Liu,
Bei Feng,
Bin Wang,
Bingxuan Wang,
Bo Liu,
Chenggang Zhao,
Chengqi Dengr,
Chong Ruan,
Damai Dai,
Daya Guo,
Dejian Yang,
Deli Chen,
Dongjie Ji,
Erhang Li,
Fangyun Lin,
Fuli Luo,
Guangbo Hao,
Guanting Chen,
Guowei Li,
H. Zhang,
Hanwei Xu,
Hao Yang,
Haowei Zhang,
Honghui Ding
, et al. (132 additional authors not shown)
Abstract:
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference…
▽ More
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.
△ Less
Submitted 19 June, 2024; v1 submitted 7 May, 2024;
originally announced May 2024.
-
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
Authors:
DeepSeek-AI,
:,
Xiao Bi,
Deli Chen,
Guanting Chen,
Shanhuang Chen,
Damai Dai,
Chengqi Deng,
Honghui Ding,
Kai Dong,
Qiushi Du,
Zhe Fu,
Huazuo Gao,
Kaige Gao,
Wenjun Gao,
Ruiqi Ge,
Kang Guan,
Daya Guo,
Jianzhong Guo,
Guangbo Hao,
Zhewen Hao,
Ying He,
Wenjie Hu,
Panpan Huang,
Erhang Li
, et al. (63 additional authors not shown)
Abstract:
The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B…
▽ More
The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.
△ Less
Submitted 5 January, 2024;
originally announced January 2024.
-
SelfCoLearn: Self-supervised collaborative learning for accelerating dynamic MR imaging
Authors:
Juan Zou,
Cheng Li,
Sen Jia,
Ruoyou Wu,
Tingrui Pei,
Hairong Zheng,
Shanshan Wang
Abstract:
Lately, deep learning has been extensively investigated for accelerating dynamic magnetic resonance (MR) imaging, with encouraging progresses achieved. However, without fully sampled reference data for training, current approaches may have limited abilities in recovering fine details or structures. To address this challenge, this paper proposes a self-supervised collaborative learning framework (S…
▽ More
Lately, deep learning has been extensively investigated for accelerating dynamic magnetic resonance (MR) imaging, with encouraging progresses achieved. However, without fully sampled reference data for training, current approaches may have limited abilities in recovering fine details or structures. To address this challenge, this paper proposes a self-supervised collaborative learning framework (SelfCoLearn) for accurate dynamic MR image reconstruction from undersampled k-space data. The proposed framework is equipped with three important components, namely, dual-network collaborative learning, reunderampling data augmentation and a specially designed co-training loss. The framework is flexible to be integrated with both data-driven networks and model-based iterative un-rolled networks. Our method has been evaluated on in-vivo dataset and compared it to four state-of-the-art methods. Results show that our method possesses strong capabilities in capturing essential and inherent representations for direct reconstructions from the undersampled k-space data and thus enables high-quality and fast dynamic MR imaging.
△ Less
Submitted 8 August, 2022;
originally announced August 2022.
-
Capacity Results for the Wiretapped Oblivious Transfer
Authors:
Tianyou Pei,
Wei Kang,
Nan Liu
Abstract:
In this paper, we study the problem of the 1-of-2 string oblivious transfer (OT) between Alice and Bob in the presence of a passive eavesdropper Eve. The eavesdropper Eve is not allowed to get any information about the private data of Alice or Bob. When Alice and Bob are honest-but-curious users, we propose a protocol that satisfies $1$-private (neither Alice nor Bob colludes with Eve) OT requirem…
▽ More
In this paper, we study the problem of the 1-of-2 string oblivious transfer (OT) between Alice and Bob in the presence of a passive eavesdropper Eve. The eavesdropper Eve is not allowed to get any information about the private data of Alice or Bob. When Alice and Bob are honest-but-curious users, we propose a protocol that satisfies $1$-private (neither Alice nor Bob colludes with Eve) OT requirements for the binary erasure symmetric broadcast channel, in which the channel provides dependent erasure patterns to Bob and Eve. We find that when the erasure probabilities satisfy certain conditions, the derived lower and upper bounds on the wiretapped OT capacity meet. Our results generalize and improve upon the results on wiretapped OT capacity by Mishra et al. Finally, we propose a protocol for a larger class of wiretapped channels and derive a lower bound on the wiretapped OT capacity.
△ Less
Submitted 5 September, 2022; v1 submitted 13 July, 2022;
originally announced July 2022.
-
Identifying Aggregation Artery Architecture of constrained Origin-Destination flows using Manhattan L-function
Authors:
Zidong Fang,
Hua Shu,
Ci Song,
Jie Chen,
Tianyu Liu,
Xiaohan Liu,
Tao Pei
Abstract:
The movement of humans and goods in cities can be represented by constrained flow, which is defined as the movement of objects between origin and destination in road networks. Flow aggregation, namely origins and destinations aggregated simultaneously, is one of the most common patterns, say the aggregated origin-to-destination flows between two transport hubs may indicate the great traffic demand…
▽ More
The movement of humans and goods in cities can be represented by constrained flow, which is defined as the movement of objects between origin and destination in road networks. Flow aggregation, namely origins and destinations aggregated simultaneously, is one of the most common patterns, say the aggregated origin-to-destination flows between two transport hubs may indicate the great traffic demand between two sites. Developing a clustering method for constrained flows is crucial for determining urban flow aggregation. Among existing methods about identifying flow aggregation, L-function of flows is the major one. Nevertheless, this method depends on the aggregation scale, the key parameter detected by Euclidean L-function, it does not adapt to road network. The extracted aggregation may be overestimated and dispersed. Therefore, we propose a clustering method based on L-function of Manhattan space, which consists of three major steps. The first is to detect aggregation scales by Manhattan L-function. The second is to determine core flows possessing highest local L-function values at different scales. The final step is to take the intersection of core flows neighbourhoods, the extent of which depends on corresponding scale. By setting the number of core flows, we could concentrate the aggregation and thus highlight Aggregation Artery Architecture (AAA), which depicts road sections that contain the projection of key flow cluster on the road networks. Experiment using taxi flows showed that AAA could clarify resident movement type of identified aggregated flows. Our method also helps selecting locations for distribution sites, thereby supporting accurate analysis of urban interactions.
△ Less
Submitted 20 August, 2021;
originally announced August 2021.
-
An adaptive Origin-Destination flows cluster-detecting method to identify urban mobility trends
Authors:
Mengyuan Fang,
Luliang Tang,
Zihan Kan,
Xue Yang,
Tao Pei,
Qingquan Li,
Chaokui Li
Abstract:
Origin-Destination (OD) flow, as an abstract representation of the object`s movement or interaction, has been used to reveal the urban mobility and human-land interaction pattern. As an important spatial analysis approach, the clustering methods of point events have been extended to OD flows to identify the dominant trends and spatial structures of urban mobility. However, the existing methods for…
▽ More
Origin-Destination (OD) flow, as an abstract representation of the object`s movement or interaction, has been used to reveal the urban mobility and human-land interaction pattern. As an important spatial analysis approach, the clustering methods of point events have been extended to OD flows to identify the dominant trends and spatial structures of urban mobility. However, the existing methods for OD flow cluster-detecting are limited both in specific spatial scale and the uncertain result due to different parameters setting, which is difficult for complicated OD flows clustering under spatial heterogeneity. To address these limitations, in this paper, we proposed a novel OD flows cluster-detecting method based on the OPTICS algorithm which can identify OD flow clusters with various aggregation scales. The method can adaptively determine parameter value from the dataset without prior knowledge and artificial intervention. Experiments indicated that our method outperformed three state-of-the-art methods with more accurate and complete of clusters and less noise. As a case study, our method is applied to identify the potential routes for public transport service settings by detecting OD flow clusters within urban travel data.
△ Less
Submitted 9 June, 2021;
originally announced June 2021.
-
A New Insight into Land Use Classification Based on Aggregated Mobile Phone Data
Authors:
Tao Pei,
Stanislav Sobolevsky,
Carlo Ratti,
Shih-Lung Shaw,
Chenghu Zhou
Abstract:
Land use classification is essential for urban planning. Urban land use types can be differentiated either by their physical characteristics (such as reflectivity and texture) or social functions. Remote sensing techniques have been recognized as a vital method for urban land use classification because of their ability to capture the physical characteristics of land use. Although significant progr…
▽ More
Land use classification is essential for urban planning. Urban land use types can be differentiated either by their physical characteristics (such as reflectivity and texture) or social functions. Remote sensing techniques have been recognized as a vital method for urban land use classification because of their ability to capture the physical characteristics of land use. Although significant progress has been achieved in remote sensing methods designed for urban land use classification, most techniques focus on physical characteristics, whereas knowledge of social functions is not adequately used. Owing to the wide usage of mobile phones, the activities of residents, which can be retrieved from the mobile phone data, can be determined in order to indicate the social function of land use. This could bring about the opportunity to derive land use information from mobile phone data. To verify the application of this new data source to urban land use classification, we first construct a time series of aggregated mobile phone data to characterize land use types. This time series is composed of two aspects: the hourly relative pattern, and the total call volume. A semi-supervised fuzzy c-means clustering approach is then applied to infer the land use types. The method is validated using mobile phone data collected in Singapore. Land use is determined with a detection rate of 58.03%. An analysis of the land use classification results shows that the accuracy decreases as the heterogeneity of land use increases, and increases as the density of cell phone towers increases.
△ Less
Submitted 23 October, 2013;
originally announced October 2013.
-
On Generation of Firewall Log Status Reporter (SRr) Using Perl
Authors:
Sugam Sharma,
Hari Cohly,
Tzusheng Pei
Abstract:
Computer System Administration and Network Administration are few such areas where Practical Extraction Reporting Language (Perl) has robust utilization these days apart from Bioinformatics. The key role of a System/Network Administrator is to monitor log files. Log file are updated every day. To scan the summary of large log files and to quickly determine if there is anything wrong with the serve…
▽ More
Computer System Administration and Network Administration are few such areas where Practical Extraction Reporting Language (Perl) has robust utilization these days apart from Bioinformatics. The key role of a System/Network Administrator is to monitor log files. Log file are updated every day. To scan the summary of large log files and to quickly determine if there is anything wrong with the server or network we develop a Firewall Log Status Reporter (SRr). SRr helps to generate the reports based on the parameters of interest. SRr provides the facility to admin to generate the individual firewall report or all reports in one go. By scrutinizing the results of the reports admin can trace how many times a particular request has been made from which source to which destination and can track the errors easily. Perl scripts can be seen as the UNIX script replacement in future arena and SRr is one development with the same hope that we can believe in. SRr is a generalized and customizable utility completely written in Perl and may be used for text mining and data mining application in Bioinformatics research and development too.
△ Less
Submitted 5 April, 2010;
originally announced April 2010.
-
On Utilization and Importance of Perl Status Reporter (SRr) in Text Mining
Authors:
Sugam Sharma,
Tzusheng Pei,
Hari Cohly
Abstract:
In Bioinformatics, text mining and text data mining sometimes interchangeably used is a process to derive high-quality information from text. Perl Status Reporter (SRr) is a data fetching tool from a flat text file and in this research paper we illustrate the use of SRr in text or data mining. SRr needs a flat text input file where the mining process to be performed. SRr reads input file and der…
▽ More
In Bioinformatics, text mining and text data mining sometimes interchangeably used is a process to derive high-quality information from text. Perl Status Reporter (SRr) is a data fetching tool from a flat text file and in this research paper we illustrate the use of SRr in text or data mining. SRr needs a flat text input file where the mining process to be performed. SRr reads input file and derives the high quality information from it. Typically text mining tasks are text categorization, text clustering, concept and entity extraction, and document summarization. SRr can be utilized for any of these tasks with little or none customizing efforts. In our implementation we perform text categorization mining operation on input file. The input file has two parameters of interest (firstKey and secondKey). The composition of these two parameters describes the uniqueness of entries in that file in the similar manner as done by composite key in database. SRr reads the input file line by line and extracts the parameters of interest and form a composite key by joining them together. It subsequently generates an output file consisting of the name as firstKey secondKey. SRr reads the input file and tracks the composite key. It further stores all that data lines, having the same composite key, in output file generated by SRr based on that composite key.
△ Less
Submitted 21 January, 2010; v1 submitted 19 January, 2010;
originally announced January 2010.
