Search Results (619)

In the complex clutter background, the clutter center frequency is not fixed, and the spectral width is wide, which leads to the performance degradation of the traditional adaptive clutter suppression method. Therefore, an adaptive clutter intelligent suppression method based on deep reinforcement learning (DRL) is proposed. Each range cell to be detected is regarded as an independent intelligence (agent) in the proposed method. The clutter environment is interactively learned using a deep learning (DL) process, and the filter parameter optimization is positively motivated by the reinforcement learning (RL) process to achieve the best clutter suppression effect. The suppression performance of the proposed method is tested on simulated and real data. The experimental results indicate that the filter notch designed by the proposed method is highly matched with the clutter compared with the existing adaptive clutter suppression methods. While suppressing the clutter, it has a higher amplitude-frequency response to signals at non-clutter frequencies, thus reducing the loss of the target signal and maximizing the output signal-to-clutter and noise rate (SCNR). Full article

Autonomous underwater vehicles (AUVs) have been increasingly applied in marine environmental monitoring. Their outstanding capability of performing tasks without human intervention makes them a popular tool for environmental data collection, especially in unknown and remote regions. This paper addresses the path planning problem when AUVs are used to perform plume source tracing in an unknown environment. The goal of path planning is to locate the plume source efficiently. The path planning approach is developed using the Double Deep Q-Network (DDQN) algorithm in the deep reinforcement learning (DRL) framework. The AUV gains knowledge by interacting with the environment, and the optimal direction is extracted from the mapping obtained by a deep neural network. The proposed approach was tested by numerical simulation and on a real ground vehicle. In the numerical simulation, several initial sampling strategies were compared on the basis of survey efficiency. The results show that direct learning based on the interaction with the environment could be an appropriate survey strategy for plume source tracing problems. The comparison with the canonical lawnmower path used in practice showed that path planning using DRL algorithms could be potentially promising for large-scale environment exploration. Full article

Massive computational resources are required by a booming number of artificial intelligence (AI) services in the communication network of the smart grid. To alleviate the computational pressure on data centers, edge computing first network (ECFN) can serve as an effective solution to realize distributed model training based on data parallelism for AI services in smart grid. Due to AI services with diversified types, an edge data center has a changing workload in different time periods. Selfish edge data centers from different edge suppliers are reluctant to share their computing resources without a rule for fair competition. AI services-oriented dynamic computational resource scheduling of edge data centers affects both the economic profit of AI service providers and computational resource utilization. This letter mainly discusses the partition and distribution of AI data based on distributed model training and dynamic computational resource scheduling problems among multiple edge data centers for AI services. To this end, a mixed integer linear programming (MILP) model and a Deep Reinforcement Learning (DRL)-based algorithm are proposed. Simulation results show that the proposed DRL-based algorithm outperforms the benchmark in terms of profit of AI service provider, backlog of distributed model training tasks, running time and multi-objective optimization. Full article

Mobile Edge Computing (MEC) is crucial for reducing latency by bringing computational resources closer to the network edge, thereby enhancing the quality of services (QoS). However, the broad deployment of cloudlets poses challenges in efficient network slicing, particularly when traffic distribution is uneven. Therefore, these challenges include managing diverse resource requirements across widely distributed cloudlets, minimizing resource conflicts and delays, and maintaining service quality amid fluctuating request rates. Addressing this requires intelligent strategies to predict request types (common or urgent), assess resource needs, and allocate resources efficiently. Emerging technologies like edge computing and 5G with network slicing can handle delay-sensitive IoT requests rapidly, but a robust mechanism for real-time resource and utility optimization remains necessary. To address these challenges, we designed an end-to-end network slicing approach that predicts common and urgent user requests through T distribution. We formulated our problem as a multi-agent Markov decision process (MDP) and introduced a multi-agent soft actor–critic (MAgSAC) algorithm. This algorithm prevents the wastage of scarce resources by intelligently activating and deactivating virtual network function (VNF) instances, thereby balancing the allocation process. Our approach aims to optimize overall utility, balancing trade-offs between revenue, energy consumption costs, and latency. We evaluated our method, MAgSAC, through simulations, comparing it with the following six benchmark schemes: MAA3C, SACT, DDPG, S2Vec, Random, and Greedy. The results demonstrate that our approach, MAgSAC, optimizes utility by 30%, minimizes energy consumption costs by 12.4%, and reduces execution time by 21.7% compared to the closest related multi-agent approach named MAA3C. Full article

This study aims to assess effective radiation doses (EDs) for pediatric computed tomography (CT) procedures in Madinah (single hospital), Saudi Arabia, and propose diagnostic reference levels (DRLs) for these procedures. This retrospective study collected data from 600 pediatric patients who underwent five CT procedures. The data were categorized by the type of CT procedure and the age of the patients. EDs and proposed DRLs for the pediatric CT procedures were computed. The highest EDs were found for abdominal (6.3 mSv) and head (4.8 mSv) CT procedures in pediatric patients aged <1 year. DRLs of the CTDI_vol and DLP for abdominal and head CT procedures in pediatric patients aged <1 year were 4.2 mGy, 94 mGy.cm and 25 mGy, 414 mGy.cm, respectively. Chest EDs had the lowest EDs among all pediatric CT procedures, with EDs of 1.93, 1.51, 1.91, and 2.05 mSv in patients aged <1, 1 ≤ to < 5, 5 ≤ to < 10, and 10 ≤ to ≤ 15 years, respectively. It can be concluded that optimization is required for abdominal and head CT procedures in pediatric patients aged <1 year. Frequent updates on ED and DRL calculations will help monitor radiation doses and minimize radiation risks for patients undergoing these procedures. Full article

The rapid expansion of renewable energy in buildings has been expedited by technological advancements and government policies. However, including highly permeable intermittent renewables and energy storage presents significant challenges for traditional home energy management systems (HEMSs). Deep reinforcement learning (DRL) is regarded as the most efficient approach for tackling these problems because of its robust nonlinear fitting capacity and capability to operate without a predefined model. This paper presents a DRL control method intended to lower energy expenses and elevate renewable energy usage by optimizing the actions of the battery and heat pump in HEMS. We propose four DRL algorithms and thoroughly assess their performance. In pursuit of this objective, we also devise a new reward function for multi-objective optimization and an interactive environment grounded in expert experience. The results demonstrate that the TD3 algorithm excels in cost savings and PV self-consumption. Compared to the baseline model, the TD3 model achieved a 13.79% reduction in operating costs and a 5.07% increase in PV self-consumption. Additionally, we explored the impact of the feed-in tariff (FiT) on TD3’s performance, revealing its resilience even when the FiT decreases. This comparison provides insights into algorithm selection for specific applications, promoting the development of DRL-driven energy management solutions. Full article

Sparse-angle X-ray Computed Tomography (CT) plays a vital role in industrial quality control but leads to an inherent trade-off between scan time and reconstruction quality. Adaptive angle selection strategies try to improve upon this based on the idea that the geometry of the object under investigation leads to an uneven distribution of the information content over the projection angles. Deep Reinforcement Learning (DRL) has emerged as an effective approach for adaptive angle selection in X-ray CT. While previous studies focused on optimizing generic image quality measures using a fixed number of angles, our work extends them by considering a specific downstream task, namely image-based defect detection, and introducing flexibility in the number of angles used. By leveraging prior knowledge about typical defect characteristics, our task-adaptive angle selection method, adaptable in terms of angle count, enables easy detection of defects in the reconstructed images. Full article

As global energy demand rises and climate change poses an increasing threat, the development of sustainable, low-carbon energy solutions has become imperative. This study focuses on optimizing shared energy storage (SES) and distribution networks (DNs) using deep reinforcement learning (DRL) techniques to enhance operation and decision-making capability. An innovative dynamic carbon intensity calculation method is proposed, which more accurately calculates indirect carbon emissions of the power system through network topology in both spatial and temporal dimensions, thereby refining carbon responsibility allocation on the user side. Additionally, we integrate user-side SES and ladder-type carbon emission pricing into DN to create a low-carbon economic dispatch model. By framing the problem as a Markov decision process (MDP), we employ the DRL, specifically the deep deterministic policy gradient (DDPG) algorithm, enhanced with prioritized experience replay (PER) and orthogonal regularization (OR), to achieve both economic efficiency and environmental sustainability. The simulation results indicate that this method significantly reduces the operating costs and carbon emissions of DN. This study offers an innovative perspective on the synergistic optimization of SES with DN and provides a practical methodology for low-carbon economic dispatch in power systems. Full article

Driven by the rise of intelligent manufacturing and Industry 4.0, the manufacturing industry faces significant challenges in adapting to flexible and efficient production methods. This study presents an innovative approach to solving the Flexible Job-Shop Scheduling Problem (FJSP) by integrating Heterogeneous Graph Neural Networks based on Relation (HGNNR) with Deep Reinforcement Learning (DRL). The proposed framework models the complex relationships in FJSP using heterogeneous graphs, where operations and machines are represented as nodes, with directed and undirected arcs indicating dependencies and compatibilities. The HGNNR framework comprises four key components: relation-specific subgraph decomposition, data preprocessing, feature extraction through graph convolution, and cross-relation feature fusion using a multi-head attention mechanism. For decision-making, we employ the Proximal Policy Optimization (PPO) algorithm, which iteratively updates policies to maximize cumulative rewards through continuous interaction with the environment. Experimental results on four public benchmark datasets demonstrate that our proposed method outperforms four state-of-the-art DRL-based techniques and three common rule-based heuristic algorithms, achieving superior scheduling efficiency and generalization capabilities. This framework offers a robust and scalable solution for complex industrial scheduling problems, enhancing production efficiency and adaptability. Full article

Intelligent scheduling of knitting workshops is the key to realizing knitting intelligent manufacturing. In view of the uncertainty of the workshop environment, it is difficult for existing scheduling algorithms to flexibly adjust scheduling strategies. This paper proposes a scheduling algorithm architecture based on deep reinforcement learning (DRL). First, the scheduling problem of knitting intelligent workshops is represented by a disjunctive graph, and a mathematical model is established. Then, a multi-proximal strategy (multi-PPO) optimization training algorithm is designed to obtain the optimal strategy, and the job selection strategy and machine selection strategy are trained at the same time. Finally, a knitting intelligent workshop scheduling experimental platform is built, and the algorithm proposed in this paper is compared with common heuristic rules and metaheuristic algorithms for experimental testing. The results show that the algorithm proposed in this paper is superior to heuristic rules in solving the knitting workshop scheduling problem, and can achieve the accuracy of the metaheuristic algorithm. In addition, the response speed of the algorithm in this paper is excellent, which meets the production scheduling needs of knitting intelligent workshops and has a good guiding significance for promoting knitting intelligent manufacturing. Full article

With the emergence of vehicular edge computing (VEC) and electric vehicles (EVs), integrating computation and charging tasks presents challenges due to limited resources and dynamic vehicular networks. This research focuses on the joint optimization of computation offloading and charging scheduling in VEC networks. Specifically, we optimize the offloading factor, charging association variable, and charging rates to minimize the system delay and energy consumption by leveraging the multi-attributes of EVs in both information and energy networks. Considering the dynamic environment, we model the problem as a Markov Decision Process, and use the Multi-Agent Reinforcement Learning (MARL) algorithm MADDPG, with its centralized training and distributed execution mechanisms. Simulation results demonstrate that this approach significantly improves utility while reducing energy consumption and latency. Full article

In the rapidly evolving domain of mobile edge–cloud computing (MECC), the proliferation of Internet of Things (IoT) devices and mobile applications poses significant challenges, particularly in dynamically managing computational demands and user mobility. Current research has partially addressed aspects of service migration and resource allocation, yet it often falls short in thoroughly examining the nuanced interdependencies between migration strategies and resource allocation, the consequential impacts of migration delays, and the intricacies of handling incomplete tasks during migration. This study advances the discourse by introducing a sophisticated framework optimized through a deep reinforcement learning (DRL) strategy, underpinned by a Markov decision process (MDP) that dynamically adapts service migration and resource allocation strategies. This refined approach facilitates continuous system monitoring, adept decision making, and iterative policy refinement, significantly enhancing operational efficiency and reducing response times in MECC environments. By meticulously addressing these previously overlooked complexities, our research not only fills critical gaps in the literature but also enhances the practical deployment of edge computing technologies, contributing profoundly to both theoretical insights and practical implementations in contemporary digital ecosystems. Full article

Automated driving is a promising development in reducing driving accidents and improving the efficiency of driving. This study focuses on developing a decision-making strategy for autonomous vehicles, specifically addressing maneuvers such as lane change, double lane change, and lane keeping on highways, using deep reinforcement learning (DRL). To achieve this, a highway driving environment in the commercial multi-body simulation software IPG Carmaker 11 version is established, wherein the ego vehicle navigates through surrounding vehicles safely and efficiently. A hierarchical control framework is introduced to manage these vehicles, with upper-level control handling driving decisions. The DDPG (deep deterministic policy gradient) algorithm, a specific DRL method, is employed to formulate the highway decision-making strategy, simulated in MATLAB software. Also, the computational procedures of both DDPG and deep Q-network algorithms are outlined and compared. A set of simulation tests is carried out to evaluate the effectiveness of the suggested decision-making policy. The research underscores the advantages of the proposed framework concerning its convergence rate and control performance. The results demonstrate that the DDPG-based overtaking strategy enables efficient and safe completion of highway driving tasks. Full article

Time-sensitive networking (TSN) technologies have garnered attention for supporting time-sensitive communication services, with recent interest extending to the wireless domain. However, adapting TSN to wireless areas faces challenges due to the competitive channel utilization in IEEE 802.11, necessitating exclusive channels for low-latency services. Additionally, traditional TSN scheduling algorithms may cause significant transmission delays due to dynamic wireless characteristics, which must be addressed. This paper proposes a wireless TSN model of IEEE 802.11 networks for the exclusive channel access and a novel time-sensitive traffic scheduler, named the wireless intelligent scheduler (WISE), based on deep reinforcement learning. We designed a deep reinforcement learning (DRL) framework to learn the repetitive transmission patterns of time-sensitive traffic and address potential latency issues from changing wireless conditions. Within this framework, we identified the most suitable DRL model, presenting the WISE algorithm with the best performance. Experimental results indicate that the proposed mechanisms meet up to 99.9% under the various wireless communication scenarios. In addition, they show that the processing delay is successfully limited within the specific time requirements and the scalability of TSN streams is guaranteed by the proposed mechanisms. Full article

Vehicle routing problems (VRPs) are challenging problems. Many variants of the VRP have been proposed. However, few studies on VRP have combined robustness and just-in-time (JIT) requirements with uncertainty. To solve the problem, this paper proposes the just-in-time-based robust multiobjective vehicle routing problem with time windows (JIT-RMOVRPTW) for the assembly workshop. Based on the conflict between uncertain time and JIT requirements, a JIT strategy was proposed. To measure the robustness of the solution, a metric was designed as the objective. Afterwards, a two-stage nondominated sorting ant colony algorithm with deep reinforcement learning (NSACOWDRL) was proposed. In stage I, ACO combines with NSGA-III to obtain the Pareto frontier. Based on the model, a pheromone update strategy and a transfer probability formula were designed. DDQN was introduced as a local search algorithm which trains networks through Pareto solutions to participate in probabilistic selection and nondominated sorting. In stage II, the Pareto frontier was quantified in feasibility by Monte Carlo simulation, and tested by diversity-robust selection based on uniformly distributed weights in the solution space to select robust Pareto solutions that take diversity into account. The effectiveness of NSACOWDRL was demonstrated through comparative experiments with other algorithms on instances. The impact of JIT strategy is analyzed and the effect of networks on the NSACOWDRL is further discussed. Full article