Introduction

South-East Asia is one of the regions of the world that suffers from serious damage of natural hazards every year. Typhoon is one of the severe weather phenomena that hits this region every year. The latest major disaster is Typhoon No. 30 in November 2013. It is reported that the total number of deaths and missing in the Philippines were about eight thousands. Damage to crops and infrastructures is estimated at 750 million US dollars since over 1.14 million houses were damaged. Palau, Vietnam, China, and Taiwan suffered from serious damage by the typhoon as well. The regional phenomena, such as floods, droughts, and heatwaves also cause significant damage. The continuous heavy rain had caused flooding across 9 provinces of southern Thailand in November 2017. The flood death toll reached five on 25 November 2017, as reported by the Department of Disaster Prevention and Mitigation (DDPM), Thailand. The floods had also affected a total of 74 districts, 113,900 households, and 385,498 people. Recently, trees and utility poles were leveled and roads were flooded on 4 January 2019 as rain, heavy wind and seawater from the tropical storm Pabuk lashed coastal villages and tourist resorts on southern Thailand’s east coast. One person was reported dead and another missing after a fishing boat encountered rough seas. Before the storm, more than 6,100 people were forced to flee their homes in four provinces, according to the DDPM, Thailand.

From the viewpoint of weather disaster mitigation, quick dissemination of weather information to all the countries in the risk region is of great importance. Meteorological satellites in geostationary orbit play an important role to provide continuous atmospheric observations both for weather forecasting and for monitoring a wide range of environmental phenomena. The third-generation meteorological satellites equipped with highly improved imager, such as Advanced Himawari Imager (AHI) on board Himawari-8 and Himawari-9 (Bessho et al. 2016) or Advanced Baseline Imager (ABI) on board Geostationary Operational Environmental Satellite (GOES)-16 (Schmit et al. 2016), yield a large amount of remotely sensed image data. Himawari-8 was successfully launched from Tanegashima Space Center in Kagoshima, Japan on 7 October 2014 and settled in the geostationary orbit on 16 October 2014. Himawari-9 was next launched on 2 November 2016 and entered operation as a backup to the currently operational Himawari-8 on 10 March 2017 after a period of in-orbit testing. It will eventually replace the Himawari-8. Both Himawari-8 and Himawari-9 are located in close proximity at around 140.7 degrees East, separated by only 0.1 degree, and observe the East Asia and Western Pacific regions for a period of 15 years. GOES-16 was launched from Cape Canaveral in Florida, United State Of America on 19 November 2016 and is a collaboration between National Oceanic and Atmospheric Administration (NOAA) and National Aeronautics and Space Administration (NASA).

After the commencement of the Himawari-8 operation, there are several websites that provide Himawari-8 remotely sensed images. Surely, these meteorological satellites bring us valuable information on weather, but it is not easy to provide the large information amount on a website in real time. Himawari-8 real-time web developed and maintained by the National Institute of Information and Communications Technology (NICT), Japan provides full-resolution remotely sensed images in real time (NICT Science Cloud 2019b; Murata et al. 2018). The Himawari-8 real-time web is now available in multiple languages, including English, Japanese, Korean, Chinese, Indonesian, Myanmar, Thai, Russian, French, and Tetum. Currently, there are international accesses to the website from over 17 countries every day. To reduce network traffic and improve the web access speed, the scalability issues such as mirror websites are considered. However, as mentioned above, the large information amount also becomes a major barrier for scalability, e.g., data processing and transferring.

Japan Gigabit Network (JGN) is the ultra high-speed research and development (R&D) network infrastructure for information and communications technology (ICT) development (NICT 2019a). It provides services to advance researchers’ R&D activities among industry, academia, and government. Asia-Pacific Advanced Network (APAN) is a non-profit international backbone network that connects the research and education networks of the Asia-Pacific region members to other research networks around the world (APAN Ltd. 2019). It aims to promote developments in network technology and advances in network-based applications and services. Both JGN and APAN are working together under collaborative environments and are expected to contribute to the disaster prevention and mitigation.

In general, a File Transfer Protocol (FTP) application, which relies on Transmission Control Protocol (TCP), is defined to provide a simple, effective, and reliable method for transferring files between any two hosts. It is expected to transfer data within required time interval. However, TCP, the most popular transport protocol, occasionally achieves unsatisfactory throughput in international networks. There are many techniques to solve this issue: Wide Area Network (WAN) accelerators such as Hitachi WAN Accelerator GX-1000 (Hitachi High-Technologies Corporation 2019) and WAN optimization (Riverbed Technology 2019), TCP variants such as HighSpeed TCP (Floyd 2003), Delay-Tolerant Networking (DTN) (Cerf et al. 2007), and Long Fat pipe TCP (LFTCP) (Geek magazine 2018), and User Datagram Protocol (UDP)-based protocols such as UDP-based transfer protocol (UDT) (Gu and Grossman 2007), Universal Network Acceleration Protocol (UNAP) (Oguchi et al. 2016), and High-performance and Flexible Protocol (HpFP) (Murata et al. 2016a). Since most of them are commercial (not open source), the detailed algorithms are not open to the public. Therefore, any customization for a specific purpose is not easy.

In this paper, we propose a mirroring system to provide a cost-effective solution for the mirror websites of the Himawari-8 real-time web. Our goal is to set up the mirror websites of the Himawari-8 real-time web in several high disaster-prone countries in Asia and Oceania region. It is necessary to develop the mirroring system that is cost-effective to provide quality service to as many people as possible. A mirroring model is introduced to avoid the big data processing in the mirror websites, while a HpPF-based file copy tool is adopted to facilitate the big data transferring. The first mirror website is set up at an institution in Thailand connected via JGN/APAN. The results show that the proposed mirroring system is able to operate effectively by reducing the central processing unit (CPU) usage in the mirror website and transferring remotely sensed image files at high speed over the international networks even under packet loss conditions.

The rest of the paper is organized as follows. Section “Himawari-8 real-time web” describes our works on the Himawari-8 real-time web. Section “?? ??” then presents the design and implementation of our mirroring system. Performance analysis results are presented in “Results”, followed by conclusions and further works in “Conclusions”.

Himawari-8 real-time web

Design

Himawari-8 real-time web, which is developed and maintained by NICT, is a web-based data visualization for Himawari-8 satellite sensed images in real time and with full resolution (NICT Science Cloud 2019b). It is supported by an ecosystem, which performs a tiled pyramid representation and a parallel processing of the data on an academic cloud (Murata et al. 2018). Both tiled pyramid image files and movie files are generated from the original Portable Network Graphics (PNG) file provided by Japanese Meteorological Agency (JMA). The coordinate system in the PNG is subject to the as-is view of the Earth from the satellite. It should be noted that the Himawari-8 real-time web does not support any standard tile image formats as well, but depends on its original one.

Data

The Himawari-8 carries the AHI scanning five areas: full disk, Japan Area, Target Area, and two Landmark Areas (Bessho et al. 2016). The full disk are images of the whole Earth as seen from the satellites. Since the objective of this paper is to provide Himawari-8 remotely sensed images for Asia and Oceania countries, we consider on the full disk data only. Figure 1 shows a screen capture image of the Himawari-8 real-time web on 4 January 2019. The image capturing range of the full disk is preliminarily fixed, thus the file size of tiled pyramid image does not change in each time step.

Fig. 1
figure 1

Full disk image provided by Himawari-8 real-time web

Full size image

The number of tiled pyramid image files in each time step is 741 for the full disk. The spatial resolution (full resolution) of the provided PNG file is 1 km in the full disk. There are 12 zoom levels for the full disk provided by the Himawari-8 real-time web. The original spatial resolution of the full disk is 11000 × 11000 pixels. The size of single tiled pyramid image is 550 × 550 pixels. The maximum and minimum file sizes in a day are about 700 KB at noon and 0.4 KB at midnight, respectively. Note that the file size changes depending on the local time of day, since the PNG compression works particularly well for black area.

International accesses

The number of international accesses in 2018 is 508,371 from 208 countries, which is as much as 2.8 times higher than in 2016. This motivates us to introduce an international mirroring system for the Himawari-8 real-time web.

Proposed mirroring system

Overview

In this section, we propose a mirroring system to provide a cost-effective solution for the mirror websites of the Himawari-8 real-time web. In general, a mirror website is a complete copy of a principal website that is placed under a different address. All data of the principal website are transferred to the mirror websites, and the same data sets are accessed on the same web applications. Figure 2 shows an ordinary mirroring model of the Himawari-8 real-time web. The mirror websites located in other countries need to quickly download the original remotely sensed image files from the HimawariCloud located in Japan (Murata et al. 2013), and concurrently process the image files to generate a set of tiled pyramid image files. The number of generated tiled pyramid images is as many as 741 in every 10 minutes, as shown in Table 1. Such big data processing requires more CPU usage and network resources, which leads to high cost. Eventually, both data processing and transferring become barriers to deploy the mirror websites of the Himawari-8 real-time web.

Fig. 2
figure 2

Ordinary mirroring model of Himawari-8 real-time web

Full size image
Table 1 Data size of remotely sensed image file and tiled pyramid image files
Full size table

Mirroring model

We introduce a novel mirroring model of the Himawari-8 real-time web to solve the data processing and transferring problems. The key idea behind this model is to eliminate the data processing in the mirror websites and transfer the generated images from the Himawari-8 real-time web instead of the original remotely sensed image files from the HimawariCloud. Figure 3 shows the novel mirroring model of the Himawari-8 real-time web. The original data set is transferred from the HimawariCloud provided by JMA to the NICT Science Cloud every 10 minutes. The data are therein converted into the tiled pyramid image files on the Himawari-8 real-time web. It should be noted that in our novel mirroring model, we utilize the NICT Science Cloud for data processing and provisions (Murata et al. 2018). The NICT Science Cloud, which is an academic cloud, provides computing resources and ecosystems for a variety of scientific workflows (Murata et al. 2014). For data processing, we exploit parallel processing to create tiled pyramid image files via a task scheduler on Gfarm file system (Tatebe et al. 2010), called the Pwrake (Tanaka and Tatebe 2012). For data provisions, we customize the World Science Data Bank (WSDBank), which is a web application designed to provide the science data (Murata et al. 2014), for the Himawari data. The set of tiled pyramid image files are then immediately transferred to the mirror websites. In this study, we locate the mirror website at National Electronics and Computer Technology Center (NECTEC) in Bangkok, Thailand (NICT Science Cloud and NECTEC 2019c). Note that since no data processing is performed on the mirror websites in Fig. 3, the small and less expensive personal computers (PCs) with a low-powered CPU and a small-sized memory are sufficient. In our novel mirroring model, we utilize the JGN/APAN, as discussed in “Introduction”.

Fig. 3
figure 3

Novel mirroring model of Himawari-8 real-time web

Full size image

File transfer tool

We adopt a HpFP-based file copy tool, named high-performance copy (hcp), to transfer the generated images from the Himawari-8 real-time web to the mirror websites. HpFP, which is implemented on UDP, puts more emphasis on latency and packet loss tolerance than on fairness and friendliness (Murata et al. 2016a). It achieves high throughput even over networks with high latency and packet loss and is succeeded in high-speed data transfer over the satellite networks (Murata et al. 2016b; Pavarangkoon et al. 2016), parallel storage system (Murata et al. 2016c), and meteorological phased array radar data transfer over long fat networks (LFNs) (Murata et al. 2016c, 2016d).

The hcp can be run from the command line on major Linux distributions. It has succeeded in transferring weather radar data files in real time (Murata et al. 2016d). Therein the total file size to be transferred is 499 MB. The hcp achieves throughput of 489.2 Mbps over the network with 42.4 ms round trip time (RTT) and 0.02% packet loss ratio (PLR).

Monitoring tool

Our mirroring system is designed and implemented for practical use. The monitoring of the remote servers, networks, and data deployed in Asia and Oceania region is important for real operation of the system. In this paper, we adopt an easy-to-use monitoring tool, namely Wide-area Observation Network Monitoring (WONM) (Murata et al. 2013). The WONM is designed as a light-weight and easy-to-use tool working on Linux, Windows, and Mac OS X. A system manager sets up the WONM application in each remote server. The agent running at a cloud system constantly collects the information of remote servers and displays the information on the screens of personal computers (PCs) or smartphones. This tool brings us the information of CPU, memory, disk, motherboard, network card, system, and uninterruptible power supply (UPS) in a certain interval such as every minute. We successfully operate a world-wide ionospheric observation network, a cloud computer network, a phased array weather radar system, and a small-sized weather radar array system using the WONM.

In this paper, we customize the WONM for monitoring the mirror websites deployed over Asia and Oceania countries, as shown in Fig. 4. Besides the basic information discussed above, we add new information of data transfer status from the Himawari-8 real-time web, access log collection from the mirror websites, and network condition using hperf (Murata et al. 2016a). The hperf, which is a HpFP-based application, is developed to measure the end-to-end throughput, the network latency, and the packet loss in LFNs.

Fig. 4
figure 4

Appearance of WONM

Full size image

Results

Experiment model

Figure 5 shows a experiment model of mirroring system for the Himawari-8 real-time web, where the mirror website is located at the NECTEC in Bangkok, Thailand. As discussed in “Mirroring model”, we develop the mirror website of the Himawari-8 real-time web using a small-size web server for real-time image view only. A set of tiled pyramid image files is generated on the Himawari-8 real-time web in Japan, and then transferred to the mirror website in Thailand over the JGN/APAN as soon as possible. On the mirror website, a web application provides the full-resolution remotely sensed images.

Fig. 5
figure 5

Experiment model of mirroring system for Himawari-8 real-time web

Full size image

Network conditions

For the successful mirroring of the Himawari-8 real-time web in Fig. 5, one of the most important keys is the capability of real-time file transfer. As mentioned in “Data”, the data amount of the Himawari-8 real-time web to be transferred to the mirror website is 370 MB in total per every 10 minutes, corresponding to 4.93 Mbps. Therefore, the high-bandwidth international network, which constantly provides more than 4.93 Mbps bandwidth, is a necessary condition. Taking into consideration other processes such as file input/output (I/O), file deployment, and other overheads, we require at least more than 10 Mbps. The JGN/APAN is adopted in our experiment model, as shown in Fig. 5. Figure 6 shows an example of daily, weekly, monthly, and yearly traffics on the JGN/APAN between Singapore and Thailand. Note that the bandwidth of this network is 1 Gbps. There are constant traffics therein, but they occupy less than 800 Mbps in most of the times. This implies that the available bandwidth is enough to transfer data for our mirroring system.

Fig. 6
figure 6

Traffic on JGN/APAN on 20 February 2019

Full size image

We investigate the network conditions between the Himawari-8 real-time web and the mirror website using the traceroute command. Figure 7 shows the result of traceroute from the principal website in Japan to the mirror website in Thailand on 20 February 2019. It is obvious that the path is over JGN/APAN with 98 ms RTT.

Fig. 7
figure 7

Traceroute from principal website in Japan to mirror website in Thailand on 20 February 2019

Full size image

Himawari-8 data transfer

At the same time, we evaluate the performance of TCP over the JGN/APAN using the iperf (Dugan et al. 2019) and measure the network conditions using hperf (Murata et al. 2016a). Note that in this study, the achievable bandwidth is fixed to 100 Mbps to avoid causing network congestion and we consider TCP CUBIC congestion control as it is widely used (Ha et al. 2008). The target throughput (-mt option) of the hperf is set to 1 Mbps for measurement purpose only. Figure 8 shows the throughputs and the network conditions from the principal website in Japan to the mirror website in Thailand. The TCP achieves less than the expected throughput (10 Mbps) in some periods of the day, as shown in Fig. 8a. This is because the TCP is not able to achieve high throughput in long-distance network with packet loss, which has the average RTT of 123 ms, as shown in Fig. 8b and the average PLR of 0.08%, as shown in Fig. 8c. This implies that it is impossible to transfer the 10-minute Himawari data using TCP-based file transfer applications (e.g., FTP) in real time.

Fig. 8
figure 8

Throughputs and network conditions between principal website in Japan and mirror website in Thailand

Full size image

We also evaluate the performance of HpFP over the JGN/APAN using the hperf. The target throughput of the hperf is set to 100 Mbps. The result shows that the HpFP achieves more than the expected throughout in the whole period of the day, as shown in Fig. 9. Based on this result, we adopt the hcp, which is the HpFP-based file copy tool, to transfer meteorological satellite data from the Himawari-8 real-time web to the mirror websites. In order to verify the performance of the hcp, we examine the throughput of Himawari data transfer from the principal website in Japan to the mirror website in Thailand. Figure 10 shows the throughput of hcp for transferring the 10-minute Himawari data. The throughput of hcp is more than 90 Mbps throughout the whole period of the day, with the mean value and standard deviation of 98.75 Mbps and 1.04 Mbps, respectively. Therefore, the proposed mirroring system is able to operate in real time.

Fig. 9
figure 9

Throughput of HpFP between principal website in Japan and mirror website in Thailand

Full size image
Fig. 10
figure 10

Throughput of hcp for transferring 10-minute Himawari data

Full size image

Web performance

We investigate the performance of our mirror website located in Thailand in terms of page load time. The page load time is a web performance metric that demonstrates the time taken to download and display the whole content of a web page in the user’s browser after the user clicks a link or makes a request, usually measured in seconds (StackPath 2019). Several different factors affect the page load time, such as web hosting server, bandwidth capacity, web page design, elements on a web page, geographical location of users, device, and type of browser used. In this experiment, we use public access Internet (such as hotel’s, airports, cafes Internet) to access both the principal website and the mirror website. Google Chrome with page load time extension is used to measure the page load time and displays it in the toolbar (Vykhodtsev 2019). The page load time is evaluated at various zoom levels of full disk. Note that we clear the browser’s cache manually every time to ensure the correctness of the page load time.

Table 2 shows the page load times of Himawari-8 real-time web accessed from Japan and Thailand. The principal website and the mirror website are located in Japan and Thailand, respectively. It is obvious that the domestic access is faster than the international access at every zoom level. The average page load time of Himawari-8 real-time web accessed from the public access Internet in Japan to the principal website in Japan and the mirror website in Thailand is about 1.45 and 7.51 seconds, respectively. On the other hand, the average page load time of Himawari-8 real-time web accessed from the public access Internet in Thailand to the principal website in Japan and the mirror website in Thailand is about 6.37 and 2.05 seconds, respectively. That is, the domestic access is at least three times faster than the international access.

Table 2 Experiment to measure page load times of Himawari-8 real-time web.
Full size table

Discussions

In this study, we transfer the tiled pyramid image files in real time from the principal website in Tokyo, Japan to the small-size mirror website in Bangkok, Thailand. Due to the limitation of CPU power on the mirror website, the image files are processed on the CPU of the small-size web server. Only the generated images on the principal website in Japan is provided in this study. In order to make a better use of the Himawari-8 real-time web, the archived data search is needed. In the operation of the principal website, the access record indicates that the searches for the past events on the web are not negligible.

We prove that the mirroring of the Himawari-8 real-time web is successful using the proposed system. It requires the high-speed international network, e.g., the JGN/APAN, for this mirroring. The JGN/APAN members include many institutions in countries (APAN Ltd. 2019; NICT 2019a). Several meteorological agencies are involved with those institutions. This suggests that there are many possible candidates to conduct our mirroring system, as indicated in Fig. 11. These techniques are devoted to improve the lives of people and industries in Asia and Oceania region. We next focus on Philippines and Taiwan, which are two major countries to suffer from typhoons frequently.

Fig. 11
figure 11

Imaginary picture of mirroring for Himawari-8 real-time web to Asia and Oceania countries

Full size image

As discussed in “Web performance”, the domestic access is at least three times faster than the international access. To satisfy user expectations as users browse our website, we consider a prioritization to domestic access. We set up a virtual uniform resource locator (URL), namely https://himawari.asia. When a user accesses the virtual URL, it is redirected to the principal website or the mirror websites automatically by considering the Internet protocol (IP) address of the user corresponded to the domestic access, as shown in Fig. 12.

Fig. 12
figure 12

Prioritization to domestic access

Full size image

For scalability issue, we address limitations and key points, such as more data sets, more mirror sites, more users, advanced services, etc. With respect to more data sets, the disk input/output (I/O) speed and the bandwidth capacity should be considered. In addition, the time interval also restricts the size of data sets. For example, in case that the bandwidth capacity is 100 Mbps and the time interval is every 10 minutes, the maximum size of transferred data sets is 7.32 GB. Concerning more mirror sites, we need to consider a cascaded data transferring among mirror sites. We first transfer the tiled pyramid image files from the principal website to some mirror websites, and then transfer them from those mirror websites to others. In terms of having more users, the total access number to the Himawari-8 real-time web in 2019 is about 3 million, where domestic (Japanese) access number is 2,083,929 and international access number is 1,037,595. It is obvious that the international access number is about one-third of the total. With coming of the mirror sites, the principal website is used for the domestic accesses, while the mirror sites are used for the international accesses. This implies that our mirroring system are able to support more users of both domestic and international accesses. For advanced services, the Himawari-8 real-time web provides data and images for research and general use. The user can download data and images since July 7, 2015 from the principal website only, since the mirror sites equipped with small disk storage cannot store all archive data. In addition, the mirror sites are also intended to support the future plan of providing an instructional media and an educational game for schools and science museums.

One of the potential developments of our mirroring system in the near future is to use a standard tiled image format as in Google Maps (Google Inc. 2019) or CESIUM websites (Analytical Graphics 2019). Figure 13 shows the different views of the area around the Indochinese Peninsula and Peninsular Malaysia between Himawari-8 real-time web and CESIUM website. The cloud images are based on the Himawari-8 band-3 data, and the date and time is on 4 April 2019 at 2pm (JST). The CESIUM view looks more familiar to the people living in this area than the active safing and isolation system (AS-IS) view provided by the Himawari-8 real-time web. It is because this area is located close to the edge of the Earth from the longitudinal location on the geostationary orbit of the Himawari-8 satellite. It is preferable that the viewpoint of the target area and country of the website is around the zenith of them. Nevertheless, it is not easy to convert every Himawari’s remotely sensed image to each country in Asia and Oceania. One of the practical ways to solve this issue is to use a Web Graphics Library (WebGL) virtual globe and map engine, such as Google Maps and CESIUM. Many smartphones recently support a function to detect their location, longitude, and latitude using Global Positioning System (GPS). We are able to set the initial location of the viewpoint of the website at the zenith on the geostationary orbit on their smartphones.

Fig. 13
figure 13

Two views of Thailand map

Full size image

In order to adapt the design and implementation of our mirroring system to other web services, we classify web services into real-time and non-real-time web services. For real-time web service, all design and implementation can be adopted to provide cost-effective solutions for the data processing and transferring in many scientific domains. For non-real-time web service, the hcp can be adopted to transfer/synchronize the data efficiently between the principal website and the mirror websites. Furthermore, the WONM can be customized for monitoring both real-time and non-real-time web services.

Conclusions

The Himawari-8 real-time web, which is conducted at the NICT, is only one website to provide full-resolution remotely sensed images in real time. This paper proposed the mirroring system to provide the cost-effective solution for the mirror websites of the Himawari-8 real-time web to Asia and Oceania countries. We introduced the mirroring model of the Himawari-8 real-time web to solve the CPU power consumption problem. The hcp, which is the HpFP-based file copy tool, plays an important role to enable the high-speed data file transfer on the international networks even under packet loss conditions. In addition, we also introduced the WONM system to ease the maintenance of remote mirror websites. The WONM system working on smartphones is convenient for such world-wide operations. The experiment results showed that our mirroring system is effective and appropriate for real use. The overall user satisfaction of our website is improved where the domestic access to the mirror website is at least three times faster than the international access to the principal website. Our first mirror website located at the NECTEC in Bangkok, Thailand was launched on 25 December 2017. This suggested that the proposed mirroring system has a potential for deployment in other Asia and Oceania countries.