NMRF / RR / 03 /2015

RESEARCH REPORT
Verification of Met Office Unified Model (UM)
quantitative precipitation forecasts during
the Indian monsoon using the
Contiguous Rain Areas (CRA) method

Raghavendra Ashrit, Elizabeth Ebert*, Ashis K. Mitra,

Kuldeep Sharma, G. R. Iyengar and E. N. Rajagopal

March 2015

National Centre for Medium Range Weather Forecasting

Ministry of Earth Sciences, Government of India
A-50, Sector-62, NOIDA, India 201 309
 Verification of Met Office Unified Model (UM) quantitative
precipitation forecasts during the Indian Monsoon using the Contiguous
Rain Area (CRA) method

Raghavendra Ashrit, Elizabeth Ebert*, Ashis K. Mitra, Kuldeep Sharma, Gopal

Iyengar and E.N. Rajagopal

National Centre for Medium Range Weather Forecasting

Noida, India

*Centre for Australian Weather and Climate Research

Melbourne, Australia

National Centre for Medium Range Weather Forecasting

Ministry of Earth Sciences
A-50, Sector 62, NOIDA-201309, INDIA

March 2015
Earth System Science Organisation
National Centre For Medium Range Weather Forecasting Document
Control Data Sheet
1 Name of the National Centre for Medium Range Weather Forecasting
Institute
2 Document NMRF/RR/03/2015
Number
3 Date of March 2015
publication
4 Title of the Verification of Met Office Unified Model (UM) quantitative
document precipitation forecasts during the Indian Monsoon using the
Contiguous Rain Area (CRA) method
5 Type of Research Report
Document
6 No.of pages & 32 Pages, 15 Figures and 3 Tables
Figures
7 Number of 31
References
8 Author (S) Raghavendra Ashrit, Elizabeth Ebert*, Ashis K. Mitra, Kuldeep
Sharma, Gopal Iyengar and E.N. Rajagopal
9 Originating NCMRWF
Unit
10 Abstract The operational medium range rainfall forecasts of the Met
(100 words) Office Unified Model (UM) are evaluated over India during six
monsoon (JJAS) seasons from 2007-2012 using the Contiguous
Rainfall Area (CRA). The forecasts show a wet bias (due to
excessive number of rainy days) and higher rainfall frequency
for thresholds of 0-20 mm d -1. Over the South-West (SW) India,
the forecasts tend to underestimate rain intensity and the events
tended to be displaced to the west and southwest of the observed
position on an average by about 1° distance. Over eastern India
(E) forecasts of lighter (heavy) rainfall events tend to be
displaced to the east on an average by about 1°(southwest by 1-
2°). In all four regions, the relative contribution to total error
due to displacement increases with increasing CRA threshold.
11 Security Non-Secure
classification
12 Distribution Unrestricted Distribution
13 Key Words
Abstract
The operational medium range rainfall forecasts of the Met Office
Unified Model (UM) are evaluated over India using the Contiguous
Rainfall Area (CRA) verification technique. In the CRA method, forecast
and observed weather systems are objectively matched to estimate
location, volume, and pattern errors. Daily rainfall forecasts from six
(2007-2012) monsoon seasons are verified against two observed
rainfall datasets, the Tropical Rainfall Measuring Mission (TRMM)
rainfall and the India Meteorological Department and NCMRWF merged
rainfall data (NSGM).
The model forecasts show a wet bias (due to excessive number of
rainy days) and higher rainfall frequency for thresholds of 0-20 mm d-1
when verified against both observed data sets. Verification against the
NSGM data consistently suggests higher skill in the forecasts as
compared to TRMM data. Forecast rain systems are also verified using
10, 20 and 40 mm d-1 CRA thresholds for four sub-regions namely (a)
north west (NW) (ii) south west (SW) (iii) eastern (E) and (d) north
east (NE) sub-region. Over the SW sub-region, the forecasts tend to
underestimate rain intensity. In the SW region, the forecast events
tended to be displaced to the west and southwest of the observed
position on an average by about 1° distance. Over eastern India (E)
forecasts of lighter (heavy) rainfall events tend to be displaced to the
east on an average by about 1°(southwest by 1-2°). In all four regions,
the relative contribution to total error due to displacement increases
with increasing CRA threshold. These findings can be useful for
forecasters and for model developers with regard to the model
systematic errors associated with the monsoon rainfall over different
parts of India.
1. Introduction

The rainfall during the monsoon (June-September, JJAS) season

contributes over 75% of the annual rainfall in most parts of the Indian

subcontinent and is the lifeline for agriculture and economy of the entire

region. Forecasting of seasonal rainfall gets great attention due to

ensuing drought (flood) conditions. The monsoon rainfall occurs in many

sporadic weather events having spatial scales from 100 to 1000 km. The

daily and weekly rainfall during the season poses a significant forecasting

challenge in Numerical Weather Prediction (NWP). This is due to complex

interactions involving topography, treatment of synoptic scale systems,

and mesoscale convective systems and non-availability of good quality

high resolution observations over land and neighbouring seas.

Numerical weather prediction (NWP) models have undergone

significant improvements in the last two decades and have demonstrated

reasonable success and skill in short to medium-range weather

forecasting (Kalnay et al. 1998; Simmons and Hollingsworth 2002; Harper

et al. 2007). However, the predictability of Indian summer monsoon

conditions is quite low (Goswami and Ajay Mohan 2001). Drivers of the

intraseasonal variability such as the Madden-Julian oscillation modulate

the frequency of occurrence of synoptic events such as lows, depressions

and tropical cyclones (Maloney and Hartmann 2000; Goswami et al.

2003; Bessafi and Wheeler 2006). Accurate quantitative precipitation

forecasting (QPF) still remains a challenge, as evidenced by the frequent

large errors in the predicted precipitation amounts and distribution from

NWP models. Factors contributing to these errors include: errors in the

initial conditions, predicted flow (dynamics), large-scale and convective

rain processes (physics), grid resolution and representation of local land

surface characteristics. As a result, the QPFs often have large errors in

predicted position of the rain system, shape and size of the rain pattern,

and magnitude or intensity of rainfall.

With the enhanced computing capability in recent years, the spatial

and temporal resolution of models has also increased. Verification of high

resolution forecasts using traditional metrics often suggests poor forecast

skill due to the lack of exact matches among the forecast/observation

pairs. Several new diagnostic spatial verification approaches have been

developed recently that better reflect the quality of the forecasts.

Feature-based methods and neighbourhood verification approaches

assess the broader forecast quality without over-penalizing the errors at

grid scale. Ebert (2008), Ebert and Gallus (2009), and Gilleland et al.

(2010) provide a detailed review and intercomparison of several spatial

verification methods. The Contiguous Rain Area (CRA) method is a

feature-based approach that isolates systems or features of interest and

evaluates their properties, namely, location, size, intensity, and pattern.

It was one of the first methods to measure errors in predicted location

and to separate the total error into components due to location, volume,

and pattern errors (Ebert and McBride 2000; Ebert and Gallus 2009).

2
 The present study forms an important component of India’s

National Monsoon Mission (NMM) Programme to investigate and quantify

the rainfall forecast biases in the Unified Model as part of an overall

scientific challenge to better predict the Indian monsoon.The medium

range rainfall forecasts over India are evaluated in this study to assess

the model performance during the monsoon season. The Met Office

Unified Model rainfall forecasts (UM hereafter) during the six monsoon

(JJAS) seasons from 2007 to 2012 are evaluated using traditional and

CRA verification techniques.

There have been quite a few studies reporting rainfall forecast

verification over India based on different models (Mandal et al. 2007, Das

et al. 2008, Ashrit and Mohandas 2010, Chakraborty 2010, Iyengar et al.

2011). In general these studies indicate that the average root mean

squared error (RMSE) of daily rainfall is high (low) in higher (lower)

rainfall regions and that RMSE increases with lead time. Spatial

distributions of various performance metrics over Indian land regions

show that models tend to have better forecast skill over northern and

north-western India (Iyengar et al. 2011). The detailed verification of the

UM forecasts and its comparison with National Centre for Medium Range

Weather Forecasting (NCMRWF) operational forecasts highlighting the

biases are reported in a series of monsoon reports (Iyengar et al, 2011,

2014). This study represents the first application of a diagnostic spatial

verification method to rainfall forecasts over India. The data and

verification methodology are described in Section 2. Section 3 gives

3
examples of CRA method applied to rain systems over India and discusses

the results in the context of regional performance. The summary and

conclusions are given in Section 4.

2. Data and Methodology

2.1 Observed Rainfall Data over India

Figure 1 shows the geographical domain chosen for the present

study, 7°-38.5°N, 67°-100.5°E, with the distribution of rain gauges in the

India Meteorological Department (IMD) network during the monsoon.

Rainfall analyses based on quality controlled observations are critical for

verification of the NWP forecasts. In this study we use two observation

data sets, (a) the Tropical Rainfall Measuring Mission (TRMM) 3B42 (V7)

daily multi-sensor 0.25° x 0.25° gridded rainfall and (b) the IMD and

NCMRWF satellite+gauge merged (NSGM) rainfall data at 1°. The TRMM

daily rainfall data accumulated at 0000 UTC is used in this study to match

with the Met Office forecast which is also accumulated at 0000 UTC.

However, the NSGM data which uses 24 hour gauge rainfall and the 3

hourly TRMM estimates is accumulated at 0300 UTC.

TRMM rainfall data have been used in numerous studies since its

launch in 1997. TRMM’s 3B42 multi-sensor precipitation estimates over

the monsoon region are very useful for monsoon studies and verification

of rainfall forecasts, since they cover large data sparse oceanic regions.

However there are biases over the land regions which require correction

(Mitra et al. 2009; Chen et al. 2013). The NSGM objectively analyses IMD

4
daily rain gauge observations onto a 1° grid using a successive

corrections technique with the TRMM 3B42 satellite precipitation providing

the first guess field, thus providing spatially continuous rainfall over land

and ocean. As noted by Mitra et al. (2009), the 1° grid resolution is

appropriate for capturing the large scale rain features associated with the

monsoon. The merging of the IMD gauge data into TRMM 3B42 not only

corrects the mean biases in the satellite estimates but also improves the

large-scale spatial patterns in the satellite field, which is affected by

temporal sampling errors (Mitra et al. 2009).

Both the rainfall analyses (TRMM and NSGM) are used in the

present study. The higher resolution of the TRMM data may detect spatial

detail and heavier rainfall that is not resolved by the coarser NSGM, as

can be seen in Figure 2 for the case of 24 hour accumulated rainfall valid

for 00Z22nd August 2012. On the other hand, the TRMM estimates

sometimes miss out on some of the heavy rain over land that is captured

by the gauge observations in the NSGM analysis (Mitra et al. 2009 and

Prakash et al. 2014).

A comparison of the frequency of occurrence of rainfall exceeding

different thresholds (1, 10, 20, 40, 80, and 160 mm d-1) is shown for both

TRMM and NSGM in Figure 3, based on data exclusively over the land

regions. While both data sets have similar rain intensity distributions,

compared to NSGM, TRMM captures fewer 1 mm d-1 rainfall events but

detects a higher frequency of 20 and 40 mm d-1 events, which can be

attributed to relatively higher grid resolution.

5
2.2 Model Rainfall Forecasts over India

The Met Office Unified Model (UM) is the numerical

modelling system developed and used at the Met Office in the United

Kingdom (UK) (Davies et al. 2005). In this 'seamless' prediction system

different configurations of the same model are used across all time and

space scales, with each configuration designed to best represent the

processes which have most influence on the timescale of interest. For

example, for accurate climate predictions the use of a coupled ocean

model is essential, while for short-range weather forecasting a higher

resolution atmospheric model may be more beneficial than running a

costly ocean component. It can be run in global and limited area domains

and can also be coupled to land surface, ocean models, wave models,

chemistry and Earth system components.

This study uses the rainfall forecasts from the Met Office operational

medium range global model configuration. The Unified Model (UM) is

continually developed, taking advantage of improved understanding of

atmospheric processes and steadily increasing supercomputer power. The

Met Office upgrades its operational NWP configurations up to four times

per year. Some of the important changes during the 2007-2012 study

period are tabulated in Table 1. The atmospheric model uses non-

hydrostatic dynamics with semi-Lagrangian advection and semi-implicit

time stepping. It is a grid point model with the ability to run with a

rotated pole and variable horizontal grid. A number of sub-grid scale

processes are represented, including convection (Gregory and Rowntree

6
1990, Gregory and Allen 1991, Grant 2001), boundary layer turbulence

(Brown et al., 2007), radiation (Edwards and Slingo 1996), cloud

microphysics (Wilson and Bollard 1999) and orographic drag (Webster et

al., 2003). The model is initialized using a state of the art global four

dimensional variational (4DVAR; Rawlins et al., 2007) data assimilation

technique. During 2007-2012, the horizontal and vertical resolution of the

global configuration improved from about 40 km and 50 levels in 2007 to

about 25 km and 70 levels in 2010.

The verification of rainfall forecasts is carried out at all lead times

Day1 (t+0 ->t+24h) through Day5 (t+96h -> t+120h). For brevity, the

results are presented for Day1 forecasts. It is found that the results for

Day2, Day3, Day4 and Day5 forecasts (not shown) are consistent with

those of Day1 forecasts. The model forecast rainfall is accumulated at

0000 UTC. The model grid resolution varies from 40 km to 25 km during

2007-2012 (Table 1).

The model forecast and TRMM rainfall are interpolated to NSGM

analysis grids (a common grid resolution of 1° x 1°) to enable the

verification of large scale daily monsoon rainfall. In this study we use

rainfall data over land only, to focus on model performance over land.

2.3 Forecast Verification Approach

The frequency of rain in the forecasts and observations are

compared as a first indication of potential biases in model rainfall

predictions. The forecast daily rainfall fields are verified using standard

7
categorical verification scores frequently employed in evaluating

precipitation, Probability of Detection (POD), Success Ratio (SR),

Probability of False Detection (POFD), Bias Score or Frequency Bias

(BIAS) , Equitable Threat Score (ETS) and Hanssen and Kuipers score (HK

Score). Descriptions of these scores can be found in standard references

on statistical methods (e.g., Jolliffe and Stephenson (2012) and Wilks

(2011)).

This is followed by verification using the CRA method to quantify the

systematic errors for rain systems. The CRA method is a feature-based

verification procedure suitable for gridded forecasts that was developed

for estimating the systematic errors in forecasts of rainfall systems (Ebert

and McBride 2000; Ebert and Gallus 2009). In addition to measuring

errors in predicted location, the CRA method decomposes the total error

into components due to errors in location, volume and pattern. The

location errors in the model forecasts suggest issues with the model

dynamics. The volume and pattern errors possibly emanate from physics

and thermodynamics. The steps involved in the CRA technique are

described in Ebert and Gallus (2009). A brief summary of the procedure is

given here.

A CRA is defined for an observation/forecast pair based on a user-

specified isohyet (rain rate contour) in the forecast and/or the

observations. It is the union of the forecast and observed rain entities as

illustrated in Figure 4. This simple approach is used to match a forecast

rain system with an observed rain system under the assumption that they

8
are associated with a common synoptic situation, which is reasonable for

monsoon rain events. During the monsoon season, large parts of India

regularly receive rainfall in the range up to 10 mm d-1. It was found that

choice of 1, 2, and 5 mm d-1 contour frequently spread the CRA across

large geographical areas, merging unrelated rain systems. CRAs defined

by higher thresholds of 10, 20, 40 and 80 mm d-1 were used to better

isolate the heavy rain events of interest in this study.

In the next step a pattern matching technique is used for estimating

the location error. Here the forecast field is horizontally translated over

the observed field until the best match is obtained. The geometric

distance between the centres of gravity (COG) in the observed and

estimated fields forms the location error or vector displacement. The best

match between the two entities can be determined either: (a) by

maximizing the correlation coefficient, (b) by minimizing the total squared

error, (c) by maximizing the overlap of the two entities, or (d) by

overlaying the centres of gravity of the two entities. For a good forecast,

all of the methods will give very similar location errors. In the present

study, the best match is determined by maximizing the correlation, as

was also done by Ebert and Gallus (2009).

The mean squared error (MSE) and its decomposition (location

error, volume error and pattern error) are computed as shown below (see

Grams et al. 2006, for details of the derivation).

MSETotal = MSEDisplacement + MSEVolume + MSEPattern(1)

where the component errors are estimated as

9
 MSEDisplacement = 2sFsO (rOPT - r),

MSEVolume = (F’ – O’), (2)

MSEPattern = 2sFsO (1 - rOPT) + (sF - sO)2

In the above expressions F’ and O’ are the mean forecast and observed

precipitation values after shifting the forecast to obtain the best match, sF

and sO are the standard deviations of the forecast and observed

precipitation, respectively, before shifting. The spatial correlation between

the original forecast and observed features (r) increases to an optimum

value (rOPT) in the process of correcting the location via pattern matching.

The number of ‘good matches’ corresponds to the number of forecasts

that matched well with observations when the optimum correlation (rOPT)

was (statistically) significantly greater than zero (accessed via two tailed

t-test).

3. Results of Rainfall Forecast Verification

3.1 Evaluation of Forecast Rain Occurrence during 2007-2012

As discussed in the introduction, forecasting of rainfall over India

and tropics in general is a challenge for the NWP models. While the

models generally capture large scale features of the monsoon rainfall

distribution, they fail to reproduce the regional peculiarities. This is

evident even in the observed and forecast seasonal mean rainfall over

India. The observed and forecast mean JJAS rainfall over India during

2012 is shown in Figure 5 for forecasts with different lead times. The

forecasts successfully capture the gross features of mean monsoon

10
rainfall in terms of high rainfall amounts (16-32 mm d-1) along the west

coast and reducing rainfall amounts eastwards over the peninsula.

Similarly, the model captures high rainfall amounts (16-32 mm d-1) over

northeast India and reducing rainfall amounts westwards over northwest

India. The model shows large biases in rainfall over northern India

adjoining the Himalayas. This feature is typical and can be seen during

each of the monsoon seasons. The low level winds (850 hPa; not shown)

over the Gangetic plains typically show strong easterly bias (Iyengar et al.

2011) which partly explains the rainfall bias over that region. To further

examine the bias in the rainfall forecast, climatologically frequency is

constructed although using the available 7 years of data. The frequency

of rainfall occurrence in excess of different thresholds is shown in Figure 6

for the Day1 forecasts and the two observation data sets (TRMM and

NSGM). The model overestimates the rainfall frequency at all thresholds

below 20 mm d-1. Figure 6 is based on all rainy days in the seven

monsoon seasons; this model behaviour was consistently seen in each of

them.

It is also found that the model features high number of rainy days

compared to observations. This is presented in the form of a spatial

distribution of the rainy day counts over India in Figure 7. The panels (a)

and (b) show the observed (NSGM) and forecast count of rainy days

(rainfall >1mm d-1). The difference between the two is striking with the

forecasts having a higher number of rainy days, particularly where the

number of rainy days is observed to be lower than average, in nearly all

11
parts of India. The forecasts have an excessive number of rainy days,

throughout the SW, E and NE regions of India. Even over most of the dry

region of NW the model predicts a relatively higher number of rainy days.

The panels (c) and (d) show the rainy day counts in the UM forecasts with

2mm d-1 and 5mm d-1 thresholds respectively. The 5mm d-1 threshold

produces a more realistic pattern overall of the rainy day counts where

the number of rainy days is observed to be higher than average. On the

other hand, large parts of the peninsula show a reduced number of rainy

days particularly where the number of rainy days is observed to be lower

than average. Over the north east (NE) the number of rainy days is still

overestimated compared to observations.

The categorical verification scores for the UM Day1 forecasts over

India are summarized using box and whisker plots in Figure 8. Scores are

computed for each rainfall threshold based on all the observation/forecast

pairs of each day during the six monsoon seasons and represent the grid

scale QPF performance that may be expected on any given day. The

verification results are presented against both TRMM and NSGM data.

The panels in the top row (Figure 8a,b) show the Probability of

Detection (POD) and Success Ratio (SR). While POD indicates the fraction

of observed ‘yes’ events forecast correctly, SR indicates the fraction of

forecast ‘yes’ events that were actually observed. Both scores range from

0 to 1 with 1 being a perfect score. Both the scores have high values for

rainfall thresholds below 20 mm d-1. The verification of forecast against

12
the NSGM shows higher scores at all thresholds compared to verification

against TRMM.

The two panels in the middle row (Figure 8c,d) show Probability of

False Detection (POFD) and Bias Score or Frequency Bias (BIAS). POFD

indicates what fraction of ‘no’ events were incorrectly forecast as ‘yes’

events. POFD varies from 0 to 1 with 0 being a perfect score. The POFD

values indicate that forecasts have high false alarms at low rainfall

thresholds. Again, the verification against the NSGM shows better score

values compared to TRMM. The Frequency Bias (BIAS) indicates how the

observed and forecast frequency of ‘yes’ events compare. BIAS varies

from 0 to ∞ with 1 indicating a perfect forecast, BIAS>1 indicating over-

forecasting and BIAS<1 indicating under-forecasting. The BIAS in Fig. 8

suggests the model is over-forecasting rain occurrence, particularly at

lower thresholds (<20mm d-1), consistent with Fig. 6.

Similarly, the panels in the bottom row (Figure 8e,f) show the box

and whisker plots for two summary scores, the Equitable Threat Score

(ETS) and Hanssen and Kuipers score (HK Score). While ETS tells how the

forecast ‘yes’ events correspond to observed ‘yes’ events (accounting for

random hits), HK Score tells how well the forecasts separate the ‘yes’ and

‘no’ events. For both scores, 0 denotes no skill and 1 means a perfect

score. The ETS and HK show low values of the score at all rainfall

thresholds. As before, relatively higher scores are obtained when the

forecasts are verified against the NSGM.

13
3.2 Example of CRA Verification over India

The forecast errors are assessed in terms of displacement, pattern,

and intensity using the CRA verification method. We first demonstrate the

application of CRA verification and discuss the interpretation of the

results. Figure 9 shows an example of CRA verification corresponding to a

heavy rain event over western India on 3rd July 2007 associated with a

low pressure system that formed over the Arabian Sea close to the Indian

coast and moved inland over the Gujarat region. The verification scores

indicate moderate skill (correlation=0.55; ETS=0.33; HK=0.61 and

POD=0.82). The RMSE was 24.5 mm d-1, about half of the observed mean

rainfall. Although the forecast underestimated the average rain rate, the

predicted rain volume was higher than the observations due to a larger

area of rain being predicted. The biases can be related to high forecast

rainfall along the Myanmar coast and along the foothills of Himalayas.

Figure 10 shows the CRA verification using a 10 mm d-1 threshold to

isolate the heavy rainfall along the west coast of India. The CRA is

bounded by the domain from 8.5° - 26°N and 69° – 78.5°E and includes

72 grid points. The scatter diagram shows the correspondence between

the forecast and observed rainfall after attaining the best match (rOPT) by

shifting the forecast slightly to the north. For this forecast, the CRA

verification shows a very modest improvement in correlation from 0.41 to

0.44. The error decomposition shows the primary contribution to the

overall error came from the pattern error (86%).

14
3.3 CRA verification results for 2007-2012

The rainfall over different parts of India can be associated with

different synoptic regimes as well as having different topography and

proximity to neighbouring seas. The four regions shown in Figure 7 can be

considered as rainfall zones for CRA verification. The rainfall over north-

eastern India (NE) and the south-western peninsula (SW) strongly reflects

the effects of the low level monsoon flow and the orographic

enhancement over the mountains. The rainfall over eastern India (E) can

be associated with the monsoon trough and south-easterly flow from the

Bay of Bengal. The monsoon trough extends from north-western India to

the head of the Bay of Bengal. The low pressure systems that develop

over the Bay of Bengal and track in the westerly and north-westerly

direction also significantly contribute to the rainfall over eastern India (E).

Some of the low pressure systems track far inland in the westerly and

north-westerly direction to produce rainfall spells over the arid and dry

regions of northwest India (NW). However the rainfall over the NW region

is some times associated with eastward passage of an upper-level

trough/low in the mid-latitude westerlies and their interaction with the

inland low pressure systems.

CRA verification results are presented for each zone based on the

central location of the CRA. These results are based on the CRA statistics

from six monsoon seasons (2007-2012) and for four different rainfall

thresholds (10, 20, 40 and 80 mm d-1). Table 2 shows the number of

CRAs in each zone and for different CRA thresholds. Verification using the

15
TRMM analysis gives a slightly higher total number (940, 770 and 85) of

CRAs (all four zones combined) compared to verification against the

NSGM analysis (901, 541 and 51) for 10, 20 and 40 mm d-1 CRAs

respectively. This is mainly reflected in three zones of NW, E and NE and

can be attributed to the TRMM rainfall data estimating a greater frequency

of heavy rain events (refer to Figure 6). However the total number of

good matches is higher for CRA verification involving NSGM analyses

(546, 217 and 18) compared to TRMM (381, 147 and 12). NSGM analyses

are smoother and more conducive to higher correlations. For the 10 mm

d-1 threshold, a good match is obtained in over 50% of the CRAs. The

percentage of good matches decreases with the increasing CRA threshold

indicating the decreasing skill of the NWP forecast model to predict the

heavy rainfall. For an 80 mm d-1 CRA threshold (not shown) the number

of good matches is so low that it does not form a good sample for

analysis. The percentage of good matches for the 10 and 20 mm d-1

threshold is high (60% and 40%, respectively) for NSGM and relatively

low (40% and 19%, respectively) for TRMM. The CRA verification results

for these thresholds can be considered robust and significant, while for

the 40 mm d-1 CRA threshold, the number of good matches is low (35%

for NSGM and 14% for TRMM) and the results of the CRA verification

should be viewed with caution.

Based on the good matches, the scatter plots in Figure 11 show the

association among the observed and forecast average rainfall for rain

systems located in each of the four zones. The difference in the CRA

16
verification results for TRMM and NSGM is rather striking in all zones and

for all thresholds. Especially over the SW region, the model bias brought

out in the NSGM (panels on right) is not seen as clearly in the TRMM

(panels on left) which has fewer samples. Similarly, Figure 12 shows the

association among the observed and forecast maximum rain in each of

the four zones. The scatter of mean rain intensity (Figure 11) and

maximum rain intensity (Figure 12) over NE India consistently suggest

overestimation of rain intensity in the forecasts. Over the SW region, the

forecasts tend to underestimate rain intensity. The nature of bias seen in

the mean rain intensity (Figure 11) is also reflected in the maximum rain

intensity (Figure 12). It is likely that the model forecasts tend to rain

more over the ocean. In the present investigation, the verification is

carried out over the land.

The average CRA verification statistics (after obtaining the best

match of the forecast with the observations) using NSGM as observation

data are compiled in Table 3. Figures 13 and 14 complement the

information in Table 3. Figure 13 (left panel) shows the scatter of position

errors (in degrees latitude and longitude) in each of the four zones in the

Day1 forecasts. Each point in scatter plots shows the east-west versus

north-south displacement in the centre of gravity (COG). Clustering of

these points indicates increased frequency. Panels on the right show the

spatial distribution of the counts of the position errors. The spatial

distribution counts are not shown for 40 mm d-1 CRAs since the available

points are too few. Similar distributions are evident in the Day3 and Day5

17
forecasts (not shown). Figure 14 shows the distributions of the position

errors and post-correction spatial correlations using box-whisker plots.

The mean pattern correlation (and the RMSE in mm d-1) achieved

after correcting for location error are presented in Table 3. The pattern

correlation (and RMSE) values range from 0.45 to 0.64 (18.3 to 30.7 mm

d-1) for lower CRA thresholds (10 and 20 mm d-1). These can be

considered robust since they are based on a large sample. For a higher

CRA threshold (40 mm d-1), the pattern correlation (0.49 to 0.68) and

RMSE values (30 to 37 mm d-1) are higher since the focus is on a smaller

area of heavier rain, but due to the much smaller sample size these mean

results contain greater uncertainty.

The mean displacement errors are given in degrees latitude and

longitude in Table 3. Positive (negative) values of x-displacement error

indicates that rain events are forecast to the east (west) of the observed

location. Similarly, positive (negative) values of y-displacement error

indicates that rain events are forecast to the north (south) of the

observed location. The largest mean x-displacement error is in the

eastern region (E) with forecasts located an average of 2° longitude

eastwards for CRAs defined by the 10 mm d-1 threshold. This is consistent

with the reported slow movement of the low pressure systems (Iyengar et

al. 2011 and 2014) in model forecasts over eastern India after landfall.

The magnitude of x-displacement error in eastern India (E) is seen to

decrease for higher CRA thresholds. This clearly suggests that in eastern

India (E), the location of heavier rain is predicted with greater accuracy

18
than the lighter rain events. The mean north-south displacement errors

given by y-displacement error are relatively moderate with mean values

less than 1° latitude.

The contributions to the total error due to displacement , volume

and pattern are also summarized in Table 3. In all four regions, the

contribution from pattern error forms the highest share for the 10 mm d-1

CRAs, which tend to have larger areas. Over E and SW regions, the

relative contribution of pattern error (displacement error) decreases

(increases) for 20 and 40 mm d-1 CRAs.

Referring to Figure 13, it can be noted that over the NW region, for

10 mm d-1 CRAs, the forecast location errors are spread in eastern

quadrant with highest counts of 6 just east of the origin. Over the SW

region, the scatter of the position errors (panels on left) shows a

systematic southeast to northwest orientation typical of the rainfall along

the west coast of India with a majority of forecasts displaced to the west

of the observed event (also seen in Figure 14b). Over eastern India (E),

the 10 mm d-1 CRAs tend to be forecast to the east of the observed

location with a high count of 21 and 19 showing displacement by 1° east

and south-eastwards respectively. Forecast rainfall in the NE region tends

to be systematically predicted to the south of the observed location with

high count of 17 at 1° south of observed location.

The vector errors are highest for 10 mm d-1 CRAs in all regions

except over the NE region where the vector errors for 10 and 20 mm d-1

CRA thresholds are comparable. These are based on fewer samples

19
compared to the 10 and 20 mm d-1 CRA results, and represent stronger

storms on average. Overall, vector errors were lowest in the SW, probably

reflecting the strong orographic influence on the rainfall in that region.

The panels in Figure 15 show the RMSE (mm d-1) and the

percentage contribution to total error due to location, volume and pattern

error. Not surprisingly, the RMSE is least for 10 mm d-1 CRAs with

generally lower spread as this sample includes a larger proportion of

lighter rainfall events. The 20 and 40 mm d-1 CRAs show higher RMSEs

and greater variability. The contribution from pattern error is dominant in

all four regions for 10 mm d-1 CRAs with the median value ranging from

60% in the NE region to 80% in the E region. The median contribution to

error due to displacement is around 15% for all regions for 10 mm d-1

CRAs. This low relative contribution from displacement in the E region is

surprising, given the large systematic eastward errors seen in Figures 13

and 14, but the pattern error in this region is very large and dominates

the total error. Contributions from volume error are generally least except

in the SW and NE regions where they are responsible for a similar

proportion as the displacement errors.

4. Summary and Conclusions

The present study forms an important component of India’s

National Monsoon Mission (NMM) Programme to investigate and quantify

the rainfall forecast biases in the Unified Model as part of an overall

scientific challenge to better predict the Indian monsoon. This study has

20
examined the performance of the Met Office Unified Model (UM) over

India for six years (2007-2012) during the monsoon season, using both

TRMM 3B42 and the IMD blended gauge analysis (NSGM). Both data sets

provide useful reference data for verification, though differences in the

detection ability, spatial structure, and intensity distributions of TRMM and

NSGM observed data sets lead to better performance of the model when

compared to NSGM as opposed to TRMM (Figure 6, Table 2).

The model forecasts show a wet bias resulting from an excessive

number of rainy days compared to observations all over India. In

particular, the model has a higher frequency of rainfall occurrence at all

thresholds from 0-20 mm d-1 compared to both sets of observations,

while the forecast frequency of rain greater than 20 mm d-1 is well

represented.

Systematic errors in the forecast rain systems are estimated using

CRA analysis with 10, 20 and 40 mm d-1 threshold for four regions: the

north-west (NW), south-west (SW), east (E) and north-east (NE). The

mean and maximum rain amounts tended to be overestimated in the E

and NE regions but underestimated in the mountainous SW region.

The displacement errors are scattered but show some systematic

trends, depending on the CRA threshold. In the NW and SW, the forecast

events are frequently displaced by about 1° to the north and 1° to the

west, respectively, of the observed position. Over eastern India (E)

forecasts for lighter rainfall events tend to be displaced about 1° to the

east while heavier forecast rainfall events are displaced slightly southwest

21
of the observed location by about 1°. Southerly forecast displacements

are most common in the NE region.

For 10 mm d-1 CRAs, the contribution from pattern error is

dominant in all four regions, and the contribution from volume error is

generally least. The relative contribution to total error due to

displacement tends to increase with increasing CRA threshold as the

relative contribution from pattern error decreases.

The information on the dominant contribution to the total error in

any region may be useful guidance for the forecaster. For example, over

the plains adjoining the Himalayas, it is often seen that the UM forecasts

produce excess rainfall mainly associated with a prominent easterly bias

in the 850 hPa winds (Iyengar et al. 2011). Another example is the

rainfall associated with the Bay of Bengal low pressure systems where the

predicted low pressure systems in the model make a rather slower than

observed west north-westerly movement. The impact of position errors

are reflected in Figures 13-15 for region E.

The detailed analysis presented in this study can help the model

developers and forecasters to understand the systematic errors

associated with forecast characteristics of monsoon rainfall over different

parts of India. A similar analysis of QPFs from other modelling systems

will provide robust measures of bias, accuracy, and relative error

components in forecast rain systems over India. Additionally, plausible

sources of forecast errors including grid resolution, model initialization,

and physical processes will be addressed in future studies.

22
Acknowledgements The authors thank Noel Davidson of CAWCR and

other colleagues in NCMRWF for fruitful interactions, scientific discussions

in addition to comments/feedback that were useful in finalizing the

manuscript. The model forecast data used in this study is obtained from

the Met Office which is duly acknowledged. This work was supported by

the Government of India, Ministry of Earth Sciences National Monsoon

Mission under project MM/SERP/CAWCR-AUS/2013/INT-6/002.

23
References:

Ashrit R. and S. Mohandas 2010: Mesoscale model forecast verification

during monsoon 2008. J. Earth Syst. Sci. 119, No. 4, 417–446

Bessafi, M. and M. C. Wheeler, 2006: Modulation of south Indian Ocean

tropical cyclones by the Madden-Julian oscillation and convectively-

coupled equatorial waves. Mon. Wea. Rev., 134, 638-656.

Brown, A.R., Beare, R.J., Edwards, J.M, Lock, A.P., Keogh, S.J., Milton,

S.F., and Walters,D.N., 2007: Upgrades to the boundary layer scheme

in the Met Office NWP model.Bound. Lay. Meteorol., 118, 117-132.

Chen, Y., E.E. Ebert, K.J.E. Walsh, and N.E. Davidson, 2013: Evaluation of

TRMM 3B42 daily precipitation estimates of tropical cyclone rainfall over

Australia. J. Geophys. Res., in revision.

Das S, Ashrit R, Gopal R Iyengar, Saji Mohandas, Das Gupta M, John P

George, Rajagopal E N and Surya Kanti Dutta 2008: Skills of different

mesoscale models over Indian region during monsoon season: Forecast

errors.J. Earth Syst. Sci. 117(5) 603–620.

Davies, T.,M. J. P. Cullen, A. J. Malcom, M. H. Mawson, A. Staniforth, A.

A. White and N. Wood 2005: A new dynamical core for the Met Office’s

global and regional modeling of the atmosphere, Q. J. R. Meteorol. Soc.

(2005), 131, pp. 1759–1782.

Ebert, E.E. and J. L. McBride, 2000: Verification of precipitation in

weather systems: Determination of systematic errors. J. Hydrol., 239,

179-202.

24
Ebert E.E., 2008: Fuzzy verification of high-resolution gridded forecasts; A

review of proposed framework. Meteorol. Appl. 15, 51-64.

Ebert, E.E. and W. A. Gallus Jr, 2009: Towards better understanding of

Contiguous Rain Areas (CRA) method of spatial verification, Weather

and Forecasting, 24, 1401-1415

Edwards J.M. and A. Slingo, 1996: Studies with a flexible new radiation

code Part I. Choosing aconfiguration for a large-scale model. Q. J. R.

Met. Soc., 122, 689-719.

Gilleland, E., D. A. Ahijevych, B. G. Brown and E. E. Ebert, 2010:

Verifying forecasts spatially. Bull. Amer. Met. Soc., 1365-1373.

Goswami, B. N. and R. S. Ajaya Mohan, 2001: Intra-seasonal oscillations

and inter-annual variability of the Indian summer monsoon. J. Climate,

14, 1180-1198.

Goswami, B. N., R. S. Ajaya Mohan, P. K. Xavier, and D. Sengupta, 2003:

Clustering of low pressure systems during the Indian summer monsoon

by intraseasonal oscillations. Geophys. Res. Lett., 30, 1431,

doi:10.1029/ 2002GL016734.

Grams, J.S., W.A. Gallus, L.S. Wharton, S. Koch, A. Loughe, and E.E.

Ebert, 2006: The use of a modified Ebert-McBride technique to evaluate

mesoscale model QPF as a function of convective system morphology

during IHOP 2002. Weather and Forecasting, 21, 288–306.

Grant, A.L.M., 2001: Cloud base mass fluxes in the cumulus capped

boundary layer. Q. J. R. Met. Soc. 127, 407-421.

25
Gregory, D. and S. Allen, 1991: The effect of convective scale

downdraughts upon NWP and Climate. Proc. 9th AMS conf on NWP,

Denver, USA, 122-123.

Gregory, D. and P.R. Rowntree, 1990: A mass flux convection scheme

with representation of cloud ensemble characteristics and stability-

dependent closure. Mon. Wea. Rev., 118,1483-1506.

Harper, K., L. W. Uccellini, E. Kalnay, K. Carey, and L. Morone, 2007:

50th anniversary of numerical weather prediction. Bull. Amer. Meteor.

Soc., 88, 639–650.

Iyengar, G., R. Ashrit, M. Das Gupta, and co-authors, 2011: NCMRWF &

UKMO Global Model forecast verification: Monsoon 2010. Monsoon

Report, NMRF/MR/02/2011.

Iyengar, G. R., Raghavendra Ashrit, Kuldeep Sharma and E.N. Rajagopal

2014: Monsoon-2013: Performance of NCMRWF Global Assimilation-

Forecast System, Monsoon Report, NMRF/MR/01/2014.

Jolliffe, I.T. and Stephenson, D.B., 2011: Introduction, in Forecast

Verification: A Practitioner's Guide in Atmospheric Science, Second

Edition (eds I. T. Jolliffe and D. B. Stephenson), John Wiley & Sons,

Ltd, Chichester, UK. doi: 10.1002/9781119960003.ch1

Kalnay, E., S. J. Lord, and R. D. McPherson, 1998: Maturity of operational

numerical weather prediction: Medium range. Bull. Amer. Meteor. Soc.,

79, 2753–2769.

26
Maloney, E. D. and D. L. Hartmann, 2000: Modulation of eastern North

Pacific hurricanes by the Madden-Julian oscillation. J. Climate, 13,

1451-1460.

Mandal, V., U.K. De and B.K. Basu 2007: Precipitation forecast verification

of Indian summer monsoon with intercomparison of the three diverse

regions. Weather and Forecasting, 22, 428-443.

Mitra A. K., A. K. Bohra, M. N. Rajeevan, T. N. Krishnamurti, 2009: Daily

Indian precipitation analysis formed from a merge of rain gauge data

with TRMM TMPA satellite-derived rainfall estimates. J. Meteor. Soc.

Japan, 87A, 265-279.

Prakash, S., V. Satyamoorthy, C. Mahesh and R. M. Gairola 2014: An

evaluation of high-resolution multisatellite rainfall products over the

Indian monsoon region. Int. J. Remote Sensing, 35(9), 3018-3035.

Rawlins F, S. P. Ballard, K.J. Bovis, A. M. Clayton, D. Li, G.W. Inverarity,

A.C. Lorenc and T.J.Payne, 2007: The Met Office global four-

dimensional variational data assimilation scheme, Q. J. R. Met. Soc.,

133, 347–362

Simmons, A.J., and A. Hollingsworth, 2002: Some aspects of the

improvement in skill of numerical weather prediction. Quart. J. Roy.

Meteor. Soc., 128, 647–677.

Stephenson D.B., B. Casati, C.A.T. Ferro and C.A. Wilson, 2008: The

extreme dependency score: a non-vanishing measure for forecasts of

rare events. Meteorol. Appl., 15, 41-50.

27
Webster S., A.R. Brown, D. Cameron and C.P.Jones, 2003: Improvements

to therepresentation of orography in the Met Office Unified Model. Q. J.

R. Met. Soc., 126, 1989-2010

Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences.

International Geophysical Series Vol. 100., Academic Press.

28
Table 1. Some of the important Unified Model (UM) changes in recent years.
UM Versions Configuration
2007 UM6.4 (Feb2007), UM6.5 (Jul2007) N320L50 (~40km in mid-latitudes)
2008 UM7.0 (Mar2008), UM7.1 (Aug2008) N320L50 (~40km in mid-latitudes)
2009 UM7.3 (Mar2009), UM7.4 (Aug2009) N320L70 (~40km in mid-latitudes)
12 min timestep
2010 UM7.6 (Apr2010), UM7.1 (Aug2010) N512L70 (~25 km in mid-latitudes)
10 min time step
4DVAR data assimilation
2011 UM7.9 (Apr2011), UM8.0 (Aug2011) N512L70 (~25 km in mid-latitudes)
10 min time step
Hybrid data assimilation
2012 UM8.2 (Apr2012) N512L70 (~25 km in mid-latitudes)
10 min time step
Hybrid data assimilation

29
Table 2. The number of CRAs verified in each zone and the number of good matches.

CRA Region NSGM NSGM TRMM TRMM

Good Good
thresholds Total CRAs matches Total CRAs matches
10mm NW 93 55 90 42
SW 280 183 154 57
E 336 227 466 210
NE 192 81 230 72
Total 901 546 940 381

20mm NW 70 21 122 30
SW 170 83 115 19
E 210 84 302 63
NE 91 29 231 35
Total 541 217 770 147

40mm NW 5 1 12 1
SW 23 9 12 1
E 20 6 43 6
NE 3 2 18 4
Total 51 18 85 12

30
Table 3. (a) Verification of Day1 rainfall forecasts (b) mean displacement errors and (c)
components of total mean squared error (MSE) over four rainfall zones and for three
CRA thresholds, as verified against NSGM data.
-1 -1 -1
10 mm d threshold 20 mm d threshold 40 mm d threshold
NW SW E NE NW SW E NE NW SW E NE
(a) Forecast Verification
Pattern
Correlation 0.54 0.63 0.45 0.48 0.58 0.64 0.55 0.48 -- 0.59 0.68 -
-1
RMSE(mm d ) 18.3 21.1 18.8 18.6 30.1 30.7 23.5 30.0 - 41.0 30.0 -

(b) Displacement error

(degrees)
x-displacement
(deg lon) 1.73 0.28 2.11 0.44 0.87 -0.15 0.75 0.29 - 0.13 0.51 -
y-displacement
(deg lat) 0.82 0.58 0.44 -0.62 0.03 0.06 0.26 -0.19 - -0.51 1.03 -

(c)% of total error due to

Displacement
error 19 19 17 20 26 17 29 23 - 20 60 -
Volume error 12 17 9 21 19 28 9 15 - 30 5 -
Pattern error 69 64 74 59 55 55 62 62 - 50 35 -

31
Figure Captions

Figure 1. Geographical domain over India used for rainfall verification showing terrain

elevation (km) and typical distribution of the rain gauge network on any day during the

monsoon season.

Figure 2. 24 hour rainfall valid for 22nd Aug 2012 in (a) TRMM (valid for 00UTC) and (b)

NSGM (valid for 03UTC).

Figure 3. Frequency distribution of observed daily rainfall (JJAS) over India during 2007-

2012. The box-whiskers compare the rainfall based on satellite estimates (TRMM) and

satellite+gauge merged (NSGM). The comparison is presented for different rainfall

thresholds. The values greater (lower) than 3/2 times the 75th percentile (25th percentile) are

outliers.

Figure 4. CRA formed by overlap of forecast and observations.

Figure 5. (a) Observed (NSGM) and UM forecast (b)Day1, (c)Day2, (d)Day3 and (e)Day5

mean JJAS rainfall (mm d-1) over India during 2012.

Figure 6. Observed and UM Day1 forecast rainfall frequency distribution over India.

Figure 7. (a) Observed (NSGM) and (b) UM Day1 forecast number of rainy (>1mm d-1) days

during JJAS 2012;(c)same as (b) with >2mm d-1as the definition of a rainy day and (d) same

as (b) with >5mm d-1as the definition of a rainy day. The boxes show the four domains that

are used to investigate regional variation in forecast performance.

Figure 8. Rainfall forecast verification scores for Day1 forecasts verified against TRMM

3B42 and NSGM rainfall analyses: (a) Probability of Detection (POD), (b) Success Ratio

(SR), (c) Probability of False Detection (POFD), (d) Frequency Bias or Bias Score (BIAS),

(e) Equitable Threat Score (ETS), and (f) Hanssen and Kuipers score (HK score).

Figure 9. Verification of forecast (left) rainfall over India valid for 3rd July 2007. Observed

(NSGM; right) rainfall and the skill scores (using raining (>1mm) grids) are also shown.

32
Figure 10. CRA verification results for forecast rainfall along the west coast of India valid for

3rd July 2007. The CRA is defined using a 10mm d-1 threshold.

Figure 11. Forecast vs observed (TRMM, left; NSGM, right) mean rain intensity over four

regions of India (NE,SW, E and NE) for three different CRA thresholds.

Figure 12. As in Figure 11 for maximum rain intensity.

Figure 13. Spatial distributions (scatter plots on left) of displacement errors and the CRA

counts (shaded grids on right) in 1 degree grid box for individual rainfall zones and CRA

thresholds. The x- and y- axes are in degrees longitude and latitude

respectively.(computations are based on NSGM data)

Figure 14. Box-whisker plots summarizing the correlation, x-, y-, and vector errors (degrees)

over (a) north west (NW) (b) south west (SW) (c) east (E) and (d) north east

(NE).(Computations are based on NSGM data).

Figure 15. Box-whisker plots summarizing the RMSE and contribution to total error from

displacement error, volume error and pattern error over (a) north west (NW) (b) south west

(SW) (c) east (E) and (d) north east (NE).(Computations are based on NSGM data)

33
Figure 1. Geographical domain over India used for rainfall verification showing
terrain elevation (km) and typical distribution of the rain gauge network on any day
during the monsoon season.
Figure 2. 24 hour rainfall valid for 22nd Aug 2012 in (a) TRMM and (b) NSGM.
(TRMM rainfall is accumulated at 00UTC while NSGM rainfall is accumulated at
03UTC)
Figure 3. Frequency distribution of observed daily rainfall (JJAS) over India during
2007-2012. The box-whiskers compare the rainfall based on satellite estimates
(TRMM) and satellite+gauge merged (NSGM). The comparison is presented for
different rainfall thresholds. The values greater (lower) than 3/2 times the 75th
percentile (25th percentile) are outliers.

Figure 4. CRA formed by overlap of forecast and observations.

Figure 5. Observed (upper left) and forecast (Day1, Day3 and Day5) mean JJAS
rainfall (mm/day) over India during 2012.
Figure 6. Observed and UKMO Day1 forecast rainfall frequency distribution over
India.

Figure 7. Observed (NSGM) and UKMO Day1 forecast number of rainy days (rainfall
> 1mm/day) during JJAS 2012. The boxes show four domains that are used to
investigate regional variation in forecast performance.
(a) (b)

(c) (d)

(e) (f)

Figure 8. Rainfall forecast verification scores for Day1 forecasts verified against
TRMM 3B42 and NSGM rainfall analyses: (a) probability of detection (POD), (b)
success ratio (SR), (c) probability of false detection (POFD), (d) extreme dependency
score (EDS), (e) equitable threat score (ETS), and (f) Hanssen and Kuipers score (HK
score).
 UKMO 00-24 forecast for 20070703 NSGM analysis for 20070703

Figure 9. Verification of forecast (left) rainfall over India valid for 3 rd July 2007.
Observed (NSGM; right) rainfall and the skill scores are also shown.

UKMO fcst 20070703 CRA 20070703

UKMO 00-24 fcst 20070702 n=1109

NSGM analysis 20070703

Figure 10. CRA verification results for forecast rainfall along the west coast of India
valid for 3rd July 2007. The CRA is defined using a 10mm d-1 threshold.
Figure 11. Forecast vs observed (TRMM, left; NSGM, right) mean rain intensity over four regions of India (NE,SW, E and NE) for three
different CRA thresholds.
Figure 12. As in Figure 11 for maximum rain intensity.
Figure 13. Spatial distributions (scatter plots on left) and frequency (shaded contour plots on right) of CRA displacements for individual rainfall
zones and CRA thresholds. The x- and y- axes are in degrees longitude and latitude respectively.
Figure 14. Box-whisker plots summarizing the correlation, x-, y-, and vector errors (degrees) over four rain zones.
Figure 15. Box-whisker plots summarizing the RMSE and contribution to total error from displacement error, volume error and pattern error
over four rain zones.