Development of A Weather Forecasting Code: A Case Study: IEEE Software August 2008
Development of A Weather Forecasting Code: A Case Study: IEEE Software August 2008
Development of A Weather Forecasting Code: A Case Study: IEEE Software August 2008
net/publication/3249531
CITATIONS READS
29 734
8 authors, including:
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Susan Squires on 05 August 2013.
Development of a
Weather Forecasting Code:
A Case Study
Richard Kendall, David Fisher, and Dale Henderson, Software Engineering Institute
Andrew Mark, Douglass Post, and Clifford E. Rhoades Jr., US Department of Defense
High Performance Computing Modernization Program
C
omputational science is increasingly providing insight into scientific phenom-
A case study of a
ena that have previously been studied only experimentally, observationally, or
weather-forecasting
theoretically. Codes, the software projects written for computational science,
code aimed to generally have three main differences from traditional software projects, such
investigate the as those from the commercial software domain. First, because these projects often per-
code development form new scientific investigations, the requirements can’t be known in advance and must
challenges, evolve over time. Second, the main driving force for these projects is producing correct,
understand the important scientific advances rather than ensuring ■ identifying issues that hardware and software
development tools software quality through formalized software engi- vendors must address to make the code devel-
neering processes. Finally, these projects’ developers opment process more productive,
used, and document tend to be domain scientists rather than software ■ developing a reference body of case studies for
the findings for engineers, so they’re less likely to use any heavy the computational science and engineering com-
other developers. software development processes.1–3 munity, and
To better understand the impact these character- ■ documenting the lessons learned from analysis
istics have on software development practices in the and personal team interviews.
computational-science domain and to document
lessons learned for similar projects’ benefit, we per- For this case study, we followed the same meth-
formed a series of case studies of computational- odology that we used in the previous case stud-
science code development projects sponsored by the ies.2 Here, we provide only a basic overview of the
Darpa High Productivity Computing Systems pro- methodology to give context for understanding the
gram. Here, we describe the sixth case study (after results. After we identified a suitable project, the
Falcon,4 Hawk,5 Condor,6 Eagle,7 and Nene8). team completed a questionnaire. We used the re-
The common objectives for all case studies were sponses to plan an on-site interview, after which
we iterated the results with the code team to ensure
■ identifying critical success factors, correctness.
July/August 2008 I E E E S o f t w a r e 61
Table 1
The Osprey team’s development practices
Practice Description Followed Comments
Collective Allow anyone to change any code anywhere in the Partially Followed to a small degree; a limited set
ownership system at any time. of developers allowed to make changes
Configuration Establish and maintain the integrity of work products Yes A key role in this project
management using configuration identification and control.
Continuous Integrate and build the system many times a day, Partially Performed weekly rather than daily
integration each time a task is completed.
Feature-driven Establish an overall architecture and feature list, Yes Project development driven by new features
development then design by feature and build by feature.
Frequent delivery/ Have many releases with short time spans; No Driven by task—no explicit goal to have
small releases implement the highest-priority functions first. small releases, although this often occurs
Onsite customer Include a real, live user on the team who is available Yes Code’s developers are also users
full time to answer questions.
Organizational Develop team members’ skills and knowledge so No Nothing formal in place
training that they can perform their roles effectively.
Pair programming Work side by side with another programmer at one No Not found to be useful
computer, collaborating on the design, algorithm,
and code.
Planning game Quickly determine the scope of the next release No Not used
with business priorities and technical estimates.
Peer reviews Review peers’ software artifacts (requirements, Partially Informal code reviews (but not
design, code) to improve quality. requirements or designs) used
Process and product Objectively evaluate adherence to process Partially Long-term roadmap for Osprey, but
quality assurance descriptions and resolve noncompliance. no formal monitoring of adherence
to the development strategy
Project monitoring Provide an understanding of the project’s Yes Internal and external reviews throughout
and control progress so that appropriate corrective actions the year; local groups meeting weekly
can be taken if progress deviates from plan. to discuss issues
Project planning Establish and maintain plans that define project Yes Planning done at the beginning of each
activities. fiscal year as part of the funding cycle
Refactoring Restructure software to remove duplication, Yes Used to improve performance and make it
improve communication, simplify, or add flexibility. easier to change the code in the future
Requirements Produce, analyze, and verify customer, project, Yes A formal process followed as part of
development and product requirements. proposal generation
Requirements Manage the project’s requirements and identify Yes Internal milestones set by each subteam
management inconsistencies with the project plan.
Retrospective Perform a postiteration review of the effectiveness Yes Performed at the annual review
of the work performed, methods used, and estimates.
Risk management Identify potential problems and adequately handle Partially Emphasis on testing and benchmarking;
them. conscious strategy for managing technology
risks; no formal management of other risks
Simple design Design only what is being developed, with little Partially There’s a long-term plan beyond the current
planning for the future. funding; the design is documented only in the
code, so features that aren’t coded haven’t been
formally designed
Tacit knowledge Maintain and update project knowledge in Yes Tacit knowledge in the scientific discipline
participants’ heads rather than in documents. is important
Test-driven Write module or method tests before and during Partially Testing integrated with development
development coding.
July/August 2008 I E E E S o f t w a r e 63
Osprey team still uses Fortran because it’s easy development teams can function well with a very
to learn compared with modern languages. This lightweight process as long as teams are small, per-
The Osprey is true also because the team has yet to encounter form adequate planning, and have good commu-
W
it isn’t the main driver, code performance is still im-
portant to the Osprey team because its customers e hope that these lessons learned will
require real-time applications that support opera- be useful to other scientific-code devel-
tional functions. In addition, as discussed earlier, opers. We realize that the Osprey proj-
maintainability and portability are also essential to ect might be unique and might not represent expe-
the Osprey code’s success. riences in other domains. Nonetheless, we believe
that many of these lessons are applicable to a broad
Agile approaches are better suited range of computational science projects.
than more traditional methodologies
In the projects we studied, including Osprey,
Acknowledgments
scientific-code teams valued agility over formalized We thank the Osprey team members for participating
processes. That is, they usually avoided rigid soft- in this case study. Air Force grant FA8750-05-1-0100
ware management approaches, but planning and supported this research in part.
Osprey is a pseudonym; we have masked or omit-
adoption of useful software practices are important ted any details that might reveal the code project’s
to the success of these projects. We’ve found that identity.
Jeffrey C. Carver is an assistant professor in the Computer Sci- Douglass Post is the chief scientist of the DoD High Perfor
ence and Engineering Department at Mississippi State University. Be- mance Computing Modernization Program. He is a permanent
ginning 16 August, he’ll be an assistant professor in the Department of member of the senior technical staff of the Software Engineering
Computer Science at the University of Alabama. His research inter- Institute. At the HPCMP, he also manages the DoD Create program to
ests include empirical software engineering, software engineering for develop large-scale computational engineering tools for designing
computational science, software architecture, and requirements engi- ships, airplanes, and RF antennas. His research interests include the
neering. Carver received his PhD from the University of Maryland. development of computational engineering and science tools and
He’s a member of the IEEE Computer Society, ACM, American Society the associated software engineering issues. Post received his PhD in
for Engineering Education, and International Software Engineering physics from Stanford University. He’s a fellow of the IEEE, American
Research Network. Contact him at [email protected] until 16 Aug., and at [email protected]. Physical Society, and American Nuclear Society. Contact him at [email protected].
edu after that date.
David Fisher is the chief engineer for the Create program Clifford E. Rhoades Jr. is the technical director of the Maui
and project manager for Create infrastructure in the Department High Performance Computing Center under an Intergovernmental
of Defense High Performance Computing Modernization Program. Personnel Act assignment from the Software Engineering Institute. His
He completed the work described in this article while he was at the research interests include computational physics, high-performance
Software Engineering Institute. His interests include computational computing algorithms, and software engineering. Rhoades received
science and engineering, high-performance computing, software his PhD in physics from Princeton University. He was one of the first
engineering of complex systems, emergent behavior, and software five American Physical Society fellows elected in computational phys-
development infrastructure. Fisher received his PhD in computer ics. He’s a member of the ACM, the Air Force Association, the Ameri-
science from Carnegie Mellon University. Contact him at david. can Institute of Aeronautics and Astronautics, the American Physical
[email protected]. Society, the IEEE, and the IEEE Computer Society. Contact him at [email protected].
Dale Henderson is retired from the Los Alamos National Susan Squires is executive director of user research at Tac-
Laboratory after a long career in basic and applied research, with tics, a user and customer research consultancy firm that specializes
some focus on large-scale computation and computer simulation. in ethnography to gain insight in connecting what people do to what
Henderson received his PhD from Cornell University in applied phys- they say. She completed the work described in this article while she
ics. He’s a member of the American Physical Society. Contact him at was at Sun. She’s a practicing anthropologist with experience in
[email protected]. customer research, strategic planning, and program management.
Squires received her PhD in anthropology from Boston University.
She is a fellow of the Society for Applied Anthropology. Contact her
at [email protected].
July/August 2008 I E E E S o f t w a r e 65