1 Software Development Cycle
The development environment for the ESMF project has several defining characteristics. First, the ESMF development team, the Joint Specification Team, is distributed. This makes incorporating simple, efficient communication mechanisms into the development process essential. Second, the development team is large, diverse, and mostly self-managed. Procedures and conventions must be clear and well documented in order for the whole team to adopt them. Third, the initial set of target applications for the ESMF, referred to as the Joint Milestone Codeset (JMC), is known. Many of those who developed the JMC are active participants in the ESMF project. We therefore have the opportunity to guarantee the ESMF's viability by allowing future users and real codes to guide framework development. For example, we prioritize requirements based on the number of JMC applications that need a capability.
1.1 Software Process Model
The ESMF software development cycle is based on the staged delivery model [10]. The steps in this software development model are:
Software Concept Collect and itemize the high-level requirements of the system and identify the basic functions that the system must perform.
Requirements Analysis Write and review a requirements document - a detailed statement of the scientific and computational requirements for the software.
Architectural Design Define a high-level software architecture that outlines the functions, relationships, and interfaces for major components. Write and review an architecture document.
Stage 1, 2, ..., n Repeat the following steps creating a potentially releasable product at the end of each stage. Each stage produces a more robust, complete version of the software.
Detailed Design Create a detailed design document and API specification. Incorporate the interface specification into a detailed design document and review the design.
Code Construction and Unit Testing Implement the interface, debug and unit test.
System Testing Assemble the complete system, verify that the code satisfies all requirements.
Release Create a potentially releasable product, including User's Guide and User's Reference Manual. Frequently code produced at intermediate stages software will be used internally.
Code Distribution and Maintenance Official public release of the software, beginning of maintenance phase.
We have customized and extended this standard model to suit the ESMF project. Subsequent sections describe how the ESMF team shall implement each development step. It is essential for process definition to be an ongoing: developers must be able to abandon processes that are not working, and to refine processes as lessons are learned during development. We therefore anticipate a period of process refinement prior to each major stage of development. We shall expedite development overall by advancing the progress of a small subset of the framework functionality before the rest. This shall allow us to experiment with document templates and procedures for a particular stage, such as detailed design, before undertaking that stage wholesale.
1.2 Software Concept
The first step in the standard staged development cycle consists of collecting and itemizing the high-level requirements of the software, and identifying the basic functions that it must perform. Participants in the ESMF project completed this exercise in the process of developing a unified set of proposals. A summary of the high-level requirements of ESMF - a statement of project scope and vision - is included in the General Requirements part of the ESMF Requirements Document[6].
1.3 Requirements Analysis
The first section of the ESMF Requirements Document outlines the major ESMF capabilities necessary to meet project milestones and achieve project goals. The second part of the document is a detailed requirements specification for each functionality class included in the framework.
1.3.1 Community Input
We encourage input from the community on requirements. An open community meeting was held on May 30, 2002 in order to obtain feedback on the requirements assembled by the ESMF team and to elicit additional requirements. Hardcopy and web-based requirements submission forms were made available to community members to facilitate the collection process. Additional requirements received are catalogued and viewable on the ESMF website, via the Development link on the navigation bar.
To submit a requirement or comment on an existing requirement on-line, visit the ESMF website and click on the Development link. Instructions for submitting forms are available on this page.
We place highest priority on the requirements imposed by our contractual obligation to NASA as described in our milestones, and by the needs of the codes in the JMC codeset. We will consider adding additional software features desired or required by the community on a case-by-case basis. Factors we would consider in making such a decision would include value to the community overall and resources required to implement the feature.
1.3.2 Requirements Review Teams and Procedures
At the beginning of the requirements process for each functionality class, the ESMF Deployment Team shall be polled to find out which JMC codes plan to use the functionality. These listings are included in the ESMF Requirements Document. The review teams for requirements must include at minimum a representative for each JMC code that is planning to use the class or utility.
Generally the review process will include at least two reviews conducted via email on the esmf_tech@ucar.edu list, and at least one teleconference or live review.
Reviews continue until they are done, or until common sense dictates that the process should stop. We are to a large extent constrained by the timetable specified in our milestones.
Typically the first documents of a given type take longer to prepare than subsequent documents, since we use the initial documents to refine format and process. The ESMF technical leads prepare review schedules based on factors such as the maturity of the document format, imminent milestones, and concurrent activities. A signoff review is held when a document is deemed complete or when additional work on it is not likely to be as productive as proceeding with another task. For example, we may temporarily halt work on a design document in order to engage in an associated prototyping activity. The appropriate time for signoff is decided by consensus.
1.3.3 Requirements Document Organization and Conventions
Requirements documents shall be organized so that specific requirements are listed under a titled topic. For example, under the title Platforms, the ability to run on an SGI Origin, IBM SP, Compaq ES, etc. are listed as individual and separately numbered requirements.
Templates for requirements documents are available in the ESMF document template set (see Section 2.3).
1.3.4 Attributes of Requirements
Each specific requirement possesses the following attributes: priority, source, verification, status, and notes, the last of which is optional. These are typical for requirements analysis [12]. We'll now look at them in more detail.
Priority
The purpose of the priority attribute is to associate each requirement with the milestones and longer term project goals that it satisfies. Each requirement is assigned a number from 1-3, with values defined as follows:
This capability is directly required for a milestone OR half or more of the JMC applications that could use the utility or class in which this capability is embedded would require this capability in order to maintain their existing functionality;
Less than half of the JMC applications that could use the utility or class in which this capability is embedded would require this capability in order to maintain their existing functionality.
This capability is desired in order to extend the existing functionality of one or more JMC codes.
If some capability merits additional explanation to describe its priority, those preparing requirements are encouraged to elaborate.
Changes to requirements' priorities during development will be reviewed by the technical leads and other members of the Joint Specification Team on an as-needed basis. General requirements will be reviewed overall before being updated and submitted for Milestones F, G and K. This occurs on roughly a yearly basis.
Source
The source attribute traces each capability to the applications to which it applies. In addition to applications particular people or organizations may be noted. This attribute helps to identify those that can provide further information and who may also be potential testers and users. It prevents the inclusion of features that have little likelihood of being used.
Verification
The verification attribute specifies an objective and quantitive strategy for assessing whether a requirement is satisfied. Typical values include code inspection, unit test and system test. Some capabilities may require the preparation of special data sets.
Status
Throughout the course of this project it will be useful for us to track what has been accomplished and to archive ideas for extensions and improvements. The status attribute identifies each capability as:
proposed; this indicates an item that has been accepted as useful, but that is not scheduled for implementation.
approved-1; this indicates an item approved for implementation as part of the 1st code release at Milestone F
approved-2; this indicates an item approved for implementation as part of the 2nd code release at Milestone G
implemented
verified
rejected (with reason cited); this indicates an item that has been actively rejected by the review team
Whether the cabability exists in other packages or models is also helpful to note.
Notes
This is a catch-all for additional information such as background, references, related design and implementation issues, risk factors, and so on.
1.4 Architectural Design
The architecture of the framework is presented in an ESMF Architecture Document in terms of an object model [5]. This document describes how the functionality specified in the ESMF Requirements Document is apportioned into classes, and how those classes interact. It also describes the overall layering strategy for the framework, the machine model and the programming model. The glossary of ESMF terms started in the ESMF Requirements Document was refined and extended in this document.
The architecture document was reviewed by all members of the Joint Specification Team, and by an ESMF Peer Review Board.
The comments received back on the Architecture Document, informally and from the Peer Review Board, indicated that the presentation of the document was ineffective at conveying how the ESMF worked. Although the document was full of detailed and complex diagrams, the terminology and diagrams were oriented to software engineers and were not especially scientist-friendly. The detailed diagrams also made the document difficult to maintain. The lessons learned during this stage of development will influence the strategy for presentation of subsequent documents.
1.5 Detailed Design
Users interact with the ESMF through its Application Programming Interface (API). The clarity, consistency, and functionality embodied by the API is a primary factor in determining whether users will be willing and able to adopt the ESMF. As such, it is exceedingly important that the design - and in particular, the API - be thoroughly reviewed by the ESMF application team before the framework is widely deployed. These reviews require iteration. The ESMF API will change rapidly over the first year of development, as we integrate and standardize classes developed by different developers, and as implementation reveals what parts of the design do and do not work well.
1.5.1 Community Input
As with requirements, the ESMF team encourages design input from the community. Comments on the emerging ESMF API were solicited at the community meeting held on May 15, 2003. Web-based forms for design input are available on the ESMF website via the Development link.
1.5.2 Initial Design Reviews
Initial design reviews were held in the early prototype stage of new classes, typically a few months after design began and when some prototype code was available. The design documents prepared included descriptions and, as appropriate, diagrams of classes, showing class relationships and interactions. The documents also included API specifications for all class methods, and descriptions of data structures. Where feasible at this stage of development, discussion of design options was included. To facilitate document development and standardization, a design template was prepared for use by all developers, and included in the set of ESMF document templates (see Section 2.3).
As with the ESMF Requirements Document, members of the Joint Specification Team prepared and reviewed the design documents.
The initial set of design reviews established general outlines for code construction. In most cases, designs changed dramatically as implementation got underway, and as ideas were prototyped. Object models were found to be ineffective compared to API inspection and code examples in conveying how a particular piece of code would work.
1.5.3 Final Interface Reviews
It is clear that another round of intensive class API reviews shall be necessary. These shall occur when the implementation of a class is complete, or close to it. Each ESMF application group shall review and sign off on the final interfaces for each class. The intent of these reviews is to ensure that the interfaces are adequate and easy to use, and to bring the interfaces into line so that they follow standard ESMF conventions for design policies, method names, and documentation procedures.
The review procedure for each class shall be as follows:
A telecon review shall be held in which the full class API and examples of use are provided.
The reviewed material and a summary of requested changes will be posted to the esmf_tech@ucar.edu list. Comments will be collected for a week, and should be sent to the same list. The commentary period may be extended as necessary.
Code and documentation will be modified to reflect the review results.
Modified examples and documentation for the class will be posted to a webpage maintained by the Core Team. The material posted will be the section of the Reference Manual that describes the class. Each application group will have two weeks to do a final review and sign off on the interfaces.
1.6 Code Construction and Unit Testing
Code shall be written in accordance with the interface specifications presented in design documents and the coding conventions described in Section 3. Code reviews shall be conducted by members of the Core Team.
Each class shall be accompanied by a suite of unit tests. The user shall have the option to build either ``sanity check'' type tests or an exhaustive suite. The exhaustive tests will include tests of all functionalities and a variety of valid and invalid input values. The sanity tests will be a minumum set of tests to indicate whether, for example, the software has been installed correctly. The NASA Computational Technologies team will review and supplement the unit tests to ensure that software requirements are met.
1.7 System Testing
System testing and integration shall be performed incrementally on the evolving body of ESMF software, in accordance with the Build and Test Plan.
The ESMF VAlidation (EVA) suite shall be used to test the use of the ESMF system as portions of it become available. For additional discussion of test and validation procedures, see Section 7.
1.8 Release and Iteration
The ESMF project has two major public software releases, one scheduled for April 2003 and one scheduled for April 2004. There are internal software releases monthly. The tagging convention for public and internal releases is described in Section 4.2.
Major releases shall be accompanied by thorough documentation, including an ESMF User's Guide and ESMF Reference Manual. User documentation shall include code excerpts, examples to demonstrate ideas, existing bugs, and restrictions. Detailed test documentation will be made available to the NASA evaluation team, to allow team members to understand the extent of testing, plan deviations, and links to requirements.
In addition, Milestone K (January 2005) includes the provision of a tutorial application that is part of the ESMF VAlidation (EVA) Suite. The entire EVA Suite itself shall be available for download via SourceForge and the ESMF website to users who wish to experiment with other application examples. Releases shall be announced via mailing lists, on the ESMF website, and at community meetings. Source code shall be available for download from the ESMF website and from the SourceForge site.
At the close of both internal and public release cycles, we plan to iterate on requirements, architecture, design, and implementation in order to refine and extend the ESMF software and documentation. The goal will be to maintain a close-to-deliverable system throughout development, with functionality and optimization increasing over time.
1.9 Maintenance and Extension
Routine maintenance of the framework will include ports to new platforms and modifications for new compilers. Extensions to the framework may occur at a wide range of different scales, from adding a new performance optimization to creating an extensive library of tools for advanced support of data assimilation systems. The version of the ESMF User's Guide delivered in January 2005 will provide instructions for routine extensions of the framework - for example, adding support for an additional grid. Other extensions to the framework will be facilitated by the system documentation provided in the User's Guide and Reference Manual.
We will consider open source contributions after our first mature software release at Milestone G (April 2004). We anticipate that if we accept code contributions from others not in our collaboration the contributors will interact with a Change Review Board. We feel that it is premature to detail those interactions at this time. We will upate this document with the specifics such interactions in the Milestone G timeframe.
NCAR has committed to providing ongoing support for maintenance and development of the framework, but the magnitude of that support, and whether it will be supplemented by contributions from other institutions or agencies, is unknown.
1.10 Schedule of Developer and User Documentation
Below is a schedule of documents that shall be prepared for the ESMF, as provided to NASA September 28, 2001 and agreed on during the NASA Site Visit on December 4-5, 2001. In response to the needs of the project some documents were prepared ahead of schedule, and information was shifted to a different document than originally planned. Notes on these changes are contained in square brackets [].
Date (Milestone) Document Contents
May 2002 (A) Draft Software Developer's Guide Recommended and required practices for those developing the core framework. Shall include coding conventions, document templates, configuration management procedures, and software quality assurance procedures. Subsequent versions of the Developer's Guide (for Milestones F and K) shall include comprehensive sections on how to extend and maintain the framework.
General Requirements Document Overall, high level requirements for the ESMF software. The General Requirements Document shall also include framework-wide requirements, such as platforms to be supported.
Aug. 2002 (H) Architecture Document Layering strategy and major classes in the ESMF. This document shall include a high-level description of how major classes in the framework interact.
Implementation Report Analysis of options for implementation language, language interoperability, programming model, and support for scalar and vector processors.
Build and Test Plan Development plan for the ESMF Core Team and ESMF Deployment Teams. Analysis of software dependencies and chart of sequence in which ESMF classes will be developed and integrated. Includes major system integration and validation events. Prepared using a tool such as Microsoft Project.
April 2003 (F) Draft Software Developer's Guide Updated Software Developer's Guide for developers transitioning from the design phase to implementation of the framework.
Draft Class Requirements Documents Analysis of requirements for each major class or set of related classes in the framework. [A draft of the class requirements documents was prepared early, at Milestone A.]
Draft Class Design Documents Detailed design including API specification for each major class or set of related classes in the framework.
April 2004 (G) Class Requirements Documents Updated requirements.
Class Design Documents Updated detailed design and API specifications.
Jan. 2005 (K) User's Guide How to install and run the ESMF, including a specification of system requirements necessary for operating the framework. Includes sufficient explanation to enable users to understand benefits the framework offers, how it is structured, and how it works. [A draft of the User's Guide was prepared early, at Milestone F.]
API Reference Description of the ESMF interface and the function of each public method.
Software Developer's Guide Updated Software Developer's Guide to enable ESMF extension and maintenance.
We note that at NASA's request the Developer's Guide has been extended to include some guidelines for developing framework applications as well as the core framework.
