Data Sources
NCEI receives data from many different sources, including individual researchers, other government organizations, and research institutions from all over the world. The data submitted to the NCEI archive are subject to one of the three following quality control procedures:
NOAA Scientists
Data from NOAA scientists and contractors are subject to the to the guidelines and best practices for Data Management Planning (archiving responsibilities), Data Documentation (metadata requirements), Data Access (discoverability), and Data Citation (credit and citations) found in the NOAA Data Management Handbook.
NOAA Funded Projects
Data providers who received NOAA funding through a grant, cooperative agreement, or contract should review project-specific guidelines for data archive responsibilities and related peer-reviewed publications. Those guidelines are in the Handbook’s Data and Publication Sharing Directive for NOAA Grants, Cooperative Agreements, and Contracts sections.
Non-NOAA Funded Data
Data collected without NOAA funding or support must undergo a scientific appraisal process to be considered for the archive and is subject to the NESDIS non-NOAA data policy upon approval. Please refer to the Handbook for more information about the appraisal process.
Submitting Your Data
NCEI follows a robust, well defined archive process to preserve data for future use. We accept data, metadata, and products that add value to the core NOAA data collection, and contribute to the long-term preservation of the global environmental data record.
Submission Methods
Contact Us
Before contributing data, first time submitters should email [email protected] to consult with an archive specialist. Our staff can help you choose the correct submission method and provide a detailed explanation of the data submission process.
ATRAC
Use the Advanced Tracking and Resource Tool for Archive Collections (ATRAC) to submit repeating or multiple delivery data, or data that exceeds 20 GB.
Data Submission Format Guidance
NCEI prefers open file formats maintained by a standards organization rather than proprietary or product-specific options. Self-describing formats like netCDF and HDF include valuable metadata that support data compatibility and long-term information preservation. PDF or PDF for archiving (PDF/A) are preferred for archived documents. Above all, choose a format that accommodates characteristics of the data being encoded. Additionally, consider the format’s long-term (decades into the future) compatibility with data access software and infrastructure. For example, tabular data in comma-separated UTF-8 or ASCII files are much more likely to be accessible to future operating systems than proprietary spreadsheet formats (e.g., Lotus 1-2-3).
NCEI recommends using the National Archives and Records Administration (NARA) transfer formats for digital data. Refer to the Library of Congress Sustainability of Digital Formats page when considering formats outside of the NARA list. All submissions are subject to review, and may need to undergo changes to format, filename, packaging, and other criteria to meet system or preservation requirements before being approved.
Frequently Used Formats
| Generic Data Types | Preferred Format(s) | Comment(s) |
|---|---|---|
| Tabular Data |
|
ASCII or UTF/UTF8/UNICODE encoding; See additional NetCDF guidance for many kinds of observation data (i.e, profiles, time series, trajectories, etc.) |
| Geospatial Information Systems (GIS) |
|
|
| Text |
|
Text documents must meet NOAA accessibility guidelines for compliance with Section 508 |
| Metadata | ISO 191xxx (series of international Geographic Metadata) (.xml) | The preferred representation of metadata in these standard formats is eXtensible Markup Language (xml) |
| Moving Images |
|
Please contact [email protected] before submitting video data. Be sure to assess potential copyright and privacy issues before archiving. |
| Still Images | TIFF | |
| Others |
|
File Naming Conventions
The names of files submitted to NCEI must:
- Be comprehensible and readable
- Exclude special characters (such as punctuation and symbols)
- Use appropriate file format extensions
Additional file naming and packaging rules may apply for data based on the submission pathway, or data that must adhere to specific program guidelines. NCEI personnel are available to review and provide feedback on files as needed.
Supporting Documentation
An archive requires sufficient information to read, understand and characterize data holdings in accordance with documentation standards. This documentation contextualizes data, allowing current and future users to:
- Read the encoded format including descriptions of the data variables/content
- Understand data lineage pertaining to how it was created
- Characterize data quality to assess its usability
- Identify and distinguish data
- Trust the data integrity
Common supporting documents include user guides, format specifications, algorithm theoretical basis documents, and read-me files. Other forms of documentation may be appropriate for archiving to support data stewardship. NCEI may request additional documentation in order to meet these requirements.
The submitter is responsible for ensuring that documents comply with Federal Section 508 accessibility requirements. The NOAA Central Library Section 508 compliance page has many resources, such as how to create 508 compliant documents and tools for checking accessibility.
Collection Metadata
NCEI requires standardized metadata for every collection or dataset to support discoverability, understandability and interoperability. This documentation includes a standard description, instructions for accessing and citing the data, and other pertinent details. Collection and metadata creation is supported through the various data submission methods.
Data Licensing
From the NESDIS/NCEI Open Data Policy:
GENERAL POLICY AND GUIDANCE
- Environmental data and information produced by NOAA or any Federal agency are available fully and openly to data users unless explicitly exempt from release by Federal law, regulation or policy. These data are in the public domain in the United States. To serve our international partners and customers, NCEI will work to ensure a Creative Commons CC0 license is applied.
- Environmental data and information collected or created under NOAA grants or cooperative agreements must be made visible, accessible, and independently understandable to general users in a timely manner (typically no later than two (2) years after the data are collected or created) (See 5d.). Except where limited by law, regulation, policy or security requirements, NCEI will strongly encourage providers to apply a Creative Commons CC0 (preferred) or CC BY 4.0 license in grant or agreement language, or upon submission.
- NCEI may archive data produced by non-Federal entities for which the creator either surrenders or maintains intellectual rights. NCEI will seek to minimize access restrictions for environmental data and information acquired from non-Federal sources, and will encourage providers to apply for a Creative Commons CC0 (preferred) or CC BY 4.0 license. If it is determined that accepting limited-access data is not in the best interest of the public, NCEI may decline to accept data. (See 5e.)
NCEI Archive Services
Data Discovery and Access
NCEI has a number of search and visualization tools that allow users to find data using custom queries that pull from the entire archive, as well as individual products and datasets. Following the archive process early in your data gathering and collection efforts can optimize its discoverability with these systems.
Data Identification and Attribution Using DOIs
NCEI may mint Digital Object Identifiers (DOIs) for data held in its archive in accordance with the NOAA Data Citation Procedural Directive. Once minted, a DOI provides a unique and persistent identifier that allows users to accurately cite and locate data obtained from NCEI. Data without an assigned DOI can be cited, but may not have a persistent URI or have citation metadata published in the DataCite discovery service. See FAQs below for more information.
Data Preservation
Data stewarded by NCEI must include metadata about provenance, authenticity, the technical environment necessary to use the data object, what preservation actions have been taken, and what intellectual property rights apply to the data object. To provide long term preservation for digital and analog data holdings, NCEI follows guidance from many best practice communities to ensure that data are accurate and authentic for the long term.
As mandated by Federal regulations, data maintained by NCEI are scheduled in accordance with the appropriate NOAA Records Schedules. These schedules identify many records series which indicate the requirements for how long those records (data and metadata) must be retained, and other conditions for retention and preservation. Each record series must be approved by the National Archives and Records Administration (NARA).
To support and ensure data preservation, NCEI routinely performs regular media migration, data integrity checks (e.g., comparing cryptographic hash values prior to and following file movement between storage media), daily and weekly media content backups, and other practices to ensure access to data objects. NCEI follows guidance from NARA for backup processes, storage media, etc.
FAQs
-
Why should I archive my data?
-
The goal of data preservation is to ensure that your data are independently understandable for future use and re-use. To assist you, NCEI follows a well defined archive process based on the NOAA Procedural Directives that contextualizes data and documentation to ensure that they remain valuable and functional for as long as possible. Because all data acquired, observed, processed or recovered by NOAA is subject to these guidelines, it’s important to incorporate them early in a project’s lifecycle to ensure data compliance and long term viability.
-
How does NCEI decide if my data will be archived?
-
NCEI generally follows the guidance outlined in the NOAA Procedure for Scientific Records Appraisal and Archive Approval (2008). NCEI also follows a policy for receiving and using non-NOAA data issued by our parent organization, the NOAA National Environmental Satellite, Data, and Information Service (NESDIS). This policy establishes criteria to authenticate incoming non-NOAA data, ensure consistency and sustainability, and maintain security. Much of the appraisal process is based on the information you provide to NCEI before submitting data. NCEI considers many factors, including but not limited to:
- What type of data is being presented to NCEI to archive? For general categories of the types of data acquired by NCEI, please refer to the NCEI Archive Collecting Policy.
- Is the original observation data and metadata to be archived, a "data product," developed from other data sources, or both? If a "data product," is the original observation data archived elsewhere and if so, where?
- Is this a one-time submission of a relatively small data product (less than about 20GB total), or would the product require a frequently-updated or repeating submission of new/revised data? This helps NCEI advise about the most appropriate data submission process for the data.
- How large (in MB/GB/TB/PB) is the data or data product to be archived?
- Are consistent file naming conventions being used? Please describe the file naming convention.
- What format is used to represent the distribution analysis product? NCEI prefers data in formats that do not require specific proprietary software for re-use.
- Is there descriptive metadata for these data and/or product(s)? NCEI uses the ISO 19115 family of metadata standards, with NASA Global Change Master Directory (GCMD) keywords, to describe NCEI archival data holdings.
- Are additional services to discover, access, visualize, or distribute the analysis product needed, or would the standard suite of NCEI discovery and access tools (e.g., https, ftp, THREDDS, Live Access Server) be sufficient?
-
How should I submit my existing relational database (RDBMS) to NCEI for archiving?
-
Relational database management systems (RDBMS) present many unique challenges for long-term preservation and access. Please contact [email protected] prior to submitting data in an RDBMS to determine what is the best solution for your RDBMS data.
-
What are the acceptable file formats for data archived at NCEI?
-
There are many "acceptable” file formats. There is no 'one size fits all' format, so it is best to use a standards-based, community-recognized, non-software specific file format for encoding the data. The NCEI list of preferred formats, which are based on guidance by the National Archives and Records Administration and the Library of Congress.
An 'ideal' format is:
- Not software specific (i.e., is not limited to use by a single, specific program or application)
- Based on well-documented national or international standards based format specifications
- Suitable for the type of data that are encoded in the format
-
Can I get a Digital Object Identifier (DOI) for data at NCEI?
-
Please notify NCEI while preparing the data for archiving that you would like to obtain a DOI. Data must be archived with NCEI in order to be eligible for a NOAA Data DOI assignment. NCEI follows the guidance in the NOAA Data Citation Procedural Directive and will work to assign a DOI, as necessary.
-
Does NCEI accept "DNA data"?
-
Not at this time due to the archive storage resource constraints. NCEI is reviewing options to better support these data. NCEI may accept summarizations of genetic information, but contact [email protected] prior to submitting genetics data.
-
Can NCEI embargo access to data I submit until my paper is published?
-
Possibly. NCEI follows the guidance in the NOAA Data Access Procedural Directive for providing access to data provided by federal staff. NCEI follows the guidance in the NOAA Data and Publication Sharing Directive for NOAA Grants, Cooperative Agreements, and Contracts for providing access to data provided by non-federal staff.
-
Does NCEI archive software that I developed for use with my data?
-
NCEI does not currently archive software, scripts or other model/product generating technologies that were not written by NCEI staff.
-
What is “Science Ready Data”?
-
NCEI refers to 'science ready data' as data that are processed from engineering units to units used in scientific analyses (e.g., voltages measuring the electrical conductivity of seawater would be engineering units, but salinity calculated from the voltages would be the 'science ready data'). Satellite data are often referred to as 'Level 0', 'Level 1', etc., so 'Level 0' are the initial values measured by the satellite based instrument and sent to Earth, where specific algorithms are applied to transform those 'engineering units' into 'science ready data'.
-
What data licenses will NCEI accept?
-
NOAA has adopted the well-known and internationally recognized licenses from Creative Commons. NCEI encourages external data submitters to assign Creative Commons 1.0 Universal Public Domain Dedication (CC0-1.0) onto their data, which ensures maximum usage of the data throughout the industry, for anyone's use and for any purpose.
Submitters who wish to maintain copyright on their data can use the Creative Commons Attribution 4.0 International License (CC BY 4.0). Under this license, the data may be used by anyone, for any purpose, so long as credit is given to the provider when using the data in its original form.
NCEI may accept data submitted under other licenses. Data submitters with questions are encouraged to contact [email protected] to consult with an archive specialist.
-
For contributed data that have not been licensed by the data submitter, will NCEI apply for a license?
-
No. Data submitters should license the data before submitting to NCEI.
-
Has NCEI retroactively licensed their data holdings?
-
No. As with new acquisitions, all existing holdings must be licensed by the data submitter or originator. Because works of the US government are in the public domain in the United States, NCEI can assign a CC0 license to that portion of its holdings. Beyond that, NCEI may reach out to other data submitters to recommend licenses for their submissions. This work has not begun.
