Using earthaccess to Search for and Download SnowEx Data from the NSIDC DAAC Server

Using `earthaccess` to Search for and Download SnowEx Data from the NSIDC DAAC Server#

Tutorial Overview#

This notebook demonstrates how to search for and download NSIDC DAAC server hosted SnowEx 2023 data using the earthaccess package. SnowEx mission data have not yet migrated to the cloud and continue to be hosted at the NSIDC DAAC server.

As an example data collection, we use SnowEx23 Mar23 IOP Snow Depth Measurements, Version 1 (Snex23_MAR23_SD) over the Alaska field sites. The data are stored in csv format with a metadata-rich header.

We use earthaccess, an open source package developed by Luis Lopez (NSIDC developer) and a community of contributors, to allow easy search of the NASA Common Metadata Repository (CMR) and download of NASA data collections. It can be used for programmatic search and access for both DAAC-hosted and cloud-hosted data. It manages authenticating using Earthdata Login credentials. earthaccess can be used to find and access both DAAC-hosted and cloud-hosted data in just three lines of code. See nsidc/earthaccess.

Learning Objectives#

In this tutorial you will learn:

how to use earthaccess to search for SnowEx data using a spatial filter and explore the search results;
how to download data granules to your hub space or local machine.

Prerequisites#

The workflow described in this tutorial forms the initial steps of a Download Model workflow that could be run on your local machine or on a AWS cloud compute resource. You will need:

a NASA Earthdata Login. If you need to register for an Earthdata Login see the Getting an Earthdata Login section of the ICESat-2 Hackweek 2023 Jupyter Book.

Highly Recommended Viewing#

earthaccess NASA Tech Spotlight video recording

Watch a coding demonstration and learn about the history of earthaccess and the community that supports it.

Credits#

This notebook is based on an NSIDC Data Tutorial originally created by Luis Lopez and Mikala Beig, NSIDC, modified by Andy Barrett, NSIDC, and updated by Jennifer Roebuck and Gail Reckase, NSIDC.

Programmatic data access via earthaccess#

earthacces is a software library that provides an easy to use Python wrapper in front of the CMR and NASA DAAC API’s. Strengths of earthdata include:

cross-DAAC data access (it’s not specific to NSIDC).
Access to DAAC server and cloud hosted data.
Easy authentication handling for Earthdata Login and AWS access keys and token.

# For searching NASA data
import earthaccess

# For nice printing of Python objects
import pprint

Step 1: Authenticate#

The first step is to get the correct authentication that will allow us to access NASA Earthdata. This is all done through Earthdata Login. For accessing data that has been migrated to NASA’s Earthdata Cloud environment the login method also retrieves and handles the correct AWS credentials.

Login requires your Earthdata Login username and password. The login method will automatically search for these credentials as environment variables or in a .netrc file, and if those aren’t available it will prompt us to enter our username and password. We use a .netrc strategy. A .netrc file is a text file located in our home directory that contains login information for remote machines. If we don’t have a .netrc file, login can create one for us.

earthaccess.login(strategy='interactive', persist=True)

auth = earthaccess.login()

Step 2: Search for Collections (Data sets)#

earthaccess leverages the Common Metadata Repository (CMR) API to search for collections and granules. Earthdata Search also uses the CMR API.

We can use the search_datasets method to search for SnowEx collections by setting keyword='SnowEx'.

This will display the number of data collections (data sets) that meet this search criteria.

Query = earthaccess.search_datasets(keyword = 'SnowEx', provider = 'NSIDC_ECS')

The search_datasets method returns a python list of DataCollection objects. We can view the metadata for each collection in long form by passing a DataCollection object to print or as a summary using the summary method. We can also use the pprint function to Pretty Print each object.

We will do this for the first 10 results (objects).

for collection in Query[:10]:
    pprint.pprint(collection.summary(), sort_dicts=True, indent=4)
    print('')
    

{   'concept-id': 'C2881748646-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_SSA_SO.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_SSA_SO+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_SSA_SO/versions/1/'],
    'short-name': 'SNEX23_SSA_SO',
    'version': '1'}

{   'concept-id': 'C2893208516-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'GeoTIFF', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_Lidar.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_Lidar+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_Lidar/versions/1'],
    'short-name': 'SNEX23_Lidar',
    'version': '1'}

{   'concept-id': 'C2893835306-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'GeoTIFF', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_Lidar_Raw.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_Lidar_Raw+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_Lidar_Raw/versions/1'],
    'short-name': 'SNEX23_Lidar_Raw',
    'version': '1'}

{   'concept-id': 'C2854618795-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'GeoTIFF', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_BCEF_TLS.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_BCEF_TLS/versions/1',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_BCEF_TLS+V001'],
    'short-name': 'SNEX23_BCEF_TLS',
    'version': '1'}

{   'concept-id': 'C2856186714-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'LAZ', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_BCEF_TLS_Raw.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_BCEF_TLS_Raw/versions/1',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_BCEF_TLS_Raw+V001'],
    'short-name': 'SNEX23_BCEF_TLS_Raw',
    'version': '1'}

{   'concept-id': 'C3041011983-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'ASCII', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_SWE.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_SWE+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_SWE/versions/1/'],
    'short-name': 'SNEX23_SWE',
    'version': '1'}

{   'concept-id': 'C3172387010-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTP'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_MAR23_SD.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_MAR23_SD+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_MAR23_SD/versions/1/'],
    'short-name': 'SNEX23_MAR23_SD',
    'version': '1'}

{   'concept-id': 'C2735033831-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_SSA.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_SSA+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_SSA/versions/1/'],
    'short-name': 'SNEX23_SSA',
    'version': '1'}

{   'concept-id': 'C3184787416-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'Binary', "
                 "'FormatDescription': 'HTTPS'}, {'FormatType': 'Native', "
                 "'Format': 'CSV', 'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CSU_GPR_Raw.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CSU_GPR_Raw/versions/1/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CSU_GPR_Raw+V001',
                    'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CSU_GPR_Raw.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CSU_GPR_Raw/versions/1/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CSU_GPR_Raw+V001'],
    'short-name': 'SNEX23_CSU_GPR_Raw',
    'version': '1'}

{   'concept-id': 'C3215566728-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CRREL_GPR.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CRREL_GPR+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CRREL_GPR/versions/1/'],
    'short-name': 'SNEX23_CRREL_GPR',
    'version': '1'}

For each collection, summary returns a subset of fields from the collection metadata and the Unified Metadata Model (UMM):

concept-id is a unique id for the collection. It consists of an alphanumeric code and the provider-id specific to the DAAC (Distributed Active Archive Center). You can use the concept_id to search for data granules.
short_name is a quick way of referring to a collection (instead of using the full title). It can be found on the collection landing page underneath the collection title after ‘DATA SET ID’. See the table below for a list of the shortnames for ICESat-2 collections.
version is the version of each collection.
file-type gives information about the file format of the collection granules.
get-data is a collection of URLs that can be used to access the data, collection landing pages and data tools.
cloud-info this is for cloud-hosted data and provides additional information about the location of the S3 bucket that holds the data and where to get temporary AWS S3 credentials to access the S3 buckets. earthaccess handles these credentials and the links to the S3 buckets, so in general you won’t need to worry about this information.

Notice that the concept-id contains information about the locatio of the data: NSIDC_ECS and NSIDC_CPRD. NSIDC_ECS refers to the NSIDC local server and NSIDC_CPRD refers to NSIDC’s Earthatcloud-hosted collections, which you will see for the ICESat-2 collections currently in the cloud.

concept-id's are unique identifiers for collections. Alternatively, you can specify a short-name (and version if there are multiple public versions of a data set).

For SnowEx, ShortNames are generally how different products are referred to.

ShortName	Product Description
SNEX21_TS_SP	SnowEx21 Time Series Snow Pits
SNEX23_SSA_SO	SnowEx23 Laser Snow Microstructure Specific Surface Area Snow-off Data
SNEX23_Lidar	SnowEx23 Airborne Lidar-Derived 0.25M Snow Depth and Canopy Height
SNEX23_Lidar_Raw	SnowEx23 Airborne Lidar Scans Raw
SNEX23_BCEF_TLS	SnowEx23 Bonanza Creek Experimental Forest Terrestrial Lidar Scans
SNEX23_BCEF_TLS_Raw	SnowEx23 Bonanza Creek Experimental Forest Terrestrial Lidar Scans Raw
SNEX23_SWE	SnowEx23 Snow Water Equivalent
SNEX23_MAR23_SD	SnowEx23 Mar23 IOP Community Snow Depth Measurements
SNEX23_SSA	SnowEx23 Laser Snow Microstructure Specific Surface Area Data
SNEX23_UW_GPR	SnowEx23 University of Wyoming Ground Penetrating Radar

Narrow Search using a Spatial Filter#

Here we are going to use a bounding box for the Alaska study areas to find SnowEx collections.

Query = earthaccess.search_datasets( 
  keyword = 'SnowEx',
     bounding_box = (-149.597,64.699,-147.49,70.085),
)

Eleven datasets were found within this bounding box.
As we did above, we will make a list of the datasets within this bounding box.

for collection in Query[:11]:
    pprint.pprint(collection.summary(), sort_dicts=True, indent=4)
    print('')
    

{   'concept-id': 'C2881748646-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_SSA_SO.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_SSA_SO+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_SSA_SO/versions/1/'],
    'short-name': 'SNEX23_SSA_SO',
    'version': '1'}

{   'concept-id': 'C2893208516-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'GeoTIFF', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_Lidar.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_Lidar+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_Lidar/versions/1'],
    'short-name': 'SNEX23_Lidar',
    'version': '1'}

{   'concept-id': 'C2893835306-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'GeoTIFF', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_Lidar_Raw.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_Lidar_Raw+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_Lidar_Raw/versions/1'],
    'short-name': 'SNEX23_Lidar_Raw',
    'version': '1'}

{   'concept-id': 'C2854618795-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'GeoTIFF', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_BCEF_TLS.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_BCEF_TLS/versions/1',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_BCEF_TLS+V001'],
    'short-name': 'SNEX23_BCEF_TLS',
    'version': '1'}

{   'concept-id': 'C2856186714-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'LAZ', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_BCEF_TLS_Raw.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_BCEF_TLS_Raw/versions/1',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_BCEF_TLS_Raw+V001'],
    'short-name': 'SNEX23_BCEF_TLS_Raw',
    'version': '1'}

{   'concept-id': 'C3041011983-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'ASCII', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_SWE.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_SWE+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_SWE/versions/1/'],
    'short-name': 'SNEX23_SWE',
    'version': '1'}

{   'concept-id': 'C3172387010-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTP'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_MAR23_SD.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_MAR23_SD+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_MAR23_SD/versions/1/'],
    'short-name': 'SNEX23_MAR23_SD',
    'version': '1'}

{   'concept-id': 'C2735033831-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_SSA.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_SSA+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_SSA/versions/1/'],
    'short-name': 'SNEX23_SSA',
    'version': '1'}

{   'concept-id': 'C3184787416-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'Binary', "
                 "'FormatDescription': 'HTTPS'}, {'FormatType': 'Native', "
                 "'Format': 'CSV', 'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CSU_GPR_Raw.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CSU_GPR_Raw/versions/1/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CSU_GPR_Raw+V001',
                    'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CSU_GPR_Raw.001/',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CSU_GPR_Raw/versions/1/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CSU_GPR_Raw+V001'],
    'short-name': 'SNEX23_CSU_GPR_Raw',
    'version': '1'}

{   'concept-id': 'C3215566728-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CRREL_GPR.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CRREL_GPR+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CRREL_GPR/versions/1/'],
    'short-name': 'SNEX23_CRREL_GPR',
    'version': '1'}

{   'concept-id': 'C3227266712-NSIDC_ECS',
    'file-type': "[{'FormatType': 'Native', 'Format': 'CSV', "
                 "'FormatDescription': 'HTTPS'}]",
    'get-data': [   'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX23_CRREL_GPR_Raw.001/',
                    'https://search.earthdata.nasa.gov/search?q=SNEX23_CRREL_GPR_Raw+V001',
                    'https://nsidc.org/data/data-access-tool/SNEX23_CRREL_GPR_Raw/versions/1/'],
    'short-name': 'SNEX23_CRREL_GPR_Raw',
    'version': '1'}

Step 3: Search for data granules (files)#

First we will search for the SnowEx 23 Mar23 IOP Snow Depth Measurement collection using its short name: Snex23_Mar23_SD.

results = earthaccess.search_data(
    short_name='Snex23_MAR23_SD',
    bounding_box=(-149.597,64.699,-147.49,70.085)
)

results

[Collection: {'EntryTitle': 'SnowEx23 Mar23 IOP Community Snow Depth Measurements V001'}
 Spatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -149.597, 'EastBoundingCoordinate': -147.49, 'NorthBoundingCoordinate': 70.08499, 'SouthBoundingCoordinate': 64.699}]}}}
 Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2023-03-07T00:00:00.000Z', 'EndingDateTime': '2023-03-16T23:59:59.999Z'}}
 Size(MB): 3.01295
 Data: ['https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX23_MAR23_SD.001/2023.03.07/SNEX23_MAR23_SD_AK_20230307_20230316_v01.0.csv']]

Step 4: Download Data Locally#

For this section, we will download the granule above from the SnowEx23 Mar23 IOP Snow Depth Measurements collection locally.

We’ll download the file into a separate folder named “tmp”, which will be created for us, if it doesn’t already exist.

downloaded_files = earthaccess.download(
    results,
    local_path='/tmp',
)