Hosted by "1PLs Agency"

Marine Data Literacy 2.0

Providing instruction for managing, converting, analyzing and displaying oceanographic station data, marine meteorological data, GIS-compatible marine and coastal data or model simulations, and mapped remote sensing imagery

 

 

 

 

Home > 8. Access & Services > 8.5 THREDDS Servers

8.5 Data Browsing/Mining in THREDDS Servers

  • Exercise Title:  Data Browsing/Mining in THREDDS Servers

  • Abstract:  This exercise is not about making any specific data product from a particular data file.  It is to demonstrate how to browse for and obtain data in THREDDS servers.  Companion exercises will be published to demonstrate similar functionality with OPeNDAP/LAS data and SOS data.

  • Preliminary Reading (in OceanTeacher, unless otherwise indicated):

    • N/A

  • Required Software:

  • Other Resources: 

    • Thematic Realtime Environmental Distributed Data Services (THREDDS) - "Middleware to bridge the gap between data providers and data users...Catalogs are the heart of the THREDDS concept. They are XML documents that describe on-line datasets...The current focus of THREDDS development is the THREDDS Data Server (TDS), which actually serves the contents of the datasets, in addition to providing catalogs and metadata for them. The TDS uses the Common Data Model to read datasets in various formats, and serves them through OPeNDAP, OGC Web Coverage Service, NetCDF subset, and bulk HTTP file transfer services." [From the website]

  • Author:  Murray Brown

  • Version:  August 2013
1.  Install Environmental Data Connector SOS Beta (EDC-SOS) stand alone
2.  Open the edcconfig.xml file in your C:\EDC folder and change <CLOSE_AFTER_PROCESSING> to false.
3.  Run EDC.  You'll see these 3 data options:
  • CATALOG URL:  A THREDDS data server address
  • DIRECT ACCESS URL:  An OPeNDAP server address
  • SENSOR OBSERVATION SERVICE (SOS):  Web service address for direct access to measurements at remote locations
4.  In Panel 42 below, find and copy the XML catalog filename for the IDV/UCAR/UNIDATA master catalog of UNIDATA's THREDDS servers.  [First one in the list]
5.  Run EDC.  Check CATALOG URL and enter the catalog filename you just found.

NOTE:  In later exercises, we'll work with DIRECT ACCSS URLs and SENSOR OBSERVATION SERVICEs.
6.  Click on CONNECT on the right side.
7.  After a short wait, if the connection is made, you'll see these items appeat in the EDC object menu.
8.  You can open the folders to see these main sub-folders.

Take a few minutes to explore these folders.  There are hundred of items.  Don't try to open any data file, just explore.
9.  Navigate down into the folders to find this one.  We're going to use it as a typical example, and you don't need to worry now about its details.
10.  When you have selected a file, then you can click on SUBSET & PROCESS to continue.
11.  If you are successfully connected, you'll see this chart.  On this page are the controls to subset by time, space, variable, etc.
12.  Read through the variables, and find this one, the average temperature at "height above ground".
13.  Find the left end of the time slider, which is the beginning of the period available for use.
14.  Find the right end of the slider, which i the end.
15.  Move the 2 sliders closer together, so you have 1 or 2 day (according to the START and STOP values).
16.  Along the top, click on AOIs, and select ENTER AN AOI MANUALLY.
17.  If this file is global, then you can enter the AOI coordinates for Liberia.

Then click on ACCEPT.
18.  Before you do anything else, save this AOI for permanent use.  Click on AOIs and select SAVE CURRENT AOI.
19.  Enter an appropriate name, such as LIBERIA.  Then click OK.
20.  Now click PROCESS (along the bottom) to perform the subsetting.
21.  When asked, you can enter a filename (using our file-naming protocol, or you can accept the automatic suggestion).

Then click OK.
22.  If everything went OK, then you'll see this map of the data subset, automatically colored.  This is just for your information, and it is not a regular "data product" for scientific use.
23.  If you look into the EDC installation folder, you'll find these 2 new file, created by the subsetting process:
  • Temperature_(Unweighted_mean_of_all_members)_@_Specified_height_level_above_ground.nc
  • Temperature_(Unweighted_mean_of_all_members)_@_Specified_height_level_above_ground.xml
24.  This is the XML file, and you can see that it contains metadata.  Note that there are multiple time steps, so an animation is possible.  On your own time, find how to make an animation in EDC.
25.  Using the standard methods you've learned in the Operational exercises, you can open the NC file in Integrated Data Viewer (IDV).  Here is the contoured plan view map.  The Liberia coastline is easily visible.
26.  This completes the basic use of EDC to get, process and save files within THREDDS servers.  But there is a complex relationship between OPeNDAP servers and THREDDS servers, that we want to explore just a little more.  In the table of Panel 42 you can see that nearly all XML catalog sites have  doppleganger sites with the extension HTML. 

Many websites list the HTML location as "THREDDS" servers, but we know this can't be true, because the URL ends with XML.  What is in these sites, and how can we use them?
27.  Return to the table of THREDDS servers, and find the HTML file just below the XML file you used above.  It has exactly the same path and filename, but the extension is HTML.
28.  Open the HTML file in a browser, and this is approximately what you'll see.

Click on the NCEP MODEL DATA link.
29.  This page is similar to the list of objects in Panel 9, above.  Let' make the same selections we did above.

Click on GEFS-GLOBAL ENSEMBLE DERIVED PRODUCTS
30.  The returned page has 3 links.  Below, we're going to see what each one reveals.
31.  The top link returns a page with very detailed metadata about the data product.
32.  The middle link, which we will examine below, contains a list of 8 access methods to the data product, supported by THREDDS.
33.  Across the top of the middle link page is this URL.  Copy it for use below.

  • http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GEFS/Global_1p0deg_Ensemble/derived/catalog.html
34.  The bottom link leads to a list of data product files.
35.  You can explore the different links and the pages they open.  There are many items that are repeated on various pages and at various browse levels.
36.  To complete this short discussion, take the XML link you copied above, and covert it to an XML link.

  • http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GEFS/Global_1p0deg_Ensemble/derived/catalog.html

  • http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GEFS/Global_1p0deg_Ensemble/derived/catalog.xml
37.  Back in EDC you can enter the XML link in the CATALOG URL space, and click CONNECT.

You'll see the same files we saw above in Panel 9.  These can be processed here, as before.

 
38.  But also, near the bottom of the middle link page is this list of ACCESS methods that are offered by THREDDS for the data products.
39.  Briefly, these methods can be summarized.

Click on each one, to see for yourself.  Visual inspection may be better than these simplistic descriptions.

  1. OPeNDAP - See the excellent summary in the THREDDS Tutorial, cited above

  2. WCS - See the excellent summary in the THREDDS Tutorial, cited above

  3. WMS - See the excellent summary in the THREDDS Tutorial, cited above

  4. NetCDF Subset - Subsetting by time, space, variable, depths, etc.

  5. CDMRemote - CDM is the Common Data Model used by UNIDATA to make conversions between HDF, NetCDF, GRIB and other major formats.  This function presents you with a CDM use metadata file for the data product

  6. NCML - "Wrapper" files that can be used for virtual (but not actual) alteration of internal items/contents of NetCDF files; for example change altitude from positive to negative;  - See the excellent summary in the THREDDS Tutorial, cited above

  7. UDDC - Generates a scoresheet on how well a NetCDF file complies with various standard specifications and desired characteristics

  8. ISO - Displays the ISO-compliant metadata for the specific file
40.  This demonstrates the direct connection between XML and HTML data products in a THREDDS server.  The lessons to take are:
  • HTML links often lead (in browsers) to pages that contain HTML links that can be converted to XML and used in EDC 
  • XML links (when converted to HTML) can lead to many informative pages that lead to other useful THREDDS functions, or can be converted back again to XML for use in EDC

42.  Example THREDDS catalogs.  Existing XML catalog files (often named CATALOG.XML) can be loaded into EDC for data discovery and capture through THREDDS.  Usually, the extension XML can be replaced with HTML, leading to an information "page" that can be viewed in any browser; it cannot be used by EDC.  The HTML pages are often cited improperly on many websites as THREDDS catalogs.