Unidata Science Gateway on the NSF Jetstream Cloud

Table of Contents

hurricane.jpg

Figure 1: Credits: NASA

1 Introduction

Welcome to the Unidata Science Gateway on the National Science Foundation Jetstream Cloud. Unidata is a diverse community of education and research institutions with the common goal of sharing geoscience data and the tools to access and visualize that data. For more than 30 years, Unidata has been providing data, software tools, and support to enhance Earth-system education and research. Funded primarily by the National Science Foundation (NSF), Unidata is one of the University Corporation for Atmospheric Research (UCAR)'s Community Programs (UCP).

As part of our current five-year plan (PDF) "Unidata 2018: Transforming Geoscience through Innovative Data Service", Unidata is exploring cloud computing technologies in the context of accessing, analyzing, and visualizing geoscience data. This site collects Unidata-related technologies hosted on Jetstream; you can make use of these resources either directly in your browser or using one of the client applications described below. Our hope is that by combining the resources available in this gateway – and perhaps coupling them with XSEDE HPC resources – community members can create end-to-end scientific computing workflows in the Jetstream Cloud.

2 JupyterHub

JupyterHub server on Jetstream (GitHub login required)

This JupyterHub is now open to anyone with an GitHub login. Also please sign up to our mailing list to be made aware of developments and down times. Subscribe at: jupyter-jetstream+subscribe@ucar.edu. After logging in, please read the README_FIRST.ipynb notebook that is periodically updated with relevant information concerning this JupyterHub installation.

JupyterHub is a technology that can serve programmatic, interactive notebooks to a class of students or for scientific researchers. The Unidata JupyterHub server running on Jetstream contains three Unidata Jupyter notebook projects:

3 THREDDS Data Server

TDS installation on Jetstream

The Unidata THREDDS Data server (TDS) is a web server that provides metadata and data access for scientific datasets, using a variety of remote data access protocols. A TDS is available on Jetstream at http://thredds-jetstream.unidata.ucar.edu/thredds/catalog.xml supplying a good portion of the data available on the IDD with a five day archive.

This TDS can be accessed from the IDV or from Python with the netCDF-Python or Siphon APIs or any THREDDS client (e.g., PyCSW).

4 THREDDS AWS Nexrad Server

THREDDS AWS Nexrad Installation on Jetstream

In collaboration with the NOAA Big Data Project and Amazon Web Services (AWS), Unidata has deployed a THREDDS AWS Nexrad Server on the Jetstream Cloud. More than twenty years of Nexrad Level II data are available from this server. The data reside on AWS, but are served out by a THREDDS Data server running on Jetstream.

5 AWIPS EDEX

EDEX installation on Jetstream: edex-cloud.unidata.ucar.edu

AWIPS is a meteorological display and analysis system used by the National Weather Service. EDEX is the server-side component of this system. Unidata maintains an EDEX data server on Jetstream to ingest and serve real-time AWIPS data for rendering by the CAVE client and the python-awips data access framework. For CAVE, simply enter edex-cloud.unidata.ucar.edu in the CAVE Connectivity Preferences dialog to connect.

6 RAMADDA Geoscience CMS

RAMADDA installation on Jetstream

RAMADDA is a geoscience content management system (CMS) originally developed at Unidata and now maintained by Geode Systems. The RAMADDA installation on Jetstream contains IDV bundles that retrieve data from Jetstream data servers.

7 LDM

LDM installation on Jetstream

The Unidata Local Data Manager (LDM) system is a peer-to-peer network for event-driven data distribution, and is the fundamental component of the Unidata Internet Data (IDD) system. Two internal LDM nodes supply data to our Jetstream resources.

8 ADDE

ADDE installation on Jetstream: adde://adde-jetstream.unidata.ucar.edu:112

ADDE (Abstract Data Distribution Environment) is a data server developed by the University of Wisconsin, Space Science Engineering Center. It primarily serves satellite and radar imagery and point observations to the IDV, McIDAS-V, McIDAS-X. An ADDE server is available at adde-jetstream.unidata.ucar.edu over (the usual) port 112.

9 IDV Resources

The Unidata Integrated Data Viewer (IDV) is a desktop application for viewing data in two and three dimensions. It excels at bringing together geoscience data from disparate sources into one integrated display.

9.1 Running the IDV on Jetstream

For those who have a Jetstream allocation (e.g., startup, education, research), an IDV image is available when selecting VMs through the Jetstream Atmosphere dashboard. If you do not have a Jetstream allocation, it is possible to obtain a Jetstream Trial Access account. When searching for images, search for "IDV". You will find the Unidata IDV <version> image. Launching that image will then allow you to "Open Web Desktop" where the IDV will be available pre-loaded with the Jetstream plugin. The catalogs available via the IDV Dashboard, Data Choosers will default to the TDS and RAMADDA servers running on Jetstream enabling data-proximate performance, especially for large datasets. You can also select image sizes that are appropriate to your goals. For example, if working with a data intensive bundle, you could choose an m1.medium VM with 6 vCPUS, 16 GB RAM.

9.2 IDV Jetstream Plugin

The IDV Jetstream plugin allows easy access to Jetstream installations of the TDS, RAMADDA and ADDE from the IDV. The plugin URL is http://science-gateway.unidata.ucar.edu/jetstream.jar. See here for how to install an IDV plugin from a URL.

10 Conference Presentations

11 Under the Hood

The implementation of this effort is described here: https://github.com/Unidata/xsede-jetstream.

12 System Status

13 Contact

If you wish to contact Unidata concerning this science gateway, please contact support@unidata.ucar.edu.

14 Acknowledgments and Bibliography

14.1 Acknowledgments

14.1.1 National Science Foundation

Funding for this project was generously provided by the National Science Foundation (Grant NSF-1344155).

14.1.2 XSEDE ECSS

We thank Brian Beck, Maytal Dahan, Jeremy Fischer, Victor Hazlewood, Peg Lindenlaub, Suresh Marru, Lance Moxley, Marlon Pierce, Semir Sarajlic, Craig Alan Stewart, George Wm Turner, Nancy Wilkins-Diehr, Nicole Wolter and Andrea Zonca for their assistance with this effort, which was made possible through the XSEDE Extended Collaborative Support Service (ECSS) program.

14.2 Bibliography

References

[1] J. Chastang. Unidata ldm docker container. https://github.com/Unidata/ldm-docker/, 2016. [ bib | http ]
[2] J. Chastang. Unidata on the XSEDE Jetstream cloud. https://github.com/Unidata/xsede-jetstream/, 2017. [ bib | http ]
[3] J. Chastang. Unidata science gateway jupyterhub. In ESIP Summer Meeting 2018, Tucson, AZ USA, 7 2018. Earth Science Information Partners. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[4] J. Chastang. Jupyterhub for atmospheric science research and education on the NSF Jetstream cloud. In 98th AMS Annual Meeting, Phoenix, Arizona, USA, Jan. 6-10 2019. [ bib | DOI | http ]
[5] J. Chastang, W. Fisher, and S. Arms. Unicloud: Docker use at unidata. In Proceedings, 32nd Conference on Environmental Information Processing Technologies, 96th AMS Annual Meeting, New Orleans, LA, 1 2016. American Meteorological Society. See also http://unidata.github.io/Unidata-Dockerfiles/index.html. [ bib | .html ]
Keywords: UPC
[6] J. Chastang and M. K. Ramamurthy. Unidata science gateway on the XSEDE Jetstream cloud. In Gateways 2017, Ann Arbor, Michigan, USA, Oct. 25-25 2017. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[7] J. Chastang and M. K. Ramamurthy. A cloud-based science gateway for the geoscience community. In OGC TC/PC Meetings, Fort Collins, Colorado, USA, Jun.  4-8 2018. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[8] J. Chastang and R. Signell. Met/ocean modeling workflows on XSEDE via HPC & cloud. In ESIP Summer Meeting 2017, Bloomington, IN USA, 7 2017. Earth Science Information Partners. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[9] J. Chastang, R. Signell, and J. L. Fischer. Reducing time to science: Unidata and JupyterHub technology using the Jetstream cloud. In 2017 AGU Fall Meeting, New Orleans, Louisiana, USA, Dec. 11-15 2017. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[10] J. Chastang, R. Signell, and J. L. Fischer. A Unidata JupyterHub server: An online PyAOS resource for students and educators. In 97th AMS Annual Meeting, Austin, Texas, USA, Jan. 22-26 2018. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[11] J. Chastang, T. Yoksas, and M. K. Ramamurthy. Geoscientific data distribution in the XSEDE Jetstream cloud. In Proceedings, 33nd Conference on Environmental Information Processing Technologies, 97th AMS Annual Meeting, Seattle, WA USA, 1 2017. American Meteorological Society. See also http://science-gateway.unidata.ucar.edu. [ bib | .html ]
[12] M. K. Ramamurthy. Unidata 2018: Transforming geoscience through innovative data services. Technical report, Unidata Program Center, UCAR Community Programs, University Corporation for Atmospheric Research, 3300 Mitchell Ln, Boulder, CO 80301 USA, 10 2013. [ bib | .pdf ]
[13] M. K. Ramamurthy. A cloud-based science gateway for the geoscience community. In ESIP Summer Meeting 2018, Tucson, AZ USA, 7 2018. Earth Science Information Partners. See also http://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[14] M. K. Ramamurthy and J. Chastang. A cloud-based science gateway for the geoscience community with end-to-end workflows on the Jetstream cloud system. In 2018 EGU Spring Meeting, Vienna, Austria, Apr.  8-13 2018. See also http://science-gateway.unidata.ucar.edu. [ bib ]
[15] M. K. Ramamurthy and J. Chastang. A cloud-based science gateway for the geoscience community with end-to-end workflows on the Jetstream cloud system. In IWSG 2018 - 10th International Workshop on Science Gateways, Edinburgh, Scotland, Jun.  13-15 2018. See also http://science-gateway.unidata.ucar.edu. [ bib ]
[16] M. K. Ramamurthy and J. Chastang. Unidata: A cloud-based science gateway for the atmospheric sciences. In Gateways 2018, Austin, Texas, USA, Sept.  25-27 2018. See also http://science-gateway.unidata.ucar.edu. [ bib ]
[17] M. K. Ramamurthy and J. Chastang. A cloud-based science gateway for enabling open and reproducible science. In 98th AMS Annual Meeting, Phoenix, Arizona, USA, Jan. 6-10 2019. [ bib | http ]
[18] M. K. Ramamurthy, J. Chastang, R. M. May, and M. James. Unidata and data-proximate analysis and visualization in the cloud. In ESIP Summer Meeting 2017, Bloomington, IN, USA, 7 2017. Earth Science Information Partners. See also http://jetstream.unidata.ucar.edu. [ bib | DOI | http ]
[19] S. Sarajlic, J. Chastang, and S. Marru. Scaling Jupyterhub using Kubernetes on Jetstream cloud: Platform as a service for research and educational initiatives in the atmospheric sciences. In PEARC18, Pittsburgh, PA, USA, Jul  22-26 2018. See also http://science-gateway.unidata.ucar.edu. [ bib ]
[20] Craig A. Stewart, Timothy M. Cockerill, Ian Foster, David Hancock, Nirav Merchant, Edwin Skidmore, Daniel Stanzione, James Taylor, Steven Tuecke, George Turner, Matthew Vaughn, and Niall I. Gaffney. Jetstream: A self-provisioned, scalable science and engineering cloud environment. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, XSEDE '15, pages 29:1--29:8, New York, NY, USA, 2015. ACM. [ bib | DOI | http ]
Keywords: atmosphere, big data, cloud computing, long tail of science
[21] John Towns, Timothy Cockerill, Maytal Dahan, Ian Foster, Kelly Gaither, Andrew Grimshaw, Victor Hazlewood, Scott Lathrop, Dave Lifka, Gregory D. Peterson, Ralph Roskies, J. Ray Scott, and Nancy Wilkins-Diehr. XSEDE: Accelerating scientific discovery. Computing in Science & Engineering, 16(5):62--74, 2014. [ bib | DOI ]
[22] K. Wilcox and J. Chastang. Unidata thredds docker container. https://github.com/Unidata/thredds-docker/, 2015. [ bib | http ]
[23] Nancy Wilkins-Diehr, Sergiu Sanielevici, Jay Alameda, John Cazes, Lonnie Crosby, Marlon Pierce, and Ralph Roskies. An overview of the XSEDE extended collaborative support program, volume 595 of Communications in Computer and Information Science, pages 3--13. Springer Verlag, Germany, 2016. [ bib | DOI ]
The Extreme Science and Engineering Discovery Environment (XSEDE) is a flagship cyberinfrastructure project funded by the US National Science Foundation (NSF). XSEDE’s Extended Collaborative Support Services (ECSS) program is a significant component of the XSEDE effort, dedicated to extended engagements with our user community which transform their research. We describe the organization, operation and some highlights of the program in this submission.

Date: 2019-02-15 13:00:04 MST

Validate