-
The Future of Astronomical Data Infrastructure: Meeting Report
Authors:
Michael R. Blanton,
Janet D. Evans,
Dara Norman,
William O'Mullane,
Adrian Price-Whelan,
Luca Rizzi,
Alberto Accomazzi,
Megan Ansdell,
Stephen Bailey,
Paul Barrett,
Steven Berukoff,
Adam Bolton,
Julian Borrill,
Kelle Cruz,
Julianne Dalcanton,
Vandana Desai,
Gregory P. Dubois-Felsmann,
Frossie Economou,
Henry Ferguson,
Bryan Field,
Dan Foreman-Mackey,
Jaime Forero-Romero,
Niall Gaffney,
Kim Gillies,
Matthew J. Graham
, et al. (47 additional authors not shown)
Abstract:
The astronomical community is grappling with the increasing volume and complexity of data produced by modern telescopes, due to difficulties in reducing, accessing, analyzing, and combining archives of data. To address this challenge, we propose the establishment of a coordinating body, an "entity," with the specific mission of enhancing the interoperability, archiving, distribution, and productio…
▽ More
The astronomical community is grappling with the increasing volume and complexity of data produced by modern telescopes, due to difficulties in reducing, accessing, analyzing, and combining archives of data. To address this challenge, we propose the establishment of a coordinating body, an "entity," with the specific mission of enhancing the interoperability, archiving, distribution, and production of both astronomical data and software. This report is the culmination of a workshop held in February 2023 on the Future of Astronomical Data Infrastructure. Attended by 70 scientists and software professionals from ground-based and space-based missions and archives spanning the entire spectrum of astronomical research, the group deliberated on the prevailing state of software and data infrastructure in astronomy, identified pressing issues, and explored potential solutions. In this report, we describe the ecosystem of astronomical data, its existing flaws, and the many gaps, duplication, inconsistencies, barriers to access, drags on productivity, missed opportunities, and risks to the long-term integrity of essential data sets. We also highlight the successes and failures in a set of deep dives into several different illustrative components of the ecosystem, included as an appendix.
△ Less
Submitted 7 November, 2023;
originally announced November 2023.
-
The demise of the filesystem and multi level service architecture
Authors:
William O'Mullane,
Niall Gaffney,
Frossie Economou,
Arfon M. Smith,
J. Ross Thomson,
Tim Jenness
Abstract:
Many astronomy data centres still work on filesystems. Industry has moved on; current practice in computing infrastructure is to achieve Big Data scalability using object stores rather than POSIX file systems. This presents us with opportunities for portability and reuse of software underlying processing and archive systems but it also causes problems for legacy implementations in current data cen…
▽ More
Many astronomy data centres still work on filesystems. Industry has moved on; current practice in computing infrastructure is to achieve Big Data scalability using object stores rather than POSIX file systems. This presents us with opportunities for portability and reuse of software underlying processing and archive systems but it also causes problems for legacy implementations in current data centers.
△ Less
Submitted 31 July, 2019; v1 submitted 26 July, 2019;
originally announced July 2019.
-
Petabytes to Science
Authors:
Amanda E. Bauer,
Eric C. Bellm,
Adam S. Bolton,
Surajit Chaudhuri,
A. J. Connolly,
Kelle L. Cruz,
Vandana Desai,
Alex Drlica-Wagner,
Frossie Economou,
Niall Gaffney,
J. Kavelaars,
J. Kinney,
Ting S. Li,
B. Lundgren,
R. Margutti,
G. Narayan,
B. Nord,
Dara J. Norman,
W. O'Mullane,
S. Padhi,
J. E. G. Peek,
C. Schafer,
Megan E. Schwamb,
Arfon M. Smith,
Erik J. Tollerud
, et al. (2 additional authors not shown)
Abstract:
A Kavli foundation sponsored workshop on the theme \emph{Petabytes to Science} was held 12$^{th}$ to 14$^{th}$ of February 2019 in Las Vegas. The aim of the this workshop was to discuss important trends and technologies which may support astronomy. We also tackled how to better shape the workforce for the new trends and how we should approach education and public outreach. This document was coauth…
▽ More
A Kavli foundation sponsored workshop on the theme \emph{Petabytes to Science} was held 12$^{th}$ to 14$^{th}$ of February 2019 in Las Vegas. The aim of the this workshop was to discuss important trends and technologies which may support astronomy. We also tackled how to better shape the workforce for the new trends and how we should approach education and public outreach. This document was coauthored during the workshop and edited in the weeks after. It comprises the discussions and highlights many recommendations which came out of the workshop.
We shall distill parts of this document and formulate potential white papers for the decadal survey.
△ Less
Submitted 17 November, 2019; v1 submitted 13 May, 2019;
originally announced May 2019.
-
Ten Year Review of Queue Scheduling of the Hobby-Eberly Telescope
Authors:
Matthew Shetrone,
Mark E. Cornell,
James R. Fowler,
Niall Gaffney,
Benjamin Laws,
Jeff Mader,
Cloud Mason,
Stephen Odewahn,
Brian Roman,
Sergey Rostopchin,
Donald P. Schneider,
James Umbarger,
Amy Westfall
Abstract:
This paper presents a summary of the first 10 years of operating the Hobby-Eberly Telescope (HET) in queue mode. The scheduling can be quite complex but has worked effectively for obtaining the most science possible with this uniquely designed telescope. The queue must handle dozens of separate scientific programs, the involvement of a number of institutions with individual Telescope Allocation…
▽ More
This paper presents a summary of the first 10 years of operating the Hobby-Eberly Telescope (HET) in queue mode. The scheduling can be quite complex but has worked effectively for obtaining the most science possible with this uniquely designed telescope. The queue must handle dozens of separate scientific programs, the involvement of a number of institutions with individual Telescope Allocation Committees as well as engineering and instrument commissioning. We have continuously revised our queue operations as we have learned from experience. The flexibility of the queue and the simultaneous availability of three instruments, along with a staff trained for all aspects of telescope and instrumentation operation, have allowed optimum use to be made of variable weather conditions and have proven to be especially effective at accommodating targets of opportunity and engineering tasks. In this paper we review the methodology of the HET queue along with its strengths and weaknesses.
△ Less
Submitted 26 May, 2007;
originally announced May 2007.
