TimeSeries.tex

\documentclass[11pt,a4paper]{ivoa} \input tthdefs

\usepackage{todonotes} \usepackage{enumitem} \usepackage{pdflscape}
\usepackage{xcolor,colortbl} \usepackage[most]{tcolorbox}
 
\usepackage{listings} \lstloadlanguages{XML,sh}
\lstset{flexiblecolumns=true,tagstyle=\ttfamily,
  showstringspaces=False,basicstyle=\footnotesize}

\usepackage{multirow}

\definecolor{LightBlue}{rgb}{0.0,0.318,0.612}
\definecolor{LighterBlue}{rgb}{0.39,0.7,1}

% Added this comand for my text to be added in blue and boldface

\newcommand\ada[1]{\textcolor{blue}{\textbf{#1}}}

%\newcommand\elem[1]{\textcolor{LightBlue}{{\tt#1}}}
\newcommand\celcol[1]{\cellcolor{LighterBlue}{\textbf{#1}}}
%\def\attr#1{{\tt{\fg{DarkRed}#1}}}

\let\A=\href \def\Aref#1{section~\ref{#1}}
\def\Arefs#1{section~\ref{#1}} \def\Arefx#1{appendix~\ref{#1}}
\def\Tref#1{Table~\ref{#1}} \def\Fref#1{Figure~\ref{#1}}
\let\fg=\color
%\topmargin=-1cm
%\raggedbottom
%\oddsidemargin=-0.8cm
%\evensidemargin=-0.8cm
%\textwidth=17.5cm
%\textheight=23.5cm
\parindent=0pt \arrayrulewidth=0.75pt\renewcommand{\arraystretch}{1.2}
\definecolor{DarkRed}{rgb}{0.5,0,0}
\definecolor{DarkBlue}{rgb}{0,0,0.5}
\definecolor{DarkPurple}{rgb}{0.3,0.1,0.5}
\definecolor{DarkGoldenrod}{rgb}{0.72,0.5,0.05} \def\slash
            {{\fg{blue}/}} \def\attr#1{{\tt{\fg{DarkRed}#1}}}
            \def\requiredattr#1{{\tt\bf{\fg{DarkBlue}#1}}}
            \def\elem#1{{\tt{\fg{DarkRed}#1}}}
            \def\attrval#1#2{{\tt{\fg{DarkRed}#1}="{\fg{DarkPurple}#2}"}}
            \def\attrvalonly#1{{\tt"{\fg{DarkPurple}#1}"}}
            \def\elemdef#1#2{{\tt\fg{blue}<}{\tt{\fg{DarkRed}#1}#2}{\tt\fg{blue}>}}
            \def\literalvalue#1{{\tt"}{{\fg{DarkPurple}#1}}{\tt"}}
            \def\order{$\oplus$ } \def\unorder{{\large $\circ$ }}
            \def\deprecated {$\dagger$ } \def\choice{{$\mapsto$ }}
            \newenvironment{plain}{\begin{quote}}{\end{quote}}


\title{Time Series: Annotation of light curves in VOTable}
\ivoagroup{Not Applicable}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/AdaNebot]{Ada Nebot}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/FrancoisBonnarel]{Francois
  Bonnarel}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/MarkusDemleitner]{Markus
  Demleitner}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/MireilleLouys]{Mireille
  Louys}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/LaurentMichel]{Laurent
  Michel}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/DaveMorris]{Dave
  Morris}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/MarkTaylor]{Mark Taylor}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/JesusSalgado]{Jesus
  Salgado}
\author[http://www.ivoa.net/twiki/bin/view/IVOA/EnriqueSolano]{Enrique 
  Solano}
\editor{Ada Nebot}

\begin{document}
\begin{abstract}
  This document describes a proposal to annotate in a VOTable time
  series data. It is limited to the most common type of time series in
  astronomy: light curves, but it can be extended to other type of
  time series data easily (e.g.\ radial velocities and positions). The
  annotation reuses parts of existing Data Models when possible and
  defines a set of new conventions. This document can be taken as a test
  case of a more general purpose model.
\end{abstract}

\section*{Acknowledgments}
This work has been supported by the ESCAPE project (the European
Science Cluster of Astronomy \& Particle Physics ESFRI Research
Infrastructures) that has received funding from the European Union's
Horizon 2020 research and innovation programme under the Grant
Agreement n.\ 824064.

\section*{Conformance-related definitions}

The words ``MUST'', ``SHALL'', ``SHOULD'', ``MAY'', ``RECOMMENDED'',
and ``OPTIONAL'' (in upper or lower case) used in this document are to
be interpreted as described in IETF standard RFC2119
\citep{std:RFC2119}.

The \emph{Virtual Observatory (VO)} is a general term for a collection
of federated resources that can be used to conduct astronomical
research, education, and outreach.  The
\href{http://www.ivoa.net}{International Virtual Observatory Alliance
  (IVOA)} is a global collaboration of separately funded projects to
develop standards and infrastructure that enable VO applications.


\section{Introduction}
This document describes how data providers can publish time series in
tabular form to the Virtual Observatory by prescribing metadata for a
minimal set of columns required to describe a (photometric) time
series.  It does not describe how data providers store or manipulate
the data, it is only concerned with how the data is exposed to the
world using well-defined metadata. We assume that the data provider
already distributes time series as a single table, where this table
may contain several columns containing photometry, and it may contain
columns not explicitly mentioned here.

% Notes don't need the architecture diagram
\section{Science use cases for Time Series}
In this section we describe a number of science use cases the proposed
annotation of time series in this document will enable. We divide the
science cases into different groups according to their common
requirements.
%What operations should clients be able to perform based on this without further user intervention?
\subsection{Case A: Light curves in same photometric band}
Common requirement: Combine photometry and light curves of a given
object/list of objects in the same photometric band.

\subsubsection{Use Case 1: Supernova classification using the light curve}
The visual light curves of the different supernova types vary in shape
and amplitude, based on the underlying mechanisms of the explosion,
the way that visible radiation is produced, and the transparency of
the ejected material.

\subsubsection{Use Case 2: Long-term analysis of variable stars}
Long-term period and light curve variations, based on the historical
data and new observations. The continuous period increase can be
interpreted as a mass transfer from the secondary to the primary
star. The most likely explanation of a sinusoidal variation would be a
light travel time effect due to the existence of a circumbinary object
as in the BX Dra triple system.

\subsubsection{Use Case 3: Discovering brown dwarfs by analyzing binary microlensing events}
Microlensing events show characteristic light curve variations which
can be used to determine the type of system producing it.
%(based on Shin et al. 2012)

%\subsubsection{Use Case 4: Eclipsing binary systems - phase}
% Based on G\'omez Maqueo et al 2009)

\subsection{Case B: light curves in different photometric bands}
Common requirement: Combine photometry and light curves of a given
object/list of objects in different photometric bands.

\subsubsection{Use Case 4: Follow-up characterisation of supernovae}
%based on Zhang et al. arXiv:1208.6078v1)
Light curves at different wavelength provide different information
allowing a better understanding of the physical processes related to
the supernovae explosion.

\section{The Time Series Structure}
\label{elem:TIMESERIES}
This document proposes the minimal metadata required in VOTables to
support our science cases. We assume that the tabular data can contain
multiple rows for each astronomical source and for mixed types of time
series (e.g., different filters, positions, radial velocities) and
sometimes even include different types of data (e.g., spectra, or
images).

We propose using some elements previously defined, such as
\elem{TIMESYS} and \elem{COOSYS} (when applicable) in VOTable starting
from version~1.4 \citep{2019ivoa.spec.1021O} and we define a \elem{GROUP}
element with \attrval{name}{PHOTCAL} following the same type of structure.
This note is inspired by
a previous annotation strategy developed for annotating photometric
data in VOTables \citep{note:seb2010-1}. The aim is to be able to combine
multiple time series when they have common elements or attributes as
described in this document. 

%\input{timesys.tex}
%\input{coosys.tex}

\subsection{The \texttt{PHOTCAL} \elem{GROUP}}
The \texttt{PHOTCAL} group contains information on the photometric
system of the observations. This element is partially asociated to the
intermediate class PhotCal of the Photometry Data Model
\citep{2013ivoa.spec.1005S}. A service providing time series of the
type light-curve (e.g., magnitudes, fluxes) \textbf{MUST} provide this
element. To reference the photometric system defined by a
\texttt{PHOTCAL} \elem{GROUP}, \elem{FIELD}s (and possibly
\elem{PARAM}s) \textbf{MUST} reference the group using the VOTable
\attr{ref} attribute. A \texttt{PHOTCAL} group referenced via a
\attr{ref} attribute \textbf{SHOULD} appear before the element that
references it.

Each photometric calibration is defined as a \elem{GROUP} with the
following terms:
\begin{description}
     \item[\attrval{name}{PHOTCAL}] The name of the \elem{GROUP}
       \textbf{MUST} be set to \verb|PHOTCAL|. We realize that
       although this proposed usage of the \attr{name} is not common,
       it is not forbidden by VOTable (\citet{2019ivoa.spec.1021O},
       section 3.2), and in this context it will help clients
       interpret the contents of the \elem{GROUP}.
     \item[\attr{ID}] This attribute is used to reference the
       \texttt{PHOTCAL} \elem{GROUP} from the elements using the
       photometric system. This attribute needs to be unique within
       the document so that it can be referenced by \elem{FIELD}s and
       is mandatory.
     \item[\attrval{utype}{PhotDM:PhotCal}] utype inherited from PhotDM,
       mandatory.
     \item[\attrval{ucd}{phot}] Recommended UCD for the GROUP.
       %\todo{Something
       %ought to be here, if nothing else the relation of this to the
       %fixed name.\ada{do we need it, we could drop it since there is
       %  the PARAM with the Obscore type/subtype}} \dots
     \item[\elem{DESCRIPTION}] This sub-element is used to describe the
       \texttt{PHOTCAL} \elem{GROUP}. This attribute is optional, but
       recommended.
\end{description}

The characteristics of this \elem{GROUP} are declared as \elem{PARAM}
child elements
containing the following information:
\begin{description}
\item[filterIdentifier:] This is the \elem{PARAM} element to
  uniquely identify the zero point assigned to a filter and to a
  specific photometric system. Its value is a string, and it MUST
  have the following attributes:
\begin{description}
    \item[\attrval{name}{filterIdentifier}]
    \item[\attrval{ucd}{meta.id;instr.filter}]
    \item[\attrval{utype}{photDM:PhotometryFilter.identifier}]
    \item[\attr{value}:] a string that SHOULD follow the syntax:
      Facility/Subcategory.Band (e.g.\ \attrval{value}{GAIA/GAIA2.G},
      \attrval{value}{Palomar/ZTF.r}).
\end{description}
\item[zeroPointFlux:] Photometric zero point associated to this
  instance. This \elem{PARAM}'s value is numeric, and it MUST
  have the following attributes:
\begin{description}
    \item[\attrval{name}{zeroPointFlux}]
    \item[\attrval{ucd}{phot.mag;arith.zp}]
    \item[\attrval{utype}{photDM:PhotCal.zeroPoint.flux.value}]
    \item[\attr{unit}:] flux units
    \item[\attr{value}:] zero point flux value
\end{description}
\item[magnitudeSystem:] Magnitude system associated to this
  instance (Vega, AB, \dots). This \elem{PARAM}'s value is a string,
  and it MUST have the
  following attributes:
\begin{description}
    \item[\attrval{name}{magnitudeSystem}]
    \item[\attrval{ucd}{meta.code}]
    \item[\attrval{utype}{photDM:PhotCal.magnitudeSystem.type}]
    \item[\attr{value}:] system name
\end{description}
\item[effectiveWavelength:] Effective wavelength of the filter
  in this instance. This \elem{PARAM}'s value is numeric,
  and it MUST have the following attributes:
\begin{description}
    \item[\attrval{name}{effectiveWavelength}]
    \item[\attrval{ucd}{em.wl.effective}]
    \item[\attrval{utype}{photDM:PhotometryFilter.spectralLocation.value}]
    \item[\attr{unit}:] wavelength units
    \item[\attr{value}:] effective wavelength value
\end{description}
\end{description}

Proposed annotation example for a \texttt{PHOTCAL} \elem{GROUP}:
\lstinputlisting[language=XML]{photcal_PARAM.xml}


While following the same approach as for \elem{COOSYS} and
\elem{TIMESYS} would result in a more compact annotation, we decided
to define this element as a \elem{GROUP} to allow for some flexibility
on the data provider side. We are aware that this implies clients to
work around more complex annotations. We hope to alleviate that by
following the exact \attr{name} and \attr{utype} convention as
proposed in this document.

The alternative to doing this, and which would simplify the
annotation, is to bring this \elem{GROUP} element closer to the
\elem{COOSYS} and \elem{TIMESYS} elements by fixing the units of the
\attr{zeroPointFlux} and the \attr{effectiveWavelength} to Jy and
Angstrom respectively and add it to future versions of VOTable. This
would look as follows:

\lstinputlisting[language=XML]{photcal_FILTERSYS.xml}


%\input{VELSYS}

For types of time series data not explicitly defined in this document
we recommend to use the appropriate UCD1+
\citep{2005ivoa.spec.0819D}. The data provider is responsible for
providing accurate metadata for the elements here described.


%\input{GROUP.tex}
\subsection{\elem{TABLE}}
%\todo{We could propose to define one \elem{TABLE} element to describe the content of each timeseries with common metadata (e.~g. one per filter) with no \elem{DATA}, and to add in a separate \elem{TABLE} element the \elem{DATA} part wich will referece the approprate \elem{TABLE} element. But to be honest I am not sure about the pros and contras of that at the moment of writing.} 

We propose to use the \elem{TABLE} element for describing the contents
of the time series data itself. 
We also propose to annotate the \elem{TABLE} as a timeseries using
\elem{PARAM}s defined as elements from Obscore
\citep{2017ivoa.spec.0509L}:

The \elem{TABLE} must include:

\begin{itemize}
\item A human-readable \attr{DESCRIPTION} that ideally conveys enough
  human-readable information that basic scientific use of the time
  series is possible without referring to literature.

\item A \elem{PARAM} describing the obscore dataproduct type with the
  following mandatory attributes:
     \begin{compactitem}
        \item \attrval{name}{dataproduct\_type}
        \item \attrval{ucd}{meta.code.class}
        \item \attrval{utype}{obscore:ObsDataset.dataProductType}
        \item \attrval{value}{timeseries}
     \end{compactitem}

\item A \elem{PARAM} describing the dataproduct subtype with the
  following mandatory attributes:

     \begin{compactitem}
        \item \attrval{name}{dataproduct\_subtype}
        \item \attrval{ucd}{meta.code.class}
        \item \attrval{utype}{obscore:ObsDataset.dataProductSubtype}
     \end{compactitem}

The parameter's \attr{value} attribute should be set to
\attrvalonly{lightcurve} for time series following this note, including the
photometric part.  Other values, for instance for radial velocity
curves, could be used in the future; a suitable controlled vocabulary
would be created at that point.

     \item One or several columns referencing a \elem{TIMESYS}
       element.  Clients can use any such column as the independent
       variable of the time series.

     \item One or more columns referencing a \texttt{PHOTCAL}
       \elem{GROUP}.  Clients can use any such column as a dependent
       variable of a time series.  Non-photometric time series will
       replace this mechanism with some other requirement.
\end{itemize}

\lstinputlisting[language=XML]{table_TIMESERIES.xml}

We explored other ways of annotating the data, for instance defining
different tables and the relation between them using foreignKey as
possible according to Section 4.10 of VOTable
\citep{2019ivoa.spec.1021O}, or even proposing a new XMLDATA
serialization. After exploring those possibilities we conclude that
the proposed solution is the most appropriate within the actual and
most accepted usage of VOTable for both existing data providers and
clients.

\section{Example Serializations}
\subsection{Case 1: lightcurves in one filter}
Let us consider the simplest time series case possible: a table
containing only three columns:
\begin{center}
   \emph{time, magnitude, magnitude error}
\end{center}

The minimum VOTable would look like this:

\lstinputlisting[language=XML]{vot-ex1-GROUP.xml}

This annotation would also work in the case of tables including
multiple light filters in the form
\begin{center}
   \emph{time\_1, magnitude\_1,magnitude\_error\_1, time\_2,
     magnitude\_2,magnitude\_error\_2}
\end{center}

\subsection{Case 2: lightcurves in multiple filters}
Let us consider now consider a more complicated case, in which the
time series includes a column with name of the photometric filter of
each row of the time series, i.~e. a table containing four columns:
\begin{center}
   \emph{time, magnitude, magnitude error, filter name}.
\end{center}
In those cases this complicates the annotation and the need of being
able to reference rows (as well as columns) becomes an issue. It is to
overcome such problem, that we propose to use different \elem{TABLE}
elements for different types of products. We propose therefore to
annotate a VOTable as shown in the following example:

\lstinputlisting[language=XML]{vot-ex2-GROUP.xml}

%To avoid referencing columns, which would make things a bit more complicated, we propose to create different \elem{TABLE} elements, one for each filter. This, although reduntant in the definition of some of the columns, will ease combining timeseries from the client side. 

%Alternative 2: Use the GROUP name=key thing and a different table to define the relation between elements. This 
%Alternative 3: Another possible solution would be to create a new XMLDATA serialization which would allow to tag rows, as proposed in Appendix A.8 of \citet{2019ivoa.spec.1021O}. But wouldn't that break the interoperability? 


%\todo{Do we have to say something about claiming a new capability in the registry? }
%\todo{Do we want to add a summary table with utype, UCD1+, Meaning, Default value, Data type, required?} 

%\input{timeseriesDM.tex}
\bibliography{ivoatex/ivoabib,ivoatex/docrepo}

\end{document}