Spectroscopy: Data Format and Location

Spectroscopy: Data Format and Location

Wavelength scale

The astute observer may notice that the dispersion axis on the raw frames is flipped, i.e. wavelength INCREASES to the LEFT. To deal with this, the pipeline DR updates the CDELT1 header early in the reduction process. The DISPERN header (the dispersion in um/pixel) remains positive, but the CDELT1 header (the wavelength co-ordinate increment, also in um/pixel) becomes negative, although it retains the same absolute value as DISPERN.

Note that, at present, orac-dr does not accurately wavelength calibrate UIST long-slit spectroscopy data (IFU data are properly calibrated). Only an estimated wavelength scale is attached to the data.

Slit orientation on the sky

A slit position angle anywhere between 90 degrees and -90 degrees may be selected in the OT. All angles are measured east-of-north. An angle of -90 degrees is recommended for point sources; this will put EAST to the TOP of the array. In the table below we list the orientation of the slit on the sky for specific position angles.

Posn. Angle
(E of N)
Top of slit is
to the…
-90 degreesEast
-45 degreesSouth-East
0 degreesSouth
+45 degreesSouth-West
+90 degreesWest

Raw files

Raw, unreduced, data files are in /ukirtdata/raw/uist/YYYYMMDD (where YYYYMMDD is the numeric UT date) or $ORAC_DATA_IN. The files are stored as starlink HDS containers (a file with multiple data arrays). Each file is equivalent to 1 observation, and as such contains a header component and 1 or more (actually NINT) integrations. Each integration is stored as an NDF component (single data array) of the HDS file. The raw filenames are uYYYYMMDD_NNNNN.sdf where NNNNN is the observation number, padded with leading zeros when necessary. Note: you may have a hard time working with this data format unless you have access to starlink and ORAC-DR at your home institute (see below).

Reduced single frame files

Reduced data files are in /ukirtdata/reduced/uist/YYYYMMDD (where YYYYMMDD is the numeric UT date) or $ORAC_DATA_OUT. The filename structure is: (PREFIX)(UTDATE)_(FRAME NUMBER)_[EXTENSION].sdf, where (THIS) is always there and [THIS] is optional. (PREFIX) is the letter u if the file contains data from a single observation. It is gu if the file contains data from a number of observations – i.e. a group (see below). _(EXTENSION) is used by individual primitives for their output files; think of a primitive as a single step within a recipe. The pipeline keeps track of passing these files between primitives; useful ones are left on the disk at the end so you can look at intermediate data products if you wish.

For example, u20000410_00123_ff.sdf would be data from a single observation, number 123, that has had all the reduction steps up to and including flat fielding applied to it (see, e.g., the table below). Reduced files can have either HDS (multiple data arrays) or NDF (single data array) format as appropriate. Extensions for single frames are listed in the table below.

_mrawA Modifiable copy of the raw data
_bpBad Pixel Mask has been applied
_rnvRead Noise Variance added
_sbfSubtracted Bias Frame (only for STARE frames)
_povIncludes Poisson Variance
_bglHow Background Limited the integration is
_ffFlat Field applied
_nfData is a Normalised Flat Field
_reoWavelength flipped so increases to right
_wceWavelength Calibrated by Estimation, equivalent of the old CGS4 ro* file
_ssSky Subtracted

Reduced group files

The pipeline adds the individual frames into a “group” file as they are processed. The group number is usually the frame number of the first frame in the group. Extensions for group frames are listed in the table below. Note that some of these data are 2-D spectral images, and some are extracted 1-D spectra; the former will be displayed in gaia and the latter in a Kapview plotting window.

File Extensions for group files

No extensionThis is simply the difference between all the main and offset beam images. These are the equivalents of the old CGS4 rg* files.
_srsvvResidual sky subtracted from the above file
_yprThe spectral image collapsed along the x (dispersion) axis
_oepThe opt-extract profiles
_oerThe opt-extract profiling residuals
_oesThe opt-extracted spectra
_ccsCross-Correlated and Shifted, spectrally aligned to beam 1
_ccfThe Cross-Correlation Functions from forming the _ccs frames
_spExtracted Spectrum – the coaddition of all the beams
_snrS/N plot of the extracted spectrum
_nsp (for the ratioing star only)_sp normalized to 1-sec exposure
_std (for the ratioing star only)_nsp divided by a blackbody curve with a temperature same as that of the ratioing star
_awsAligned with Standard – spectrally aligned with the standard star
_scfStandard cross-correlation function from forming the _aws frames
_dbsDivided by a Standard star (including the standard star black body model)
_fcFlux Calibrated

So which data products should I work with?

Once you have transferred the raw data to your home institute, you can run ORAC-DR locally to produce reduced and semi-reduced data products, and to get a first-look at the data.

For post-reduction with Starlink software or IRAF you probably only need the individual _wce spectral images; from these you can do your own “object-sky” differencing and subsequent coaddition of the data to produce a “group” spectral image (that contains all of the data on any one target). Alternatively, if the weather was stable and all data are usable, you may be able to simply work with the reduced group image produced by ORAC-DR – this is the group file in the above table with no extension. Either way, the arc spectra (also stored as _wce files) may then be used to accurately calibrate the dispersion axis of the 2-D image or of extracted spectra. An extracted spectrum from the “group” spectral image of your standard star can then be used to flux-calibrate and correct for telluric absorption lines.

Arguably, the “most useful” data products are highlighted in red, though this is really up to you…

Finally, note that each time you run ORAC-DR a “hidden” log file is written called .oracdr-number.log. If you type ls -a you’ll see these. Have a look at one of them; it’ll list everything the DR did during a particular session. This could be useful for following DR steps or identifying some of the more obscure files.

NOTE: At present, long-slit spectroscopy data only have an ESTIMATED wavelength scale (IFU data are the exception; these have a proper wavelength calibration).

Converting to fits

Non-Starlink users may convert their data to fits format with the Starlink routines:

     > convert
     > ndf2fits "*" "*"

or, to get .fits as the file extension (instead of .fit)

     > ndf2fits "*" "*|.fit|.fits|"

IRAF users having problems reading pipeline data into iraf should check the notes on multi-component data arrays at the bottom of the previous page.