General policy

Each night, WFCAM observers usually acquire very large amounts of data (hundreds of Gbytes). We therefore recommend that ALL users obtain their WFCAM data, fully reduced, from the WFCAM Science Archive (see links on previous page). If you are happy with this arrangement, you do not need to read the rest of this page.

However, we recognise that in some exceptional cases there may be good reason why observers need immediate access to raw data. Unfortunately, due to staffing shortages and other priorities, UKIRT scientists are NOT available to copy raw data to tape and/or disk. Instead, this job is the responsibility of the observer.

All observers wishing to take raw data away with them after their run must do one of the following:

• Observers bring their own USB-2 interfaced hard disks (or lap-top with sufficient disk space) and connect this to a summit machine at UKIRT. Please read very carefully the notes on our disk connection policy.
• Observers bring their own hard disks or lap-top and copy data when they are back down in Hilo AFTER their run. Suitable workstations are available. Obviously observers MUST allow sufficient time after their run to copy their data; for a typical WFCAM run, we recommend at least TWO DAYS to complete this process.
• Observers FTP their data to their home institute while at the summit, or from Hilo after their run. Note, however, that ALL FTP jobs from the summit MUST be completed or halted before the end of the normal observing night, and certainly before the 14 hr summit rule is broken. Observers should not expect UKIRT staff to stay late at the telescope, just so they can FTP their data.

Finally, if you do want access to raw data, it is extremely important that you discuss your plans with the Support Scientist assigned to your project BEFORE you come to Hawaii. In particular, observers should check the suitability of the laptop and/or USB disk they intend to use for data transfer.

Data format and conversion

At the summit, raw WFCAM data are stored as Starlink-format NDF files (.sdf extension). WFCAM writes four files at a time – one for each array. Each NDF file is 17 Mb in size, and on a good night 2,000-3,000 observations (8,000-12,000 files) may be acquired!

These NDF files are converted to smaller, rice-compressed FITS files (4 Gb each) at the end of the night, and are ftp-ed directly to CASU in the U.K. These smaller FITS files are therefore only available on disk AT THE SUMMIT for a period of about 30 hours after they are created.

Raw NDF files are copied to Hilo on a daily basis. These data remain on disk, in Hilo, for a period of 2-3 weeks.

Note that the raw data format – Starlink NDF (.sdf files)- will need conversion to FITS at the PI’s home institution, unless Starlink is to be used for data processing. If you do not have a copy of the Starlink software package an ISO image is available upon request. This software is supported for UKIRT users by the JAC.