Mechanical

Mechanism fails to find datum: After a reboot of the cryostat control system (CCS), all the mechanisms will first have to find their reference point, or datum, before they can move to a specified position. Although rare, a mechanism occasionally fails to find datum and will usually time out, an error status message appearing in the status DM screen (see Running Michelle). The first thing to try is to move the mechanism from the CCS DM screen while watching the data switch fields in the relevant DM screen. If the datum switches move (they switch between closed and open) and the mechanism still fail to find the datum, there could be a software, firmware, or communication problem. You could try rebooting the CCS via the CCS serial console. If this fails to cure the problem, you will need to call for help. If the datum switches fail to operate, the microswitch could be faulty, and the backup switch will need to be used. This requires a change to the CCS software. Call for assistance.

The illuminated area on the array has shifted (spectroscopy path): This is rare but has occurred in the past when doing echelle spectroscopy. If the instrument has been moved or worked upon during the day and received a minor shock, the grating drum can shift slightly, causing the illuminated area on the area to move. If this occurs, go into the grating DM screen via the CCS DM window, select “force,” and hit the start button. This will force the mechanism to find data and fix the problem. Do not forget to switch back to “discretionary” after the move; otherwise, the grating will re-find datum every time a move is requested. See 20020813.012.

Large “blobs” seen in raw chopped frames: If the Michelle sacrificial or main window is exposed to water, it will become damaged. Over time this damage is probably unavoidable on the sacrificial window, although the main cryostat window should remain protected or dry. If either window becomes water-damaged, blobs of high emissivity (or high counts) become obvious in the imaging mode’s raw chopped frames. If the damage is slight, these emissive regions subtract out when chopping and are not a big issue. However, if the damage is extensive, the regions cause the array to saturate, and the images become badly affected. The only recourse is to change the window, which requires daytime engineering effort. See www.ukirt.hawaii.edu/~tkerr/window_damage.html for an example of a badly damaged window.

Unstable array temperature: Small variations in the array temperature are normal when switching between imaging filters — the thermal load on the array changes as the flux incident on the array changes. However, if you see changes in temperature of 1-2K or greater over a time period of an hour or two, this could indicate a blocked JT valve. Call for assistance. If the JT valve is becoming blocked, a temporary fix to open the JT valve to 5 turns and then quickly return the valve to a setting of 0.7 turns. This will move the obstruction away from the valve, and the JT temperature should return to normal after twenty minutes or so. However, this does not clear the contaminants from the JT system, and the blockage will return in a few hours (less if the contamination is severe). To clear the blockage, a full JT purge is required, which requires daytime engineering effort. See 20020211.003.

Electronic (Including Edict)

Array counts are 0 or > 65000: This probably means the array has not been enabled. Try enabling the array from the mich_oper DM screen (see Running Michelle). If enabling the array does not work (i.e., the indicator light on the DM screen remains yellow or the temperatures fail to rise after a few seconds), then the array safety software may be preventing the array from being enabled. This occurs when the array temperature is too warm for the array to be used safely (the current cut-off temperature is 12K). It can also occur if the safety software has failed to load. If the temperatures look normal, you can attempt to load the arraySafe software from the WFG serial console. Open up the console and type isArraySafe. The response to this command might give some clues as to what the problem is. If things look normal, try starting the software with the command arraySafeStart, waiting a few seconds (try 30 sec), and then enabling the array again. If the array still fails to enable, you may have to reload a valid waveform. To do this, you need to open or reopen the WFG serial console and type the following: wfgClear <return> followed by setIdle <return>. The setIdle command will prompt you for four responses, which are: 1, 0.03, 10, 0. After this, type wfgLoad “mch_str_bw” <return>. Now try enabling the array again. If this fails, call for help. 

The array is saturated: Saturation is rarely an issue in spectroscopy mode but can be imaging, especially through the broadband filters. The data reduction automatically detects possible contamination in the individual chop frames and should warn you if saturation is approached. If this occurs, check the raw chop frames using gaia to see the counts, which should be lower than 50,000. Typical Si filter counts are ~35,000, but the broadband filters can result in counts of around 48,000 to 50,000, depending on atmospheric conditions. If the CSO tau is high, likely, the conditions are not good enough for imaging, and you should switch to another project. If you are desperate, you can change the OT program’s exposure times, but this is not recommended. A list of working exposure times is available.
    If conditions are good, also check that you haven’t left the Michelle window shutter closed!

Noisy array: Noise on the Michelle array is generally caused by an electronic pickup. This could, in turn, be caused by poor grounding, loose cables or boards, dirty connections, etc. If the array looks noisy, you can run a test to measure the read noise. From the sequence console, go into the File menu and then choose load. You should then select the sequence called Mich_read_noise.exec. Run this sequence. The DR will report an average standard deviation for each of the 10 pairs of data taken. Note that the array is capable of a read noise of 3500 electrons in deep well STARE mode. Note the reported standard deviation and call for help if the noise is significantly higher than around 4000 electrons. 

One quadrant “missing” on the array: A channel or “quadrant” on the Michelle array consists of four contiguous “subchannels” or outputs, each of these consisting of 20-pixel columns. Occasionally after the Edict crate has been powered up, one of the four quadrants is missing from the data files. Although a soft reset of the Edict may fix this, the Edict crate usually requires power-cycling using the green rocker switch at the bottom of the crate. If this does not fix the problem, you will need to call for assistance as one of the Edict slaves may have a problem. Look to see if any of the Edict slaves are showing red error lights in the crate.

Noisy or intermittent subchannel: Loose cabling or faulty connections can cause one subchannel or output to behave erratically, showing few counts when the array is enabled, yet appearing quite active even when the array is disabled. The problem can come and go with telescope slews as well. See 20040208.003. Please call for assistance first, but the place to start checking is the front-end Edict electronics underneath the main cryostat. Please ensure that you wear a grounding strap when checking cables in this area, otherwise, there is potential for array damage to occur.

Edict reports a damaged waveform: This problem will result in a MOCS error, which is reported via the sequence console in normal observing procedures. If you see a MOCS error, check both the WFG and CCS serial consoles for error messages. If the WFG serial console reports a series of error messages, and amongst them is “damaged waveform,” then a waveform has failed to load correctly. Although this problem should not occur anymore since the waveforms’ location has changed, you should try and do a soft reset of Edict if you do see this problem. If this doesn’t fix the problem, power cycle the Edict crate. Call for help if neither of these attempts cures the problem.
    You will also very likely have to run the OCS software down before you can retake data, or at least do a kill_inst Michelle from the sequence console and then restart the connection to the instrument. 20020901.005.

Array doesn’t enable or temperature fails to rise on enable command: See Array counts are 0 or > 65000.

Edict slaves report “read 0” error: The symptoms of this may include a sequence console MOCS error, a simple failure to take a data frame, or movie failing to produce a quick look image. If the array appears to be enabled and the reason for a failure is not apparent, open all the Edict serial consoles. If one or all the slaves are reporting something along the lines of “0x80f662a0 (tAccum): daqAccumThread: Failed waiting for read 0” then Edict is in a bad state and needs rebooting. You should probably go ahead and power cycle the crate to save time, as a soft reset rarely fixes this problem. See 20020901.005.

Array doesn’t enable, or temperature fails to rise on enable command: See Array counts under “Electronic” (above)

Unable to run the WFG Serial console or open the slaves: Only one occurrence of the Edict or CCs serial consoles can run at any given time. If someone else is running them, you will not be able to successfully telnet to the console. Unless you happen to be running the consoles yourself elsewhere, you should log into Ohi as observer and type ps -ef and look for an observer process similar to “telnet ukirt9 3008”. The last four digits may be different, but it’s ukirt9 that you need to connect to. You can kill these processes with the “kill -9 processid” command and then try to reopen the consoles. Alternatively, after killing the process, you can log directly into the console with “telnet ukirt09 ****,” where **** is the 4 digit number of a process you just killed. Remember to disconnect before trying to rerun the consoles. 

Sequence Console

The dreaded MOCS error message: See 20040204.009 for an extensive discussion. Although various problems may cause the sequence console to produce a MOCS error, it is usually caused by either a problem with Edict or the CCS and is probably a communications issue. Firstly, it is rare that you can acknowledge the error message the sequence console produces and continue. However, on the odd occasion, this does work and probably depends on the specific underlying problem. Usually, it would be best to run down the OCS software and then either reset Edict or the CCS. If you look at the Edict serial consoles and see an error message (usually a damaged waveform reported in the WFG serial console or read 0 errors in the slaves), then reset Edict from the WFG serial console, restart the OCS, and retry the current sequence. If Edict reports no error, then the chances are the CCS needs rebooting from the CCS serial console. You may look for an error message in this console, but more often than not, one isn’t obvious. After the CCS is rebooted, run the OCS and retry the sequence.
    Fortunately, for a reason, no one yet understands, these problems seem to have stopped occurring. Therefore, if you do see one, it’s important to report the problem with as much relevant information as possible, including what’s reported in the Michelle serial consoles, in case the problem indicates that another batch of MOCS errors are about to occur.

Movie

Movie fails to take data: Probably due to a race condition, when doing target acquisition with Michelle, or peakup up through a slit using movie, the first movie attempt fails. The sequence console will display “Failure taking movie image” in the status field, and the quick look display will no longer update. Click on the stop button in the movie window and then hit start. Unless Edict has been given an inappropriate configuration (unlikely, unless the observer has specified their own exposure times), movie should work the second time around. There is no need to close the movie window and open it again. Hitting stop and start should fix the problem. Remember to stop movie before continuing with the sequence!

The movie quick look display is garbage: This has only happened once in my experience but is something to watch for if the observer account is being used on both Ohi and Kauwa. I believe that problem is fixed, but… 
    If gaiadisp is used to display an image on Kauwa, it can display the image on the quick look display on Ohi instead and get you into quite a mess. The quick look display on Ohi will display garbage (pixels with counts of 1e-38, for instance) because of what’s known as a byte-to-byte error. If you see this problem, the observer must kill whatever process is being run on Kauwa to display the image, and the OCS software will almost certainly have to be restarted.
    If you are stuck with running the observer account on both Kauwa and Ohi, an image needs to be displayed on a gaia display on Kauwa, start a gaia session and open the relevant file from within the gaia menu.

The movie quick look display shows data but is very noisy: This can occur for numerous reasons, but the most likely is that too few coadds are being used for successful chop subtraction. If only one coadd is used, the resulting image can be extremely messy since the array often needs a few null reads or exposures at the beginning of an observation. Check to see how many coadds are being used in the sequence console and alter the OT program if possible.
    If the array is saturated (see “The array is saturated”), this can also result in an extremely noisy looking difference frame. Call for assistance if the reason for noisy movie frames is unclear.

CCS

The mechanisms are taking a long time to move: After the CCS is rebooted, all the mechanism must find their datum position the first time a request is sent for them to move to a position. Although all the mechanisms move simultaneously (unlike CGS4, for instance), two or three mechanisms take a minute or two to find their reference position. This is necessary as these mechanisms either need to be positioned extremely accurately to find datum in a rather tedious but necessary manner or require other mechanisms to move before they can find datum successfully.
    If you feel the mechanisms are slow to move in normal operations, then check the status DM screen to see which mechanism is moving (“configuring” and “busy” will be displayed on the screen against the relevant mechanism). If you are switching gratings, then it can take a minute or two for the mechanism to find its new position – this is normal. If a mechanism takes a long time and eventually turns out, this indicates a more serious problem. See “Mechanism fails to find datum.”

Checking the current status: See “Running Michelle” to find out how to open the Status DM window. In usual operations, the mich_oper DM window reports where the mechanisms are, e.g., whether you are in spectroscopy or imaging mode, which grating is being used, the filter name, etc. When mechanisms are moving, this DM window will not display the next setting until the relevant mechanism is in place but will report that the CCS state is “BUSY” if mechanisms are moving or “IDLE” if they are in position. Likewise, the mich_oper window also reports if Michelle is taking data, with “Busy” appearing in the Edict state field when taking a frame and “Idle” when no exposure occurs.
    When mechanisms are moving, you can launch the status DM window via the STATUS button on the mich_oper screen. This screen will indicate which mechanism is moving (it will say “Configuring” and “busy”) and which are idle (“Running” and “Idle). Note that when “Running” is displayed, this actually means the mechanism is healthy rather than moving. If any mechanism reports an error state, it will be reported on this screen. If you see a red line next to a mechanism, this means that the mechanism has yet to find a datum.

Telescope-related issues

Top-end doesn’t chop: Firstly, check the topend.dl window. If a Michelle sequence has been executed from the sequence console (excepting echelle sequences), then the chop should automatically be turned on and the chop set to external so Michelle can drive the secondary. Bear in mind that these will not be set until the ChopOn command has been reached in the sequence (the Chopon command can only be seen when the engineering display is turned on). If the sequence is running but one of these fields is incorrect, check the sequence itself to see if chopping has been set up (look for commands such as SET_CHOPTHROW, SET_CHOPPA). If the sequence doesn’t contain these commands, it’s likely that chopping hasn’t been selected in the OT program target component (see “Observing Preparation“).
    If all looks well with the sequence, check the CCS cabinet’s front towards the bottom right. There should be a cable leading from the JK2 output on the Xycom card to the UKIRT chop connection a few inches to the left and below. If the cable is there, check the connections. If the cable is missing, call for help. 

The chop throw looks wrong on the guider draw window: If a new sequence is run from the sequence console, which uses a different chop throw, the guider draw display does not automatically update. You will need to click on the fast guider button for the window to update with the new chop throw. (NB. It is important to have the correctly sized draw window – guiding will fail if the size is incorrect). If this does not fix the problem, check the topend.dl DM window and make sure the chop and throw settings match those of the actual sequence. If they don’t, then for some reason, the settings have not been passed onto the TCS. You will have to stop the sequence and restart. If this does not fix the problem, run down the OCS and start again. If the problem persists, call for help.

The telescope doesn’t slew when the slew command is reached in the sequence console: Remember that the sequence pauses on the first slew command in a sequence. If you expected the telescope to slew, check to see if the observer has hit the continue button in the sequence console. If the telescope still fails to slew, then the sequence may have been “sent to engineering” rather than to the queue, and in this case, telescope slew commands are disabled. Stop the sequence, requeue it, and send for execution from the queue monitor and restart it from the top. 

“I put the star in the guide-box, but it moved away a few seconds later”: This always catches people out when Michelle hasn’t been used for a while. When executing a Michelle sequence, the chop throw, chop angle, and the chop beam need to be set up by the TCS after the sequence console sends the instructions. The important thing to remember is not to turn guiding until the telescope is in CHOPBEAM A and the sequence has gone through this point. If you turn guiding on before this command is reached, the star will move out of the guide box when the telescope offsets to chopbeam A. Also, remember that it takes a little while for the guider camera to rotate if the chop angle changes, so you need to wait for this to finish as well.

Inability to focus – only 2 images of the star appear: Currently, we are only able to focus on a CMC or guidestar when the guider camera angle is 0. It may well be possible to focus when the guider is at an angle of 45 degrees or more, but so far, we have not been able to focus if the guider is at 90 or -90 degrees. Although the reason for this is not certain, it looks very much as if the CCD is vignetted slightly at these angles, so all four images of the star do not appear. To focus, you will have to manually set the camera angle to 0 in the topend.dl screen.

Engineering mode – why isn’t the telescope chopping?: This will probably only occur when Tom Kerr is observing and is being a pain by running Michelle via the engineering interface. Michelle only sends out a chop signal; it cannot send the chop throw or chop angle (during normal queue observing, it is the OCS software that sends the throw and angle to the TCS). In engineering mode, the chop throw and position angle will have to be set manually in the topend.dl screen. You will also probably have to set up the same parameters using the  TCS NODSET command. I have no idea why the same numbers have to be set in two different places. You will also have to manually turn on chopping in the topend .dl screen and set the mode to external.