Using MSCRED with Generic Mosaic Data

Introduction

This document illustrates how to use MSCRED with generic CCD mosaic data; that is, data that does not contain any header information used by the various tasks. Data generated by the NOAO Mosaic imager systems already contain the desired information, so this document is for other sites producing mosaic data and wishing to use the facilities of the MSCRED package and for observers with such data.

This document discusses the header information and is not a guide to using the MSCRED tasks. That is described in the Guide to the NOAO Mosaic Data Handling Software . A list of all the useful information about the NOAO Mosaic Project and the MSCRED design and software may be found at http://iraf.noao.edu/projects/mosaic .

The approach taken in this document is to show examples of starting with data with no header information and adding keywords to allow certain tasks to be used. The example mosaic data has 4 CCDs arranged in a 2x2 grid. Each CCD is readout with 2 amplifiers. This produces 8 subimages per observation. In the exmaple the amplifier readouts are shown as 100 x 200 with 5 pixels of overscan. Any of these numbers can be adjusted as needed. The examples will show only the keywords relevant to the MSCRED package. Actual data will have other keywords.

Multiextension FITS Format

The MSCRED tasks operate on multiextension FITS format files (MEF files). This format is described in greater detail elsewhere. In this section we begin with data which is in separate single images and show how to make the MEF format. The keywords that will be defined at this stage are the extension names and image identifications.

Figure 1 shows a directory containing the eight individual images from a mosaic readout. The format is shown as FITS but it could be any IRAF image format. The image header is completely empty except for the keywords required by the format.

Figure 1: Individual Images

ms> dir
generic1.fits   generic3.fits   generic5.fits   generic7.fits   
generic2.fits   generic4.fits   generic6.fits   generic8.fits   
ms> imhead generic* l-
generic1.fits[105,200][short]: 
generic2.fits[105,200][short]: 
generic3.fits[105,200][short]: 
generic4.fits[105,200][short]: 
generic5.fits[105,200][short]: 
generic6.fits[105,200][short]: 
generic7.fits[105,200][short]: 
generic8.fits[105,200][short]: 
ms> imhead generic1 l+
generic1[105,200][short]: 
No bad pixels, min=0., max=0. (old)
Line storage mode, physdim [105,200], length of user area 1215 s.u.
Created Wed 13:48:48 17-Mar-1999, Last modified Wed 13:48:48 17-Mar-1999
Pixel file "generic1.fits" [ok]
EXTEND  =                    F / File may contain extensions
ORIGIN  = 'NOAO-IRAF FITS Image Kernel Aug 1 1997' / FITS file originator
DATE    = '17/03/99  '         / Date FITS file was generated
IRAF-TLM= '13:48:48 (17/03/1999)' / Time of last modification

To create the a single MEF file you need to define the extension names for each piece. It is recommended that these be short. They must not begin with a numeral due to the way the IRAF FITS syntax works. They would usually have a numberic component and that would generally be directly related to the position of the extension in the file. However, any extension name may be used.

Because it is easy to modify the extension name the keyword IMAGEID is set to be permanently associated with each piece. This can be anything also but it is recommended that it be a simple numeral.

Figure 2: Creating a MEF File with EXTNAME and IMAGEID

ms> imcopy generic1 generic[im1]
ms> imcopy generic2 generic[im2]
ms> imcopy generic3 generic[im3]
ms> imcopy generic4 generic[im4]
ms> imcopy generic5 generic[im5]
ms> imcopy generic6 generic[im6]
ms> imcopy generic7 generic[im7]
ms> imcopy generic8 generic[im8]
ms> hedit generic[im1] imageid 1 add+ verify- show- update+
ms> hedit generic[im2] imageid 2 add+ verify- show- update+
ms> hedit generic[im3] imageid 3 add+ verify- show- update+
ms> hedit generic[im4] imageid 4 add+ verify- show- update+
ms> hedit generic[im5] imageid 5 add+ verify- show- update+
ms> hedit generic[im6] imageid 6 add+ verify- show- update+
ms> hedit generic[im7] imageid 7 add+ verify- show- update+
ms> hedit generic[im8] imageid 8 add+ verify- show- update+
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
No bad pixels, min=0., max=0. (old)
Line storage mode, physdim [105,200], length of user area 1215 s.u.
Created Wed 14:19:00 17-Mar-1999, Last modified Wed 14:08:57 17-Mar-1999
Pixel file "generic.fits" [ok]
PCOUNT  =                    0 / No 'random' parameters
GCOUNT  =                    1 / Only one group
EXTNAME = 'im3     '           / Extension name
ORIGIN  = 'NOAO-IRAF FITS Image Kernel Aug 1 1997' / FITS file originator
INHERIT =                    F / Inherits global header
DATE    = '17/03/99'           / Date FITS file was generated
IRAF-TLM= '14:08:57 (17/03/1999)' / Time of last modification
IMAGEID =                    3

MSCDISPLAY

One of the first things you want to do with a mosaic observation is to display it as a single picture. This is done with the task MSCDISPLAY. This requires defining what portion of each image extension contains the data as opposed to overscan or prescan. The location of the data may change depending on which amplifier is readout and whether portions of the readout are flipped by the data acquisition system.

Figure 3 shows the arrangement of each extension including the overscan (the thin grey band). The figure shows how each extension image is actually stored in the file. Sometimes the overscan appears at the left and other times at the right depending on how the data is readout and stored in the extension. The labels were written into the data so they appear flipped in x, y, and both.

Figure 3: Orientation of Raw Data

DATASEC identifies where the data is in the image extensions.

Figure 4: Adding DATASEC

ms> hedit generic[im1] datasec [1:100,1:200] add+ verify- show- update+
ms> hedit generic[im2] datasec [6:105,1:200] add+ verify- show- update+
ms> hedit generic[im3] datasec [6:105,1:200] add+ verify- show- update+
ms> hedit generic[im4] datasec [1:100,1:200] add+ verify- show- update+
ms> hedit generic[im5] datasec [1:100,1:200] add+ verify- show- update+
ms> hedit generic[im6] datasec [6:105,1:200] add+ verify- show- update+
ms> hedit generic[im7] datasec [6:105,1:200] add+ verify- show- update+
ms> hedit generic[im8] datasec [1:100,1:200] add+ verify- show- update+
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
ms> msccmd "hselect $input $I,datasec yes" input=generic 
generic[im1]    [1:100,1:200]
generic[im2]    [6:105,1:200]
generic[im3]    [6:105,1:200]
generic[im4]    [1:100,1:200]
generic[im5]    [1:100,1:200]
generic[im6]    [6:105,1:200]
generic[im7]    [6:105,1:200]
generic[im8]    [1:100,1:200]

The keyword DETSEC identifies where in the "detector" picture each piece should go. A pixel coordinate system is defined for the entire detector or picture. Don't worry about gaps as these can be added by MSCDISPLAY if desired. In the example the detector effectively covers an area of 400x400.

In addition DETSEC can be given with flips to allow for flips in the way the pixels are written to the image extension relative to the final picture. Some mosaic detector systems flip the data into the standard orientation during readout and others do not. In the example we show various flips. A final display of figure 3 should show all the labels correctly. This means all the top extensions have to be flipped in y and the second and third from the left have to be flipped in x.

hedit generic[im1] detsec [1:100,1:200] add+ verify- show- update+
hedit generic[im2] detsec [200:101,1:200] add+ verify- show- update+
hedit generic[im3] detsec [300:201,1:200] add+ verify- show- update+
hedit generic[im4] detsec [301:400,1:200] add+ verify- show- update+
hedit generic[im5] detsec [1:100,400:201] add+ verify- show- update+
hedit generic[im6] detsec [200:101,400:201] add+ verify- show- update+
hedit generic[im7] detsec [300:201,400:201] add+ verify- show- update+
hedit generic[im8] detsec [301:400,400:201] add+ verify- show- update+
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
ms> msccmd "hselect $input $I,detsec yes" input=generic
generic[im1]    [1:100,1:200]
generic[im2]    [200:101,1:200]
generic[im3]    [300:201,1:200]
generic[im4]    [301:400,1:200]
generic[im5]    [1:100,400:201]
generic[im6]    [200:101,400:201]
generic[im7]    [300:201,400:201]
generic[im8]    [301:400,400:201]

Figure 3: Display with DETSEC Defined

In this example we have two amplifiers per CCD. In this case there really should not be a gap between the pieces from the same CCD. MSCDISPLAY can handle this if it knows which extensions come from the same CCD. This is done with the keyword CCDNAME. Each extension should be labeled by a CCD identifier. This might be the actual detector serial number. To identify amplifiers from the same CCD the CCDNAME keyword would be the same in all extensions from the same CCD. This way of identifying amplifiers from the same CCD is also used during data reductions.

ms> hedit generic[im1] ccdname ccd1 add+ verify- show- update+
ms> hedit generic[im2] ccdname ccd1 add+ verify- show- update+
ms> hedit generic[im3] ccdname ccd2 add+ verify- show- update+
ms> hedit generic[im4] ccdname ccd2 add+ verify- show- update+
ms> hedit generic[im5] ccdname ccd3 add+ verify- show- update+
ms> hedit generic[im6] ccdname ccd3 add+ verify- show- update+
ms> hedit generic[im7] ccdname ccd4 add+ verify- show- update+
ms> hedit generic[im8] ccdname ccd4 add+ verify- show- update+
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '

The following figure show the result of MSCDISPLAY with a 20 pixel gap in x and y. Note that the gaps between pairs of extensions from the same CCD no longer appear. This example figure is not completely representative of real data in that there is no variation in gain or bias levels.

Figure 4: Final Display

If the CCD detectors are read using on-chip binning then one needs to specify the binning with the CCDSUM parameter. For 2x2 binning the keyword value would be "2 2". The first number is the binning in the horizontal shift register coordinate and the second in the parallel shift direction. Normally data is written to the image with the horizontal shift dimension along the first image axis.

It is important to realize that the detector coordinate system described by the DETSEC keyword is based on unbinned pixels. Therefore, this keyword does not change when binning. Instead CCDSUM is set and the DATASEC regions are changed appropriately. In the example if the data is readout with 2x2 binning then you would have the following header.

ms> imhead binned[im3] l+
binned[im3][55,100][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:55,1:100]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '
CCDSUM  = '2 2     '

Sometimes one might want to trim some bad data from the edges of the extension images. While modifying DATASEC and DETSEC consistently would do this it is better to use the keyword TRIMSEC and not modify the other two. TRIMSEC is like DATASEC except it can define a different region of the image to use for the good data. Normally the trim section would be the same or smaller and interior to the data section. For example, to specify exclusion of the one pixel along each edge of extension im3 in the example data the trim section would be [7:104,2:199]. Just for documentation and for later use during reductions is a good idea to define this keyword if only to make it the same as the data section. The MSCCMD command below sets the trim section to be the same as the data section.

ms> msccmd
msccmd: hedit $input trimsec "(datasec)" add+ verify- show- update+
Input files: generic
msccmd: q
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '
CCDSEC  = '[100:1,1:200]'
TRIMSEC = '[6:105,1:200]'

If TRIMSEC is present MSCDISPLAY will display only that section but keep the DETSEC region the same; i.e. the image will not expand to fill the specified detector region. The commands and display below give a gross example.

ms> hedit generic[im1] trimsec [11:100,15:170] add+ verify- show- update+
ms> hedit generic[im2] trimsec [20:105,15:170] add+ verify- show- update+
ms> imehad generic[im1] l+
generic[im1][105,200][short]: 
EXTNAME = 'im1     '           / Extension name
IMAGEID =                    1
DATASEC = '[1:100,1:200]'
DETSEC  = '[1:100,1:200]'
CCDNAME = 'ccd1    '
CCDSEC  = '[1:100,1:200]'
TRIMSEC = '[11:100,15:170]'
ms> mscdisplay generic 1 xgap=20 ygap=20

MSCDISPLAY also has the ability to do quick look calibration consisting of bias subtraction from the prescan or overscan region and flat fielding with a specially constructed flat field. The bias subtraction requires specification of the BIASSEC keyword as described in the next section. A FILTER keyword might also be used to select a flat field calibration file.

CCDPROC

The basic operations of CCD reductions are performed by the CCDPROC task. CCDPROC is also called by the various calibration combining tasks such as FLATCOMBINE. See the MSCRED User's Guide for a discussion of this data reduction process.

CCDPROC can be used with data having no further header information. However, the task is easier to use and provides some consistency checks if there is further information in the header.

The CCD processing recognizes a number of standard types of calibration exposures. The main ones are zero level (also called bias), dark count, and flat field. In addition there is the actual science or object exposure. If the OBSTYPE keyword is defined with values identifying the type of exposure then CCDPROC and COMBINE and related tasks can search for the appropriate types and check against misprocessing the data.

The values for the OBSTYPE keyword as understood by default are "zero", "dark", flat", and "object". It is possible to use other values if you define a header translation file described elsewhere. For this example we use the default values.

ms> msccmd
msccmd: hedit $input obstype object add+ verify- show- update+
Input files: generic
msccmd: q
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '
CCDSEC  = '[100:1,1:200]'
TRIMSEC = '[6:105,1:200]'
OBSTYPE = 'object  '

Some calibration operations are filter dependent. For instance, flat fielding should use a flat field calibration of the same filter as the object exposure. If you are careful in calling the tasks you can do this yourself. But it is easier and safer if you can let the tasks check for matching filters. To do this requires a FILTER keyword (again the actual keyword can be translated if desired). The value of the keyword can be anything as long as it is used consistently. It is also useful to use a short common value for the first word of the keyword value.

ms> msccmd
msccmd: hedit $input filter V add+ verify- show- update+
Input files: generic
msccmd: q
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '
CCDSEC  = '[100:1,1:200]'
TRIMSEC = '[6:105,1:200]'
OBSTYPE = 'object  '
FILTER  = 'V       '

The task CCDLIST is useful for check that the MSCRED tasks will correctly identify the exposure type and filter.

ms> ccdlist generic
generic[im1][105,200][short][object][1][V]:
generic[im2][105,200][short][object][2][V]:
generic[im3][105,200][short][object][3][V]:
generic[im4][105,200][short][object][4][V]:
generic[im5][105,200][short][object][5][V]:
generic[im6][105,200][short][object][6][V]:
generic[im7][105,200][short][object][7][V]:
generic[im8][105,200][short][object][8][V]:

The CCDSEC parameter is useful for documenting the relationship between the image pixels and the CCD pixels. It is also important in handling binned and subregion readouts. The CCDSEC parameter matches the data pixels to the CCD pixels by specifying a section of the CCD corresponding to the data pixels. CCDPROC will check that the CCD pixels are correctly matched up for calibration operations such as flat fielding. It will also maintain a coordinate system that maps the image pixels to the CCD pixels.

CCDSEC is always given in unbinned pixels. The choice of the CCD pixel coordinate system is arbitrary though it should normally be defined in the same orientation as the detector system used by DETSEC. In most cases CCDSEC and DETSEC will be related by simple offsets. In the example of a 200 x 200 CCD with two amplifiers the sections would be

ms> hedit generic[im1] ccdsec [1:100,1:200] add+ verify- show- update+
ms> hedit generic[im2] ccdsec [200:101,1:200] add+ verify- show- update+
ms> hedit generic[im3] ccdsec [100:1,1:200] add+ verify- show- update+
ms> hedit generic[im4] ccdsec [101:200,1:200] add+ verify- show- update+
ms> hedit generic[im5] ccdsec [1:100,200:1] add+ verify- show- update+
ms> hedit generic[im6] ccdsec [200:101,200:1] add+ verify- show- update+
ms> hedit generic[im7] ccdsec [100:1,200:1] add+ verify- show- update+
ms> hedit generic[im8] ccdsec [101:200,200:1] add+ verify- show- update+
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '
OBSTYPE = 'object  '
FILTER  = 'V       '
CCDSEC  = '[100:1,1:200]'
ms> msccmd "hselect $input $I,ccdsec,detsec yes" input=generic
generic[im1]    [1:100,1:200]   [1:100,1:200]
generic[im2]    [200:101,1:200] [200:101,1:200]
generic[im3]    [100:1,1:200]   [300:201,1:200]
generic[im4]    [101:200,1:200] [301:400,1:200]
generic[im5]    [1:100,200:1]   [1:100,400:201]
generic[im6]    [200:101,200:1] [200:101,400:201]
generic[im7]    [100:1,200:1]   [300:201,400:201]
generic[im8]    [101:200,200:1] [301:400,400:201]

Two of the processing step in CCDPROC are removing a bias using an overscan or prescan bias region and then trimming the image to remove the bias region and any edge regions of the readout that are not good. This can be done by specifying the bias region and trim section explicitly. However it is useful to have this information in the image headers. The two keywords for this information is BIASSEC and TRIMSEC. We have already described TRIMSEC in connection with trimming during display. BIASSEC was also mentioned as used for quick look display calibration.

The bias section is given as a section relative to the image. It can be the whole bias region or a smaller section. The example below shows specifying the whole bias section.

ms> hedit generic[im1] biassec [101:105,1:200] add+ verify- show- update+
ms> hedit generic[im2] biassec [1:5,1:200] add+ verify- show- update+
ms> hedit generic[im3] biassec [1:5,1:200] add+ verify- show- update+
ms> hedit generic[im4] biassec [101:105,1:200] add+ verify- show- update+
ms> hedit generic[im5] biassec [101:105,1:200] add+ verify- show- update+
ms> hedit generic[im6] biassec [1:5,1:200] add+ verify- show- update+
ms> hedit generic[im7] biassec [1:5,1:200] add+ verify- show- update+
ms> hedit generic[im8] biassec [101:105,1:200] add+ verify- show- update+
ms> imhead generic[im3] l+
generic[im3][105,200][short]: 
EXTNAME = 'im3     '           / Extension name
IMAGEID =                    3
DATASEC = '[6:105,1:200]'
DETSEC  = '[300:201,1:200]'
CCDNAME = 'ccd2    '
OBSTYPE = 'object  '
FILTER  = 'V       '
CCDSEC  = '[100:1,1:200]'
TRIMSEC = '[6:105,1:200]'
BIASSEC = '[1:5,1:200]'