.. _msa_metadata: MSA Metadata File: ``msa`` ^^^^^^^^^^^^^^^^^^^^^^^^^^ While it doesn't contain any actual science data, the NIRSpec MSA metadata file is nonetheless a crucial component of calibration processing for NIRSpec MOS exposures. It contains all the slitlet, shutter, and source configuration information that's needed by the :ref:`calwebb_spec2 ` pipeline to process a MOS exposure. These metadata are originally generated by the `MSA Planning Tool `_ (MPT) in APT and passed through the ground system to populate the FITS metafile. Details of the metadata and how it is used in calibration processing are given below. File names for MSA metadata files are of the form: jw__msa.fits where - ppppp: program ID number - ooo: observation number - vvv: visit number - cfg: MSA configuration such as `jw01180025001_01_msa.fits`. Each MOS science exposure contains a primary header keyword `MSAMETFL` that gives the name of the MSA metadata file to be used when processing the exposure. The values of the primary header keywords `MSAMETID` and `PATT_NUM` are also used when retrieving relevant rows of data from the tables in the MSA metadata file. If any of these keywords is missing from the input science exposure, processing will not be able to proceed and an error will be raised. File Structure -------------- The overall structure of the MSA FITS file is as follows: +-----+---------------+----------+-----------+--------------------+ | HDU | EXTNAME | HDU Type | Data Type | Dimensions | +=====+===============+==========+===========+====================+ | 0 | N/A | primary | N/A | N/A | +-----+---------------+----------+-----------+--------------------+ | 1 | SHUTTER_IMAGE | IMAGE | float32 | 342 x 730 | +-----+---------------+----------+-----------+--------------------+ | 2 | SHUTTER_INFO | BINTABLE | N/A | variable x 13 cols | +-----+---------------+----------+-----------+--------------------+ | 3 | SOURCE_INFO | BINTABLE | N/A | variable x 8 cols | +-----+---------------+----------+-----------+--------------------+ - PRI: The primary data array is empty. The primary header contains a small amount of metadata related to the observation and MSA configuration. - SHUTTER_IMAGE: 2-D data array containing a map of MSA shutters that've been commanded open for this MSA configuration. - SHUTTER_INFO: 2-D FITS binary table containing metadata for each shutter involved in this MSA configuration. One table row per shutter. - SOURCE_INFO: 2-D FITS binary table containing metadata from the source catalog used to create the MSA configuration. One table row per source. The SHUTTER_INFO Metadata ------------------------- The structure of the ``SHUTTER_INFO`` table extension is as follows: +-------------------------------+-----------+----------------------+ | Column Name | Data Type | Contents | +===============================+===========+======================+ | SLITLET_ID | int16 | Slitlet ID number | +-------------------------------+-----------+----------------------+ | MSA_METADATA_ID | int16 | MSA metadata ID | +-------------------------------+-----------+----------------------+ | SHUTTER_QUADRANT | int16 | MSA Quadrant number | +-------------------------------+-----------+----------------------+ | SHUTTER_ROW | int16 | Shutter row index | +-------------------------------+-----------+----------------------+ | SHUTTER_COLUMN | int16 | Shutter column index | +-------------------------------+-----------+----------------------+ | SOURCE_ID | int32 | Source ID number | +-------------------------------+-----------+----------------------+ | BACKGROUND | string | Background flag | +-------------------------------+-----------+----------------------+ | SHUTTER_STATE | string | Shutter state | +-------------------------------+-----------+----------------------+ | ESTIMATED_SOURCE_IN_SHUTTER_X | float32 | Source x position | +-------------------------------+-----------+----------------------+ | ESTIMATED_SOURCE_IN_SHUTTER_Y | float32 | Source y position | +-------------------------------+-----------+----------------------+ | DITHER_POINT_INDEX | int16 | Dither/nod index | +-------------------------------+-----------+----------------------+ | PRIMARY_SOURCE | string | Primary source flag | +-------------------------------+-----------+----------------------+ | FIXED_SLIT | string | Fixed slit name | +-------------------------------+-----------+----------------------+ - SLITLET_ID: integer ID of each slitlet consisting of one or more open shutters. - MSA_METADATA_ID: integer ID corresponding to a particular MSA configuration or MPT plan (typically the same for all rows in a given metafile). - SHUTTER_QUADRANT: integer ID of the MSA quadrant in which the open shutter is located. - SHUTTER_ROW: integer row index of the shutter within the MSA quadrant. - SHUTTER_COLUMN: integrer column index of the shutter within the MSA quadrant. - SOURCE_ID: unique integer ID for each source in each slitlet, used for matching to an entry in the ``SOURCE_INFO`` table. - BACKGROUND: Y or N. Y indicates that the shutter is open to background only. - SHUTTER_STATE: OPEN or CLOSED. Generally will always be OPEN, unless the shutter is part of a long slit that is not contiguous. - ESTIMATED_SOURCE_IN_SHUTTER_X/Y: the position of the source within the shutter in relative units, where 0,0 is upper-right, 0.5,0.5 is center, and 1,1 is bottom-left, as planned in the MPT. - DITHER_POINT_INDEX: integer index of the nod sequence; matches to header keyword `PATT_NUM`. - PRIMARY_SOURCE: Y or N. Y indicates that the shutter contains the primary science source. - FIXED_SLIT: string name of a fixed slit containing the source; set to NONE for MSA slitlets. This column may not appear in older versions of the MSA metadata files. It is the :ref:`assign_wcs ` step in the :ref:`calwebb_spec2 ` pipeline that opens and loads all of the necessary information from the MSA metadata file. For each slitlet that it identifies as being used in the current exposure, a ``Slit`` object is created and populated with all the shutter and source information needed in subsequent processing steps. The list of ``Slit`` objects is attached to the data model for the exposure being processed and passed along to all subsequent calibration steps. For a given exposure, only the rows of the shutter table where the values of `MSAMETID` and `PATT_NUM` in the science exposure match the values of `msa_metdata_id` and `dither_point_index`, respectively, are loaded. Slitlets with a catalog source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To better understand the ways in which these metadata are used, it's useful to reference a hypothetical example of data within a ``SHUTTER_INFO`` table. The table below shows the first 9 rows of a ``SHUTTER_INFO`` table for a MOS exposure that's part of a standard 3-point nod MOS observation, in which the sources are moved to different shutters within each slitlet from one exposure to another in the observation. Only the data for slitlet 2 are shown. In this example, slitlet 2 is comprised of 3 shutters. Because a 3-point nod pattern has been used, there are 3 different sets of metadata for each slitlet (one set for each dither/nod position) and hence a total of 9 entries (3 shutters x 3 dithers). +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | Slit | Meta | | | | Src | | | X | Y | Dith | Pri | Fxd | | | | | | | | | | | | | | | | ID | ID | Quad | Row | Col | ID | Bkg | State | pos | pos | Pt | Src | Slit | +======+======+======+=====+=====+========+=====+=======+=======+=======+======+=====+======+ | 2 | 1 | 2 | 10 | 154 | 0 | Y | OPEN | NaN | NaN | 1 | N | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 155 | 42 | N | OPEN | 0.399 | 0.702 | 1 | Y | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 156 | 0 | Y | OPEN | NaN | NaN | 1 | N | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 154 | 42 | N | OPEN | 0.410 | 0.710 | 2 | Y | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 155 | 0 | Y | OPEN | NaN | NaN | 2 | N | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 156 | 0 | Y | OPEN | NaN | NaN | 2 | N | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 154 | 0 | Y | OPEN | NaN | NaN | 3 | N | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 155 | 0 | Y | OPEN | NaN | NaN | 3 | N | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ | 2 | 1 | 2 | 10 | 156 | 42 | N | OPEN | 0.389 | 0.718 | 3 | Y | NONE | +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+ The values in the `slitlet_id` column show that we're only looking at table rows for slitlet 2, all of which come from MSA configuration (`msa_metadata_id`) 1. The shutters that make up slitlet 2 are all contained in MSA quadrant 2 and are located in row 10 of that quadrant. The 3 shutters making up slitlet 2 span MSA columns 154, 155, and 156. The remaining table entries in each row vary depending on whether a source is located in a given shutter for each dither/nod position. You can see that the source is located in shutter column 155 in dither position 1, 154 in dither position 2, and 156 in dither position 3. The `source_id` column shows that source 42 is in those shutter positions for those dithers. The `background` column, meanwhile, indicates whether a given shutter only has background signal in it for each dither position. When there is a source in a shutter, the `estimated_source_in_shutter_x` and `estimated_source_in_shutter_y` columns are populated with numerical values that estimate the source location in the shutter. These are fractional position values that run from 0,0 at the lower-left corner of a shutter to 1,1 at the upper-right corner (hence 0.5,0.5 indicates the shutter center). These source positions are used in calibration steps such as :ref:`wavecorr ` and :ref:`pathloss ` to compute the appropriate corrections. The flags in the `primary_source` column are used to identify which shutter within each slitlet should be considered the "primary" shutter. This is especially important for slitlets that contain extended sources and hence the `source_id` and `background` entries may indicate that the source is present in multiple shutters. .. _msa_background_and_virtual_slits: Slitlets without a catalog source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible for users to define slitlets that do not contain a source that's defined in the MPT catalog when constructing an MSA configuration for an observation. These kinds of slitlets can be used for two purposes. First, slitlets in which all of the constituent shutters only contain background can be used to perform "master background" subtraction during the :ref:`calwebb_spec2 ` pipeline processing (see :ref:`NIRSpec Master Background ` for more details). Second, slitlets made up of open shutters that may contain signal from some uncataloged source in the field can also be created. These are referred to as "virtual" slitlets. Background and virtual slitlets have unique metadata in the shutter information table. The primary defining piece of data is their assigned `source_id` value, because these slitlets don't have a corresponding source listed in the source information table. During creation with the MPT, all background slitlets are given a `source_id` of zero. Virtual slitlets, on the other hand, are assigned *negative* `source_id` values, starting with -1 and counting downwards from there (i.e. each virtual slit has a unique negative `source_id` value). During the parsing of shutter information described in the previous section, when a slitlet is found that has no shutters with a primary source (i.e. no shutters having `primary_source` = "Y"), it is recognized as a background slitlet. In order to be able to track multiple background slitlets through the remaining processing, they are reassigned a new `source_id` value equal to their `slitlet_id`. Virtual slitlets, meanwhile, retain their unique negative `source_id` values throughout processing. During Stage 3 processing with the :ref:`calwebb_spec3 ` pipeline, unique source-based product file names will be created that distinguish data from the three different kinds of slitlets: source, background, and virtual. As described in :ref:`source-based file names `, the `SourceID` field of Stage 3 file names consists of the 9-digit `source_id` number assigned to each MOS slitlet, preceded by one of the three characters "s", "b", or "v", which identifies whether the data are from a source, background, or virtual slitlet, respectively. Note that, as described above, the `source_id` number used here for background slitlets is a copy of their `slitlet_id` number. For example, a Stage 3 file name for data taken from a virtual slitlet with `source_id` = -42 will look like: jw12345-o066_v000000042_nirspec_f170lp_g235m_x1d.fits Fixed Slits ~~~~~~~~~~~ It is possible to plan fixed slit sources alongside standard MOS targets. In this case, a unique `slitlet_id` is not assigned in the MSA file. Instead, the slit is identified by the value of the `fixed_slit` column. This value may be set to any of the NIRSpec fixed slit names used for science: S200A1, S200A2, S400A1, or S1600A1. Fixed slit targets must always have `primary_source` = "Y" and `background` = "N". They will never be extracted as background sources. The `shutter_quadrant`, `shutter_row`, and `shutter_column` fields are set to placeholder values. All other values have the same meaning and values as for MSA slitlets. The SOURCE_INFO Metadata ------------------------ The structure of the ``SOURCE_INFO`` table extension is as follows: +-------------+-----------+----------------------+ | Column Name | Data Type | Contents | +=============+===========+======================+ | PROGRAM | int32 | Program ID number | +-------------+-----------+----------------------+ | SOURCE_ID | int32 | Source ID number | +-------------+-----------+----------------------+ | SOURCE_NAME | string | Source name | +-------------+-----------+----------------------+ | ALIAS | string | Source alias | +-------------+-----------+----------------------+ | RA | float64 | Source RA | +-------------+-----------+----------------------+ | DEC | float64 | Source Dec | +-------------+-----------+----------------------+ | PREIMAGE_ID | string | Pre-image ID | +-------------+-----------+----------------------+ | STELLARITY | float64 | Source stellarity | +-------------+-----------+----------------------+ - PROGRAM: 5-digit JWST program number. - SOURCE_ID: unique integer identifier for each source in MPT catalog. - SOURCE_NAME: source name, typically constructed as the concatenation of program number and source ID number. - ALIAS: alternate source identifier from the MPT catalog. - RA/Dec: catalog source coordinates, in decimal degrees. - PREIMAGE_ID: name of a NIRCam mosaic used to determine the source catalog, if one was used. - STELLARITY: DAOphot-style stellarity value for the source, where 0 is fully extended and 1 is an unresolved point source. The table below shows an example of a few rows of hypothetical source metadata, consisting of the program ID, source ID, source name, source alias, source RA and Dec, pre-image ID, and source stellarity. +------+------+-----------+-------+------------+-------------+--------------+------------+ | | Src | Src | | | | Pre | | | | | | | | | | | | PID | ID | Name | Alias | RA | Dec | Image | Stellarity | +======+======+===========+=======+============+=============+==============+============+ | 1180 | 42 | 1180_0042 | Bob | 53.1456291 | -27.7674976 | 95065001_001 | 1.00 | +------+------+-----------+-------+------------+-------------+--------------+------------+ | 1180 | 1001 | 1180_1001 | Sue | 53.1435047 | -27.7689669 | 95065001_001 | 0.00 | +------+------+-----------+-------+------------+-------------+--------------+------------+ | 1180 | 3333 | 1180_3333 | Erin | 53.1485349 | -27.7696165 | 95065001_001 | 0.23 | +------+------+-----------+-------+------------+-------------+--------------+------------+ | 1180 | 9876 | 1180_9876 | Dave | 53.1461433 | -27.7617165 | 95065001_001 | 0.88 | +------+------+-----------+-------+------------+-------------+--------------+------------+ For each slitlet identified as having a source assigned to it in the shutter metadata, the source name, alias, RA, Dec, and stellarity are retrieved from the `SOURCE_INFO` table and stored with the ``Slit`` object created in the calibration software. The stellarity values are used in the :ref:`source type ` step to determine whether the source should be treated as point-like or extended.