HISTORY.txt changes to PGE02/MOD_PR02 (Level 1B Calibration) This file shows the following: (a) What was changed in the PGE (b) Why was it changed (c) How the output product will be affected by the change (d) The date of the change. v2.5.5 01/12/2001 ============================================================================ (Principal changes) 1. In previous versions, aggregated images were misregistered with respect to the native resolution images. The aggregation algorithm was changed to correct this misregistration. This affects all aggregated images (250_to_500, 250_to_1km, 500_to_1km). Correlation coefficients are now typically 98 to 99% whereas before they were typically 80 to 90%. 2. The output products follow new file spec versions: V2.5.1, 1/12/2001. The main changes in the file specs were related to adding the new QA and uncertainty attributes described in "Other Changes", below. (Other changes) 1. Added a global attribute "Granule Average QA Values" (float32, count=50) to hold selected temperatures and voltages that MCST will use in performing QA on the product. (CCR 528). 2. Added a new category of scaled integer unusable data value: 65529, which will be set in the product EV scaled integers for thermal bands if the radiance exceeds the maximum used for scaling to the scaled integers or for reflective bands if dn** exceeds the maximum used for scaling. Previously, the scaled integer would have been set to 32767, a number in the valid range. However, that is not appropriate for these cases because the number is actually invalid. Now, those instances will have a value that is not in the valid range. (CCR 528) 3. Added the following attributes to each uncertainty index (UI) SDS: "specified_uncertainty", float32, count = number of bands in the SDS "scaling_factor", float32, count = number of bands in the SDS "uncertainty_units", string, value = "percent". These may be used to convert the UI into uncertainty in percent. 4. Obsolete code lines related to "DN_sat_prime" were removed. Also, since the new SWIR correction algorithm implemented in V2.5.4 is now a simple one or two line algorithm, it is in-lined in Reflective_Cal. There are no changes in the product. 5. Added global attribute "Solar Irradiance on RSB Detectors over pi", which records, for reflective detectors, the RSR-weighted solar irradiance for reflective bands divided by pi, or "Esun". Also, the computation of the band-dependent radiance_scales for RSBs now uses the average of Esun over the detectors within a band rather than the value for one of the detectors in the band. While this causes no change to the product scaled integers, the computed radiances from those scaled integers may be changed slightly (< 0.1 %). 6. Obsolete code related to overlap scans in emissive preprocessing was deleted. As part of this, LUT "Number of overlap scans for temperatures" was deleted and the LUT name "Number of overlap scans for b1 and T BB" was changed to "num_overlap_scans_b1". There was no change in any product. 7. A character buffer was added in function "read_attribute" to correct a bad coding practice that was allowed on some machines but not others. There was no change to the product. v2.5.4 12/01/2000 ============================================================================ 1. The SWIR OOB leak correction algorithm, implemented in function SWIR_out_of_band_correction, was changed. The new algorithm comes from a memo by Jack Xiong, 12/01/2000. The correction to SWIR band dn is now based on a quadratic equation in terms of dn_28, the dn determined for band 28. The coefficients of the quadratic relation are input as LUTs. As part of the new algorithm, there were structural changes to the Reflective LUTs file. All SWIR correction LUTs except the switch were removed and three new SWIR correction LUTs were added. This change is expected to improve calibration of the SWIR bands (5,6,7,26) when the leak correction switch is turned ON. No other bands are affected. v2.5.3 (NOT USED IN OPERATIONS) 11/22/2000 ============================================================================ 1. In response to a failure of the GDAAC "PDPS" system reading the PGE02 V2.5.2 PCF file, the L1B code was changed to avoid reading a long, exact string within the line beginning with LUN 800510. The function "PGS_MET_GetConfigData" was replaced with "PGS_PC_GetConfigData" for reading the value associated with LUN 800510. There are no changes to the actual L1B products. v2.5.2 (NOT USED IN OPERATIONS) 11/01/2000 ============================================================================ 1. Incorporate new MCFs received from ECS (Descriptor revision = "1.2"). This version conforms to the new Terra/Aqua product format of file specifications: V2.5.0, Release 2, 8/18/2000. 2. Added a QA LUT "Control Options" to allow disabling checks for split scans and invalid scan quality array values. In the new L1A output product, there are a significant number of the latter (everywhere that split scans were fixed there now are invalid values in the second element of Scan quality array). However, these invalid values are most likely due to a counting bug in L1A rather than bad data within the scan. Thus, the LUT will be set to OFF (0) to maximize the amount of data calibrated. v2.4.4 09/26/2000 ============================================================================ The product format for this version follows the L1B V2.4.1, 07/10/2000, file specifications. This version is intended for Terra operations only. The changes listed below are changes from V2.4.3. 1. The HDF-EOS Dimension Name for MOD021KM Geo Data Sets was changed from "Max_EV_frames/5+1" to "1KM_geo_dim". This corrects the problem where "/" is an illegal character in the dimension name. 2. Changes were made in emissive bands pre-processing to correct two situations where emissive band radiances may contain errors: a) in the 40 scans immediately before or after a sector rotation, and b) if an ECAL ends in the last 40 scans of the leading granule or begins in the 1st 40 scans of the trailing granule. The magnitude of the errors produced in the above situations could be quite large. However, since sector-rotation events and ECALs are rare, the total number of scans affected by these situations has been very small compared to the total number of scans collected thus far by MODIS. 3. An indexing bug in the SWIR out-of-band correction algorithm was corrected. The bug caused the code to improperly use the band 29 radiance in the algorithm rather than the band 28 radiance. This bug will cause errors in radiances and reflectances of all SWIR bands (bands 5,6,7,26) if the SWIR OOB correction switch is turned ON (value = 1 of the LUT "SWIR_OOB_correction_switch"). Currently, this switch is set to OFF (value = 0). However, products produced with LUTs prior to V2.4.3.1 had the SWIR correction switch turned "ON", and thus contain errors. 4. A new algorithm was implemented to compute the average space view (SV) DN for emissive bands when the moon is in the SV port. Previously, if the moon was determined to be in the SVP, emissive bands were not calibrated (the scaled integers were set to the value 65526). Now, the emissive bands will be calibrated when the moon is in the SV port. A new LUT was added to define the number of frames to include. 5. A bug was corrected in emissive bands preprocessing when calculating the signal to noise ratio (SNR). This bug caused all values of SNR to be zero. The effects of the bug were twofold: (1) One term in the calculation of emissive bands uncertainty (NEdL/L_ev) was always zero. Thus, some values in "EV_1KM_Emissive_Uncert_Indexes" will be increased as a consequence of correcting this bug. (2) The values of the metadata "Noise in Thermal Detectors" were always zero. After correcting the bug, non-zero values appear in this SDS. 6. A bug was corrected when calculating values for the metadata "Change in relative responses of thermal detectors". This bug occurred only if missing scans were present in the granule. The effect of the bug was to cause some values of this metadata to be erroneously set to zero if there were any missing scans in the granule. After correcting this bug, it is expected that all values of this metadata will be non-zero. 7. A bug in function "Determine_Split_Scans" and its subordinate function was corrected. The bug applied to the relatively rare case when an abnormally short time gap in "SD start time" was present in the data but was NOT associated with a split scan. The bug caused MOD_PR02 to fail catastrophically (probable core dump). Three such failures have been identified during the past month. 8. A bug in determining the metadata "% L1A EV RSB DN Not in Day Mode" was corrected. Erroneous values were present in cases where there were split scans or a bad scan quality flag in the L1A granule. 9. The algorithm used to set bits 18-19 (SRCA calibration mode) in the Bit QA Flags was changed to use the macro ID. The previous method used the SDS "SRCA calibration mode", which was unreliable on a scan-by-scan basis. Now when bits 18-19 are set to 1-1, this indicates that there is no SRCA macro running. Whether the SRCA is actually on for these scans may be determined by examining bits 8,9 of the second word of the Electronics Redundancy Vector. v2.5.1 (NOT USED IN OPERATIONS) 08/24/2000 ============================================================================ 1. A minor bug was fixed in function "Open_and_Read_L1A". In the previous version, the return status not properly set, which caused a spurious error message to appear in the LogReport file (no other output files were affected). After correction, the spurious message was removed. v2.5.0 (NOT USED IN OPERATIONS) 08/15/2000 ============================================================================ 1. Changes were made for Aqua compatibility. Checking the consistency of all the input files and ancillary files in terms of satellite instrument were added in the code. Some inventory metadata are different for Terra and Aqua in the output products. These differences require that execution of the code with Aqua data must use a different set of MCF files from an execution with Terra data. The structure of the AssociatedPlatformInstrumentSensor group changed in both the Terra and Aqua MCF files. Those values are now defined in the MCF file rather than in the code. An input run-time parameter (LUN = 800510) was added to the PCF file to identify the satellite. 2. The HDF-EOS Dimension Name for MOD021KM Geo Data Sets was changed from "Max_EV_frames/5+1" to "1KM_geo_dim" based on CCR 525. There are same changes in the output products. 3. The following obsolete metadata were removed based on CCR 526: "Post Processing Indicates Bad data" "Managers VIS/NIR Quality Index" "Managers SWIR Quality Index" "Managers MWIR/LWIR Quality Index" "Discontinuities in linear gain b1 across granules" The following obsolete SDS attributes were removed also based on the same CCR: "corrected_counts..." attributes of EV_1KM_Emissive SDS All these data are actually removed from the output products. 4. The algorithm used to set bits 18-19 (SRCA calibration mode) in the Bit QA Flags was changed to use the macro ID. The previous method used the SDS "SRCA calibration mode", which was unreliable on a scan-by-scan basis. Now when bits 18-19 are set to 1-1, this indicates that there is no SRCA macro running. Whether the SRCA is actually on for these scans may be determined by examining bits 8,9 of the second word of the Electronics Redundancy Vector. v2.4.3 07/25/2000 ============================================================================ 1. Logic for invalid values of "Scan quality array" was changed. Previously, an invalid value of Scan quality array would result in a MOD_PR02 error exit (whole granule was lost). Now, if an invalid value is encountered, the scan is treated as missing and the rest of the scans of the granule are processed normally. 2. A logic error in Preprocess was corrected for saving the value of Band 31 dn_bb values. Previously, the value from the wrong scan and detector was being used for the PC bands cross-talk correction algorithm. Since the values are fairly stable from detector to detector and scan to scan, there will only be a small change in the Emissive bands. In a representative test case, the maximum change was about 0.5%. 3. A function was added to fill in interpolated scaled integer (SI) values for pixels that correspond to a dead detector (as flagged in the QA LUT "Detector Quality Flag Values"). The value is a linear average of valid SI values from the two nearest live detectors within the same scan. If two valid values are not available for the average, then one valid value is used (simply copied). If no valid values are available, then the pixel is set to 65531 (as in previous versions). Users of the product may identify scan lines that are affected by looking at the attribute "Dead Detector List". A side effect is that the value of the ECS core metadata field "QAPERCENTINTERPOLATEDDATA" may now be non-zero. The file specifications were updated to describe this change. 4. The LUT ingest architecture was changed to incorporate data-collection- time-dependent LUTs. The LUT file architectures were changed such that scalar, numerical LUTs implemented as global attributes were changed to be SDSs. There were several such changes in each LUT HDF file. When the values of the LUTs remain the same, there are no changes in the output products. 5. In Granule.c, one instance of function "sqrtf" was changed to "sqrt" because some compilers do not support the sqrtf function. There was no change in the product. v2.4.2 06/09/2000 ============================================================================ 1. A bug in Emissive_Cal was fixed. The indexing of Planck_mir was B_emiss but should have been D_emiss. This was causing erroneous data in all emissive bands radiances. Thus, there will be substantial changes in all emissive bands radiances. v2.4.1 (NOT USED IN OPERATIONS) 05/05/2000 ============================================================================ The macros controlling the maximum number of scans (Granule.h) were raised to allow processing of a MOD01 granule with up to 208 scans. Also, an algorithm was included to detect occurances of "split" scans, which are the root cause of the number of scans in a MOD01 granule exceeding 204. When a "split" scan is detected, the output product scaled integers for that scan will be set to the fill value of 65535. v2.4.0 (NOT USED IN OPERATIONS) 04/24/2000 ============================================================================ 1. New MCFs were delivered by ECS and incorporated into this version. There were no code changes required for the new MCFs. The effect on the product was: (a) a default value of "Not Investigated" was set in the MCF for the core metadata field ScienceQualityFlag, (b) a new archive metadata field, DESCRREVISION, was added and a a value of "1.0" was set in the MCF file. The file specs were changed to reflect these changes. 2. The production rules (MOD_PR02_production_rules.txt) were changed in accordance with CCR 519. This change formally implements the change described below under version "v2.3.2 (emergency makefile modification)". The makefile "MOD_PR02.mk" was modified as described in that section. The possible changes in the product are described in that section. 3. A new MOD_PR02 failure exit code was added as approved in CCR 520. Previously, the only failure exit code was a "1". Now, when the current MOD01 granule is found to have no scans (empty granule), MOD_PR02 will exit with an exit code of 233. This failure exit code is passed out as the PGE02 exit code. 4. A check on the MOD01 Number of Scans was added to Read_L1A_OBCEng to enable detection of empty MOD01 current granules and to produce a clear error message. There were no other changes to the products. 5. Several QA attributes approved in CCR 508 were added. These do not change the product instrument and uncertainty SDSs. The file specs were changed to reflect these new attributes. 6. A bug in the algorithm for determining the moon in the SV port was corrected. Previous instances where the moon was flagged to be in the SV port for thermal bands were erroneous. Other than the periods when the moon was erroneously flagged to be in the SV port, there are no changes to the product due to this change. v2.3.2 (emergency makefile modification) 02/25/2000 ============================================================================ Because of the high number of PGE01 production failures during the checkout phase, some MOD01 data could not be processed under the current production rules of requiring 3 successive MOD01 granules. To try to maximize the data generated, an emergency change to the V2.3.2 makefile was made to allow processing without a previous or following MOD01 granule if the granule is not available at the time of execution for MOD_PR02. The following is the text of the change as sent to the GDAAC: The change is to the MOD_PR02 makefile: MOD_PR02.mk which is not included in the tar file. Only the comparison input and output files are included. GDAAC personnel will modify the MOD_PR02 makefile as follows. Add the following to the "ADD_CFLAGS" macro: -DALLOW_MISSING_GRANULES MOD_PR02 should then be completely recompiled and re-linked. If a previous or following MOD01 granule is not available, then do not define those LUNs in the PCF. This instruction applies to LUNs 500000 (previous MOD01 granule) and 500002 (following MOD01 granule). Set the wait timers to zero for both the previous and following granules. In the comparison files supplied with this patch, the LUNs 500000 (previous MOD01 granule) and 500002 (following MOD01 granule) were not defined in the PCF. If the previous and following granules are not included in the execution, there is no effect on the product for 250m and 1km reflective solar bands. For thermal emissive bands there is a small effect on scans near the granule boundary. To date, using simulated data, typical maximum percentages changes in the "EV_1KM_Emissive" SDS between a product calculated using all three MOD01 granules and a product calculated with the previous and following granules missing are less than 0.1 %. For the 500m SWIR reflective solar bands, there is an effect on the product because the algorithm uses the band 28 radiance. v2.3.2 01/05/2000 ============================================================================ A printf inside a preprocessor #ifdef block was removed from Granule.c. There were no changes to the product from this code change. v2.3.1 (NOT USED IN OPERATIONS) 12/28/1999 ============================================================================ 1. The meaning of _FillValue for scaled-integer, uncertainty and samples- used EV SDSs was changed to include reflective band data at night. This change actually restores the meaning of _FillValue for those SDSs to the meaning prior to V2.3.0. In V2.3.0, implementation of a unique value for RSB at night (as per CCR 505) resulted in the night mode granules becoming the same size as the day mode granules. By restoring the meaning of _FillValue to include reflective band data at night, the night-mode granule sizes prduced by V2.3.1 are restored to their reduced sizes of 21 MB for the 250m and 500m products, and 136 MB for the 1km product. v2.3.0 (NOT USED IN OPERATIONS) 12/14/1999 ============================================================================ 1. The current version reflects changes implemented for the following CCRs which were approved: 470, 491, 498, 501, 502, 505, and part of 508. The following CCRs were rejected but notes were added to the file specs alerting users that certain data items will be removed after launch + 6 months: 485, 488, 504, part of 508. Note that for CCR 508, several new attributes were approved to be added and two bits redefined in the BIT QA flags. Only the redefinition of the two bits was implemented in this release. 2. Both TEB and RSB uncertainty algorithms were substantially revised. Also, bugs in converting uncertainty to uncertainty index were corrected. Thus, there may be substantial changes in the uncertainty index SDSs for this release compared to the last release. 3. Several new LUTs were added, several old LUTs were deleted and several LUTs had name changes. Thus, the LUT files (Emissive_Lookup_Tables_file, Reflective_Lookup_Tables_file and QA_Lookup_Tables_file) for previous releases are not compatible with this release. 4. The reflective calibration algorithm changed, resulting in small changes to the reflective bands scaled integer SDSs and their scaling attributes. However, there should be relatively little change in the reflectance and radiance products (except for SWIR bands, described below). 5. A new SWIR band calibration algorithm was implemented. Thus, there may be substantial changes in the SWIR band reflectance and radiance from previous releases. 6. Error messages were upgraded to include the LUN of the input or output file when trying to open/close files and for nearly all I/O operations. Fatal error messages also write out operator actions. 7. The operator actions (in file "error_messages.txt") were updated. 8. The computation of variance for computing "Noise in ..." metadata was found to be inaccurate at the float32 precision level and was upgraded. Thus, some substantial changes in these metadata items may occur from the previous release. 9. Added logic to allow the L1A sector DN values to be the HDF fill value of -32767 as an alternative missing DN flag. This is due to a defect in the MOD_PR01 code when there are completely missing scans (they do not set the value of DN to -1 as described in file specs). 10.Implemented code which assumes that the SD Sun azimuth and SD Sun zenith angles in the MOD03 product are in radians rather than in degrees. This is because the current MOD03 product writes these angles in radians, which conflicts with the MOD03 file spec. CCR 512 has been filed to change the file spec to describe these angles in radians rather than degrees. Depending on the resolution of CCR 512, the assumption may be incorrect. The only item in the L1B output products that may be affected is bit 11 of the Bit QA Flags. v2.2.0 5/30/1999 ============================================================================ The algorithms for calculate average background DN were changed. The OBC file specification was changed. The uncertainty algorithms was changed. Algorithm about saturation of reflective band detectors was added. Reflectance and radiance calibration scales of reflective bands algorithm was changed. LUTs were changed due to the new algorithms. The scientists in MCST have better understanding of the system and believe these changes will make the calibration more accurate. The output product for EV files is the same. v2.1.5 4/28/1999 ============================================================================ 1. Fixed a bug by changing some variable type from float32 to int32. The output QAPERCENTMISSINGDATA, QAPERCENTINTERPOLATEDDATA and QAPERCENTOUTOFBOUNDSDATA will be within the range[0,100] after this bug was fixed. v2.1.4 4/2/1999 ============================================================================ 1. Changed swir band lookup table values to turn off the swir band correction. v2.1.3 3/30/1999 ============================================================================ 1. Assigned realistic numbers in the LUTs so that the output L1B products have reasonable radiances. Output product changed. (a) Incorporated new R_star and L_sd values in LUTs. (b) DN_to_DN_star: The averaged space-view DN is now being subtracted from DN'. The RVS term involving the space view was removed. (c) Emissive_Cal: There was a bug in initializing dn_ev involving the block for ADC correction. After the "if () { ...}", we added: else { dn_ev = DN_ev; } which resolved the issue. 2. PreprocessP.h: Reset macro "FIRST_SV_1km_FRAME_TO_USE from 0 to 10. (no significant change to product). 3. Get_All_Emiss_Coeff: There was a bug in the computation of average b1 for each scan: it was missing the divide by num_scans. This has been fixed. (no significant change to product). 4. Get_L1B_SD_SRCA_BB_SV: There was an indexing problem in how the arrays of the DNs were being filled. This has been corrected. As a consequence of fixing this, we removed an extremely inefficient array so that our stack size requirement dropped from 161 MB to 114 MB. This changes the OBC product only. 5. MCF Files: Made minor changes to the MCF file in order to clean up some Metadata. 6. Metadata.c: The name of the global attribute: "Post Processing Indicates Bad Data" (the way the code implements) was changed to "Post Processing Indicates Bad data" (the way the file specs read)