Creating RGB imagery using SIFT and Geo2Grid

July 8th, 2021 |

The use of routine multispectral geostationary satellite imagery over the United States has increased the routine use of Red/Green/Blue composite imagery to describe and evaluate surface and atmospheric conditions. This blog post will detail how to create new (or old) RGB composites using two UW-Madison/CIMSS/SSEC-developed tools: The Satellite Information and Familiarization Tool (SIFT; Journal article link) and Geo2Grid (Previous blog posts showing Geo2Grid examples are here). The scene to be highlighted is shown above in the GOES-16 Cirrus Band; it was chosen because of the interesting parallel bands in the Cirrus, features that can identify regions of turbulence. A larger-scale view of the data (created using CSPP Geosphere) is here (for the 1.37 µm Cirrus band) or here (for True Color).

SIFT has a very useful (and easy!) RGB generator.  For this case involving cirrus, I decided to create an RGB using the Split Window Difference (10.3 µm – 12.3 µm, Band 13 – Band 15) (shown here) that has been used to identify cirrus for quite a while (link to journal article), the cirrus band 4, and also the Snow/Ice channel Band 5 (1.61 µm).  After downloading SIFT and importing the data (and creating the split window difference field — here’s a blog post that describes how to do that), a SIFT user can create an RGB and tinker with the bounds.  Changing the bounds and the gamma causes a simultaneous change in the RGB in the SIFT display window, so it’s not difficult to iterate to a satisfactory solution.  As shown below, the RGB created has the Split Window Difference as the red component, with values from 0 (no red) to 12.0 (saturated red) and a Gamma value of 2;  the cirrus channel (C04) is the green component with values from 0.27 (no green) to 0 (saturated green) and a Gamma value of 2;  the snow/ice channel (C05) is the blue component with values from 0.0 (no blue) to 0.40 (saturated blue) and a Gamma value of 1.

SIFT RGB Creation window

The RGB created in SIFT using these values is shown below.  Maybe using maximum green — a color one’s eyes are usually particularly adept at viewing — for no signal in the cirrus channel was not the best choice.  But there is nice contrast between the background and the thin cirrus, and an obvious difference between the parallel lines of cirrus in the middle of the image and other clouds, such as the cirrus at the western edge of the image!

“Cirrus” RGB at 1411 UTC on 8 July 2021 (click to enlarge)


How do you create something similar using Geo2Grid?  Step 1, of course, is always to download and install the software package.  To see what products can be created with geo2grid, enter this command:  ./geo2grid.sh -r abi_l1b -w geotiff --list-products -f /path/to/the/directory/holding/GOESR/Radiance/Files/*syyyydddhhmm*.nc .  Let’s assume all 16 channels from ABI are available.  Important caveat: Geo2Grid will only work on one data time at a time, so specify your year/julian day/hour/minute with sufficient stringency.

RGB product definitions are found in yaml files within the Geo2Grid directory. Ones for abi in particular are found in $GEO2GRID_HOME/etc/satpy/composites/abi.yaml in which file you would enter something what is shown below for a product called ‘cirrustest’;  note that it has three channels:  the first is a difference between C13 and C15 (that is, the Split Window Difference);  the second is C04 (cirrus channel) and the third is C05 (snow/ice channel). This is the same as in the SIFT definitions.

Within $GEO2GRID_HOME/etc/satpy/enhancements/abi.yaml there is a further definition of this RGB.  The crude stretch defines the bounds of the RGB:  Red includes values from 0 – 12;  Green from 27 — that is, a reflectance of 0.27, or 27% — to 0 (note that it is inverted);  Blue from 0 to 40.  In addition, Gamma values are specified:  0.5, 0.5 and 1.

Two important things to note:  Gamma in SIFT follows National Weather Service and JMA conventions.  Gamma in Geo2Grid follows EUMETSAT conventions. Thus, one is the reciprocal of the other.  Also, note the _abi suffix in the abi.yaml file name in enhancements, i.e., cirrustest_abi, to specify the satellite.

After making these changes to the two abi.yaml files, and rerunning this command:  ./geo2grid.sh -r abi_l1b -w geotiff --list-products -f /path/to/the/directory/holding/GOESR/Radiance/Files/*syyyydddhhmm*.nc, you should see a new possibility: cirrustest (or whatever you have named your new RGB). Then you run Geo2Grid commands to create the cirrustest RGB (with the -p cirrustest flag.  The commands below sequentially create the grid for the analysis, create the tiff file, georeference it with coastlines (none, in this case over the Gulf) and latitude/longitude lines, and annotate it.

../p2g_grid_helper.sh CIRRUSRGBtest -88.3 26.6 500 -500 960 720 > $GEO2GRID_HOME/CIRRUSRGBtest.conf
#
../geo2grid.sh -r abi_l1b -w geotiff -p cirrustest C04 -g CIRRUSRGBtest --grid-configs $GEO2GRID_HOME/CIRRUSRGBtest.conf --method nearest -f /arcdata/goes_restricted/grb/goes16/2021/2021_07_08_189/abi/L1b/RadC/*s20211891411*.nc
../add_coastlines.sh --add-borders --borders-outline='blue' --borders-resolution=f --add-grid --grid-text-size 20 --grid-d 5.0 5.0 --grid-D 5.0 5.0 GOES-16_ABI_RadC_cirrustest_20210708_1411??_CIRRUSRGBtest.tif
convert GOES-16_ABI_RadC_cirrustest_20210708_1411??_CIRRUSRGBtest.png -gravity Southwest -fill yellow -pointsize 14 -annotate +8+24 "1411 UTC 8 July 2021 Cirrus RGB" GOES-16_ABI_RadC_cirrustest_20210708_1411_CIRRUSRGBtest_annot_2.png

The final image from Geo2Grid is shown below. Its geographic coverage is slightly different than in SIFT, above, but the two RGBs have similar looks.

‘Cirrustest’ RGB at 1411 UTC on 8 July 2021 (Click to enlarge)

Geostationary Lightning Mapper (GLM) data displayed with Geo2Grid

January 30th, 2021 |

GOES-16 ABI CONUS sector band 13 (Clean Window, 10.3 µm) infrared imagery, 1901-2001 UTC on 30 January 2021

Geo2Grid is a scripting tool that accesses various Python packages to display Geostationary Satellite data, described on this blog before here, here and here (Polar2Grid is a similar package for Low Earth Orbit satellite data).  The animation above shows GOES-16 Band-13 (Clean Window, 10.3 µm) infrared data for an hour over Oklahoma/Kansas/Missouri/Arkansas during a time when tornadoes occurred (imagery was produced using Geo2Grid and GOES-16 level-1b radiance files).  (SPC Storm Reports).

Gridded GLM data are available at this website;  both CONUS and Full Disk domains are available, CONUS data are a simple subset of the Full Disk imagery.  These netCDF files (with ‘GLMC’ in the filename) are available each minute, and contain a variety of gridded GLM products, some of which as distributed to National Weather Service forecast offices. By using the ‘glm_l2’ reader in Geo2Grid, data can be plotted, and subsequently overlain on top of the ABI imagery, as shown below.

GOES-16 ABI CONUS sector band 13 (Clean Window, 10.3 µm) infrared imagery, 1901-2001 UTC on 30 January 2021, overlain with GLM Total Optical Energy at 1-minute time steps (Click to animate)

Applying enhancements to Geo2Grid imagery

July 3rd, 2020 |

GOES-16 ABI Band 13 (10.3 µm) Infrared Imagery from 1550 UTC on 2 July 2020, with color enhancement added, and with annotated colorbar (Click to enlarge)

geo2grid (see this recent blog post) is a handy software package that allows anyone to create useful satellite imagery. Recent blog posts that document how to use geo2grid have shown visible or red-green-blue (RGB) composites. Individual infrared bands can also provide useful information, and this blog posts describes how you can create infrared imagery to which a useful color enhancement has been applied.

Step 1, as always, is to find data. NOAA CLASS has archived Level-1b Radiance files; these are the data that geo2grid expects, and the naming convention of the files is also what geo2grid expects. Amazon Cloud Services can also offer correctly-named data. (This blogger is aware of some users who have retrieved data from Google, and those files did not follow the expected naming convention).

The animation above compares an infrared image from 1550 UTC on 2 July over the northern part of the United States. The grid is described in this blog post and was used for convenience. (Of some importance: The grid created has dimensions of 1500×800). The original grey-scaled image is shown (with coastlines overlain), then a color enhancement is applied, and finally a script is run to annotate the colorbar.

The geo2grid command used to start the image creation was:

./geo2grid.sh -r abi_l1b -w geotiff -p C13 -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest -f /arcdata/goes_restricted/grb/goes16/2020/2020_07_02_184/abi/L1b/RadF/OR_ABI-L1b-RadF-M6C13_G16_s20201841550211_e20201841559531_c20201841600013.nc

This creates a Band 13 (the -p C13 flag) grey-scaled geotiff (-w geotiff) from data at the archive site specified ( -f /arcdata/goes_restricted/..../OR_ABI-L1b-RadF-M6C13_G16_s20201841550211_e20201841559531_c20201841600013.nc ), reading Advanced Baseline Imagery Level-1b imagery (the -r abi_l1b flag), and using nearest-neighbor interpolation to a pre-defined grid (“PADERECHO”) to transform the projection from the default satellite projection (That’s this part of the command: -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest). Note that by default, the geotiff created is 8-bit (in contrast to the 11-bit imagery; some information is lost.) You can create geotiffs with full bit depth. For this example an 8-bit image is being created.

The add_colorbar shell applies a color enhancement to the grey-scaled geotiff image that geo2grid creates. For the image in this blog post, I used this invocation:

./add_colormap.sh ../../enhancements/IR4AVHRR100_colortable.txt GOES-16_ABI_RadF_C13_20200702_155021_PADERECHO.tif

This colorbar must be pre-defined by the user. The format of the colortable (“IR4AVHRR100_colortable.txt”) is a comma-separated list of greyscale values (0-255 for the 8-bit geotiff that is created) followed by Red, Green, Blue and Transparency values. (Documentation is here. The file used for this example is here — courtesy of Jim Nelson, CIMSS) The colortable is applied to the geotiff image and a separate image (.png in this case) is created. You will note that this bi-linear enhancement is various shades of grey for bit values 0-152, then different colors for values 153-255.

How do you associate the color with a temperature? For a bi-linear enhancement as used for Band 13, this is not simple. With knowledge of the calibration used, or with access to a system (such as McIDAS-X or McIDAS-V) that includes the calibration, it is possible to associate the greyscale values with temperature values and then use annotation software such as ImageMagick to write in values, with a command such as:

convert INPUTFIELD_with_Coastline_Colorbar_Enhanced.png -fill white -pointsize 24 -annotate +997+28 "-30" InputField_with_CoastLine_Colorbar_enhanced_-30Label.png

This must be done for each label you want to add to the image, but a shell script does this easily, and once done, you have something that you can use for any image that is the same size: 1500 pixels wide. The spacing between the -30ºC, -40ºC, -50ºC… labels on the color bar is constant in this example. Thus, once two values are known and placed, it’s simple to compute the offsets for the other temperature values.

Polar2Grid colorbar functionality is a bit more comprehensive than Geo2Grid. When a colorbar is created in Polar2Grid, metadata on the minimum and maximum values are inserted into the geotiff. This allows automatic generation of colorbar labels (for a linear scaling only) as shown in this excellent example of documentation!

Using geo2grid to create animations

June 5th, 2020 |

GOES-16 Airmass RGB, 1400 UTC on 2 June 2020 to 1640 UTC on 3 June 2020

Severe weather occurred over the upper Midwest on 2 June 2020, and a derecho occurred over the mid-Atlantic States on 3 June 2020. These two events were linked, and the animation above shows the system moving and redeveloping from Minnesota on 2 June 202 eastward to Pennsylvania on 3 June 2020. How was this animation created?

geo2grid is a software packaged developed at CIMSS. It is designed to produce high-quality full-resolution imagery from archived (or real-time) GOES-16/GOES-17 Level-1b imagery. This blog post (a follow-up to this one) will outline how to use the software package to create imagery.  Note that geo2grid runs on CentOS6-compatible Linux.  There is no Windows version.  Here are the steps used to create the imagery above.

    1. geo2grid software can be downloaded as a g-zipped tarball from this link. (You may need to register — for free! — before downloading). Version 1.0.1 was released in March of 2020. You can also download documentation (always a good idea) as a pdf file from the download site, or you can access it online.  Once you have unzipped and untarred the software package, you’re ready to begin.  Your unix system must know where the software package sits, and that’s through this command: EXPORT GEO2GRID_HOME=/path/to/Geo2GridSoftwareLocation/
    2. Of course, you will also need data.  For Advanced Baseline Imager (ABI) data, geo2grid expects Radiance fields.  These can be accessed via NOAA CLASS, or via “The Cloud”, or from a GRB Receiving antenna.  (some data sources are listed here)  NOAA CLASS ABI L1b Radiances data (ordered under “GOES-R Series ABI Products (GRABIPRD) (partially restricted L1b and L2+ Data Products) ”  )  will have a file format that looks something like this:  OR_ABI-L1b-RadF-M6C08_G16_s20201542340169_e20201542349477_c20201542349544.nc ; that particular file listed is Full Disk (the ‘F’ in ‘RadF’) when GOES-16 was in Mode 6 (M6) scanning, and it contains Band 8 (C08);  the start time is Year 2020, Day 154 (June 2 2020) at 23:40:16.
    3. As noted in this blog post, you can specify which fraction of the domain to display. geo2grid also has a mapping routine if you want to define your own domain (i.e., not a satellite projection, or a subset of that satellite projection), and documentation is under ‘Utility Scripts’ at the geo2grid documentation site.  To define a domain near the Great Lakes, for example, I used this command:   $GEO2GRID_HOME/bin/p2g_grid_helper.sh PADERECHO -83.0 45.0 2000 -2000 1500 800 > $GEO2GRID_HOME/PADERECHO.conf ;  the shell scripts takes a grid name (PADERECHO in this case), a center longitude/latitude (83.0 degrees West, 45.0 Degrees North), x- and y- grid-spacing in meters (2 km for this case), and the number of points in the x- and y- directions.  These mapping instructions are placed in a configuration file;  cat $GEO2GRID_HOME/PADERECHO.conf will show this line in the file:  PADERECHO, proj4, +proj=lcc +datum=WGS84 +ellps=WGS84 +lat_0=45.00000 +lat_1=45.00000 +lon_0=-83.00000 +units=m +no_defs, 1500, 800, 2000.00000, -2000.00000, -104.24668deg, 50.40805deg .  It might take some trial and error using geo2grid to find the map domain that suits you best, but it is a simple matter to iterate to a solution.
    4. Next, run geo2grid.   The command I invoked (with small changes for each time) is here:  $GEO2GRID_HOME/bin/geo2grid.sh -r abi_l1b -w geotiff -p airmass -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest --cache-dir=$GEO2GRID_HOME/datacache/ -f /data-hdd/PADerecho/Day154/14/*1450*.nc   There are several flags included in this call, and they are explained in the documentation, but also here.
      • -r:  What kind of data are being read?  In this case, level-1b data from ABI
      • -w:  Output format (geotiff).  That is the only option for imagery
      • -p:  What imagery should be created?  In this case, airmass RGB.  To find out what can be computed given the data present, user the –list-products flag.  For the data I have downloaded, this returned C08, C10, C12, C13, C15, airmass:  I had downloaded to the specified directory the components necessary to compute the airmass RGB (plus band 15).  If you wanted to create imagery for all these channels in addition to the airmass RGB, -p airmass C08 C10 C12 C13 C15 would work.  (Note:  C15 is not actually needed in the computation of the airmass RGB).
      • -g:  to what grid should these data be interpolated?  The answer: the PADERECHO grid that is defined and specifed where –grid-configs points to:  $GEO2GRID_HOME/PADERECHO.conf;
      • –method:  what interpolation method should be used?  For this case, nearest neighbor is specified.
      • –cache-dir:  This is used to speed processing.  If you have multiple files being interpolated to the same grid, it speeds things to save the interpolation methods/points.  You have to specify the directory where this file will sit.
      • -f:  Where do the Radiance files sit?  In this example I have placed different times in separate directories.  I haven’t yet figured out how to use Geo2Grid that points to a directory where multiple times sit.
    5. geo2grid includes scripts (“add_coastlines.sh“;  for full documentation, see the ‘Utility Scripts’ section in the geo2grid documentation)  that will add maps to the imagery.  $GEO2GRID_HOME/bin/add_coastlines.sh --add-coastlines --coastlines-resolution=h --coastlines-level=4 GOES-16_ABI_RadF_airmass_20200602_152017_PADERECHO.tif  This command invoked only a few of the mapping additions, as described below.  For a full list, refer to the documentation.
      1. –add-coastlines ;  as you might expect this flag adds coastlines.  In the animation above, you’ll see that the coastlines include lakes.  You can also add borders, grids (lat/lon lines) and other features.
      2. –coastline-resolution=h ; I have asked for high resolution.
      3. –coastlines-level=4 ; This is the default level of detail in the lines.
      4. The .tif is the geotiff created from Geo2Grid.  When add_coastlines.sh is finished, the filename GOES-16_ABI_RadF_airmass_20200602_152017_PADERECHO.png is created.
    6. geo2grid does not annotate imagery.  I use ImageMagick for that.  (something like this:  convert GOES-16_ABI_RadF_airmass_20200603_163017_PADERECHO.png -gravity Northeast -fill white -pointsize 32 -annotate +8+40 "3 June 2020 1630 UTC" GOES-16_ABI_RadF_airmass_20200603_163017_PADERECHOannotate.png )

A challenge in scripting geo2grid is that the start time of the files is not constant.  For example, on 2 June, the 15:20 image starts at 15:20:17.1 (as shown above, the file name includes 152017) ; the 17:20 image starts at 17:20:16.9 and the created filenames include 172016.

The completed animation is shown above.