Applying enhancements to Geo2Grid imagery

July 3rd, 2020 |

GOES-16 ABI Band 13 (10.3 µm) Infrared Imagery from 1550 UTC on 2 July 2020, with color enhancement added, and with annotated colorbar (Click to enlarge)

geo2grid (see this recent blog post) is a handy software package that allows anyone to create useful satellite imagery. Recent blog posts that document how to use geo2grid have shown visible or red-green-blue (RGB) composites. Individual infrared bands can also provide useful information, and this blog posts describes how you can create infrared imagery to which a useful color enhancement has been applied.

Step 1, as always, is to find data. NOAA CLASS has archived Level-1b Radiance files; these are the data that geo2grid expects, and the naming convention of the files is also what geo2grid expects. Amazon Cloud Services can also offer correctly-named data. (This blogger is aware of some users who have retrieved data from Google, and those files did not follow the expected naming convention).

The animation above compares an infrared image from 1550 UTC on 2 July over the northern part of the United States. The grid is described in this blog post and was used for convenience. (Of some importance: The grid created has dimensions of 1500×800). The original grey-scaled image is shown (with coastlines overlain), then a color enhancement is applied, and finally a script is run to annotate the colorbar.

The geo2grid command used to start the image creation was:

./ -r abi_l1b -w geotiff -p C13 -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest -f /arcdata/goes_restricted/grb/goes16/2020/2020_07_02_184/abi/L1b/RadF/

This creates a Band 13 (the -p C13 flag) grey-scaled geotiff (-w geotiff) from data at the archive site specified ( -f /arcdata/goes_restricted/..../ ), reading Advanced Baseline Imagery Level-1b imagery (the -r abi_l1b flag), and using nearest-neighbor interpolation to a pre-defined grid (“PADERECHO”) to transform the projection from the default satellite projection (That’s this part of the command: -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest). Note that by default, the geotiff created is 8-bit (in contrast to the 11-bit imagery; some information is lost.) You can create geotiffs with full bit depth. For this example an 8-bit image is being created.

The add_colorbar shell applies a color enhancement to the grey-scaled geotiff image that geo2grid creates. For the image in this blog post, I used this invocation:

./ ../../enhancements/IR4AVHRR100_colortable.txt GOES-16_ABI_RadF_C13_20200702_155021_PADERECHO.tif

This colorbar must be pre-defined by the user. The format of the colortable (“IR4AVHRR100_colortable.txt”) is a comma-separated list of greyscale values (0-255 for the 8-bit geotiff that is created) followed by Red, Green, Blue and Transparency values. (Documentation is here. The file used for this example is here — courtesy of Jim Nelson, CIMSS) The colortable is applied to the geotiff image and a separate image (.png in this case) is created. You will note that this bi-linear enhancement is various shades of grey for bit values 0-152, then different colors for values 153-255.

How do you associate the color with a temperature? For a bi-linear enhancement as used for Band 13, this is not simple. With knowledge of the calibration used, or with access to a system (such as McIDAS-X or McIDAS-V) that includes the calibration, it is possible to associate the greyscale values with temperature values and then use annotation software such as ImageMagick to write in values, with a command such as:

convert INPUTFIELD_with_Coastline_Colorbar_Enhanced.png -fill white -pointsize 24 -annotate +997+28 "-30" InputField_with_CoastLine_Colorbar_enhanced_-30Label.png

This must be done for each label you want to add to the image, but a shell script does this easily, and once done, you have something that you can use for any image that is the same size: 1500 pixels wide. The spacing between the -30ºC, -40ºC, -50ºC… labels on the color bar is constant in this example. Thus, once two values are known and placed, it’s simple to compute the offsets for the other temperature values.

Polar2Grid colorbar functionality is a bit more comprehensive than Geo2Grid. When a colorbar is created in Polar2Grid, metadata on the minimum and maximum values are inserted into the geotiff. This allows automatic generation of colorbar labels (for a linear scaling only) as shown in this excellent example of documentation!

Using geo2grid to create animations

June 5th, 2020 |

GOES-16 Airmass RGB, 1400 UTC on 2 June 2020 to 1640 UTC on 3 June 2020

Severe weather occurred over the upper Midwest on 2 June 2020, and a derecho occurred over the mid-Atlantic States on 3 June 2020. These two events were linked, and the animation above shows the system moving and redeveloping from Minnesota on 2 June 202 eastward to Pennsylvania on 3 June 2020. How was this animation created?

geo2grid is a software packaged developed at CIMSS. It is designed to produce high-quality full-resolution imagery from archived (or real-time) GOES-16/GOES-17 Level-1b imagery. This blog post (a follow-up to this one) will outline how to use the software package to create imagery.  Note that geo2grid runs on CentOS6-compatible Linux.  There is no Windows version.  Here are the steps used to create the imagery above.

    1. geo2grid software can be downloaded as a g-zipped tarball from this link. (You may need to register — for free! — before downloading). Version 1.0.1 was released in March of 2020. You can also download documentation (always a good idea) as a pdf file from the download site, or you can access it online.  Once you have unzipped and untarred the software package, you’re ready to begin.  Your unix system must know where the software package sits, and that’s through this command: EXPORT GEO2GRID_HOME=/path/to/Geo2GridSoftwareLocation/
    2. Of course, you will also need data.  For Advanced Baseline Imager (ABI) data, geo2grid expects Radiance fields.  These can be accessed via NOAA CLASS, or via “The Cloud”, or from a GRB Receiving antenna.  (some data sources are listed here)  NOAA CLASS ABI L1b Radiances data (ordered under “GOES-R Series ABI Products (GRABIPRD) (partially restricted L1b and L2+ Data Products) ”  )  will have a file format that looks something like this: ; that particular file listed is Full Disk (the ‘F’ in ‘RadF’) when GOES-16 was in Mode 6 (M6) scanning, and it contains Band 8 (C08);  the start time is Year 2020, Day 154 (June 2 2020) at 23:40:16.
    3. As noted in this blog post, you can specify which fraction of the domain to display. geo2grid also has a mapping routine if you want to define your own domain (i.e., not a satellite projection, or a subset of that satellite projection), and documentation is under ‘Utility Scripts’ at the geo2grid documentation site.  To define a domain near the Great Lakes, for example, I used this command:   $GEO2GRID_HOME/bin/ PADERECHO -83.0 45.0 2000 -2000 1500 800 > $GEO2GRID_HOME/PADERECHO.conf ;  the shell scripts takes a grid name (PADERECHO in this case), a center longitude/latitude (83.0 degrees West, 45.0 Degrees North), x- and y- grid-spacing in meters (2 km for this case), and the number of points in the x- and y- directions.  These mapping instructions are placed in a configuration file;  cat $GEO2GRID_HOME/PADERECHO.conf will show this line in the file:  PADERECHO, proj4, +proj=lcc +datum=WGS84 +ellps=WGS84 +lat_0=45.00000 +lat_1=45.00000 +lon_0=-83.00000 +units=m +no_defs, 1500, 800, 2000.00000, -2000.00000, -104.24668deg, 50.40805deg .  It might take some trial and error using geo2grid to find the map domain that suits you best, but it is a simple matter to iterate to a solution.
    4. Next, run geo2grid.   The command I invoked (with small changes for each time) is here:  $GEO2GRID_HOME/bin/ -r abi_l1b -w geotiff -p airmass -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest --cache-dir=$GEO2GRID_HOME/datacache/ -f /data-hdd/PADerecho/Day154/14/*1450*.nc   There are several flags included in this call, and they are explained in the documentation, but also here.
      • -r:  What kind of data are being read?  In this case, level-1b data from ABI
      • -w:  Output format (geotiff).  That is the only option for imagery
      • -p:  What imagery should be created?  In this case, airmass RGB.  To find out what can be computed given the data present, user the –list-products flag.  For the data I have downloaded, this returned C08, C10, C12, C13, C15, airmass:  I had downloaded to the specified directory the components necessary to compute the airmass RGB (plus band 15).  If you wanted to create imagery for all these channels in addition to the airmass RGB, -p airmass C08 C10 C12 C13 C15 would work.  (Note:  C15 is not actually needed in the computation of the airmass RGB).
      • -g:  to what grid should these data be interpolated?  The answer: the PADERECHO grid that is defined and specifed where –grid-configs points to:  $GEO2GRID_HOME/PADERECHO.conf;
      • –method:  what interpolation method should be used?  For this case, nearest neighbor is specified.
      • –cache-dir:  This is used to speed processing.  If you have multiple files being interpolated to the same grid, it speeds things to save the interpolation methods/points.  You have to specify the directory where this file will sit.
      • -f:  Where do the Radiance files sit?  In this example I have placed different times in separate directories.  I haven’t yet figured out how to use Geo2Grid that points to a directory where multiple times sit.
    5. geo2grid includes scripts (““;  for full documentation, see the ‘Utility Scripts’ section in the geo2grid documentation)  that will add maps to the imagery.  $GEO2GRID_HOME/bin/ --add-coastlines --coastlines-resolution=h --coastlines-level=4 GOES-16_ABI_RadF_airmass_20200602_152017_PADERECHO.tif  This command invoked only a few of the mapping additions, as described below.  For a full list, refer to the documentation.
      1. –add-coastlines ;  as you might expect this flag adds coastlines.  In the animation above, you’ll see that the coastlines include lakes.  You can also add borders, grids (lat/lon lines) and other features.
      2. –coastline-resolution=h ; I have asked for high resolution.
      3. –coastlines-level=4 ; This is the default level of detail in the lines.
      4. The .tif is the geotiff created from Geo2Grid.  When is finished, the filename GOES-16_ABI_RadF_airmass_20200602_152017_PADERECHO.png is created.
    6. geo2grid does not annotate imagery.  I use ImageMagick for that.  (something like this:  convert GOES-16_ABI_RadF_airmass_20200603_163017_PADERECHO.png -gravity Northeast -fill white -pointsize 32 -annotate +8+40 "3 June 2020 1630 UTC" GOES-16_ABI_RadF_airmass_20200603_163017_PADERECHOannotate.png )

A challenge in scripting geo2grid is that the start time of the files is not constant.  For example, on 2 June, the 15:20 image starts at 15:20:17.1 (as shown above, the file name includes 152017) ; the 17:20 image starts at 17:20:16.9 and the created filenames include 172016.

The completed animation is shown above.

When is an ABI hot (bright) spot not a fire?

May 30th, 2020 |

An ABI hot (bright) spot is not a fire when it’s a fleet of solar farms. For example, recall the CIMSS Satellite Blog entry regarding solar farms in California. 

ABI band 2 visible

ABI band 2 visible animation on May 30, 2020 (mostly) in southeastern Minnesota. Click to play mp4.

Note how some reflections are so bright that the ABI reports dark surrounding pixels. This is part of the remapping process from detector to pixel space. 



A multiple-spectral ABI comparison on May 30, 2020. The rows are: band 2, band 5, band 6 band 7, band 7 – 14 brightness temp, band 14 fire mask, band 7-14 radiance difference, band 7-14 radiance difference minus the rolling average

From left to right, top to bottom the panels are:
1) ABI band 2 reflectance, dynamically scaled to enhance contrast (will appear to flicker)
2) ABI band 5 reflectance, dynamically scaled to enhance contrast (will appear to flicker)
3) ABI band 6 reflectance, dynamically scaled to enhance contrast (will appear to flicker)
4) ABI band 7 brightness temperature, dynamically scaled to enhance contrast (will appear to flicker)
5) ABI band 7 minus band 14 brightness temperature. Red indicates positive values (extra thermal energy due to the sun and fires, if present), dynamically scaled to enhance contrast (will appear to flicker)
6) ABI band 14 brightness temperature, dynamically scaled to enhance contrast (will appear to flicker)
7) ABI Fire Detection and Characterization Algorithm (FDCA, aka WFABBA) fire detection metadata mask.  Fires are red, orange, magenta, and shades of blue indicating different confidence levels.  Green indicates fire-free land, shades of gray indicate clouds, dark  blue indicates water.
8) Radiance difference of band 7 minus band 14 radiance in band 7 space.  Red indicates positive values (extra thermal energy due to the sun and fires, if present), dynamically scaled to enhance contrast (will appear to flicker)
9) Radiance difference of band 7 minus band 14 radiance in band 7 space minus a rolling average of the 5 prior frames, to highlight changes. Red indicates positive values (extra thermal energy due to the sun and fires, if present), dynamically scaled to enhance contrast (will appear to flicker).

Aside from the solar farms, water clouds show up in the difference panels due to their reflection of shortwave radiation. 

H/T to Chris Schmidt for the 9-panel ABI imagery.  More about quantitative ABI products, including fire detection. 

The original tweet from the La Crosse WFO: “We saw some awfully bright looking “clouds” showing up via satellite in southeast Minnesota earlier this afternoon. Well after some investigation, we were able to determine they were actually solar panel arrays that the sun was hitting just right!”

NWS tweet

Solar farms and GOES-16 ABI visible imagery from the La Crosse NWS WFO.

Using NUCAPS to help nowcast Midwest convection

May 23rd, 2020 |

GOES-16 ABI Band 2 (0.64 µm) visible imagery at 1821 UTC 23 May 2020 (Click to enlarge)

The NOAA Storm Prediction Center’s Convective Outlook (graphic) from 1630 UTC 23 May 2020 shows an enhanced risk of Severe Thunderstorms in northern Illinois with a sharp cutoff in probabilities to the north in southern Wisconsin. The imagery above shows the GOES-16 ABI Band 2 (0.64 µm) visible imagery (click here for Band 13 (10.3 µm) Infrared Imagery), and it shows convection over northern Illinois/southern Wisconsin ahead of an obvious vorticity center in Iowa. How far north will the convection build?

This question can be answered by considering lower-tropospheric (900-700) and mid-tropospheric (850-500) lapse rates diagnosed from the NUCAPS Profiles produced from the NOAA-20 pass over the Upper Midwest at ~1818 UTC (NOAA-20 orbit for this day shown here); these data were available within AWIPS by 1915 UTC! 

The toggle below shows ABI Bands 2 and 13 (0.64 µm and 10.3 µm, respectively) as well as the lapse rates. The Lower Tropospheric lapse rates show a north-south gradient in stability, with air that is more stable over Wisconsin; convection over Illinois should weaken as it moves north. Mid-tropospheric lapse rates show a similar gradient in stability albeit less pronounced. (Click here to view the lower-tropospheric lapse rates overlain with NUCAPS sounding availability points from AWIPS, color-coded to show whether or not Infrared and/or microwave retrievals converged. Over southwest Wisconsin/northwest Illinois, perhaps you can argue that the gradient is influenced by soundings that did not converge; that argument would be harder to make over northcentral/northeast Illinois and southcentral/southeast Wisconsin).

GOES-16 ABI Bands 2 (0.64) and 13 (10.3) at 1821 UTC and Lower- and Mid-Tropospheric Layer Lapse Rates (900-700 and 850-500 mb, respectively) at nominal times of 1818 UTC over the upper Midwest (Click to enlarge)

This post is to remind you that satellite-derived retrieval data (independent of model data) is available in AWIPS in a timely manner to help you diagnose the thermodynamic state of the atmosphere. (Added: Storm reports for this event are here. Severe weather did cross into Wisconsin but was very close to the Illinois border).