Creating sector-spanning imagery from VIIRS today imagery

August 5th, 2020 |

NOAA-20 True-Color Imagery showing all five Great Lakes, 3 August 2020 (Click to enlarge)

VIIRS Today, like MODIS today, subsects the US into 8 sectors of equal size. If you live in central South Carolina, or in the middle of Florida, for example, the Sector 7/Sector 8 seam is right on top of you. Perhaps you want to view the entire Great Lakes in one image, rather than a split between Sector 3 and Sector 4. (This image, from this website, shows the 8 sectors over the United States).  There are easy ways to create seam-spanning imagery.

Because the sectors are of fixed and unchanging locations and sizes (1300×1500 for 1-km resolution, 5200×6000 for 250-m resolution), it’s a simple matter to create a script to access the imagery, cut parts out, and paste them back together. For this task, I use ImageMagick, a useful, scriptable image manipulation package. The two grid sizes (950x1100 for sector USA3, 700x1100 for sector USA4) and offsets (+351+101 for sector USA3,+0+101 for sector USA4) were determined by trial and error that took about 5 minutes.

The first script below is for the 1-km resolution images. The second script below is for 250-m resolution images; it has a multiplication factor of 4 applied. These scripts are on a unix system, and it would be simple enough to put some date-enabling at the front end to make this useful in a cron. That exercise, as they say, is left to the reader.

wget http://ge.ssec.wisc.edu/viirs-today/images/noaa20/true_color/2020_08_03_216/n2.20216.USA3.143.1000m.jpg
wget http://ge.ssec.wisc.edu/viirs-today/images/noaa20/true_color/2020_08_03_216/n2.20216.USA4.143.1000m.jpg
convert -crop 950x1100+351+101 n2.20216.USA3.143.1000m.jpg test1_1000.jpg
convert -crop 700x1100+0+101 n2.20216.USA4.143.1000m.jpg test2_1000.jpg
convert test1_1000.jpg test2_1000.jpg +append n2.20216.GreatLakes.143.1000m.jpg

wget http://ge.ssec.wisc.edu/viirs-today/images/noaa20/true_color/2020_08_03_216/n2.20216.USA3.143.250m.jpg
wget http://ge.ssec.wisc.edu/viirs-today/images/noaa20/true_color/2020_08_03_216/n2.20216.USA4.143.250m.jpg
convert -crop 3800x4400+1401+401 n2.20216.USA3.143.250m.jpg test1_250.jpg
convert -crop 2800x4400+0+401 n2.20216.USA4.143.250m.jpg test2_250.jpg
convert test1_250.jpg test2_250.jpg +append n2.20216.GreatLakes.143.250m.jpg

The 1-km resolution True-Color imagery from NOAA-20 (‘n2’ in the file name; ‘n1’ is Suomi-NPP) for the Great Lakes on day 216 of the years 2020 (i.e., 3 August 2016, represented as 20216 in the url) is shown above; it spans sectors USA3 and USA4. The 250-m resolution image is here.


From an email received from South Carolina:  Our area is always split between USA7 and USA8, making it hard to use the sectorized imagery.

A script for that region, in this case for Day Night Band imagery, is below. This script accesses both Suomi-NPP and NOAA-20 data.

wget http://ge.ssec.wisc.edu/viirs-today/images/noaa20/night_dynamic/2020_08_05_218/n2.20218.USA8.DNBD.1000m.jpg
wget http://ge.ssec.wisc.edu/viirs-today/images/noaa20/night_dynamic/2020_08_05_218/n2.20218.USA7.DNBD.1000m.jpg
convert -crop 800x750+501+0 n2.20218.USA7.DNBD.1000m.jpg test1_1000.jpg
convert -crop 800x750+0+0 n2.20218.USA8.DNBD.1000m.jpg test2_1000.jpg
convert test1_1000.jpg test2_1000.jpg +append n2.20218.SOUTHCAROLINA.DNBD.1000m.jpg
wget http://ge.ssec.wisc.edu/viirs-today/images/snpp/night_dynamic/2020_08_05_218/n1.20218.USA8.DNBD.1000m.jpg
wget http://ge.ssec.wisc.edu/viirs-today/images/snpp/night_dynamic/2020_08_05_218/n1.20218.USA7.DNBD.1000m.jpg
convert -crop 800x750+501+0 n1.20218.USA7.DNBD.1000m.jpg test1_1000.jpg
convert -crop 800x750+0+0 n1.20218.USA8.DNBD.1000m.jpg test2_1000.jpg
convert test1_1000.jpg test2_1000.jpg +append n1.20218.SOUTHCAROLINA.DNBD.1000m.jpg

The Day Night Band Composite from NOAA-20 is shown below. Click here to see the Suomi-NPP image.

NOAA-20 Day Night Band composite, 5 August 2020 (Click to enlarge)

This technique would work with MODIS Today imagery as well.

Applying enhancements to Geo2Grid imagery

July 3rd, 2020 |

GOES-16 ABI Band 13 (10.3 µm) Infrared Imagery from 1550 UTC on 2 July 2020, with color enhancement added, and with annotated colorbar (Click to enlarge)

geo2grid (see this recent blog post) is a handy software package that allows anyone to create useful satellite imagery. Recent blog posts that document how to use geo2grid have shown visible or red-green-blue (RGB) composites. Individual infrared bands can also provide useful information, and this blog posts describes how you can create infrared imagery to which a useful color enhancement has been applied.

Step 1, as always, is to find data. NOAA CLASS has archived Level-1b Radiance files; these are the data that geo2grid expects, and the naming convention of the files is also what geo2grid expects. Amazon Cloud Services can also offer correctly-named data. (This blogger is aware of some users who have retrieved data from Google, and those files did not follow the expected naming convention).

The animation above compares an infrared image from 1550 UTC on 2 July over the northern part of the United States. The grid is described in this blog post and was used for convenience. (Of some importance: The grid created has dimensions of 1500×800). The original grey-scaled image is shown (with coastlines overlain), then a color enhancement is applied, and finally a script is run to annotate the colorbar.

The geo2grid command used to start the image creation was:

./geo2grid.sh -r abi_l1b -w geotiff -p C13 -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest -f /arcdata/goes_restricted/grb/goes16/2020/2020_07_02_184/abi/L1b/RadF/OR_ABI-L1b-RadF-M6C13_G16_s20201841550211_e20201841559531_c20201841600013.nc

This creates a Band 13 (the -p C13 flag) grey-scaled geotiff (-w geotiff) from data at the archive site specified ( -f /arcdata/goes_restricted/..../OR_ABI-L1b-RadF-M6C13_G16_s20201841550211_e20201841559531_c20201841600013.nc ), reading Advanced Baseline Imagery Level-1b imagery (the -r abi_l1b flag), and using nearest-neighbor interpolation to a pre-defined grid (“PADERECHO”) to transform the projection from the default satellite projection (That’s this part of the command: -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest). Note that by default, the geotiff created is 8-bit (in contrast to the 11-bit imagery; some information is lost.) You can create geotiffs with full bit depth. For this example an 8-bit image is being created.

The add_colorbar shell applies a color enhancement to the grey-scaled geotiff image that geo2grid creates. For the image in this blog post, I used this invocation:

./add_colormap.sh ../../enhancements/IR4AVHRR100_colortable.txt GOES-16_ABI_RadF_C13_20200702_155021_PADERECHO.tif

This colorbar must be pre-defined by the user. The format of the colortable (“IR4AVHRR100_colortable.txt”) is a comma-separated list of greyscale values (0-255 for the 8-bit geotiff that is created) followed by Red, Green, Blue and Transparency values. (Documentation is here. The file used for this example is here — courtesy of Jim Nelson, CIMSS) The colortable is applied to the geotiff image and a separate image (.png in this case) is created. You will note that this bi-linear enhancement is various shades of grey for bit values 0-152, then different colors for values 153-255.

How do you associate the color with a temperature? For a bi-linear enhancement as used for Band 13, this is not simple. With knowledge of the calibration used, or with access to a system (such as McIDAS-X or McIDAS-V) that includes the calibration, it is possible to associate the greyscale values with temperature values and then use annotation software such as ImageMagick to write in values, with a command such as:

convert INPUTFIELD_with_Coastline_Colorbar_Enhanced.png -fill white -pointsize 24 -annotate +997+28 "-30" InputField_with_CoastLine_Colorbar_enhanced_-30Label.png

This must be done for each label you want to add to the image, but a shell script does this easily, and once done, you have something that you can use for any image that is the same size: 1500 pixels wide. The spacing between the -30ºC, -40ºC, -50ºC… labels on the color bar is constant in this example. Thus, once two values are known and placed, it’s simple to compute the offsets for the other temperature values.

Polar2Grid colorbar functionality is a bit more comprehensive than Geo2Grid. When a colorbar is created in Polar2Grid, metadata on the minimum and maximum values are inserted into the geotiff. This allows automatic generation of colorbar labels (for a linear scaling only) as shown in this excellent example of documentation!

Using geo2grid to create animations

June 5th, 2020 |

GOES-16 Airmass RGB, 1400 UTC on 2 June 2020 to 1640 UTC on 3 June 2020

Severe weather occurred over the upper Midwest on 2 June 2020, and a derecho occurred over the mid-Atlantic States on 3 June 2020. These two events were linked, and the animation above shows the system moving and redeveloping from Minnesota on 2 June 202 eastward to Pennsylvania on 3 June 2020. How was this animation created?

geo2grid is a software packaged developed at CIMSS. It is designed to produce high-quality full-resolution imagery from archived (or real-time) GOES-16/GOES-17 Level-1b imagery. This blog post (a follow-up to this one) will outline how to use the software package to create imagery.  Note that geo2grid runs on CentOS6-compatible Linux.  There is no Windows version.  Here are the steps used to create the imagery above.

    1. geo2grid software can be downloaded as a g-zipped tarball from this link. (You may need to register — for free! — before downloading). Version 1.0.1 was released in March of 2020. You can also download documentation (always a good idea) as a pdf file from the download site, or you can access it online.  Once you have unzipped and untarred the software package, you’re ready to begin.  Your unix system must know where the software package sits, and that’s through this command: EXPORT GEO2GRID_HOME=/path/to/Geo2GridSoftwareLocation/
    2. Of course, you will also need data.  For Advanced Baseline Imager (ABI) data, geo2grid expects Radiance fields.  These can be accessed via NOAA CLASS, or via “The Cloud”, or from a GRB Receiving antenna.  (some data sources are listed here)  NOAA CLASS ABI L1b Radiances data (ordered under “GOES-R Series ABI Products (GRABIPRD) (partially restricted L1b and L2+ Data Products) ”  )  will have a file format that looks something like this:  OR_ABI-L1b-RadF-M6C08_G16_s20201542340169_e20201542349477_c20201542349544.nc ; that particular file listed is Full Disk (the ‘F’ in ‘RadF’) when GOES-16 was in Mode 6 (M6) scanning, and it contains Band 8 (C08);  the start time is Year 2020, Day 154 (June 2 2020) at 23:40:16.
    3. As noted in this blog post, you can specify which fraction of the domain to display. geo2grid also has a mapping routine if you want to define your own domain (i.e., not a satellite projection, or a subset of that satellite projection), and documentation is under ‘Utility Scripts’ at the geo2grid documentation site.  To define a domain near the Great Lakes, for example, I used this command:   $GEO2GRID_HOME/bin/p2g_grid_helper.sh PADERECHO -83.0 45.0 2000 -2000 1500 800 > $GEO2GRID_HOME/PADERECHO.conf ;  the shell scripts takes a grid name (PADERECHO in this case), a center longitude/latitude (83.0 degrees West, 45.0 Degrees North), x- and y- grid-spacing in meters (2 km for this case), and the number of points in the x- and y- directions.  These mapping instructions are placed in a configuration file;  cat $GEO2GRID_HOME/PADERECHO.conf will show this line in the file:  PADERECHO, proj4, +proj=lcc +datum=WGS84 +ellps=WGS84 +lat_0=45.00000 +lat_1=45.00000 +lon_0=-83.00000 +units=m +no_defs, 1500, 800, 2000.00000, -2000.00000, -104.24668deg, 50.40805deg .  It might take some trial and error using geo2grid to find the map domain that suits you best, but it is a simple matter to iterate to a solution.
    4. Next, run geo2grid.   The command I invoked (with small changes for each time) is here:  $GEO2GRID_HOME/bin/geo2grid.sh -r abi_l1b -w geotiff -p airmass -g PADERECHO --grid-configs $GEO2GRID_HOME/PADERECHO.conf --method nearest --cache-dir=$GEO2GRID_HOME/datacache/ -f /data-hdd/PADerecho/Day154/14/*1450*.nc   There are several flags included in this call, and they are explained in the documentation, but also here.
      • -r:  What kind of data are being read?  In this case, level-1b data from ABI
      • -w:  Output format (geotiff).  That is the only option for imagery
      • -p:  What imagery should be created?  In this case, airmass RGB.  To find out what can be computed given the data present, user the –list-products flag.  For the data I have downloaded, this returned C08, C10, C12, C13, C15, airmass:  I had downloaded to the specified directory the components necessary to compute the airmass RGB (plus band 15).  If you wanted to create imagery for all these channels in addition to the airmass RGB, -p airmass C08 C10 C12 C13 C15 would work.  (Note:  C15 is not actually needed in the computation of the airmass RGB).
      • -g:  to what grid should these data be interpolated?  The answer: the PADERECHO grid that is defined and specifed where –grid-configs points to:  $GEO2GRID_HOME/PADERECHO.conf;
      • –method:  what interpolation method should be used?  For this case, nearest neighbor is specified.
      • –cache-dir:  This is used to speed processing.  If you have multiple files being interpolated to the same grid, it speeds things to save the interpolation methods/points.  You have to specify the directory where this file will sit.
      • -f:  Where do the Radiance files sit?  In this example I have placed different times in separate directories.  I haven’t yet figured out how to use Geo2Grid that points to a directory where multiple times sit.
    5. geo2grid includes scripts (“add_coastlines.sh“;  for full documentation, see the ‘Utility Scripts’ section in the geo2grid documentation)  that will add maps to the imagery.  $GEO2GRID_HOME/bin/add_coastlines.sh --add-coastlines --coastlines-resolution=h --coastlines-level=4 GOES-16_ABI_RadF_airmass_20200602_152017_PADERECHO.tif  This command invoked only a few of the mapping additions, as described below.  For a full list, refer to the documentation.
      1. –add-coastlines ;  as you might expect this flag adds coastlines.  In the animation above, you’ll see that the coastlines include lakes.  You can also add borders, grids (lat/lon lines) and other features.
      2. –coastline-resolution=h ; I have asked for high resolution.
      3. –coastlines-level=4 ; This is the default level of detail in the lines.
      4. The .tif is the geotiff created from Geo2Grid.  When add_coastlines.sh is finished, the filename GOES-16_ABI_RadF_airmass_20200602_152017_PADERECHO.png is created.
    6. geo2grid does not annotate imagery.  I use ImageMagick for that.  (something like this:  convert GOES-16_ABI_RadF_airmass_20200603_163017_PADERECHO.png -gravity Northeast -fill white -pointsize 32 -annotate +8+40 "3 June 2020 1630 UTC" GOES-16_ABI_RadF_airmass_20200603_163017_PADERECHOannotate.png )

A challenge in scripting geo2grid is that the start time of the files is not constant.  For example, on 2 June, the 15:20 image starts at 15:20:17.1 (as shown above, the file name includes 152017) ; the 17:20 image starts at 17:20:16.9 and the created filenames include 172016.

The completed animation is shown above.

When is an ABI hot (bright) spot not a fire?

May 30th, 2020 |

An ABI hot (bright) spot is not a fire when it’s a fleet of solar farms. For example, recall the CIMSS Satellite Blog entry regarding solar farms in California. 

ABI band 2 visible

ABI band 2 visible animation on May 30, 2020 (mostly) in southeastern Minnesota. Click to play mp4.

Note how some reflections are so bright that the ABI reports dark surrounding pixels. This is part of the remapping process from detector to pixel space. 

 

9-panel

A multiple-spectral ABI comparison on May 30, 2020. The rows are: band 2, band 5, band 6 band 7, band 7 – 14 brightness temp, band 14 fire mask, band 7-14 radiance difference, band 7-14 radiance difference minus the rolling average

From left to right, top to bottom the panels are:
1) ABI band 2 reflectance, dynamically scaled to enhance contrast (will appear to flicker)
2) ABI band 5 reflectance, dynamically scaled to enhance contrast (will appear to flicker)
3) ABI band 6 reflectance, dynamically scaled to enhance contrast (will appear to flicker)
4) ABI band 7 brightness temperature, dynamically scaled to enhance contrast (will appear to flicker)
5) ABI band 7 minus band 14 brightness temperature. Red indicates positive values (extra thermal energy due to the sun and fires, if present), dynamically scaled to enhance contrast (will appear to flicker)
6) ABI band 14 brightness temperature, dynamically scaled to enhance contrast (will appear to flicker)
7) ABI Fire Detection and Characterization Algorithm (FDCA, aka WFABBA) fire detection metadata mask.  Fires are red, orange, magenta, and shades of blue indicating different confidence levels.  Green indicates fire-free land, shades of gray indicate clouds, dark  blue indicates water.
8) Radiance difference of band 7 minus band 14 radiance in band 7 space.  Red indicates positive values (extra thermal energy due to the sun and fires, if present), dynamically scaled to enhance contrast (will appear to flicker)
9) Radiance difference of band 7 minus band 14 radiance in band 7 space minus a rolling average of the 5 prior frames, to highlight changes. Red indicates positive values (extra thermal energy due to the sun and fires, if present), dynamically scaled to enhance contrast (will appear to flicker).

Aside from the solar farms, water clouds show up in the difference panels due to their reflection of shortwave radiation. 

H/T to Chris Schmidt for the 9-panel ABI imagery.  More about quantitative ABI products, including fire detection. 

The original tweet from the La Crosse WFO: “We saw some awfully bright looking “clouds” showing up via satellite in southeast Minnesota earlier this afternoon. Well after some investigation, we were able to determine they were actually solar panel arrays that the sun was hitting just right!”

NWS tweet

Solar farms and GOES-16 ABI visible imagery from the La Crosse NWS WFO.