Direct Broadcast software packages – level 0 to level 2 and visualisation

Direct Broadcast packages, level 0 to level 1

Name	Version	Capabilities	Output format	Reference
RT-STPS	6.0	CCSDS data processing package, Java application. Outputs Raw Data Records (RDRs). Supports Aqua, Terra, NPP, JPSS series	hdf5	NASA Direct Readout Laboratory
metopizer	3.51	The Metopizer is a collection of tools for manipulating CCSDS Instrument Source Packets and other related data types (CADU, t-VCDU packets, Metop L0 products).	EPS Level 0	EUMETSAT – support software
MODISL1DB_SPA	1.8	Level 1 processing (Science Processing Algorithm) for Aqua/Terra MODIS from NASA DRL. Uses the v6.1 series LUTs but note that these do not reflect the latest Terra calibration.	hdf-eos	NASA Direct Readout Laboratory
ocssw_7.5 (part of SeaDAS)	7.5	Level 1 processing for Aqua/Terra MODIS from SeaDAS. A couple of issues: (i) the supplied version fails to download the LUTs through some proxy servers; see Appendix A for a fix, (ii) Land sea mask is empty in the MOD03. A workaround for (ii) is to use MODISL1DB to generate the L1A and ocssw_7.5 to process to L1B.	hdf-eos	NASA SeaDAS
IMAPP: Aqua AIRS/AMSU/HSB Processing Package for Direct Broadcast	6.0	Level 1 processing for Aqua AIRS and AMSU	hdf-eos or BUFR	CIMSS – IMAPP
IMAPP: Aqua AMSR-E Calibrated and Geolocated Antenna Temperatures	1.0	Processes Aqua direct broadcast AMSR-E data from raw packets in PDS (Production Data Set) format into Level 2A calibrated data. No longer applicable to real-time DB, due to failure of instrument	hdf-eos	CIMSS – IMAPP
Community Satellite Processing Package (CSPP) – SDR software	3.1.1	Level 1 processing for Suomi NPP and JPSS series: ATMS, CrIS, VIIRS. Based on Raytheon’s ADL software, with python and bash wrappers.	hdf5	CIMSS – CSPP
International Polar Orbiter Processing Package (IPOPP)	3.0	Level 1 processing for Suomi NPP and JPSS series: ATMS, CrIS, VIIRS, OMPS, and Aqua/Terra MODIS. NPP/JPSS processing is based on Raytheon’s ADL software	hdf5	NASA Direct Readout Laboratory
ATOVS and AVHRR Pre-processing Package (AAPP)	8.3	Level 0 to level 1 processing for NOAA POES and Metop: AMSU, MHS, HIRS, AVHRR, IASI. Also provides re-mapping and pre-processing for S-NPP/JPSS and FY-3 sounders, and VIIRS/AVHRR cloud masks.	Binary fixed length records (1b formats are similar to NOAA’s) or BUFR or hdf5	NWP SAF – AAPP
FY3L0pp FY3CL0pp	1.4 1.1.1	CADU to level 0 processing for FY-3 MWTS, MWHS, IRAS, MWRI, VIRR, MERSI. Note: assumes that the incoming CADU data have been de-randomised. See Appendix B below.	hdf5	CMA – NSMC >Support>Tools or CMA Portal >Tools>Softwares
FY3L1pp FY3CL1pp	1.4 1.2.0 patch 1	level 0 to level 1 processing for FY-3 MWTS, MWHS, IRAS, MWRI, VIRR, MERSI	hdf5	As for FY3L0pp

Table last updated: 15th January 2019

Other level 1 processing packages (not DB)

Name	Version	Capabilities	Output format	Reference
SSMIS Unified Pre-processor (UPP)	N/A	SSMIS re-mapping to common field of view, and removal of instrument biases due to solar intrusions and antenna emmisivity. Joint Met Office/NRL development. Run operationally by NRL, Monterey.	BUFR
SSMIS_PP	1.1	The original Met Office processor for SSMIS. Supported DMSP F16 only. Now largely superseded by the UPP.	BUFR	NWP SAF – SSMIS_PP
SSMIS_PP UPP averaging module	1.1	Super-obbing module for SSMIS. Supports DMSP F16, 17, 18, 19. Intended to be run by users of the unaveraged UPP data	BUFR	NWP SAF – SSMIS_PP
MWIPP		Microwave imager processor from the NWP SAF, currently supporting SSMIS, AMSR-2, GMI, MWRI. With averaging and format conversion capabilities. Initial release in early 2019.	BUFR, hdf5	NWP SAF – MWIPP

Table last updated: 15th January 2019

Level 2 processing packages

Name	Version	Capabilities	Reference
International ATOVS Processing Package (IAPP)	1.1	Temperature and humidity retrievals from ATOVS. Uses HIRS level 1d (from AAPP) as input. Released as part of CSPP.	CIMSS – CSPP
CrIS, AIRS and IASI Hyperspectral Retrieval Software	2.0	Dual-regression retrieval algorithm developed at CIMSS. Released as part of CSPP.	CIMSS – CSPP
NOAA/NESDIS/STAR Microwave Integrated Retrieval System (MIRS)	2.2	Retrievals from AMSU, MHS and ATMS. Released as part of CSPP.	CIMSS – CSPP
NOAA Unique CrIS ATMS Processing System (NUCAPS)	2.1	Retrievals from CrIS/ATMS, IASI/AMSU, AIRS. Released as part of CSPP.	CIMSS – CSPP
VIIRS EDRs	2.0	VIIRS SDR to EDR. Released as part of CSPP.	CIMSS – CSPP
NOAA Clouds from AVHRR Extended (CLAVR-x) Retrieval Software Package in Support of VIIRS, MODIS and AHVRR	2.0	CLAVRx software produces single Field-of-View (FOV) cloud retrievals in HDF4 format from input AVHRR, MODIS, and VIIRS Imager calibrated/geolocated data files. Released as part of CSPP.	CIMSS – CSPP
NOAA/STAR Advanced Clear-Sky Processor for Oceans (ACSPO)	1.1	For retrieving sea surface temperature from direct broadcast (DB) AVHRR, MODIS and VIIRS sensor data records. Released as part of CSPP.	CIMSS – CSPP
CSPP NOAA JPSS VIIRS Active Fire Software	1.0	NOAA operational version of the Active Fire software ported for application to direct broadcast inputs. Released as part of CSPP.	CIMSS – CSPP
CSPP VIIRS Flood Detection Software	1.1	Automatic flood detection using Suomi-NPP and NOAA-20 VIIRS Sensor Data Record (SDR) inputs, as well as the S-NPP VIIRS Cloud Mask Environmental Data Record (EDR) if it is available.	CIMSS – CSPP
International MODIS/AIRS Processing Package (IMAPP)	Various	Produces a range of MODIS, AIRS and AMSU products. Available as Linux binaries or as a Virtual Appliance for Windows, OS X, and Linux. Note that the IMAPP project ended in 2017 but the web site will remain available.	CIMSS – IMAPP
Scatterometer wind processors	AWDPv3.2, PenWPv2.2	Wind processors for ASCAT, OSCAT, HSCAT, RapidScat and SeaWinds. Input is the level 1b Normalized Radar Cross Section.	NWP SAF – Scatterometer
1D-Var schemes	1.1.1	The NWPSAF 1D-Var package is a stand-alone system for doing 1D-Var retrievals with user-provided files of observation and background error covariances. The package is configured to support ATOVS, AIRS, IASI, ATMS, CrIS and SSMIS sensors, and is capable of supporting any nadir-sounding sensor for which a suitable radiative transfer model is available.	NWP SAF – 1DVAR
Inversion Coupled with Imager (ICI)		Temperature and humidity retrievals, making use of AVHRR cloud mask. Package was developed by Meteo-France, and was distributed by EUMETSAT prior to 2004, but no longer supported.	See ITWG description
Improved Initialization Inversion system (3I)		HIRS and MSU retrieval system, developed by LMD. The 3I Algorithm is available as a freeware product, for academic use or scientific research.	LMD – 3I
International TOVS Processing Package (ITPP)		Predecessor of IAPP – for TOVS.	No longer available

Table last updated: 30 January 2019

Visualisation and analysis tools

Name	Version	Capabilities	Supported formats and sensors	Reference
McIDAS (Man computer Interactive Data Access System)	McIDAS-V 1.8 McIDAS-X 2018.2	McIDAS displays weather satellite (including hyperspectral) and other geophysical data in 2- and 3-dimensions, and can be used to analyze and manipulate the data with its powerful mathematical functions	List of supported data formats and products in the User Guide : section Data Formats and Sources	SSEC – McIDAS
OpenADDE	–	This software is discontinued		SSEC
VisAD (“Visualization for Algorithm Development” )	Now part of McIDAS-V	Java component library for interactive and collaborative visualization and analysis of numerical data: A general mathematical data model that can be adapted to virtually any numerical data, that supports data sharing among different users, different data sources and different scientific disciplines, and that provides transparent access to data independent of storage format and location (i.e., memory, disk or remote). A general display model that supports interactive 3-D, data fusion, multiple data views, direct manipulation, collaboration, and virtual reality. The display model has been adapted to Java3D and Java2D and used in an ImmersaDesk virtual reality display. Data analysis and computation integrated with visualization to support computational steering and other complex interaction modes.	netCDF, HDF-5, FITS, HDF-EOS, McIDAS, Vis5D, GIF, JPEG, TIFF, QuickTime, ASCII and many other file formats	SSEC – VisAD
Hydra2	2.0	HYDRA2 Multispectral Data Analysis Toolkit for the visualization and analysis of Suomi-NPP, JPSS and NASA EOS Aqua and Terra instrument data. This package is in support of Direct Broadcast users, and is co-sponsored by the NOAA Community Satellite Processing Package (CSPP) and the NASA International MODIS/AIRS Processing Package (IMAPP). HYDRA2 is based on VisAD. Written in Java/Jython.	S-NPP/NOAA-20: VIIRS, CrIS and ATMS Aqua: MODIS and AIRS Terra: MODIS Metop-A and B: IASI	SSEC- CIMSS – CSPP wep pages
polar2grid	2.2.1	software package providing scripts, utilities, and a series of functions and classes for reading polar-orbiting satellite data, remapping or gridding that data, and writing it to a file format to be used by other software	VIIRS (SDR, hdf5), MODIS (Level 1, hdf), corrected reflectance (corrected reflectance files created from above two)	SSEC – polar2grid
CSPP Sounder QuickLook (QL) Software	1.0	Plots for level 2 products of the following CSPP packages: NUCAPS, HSRTV, MIRS, IAPP		SSEC – sounder_ql
IDV (Integrated Data Viewer)	5.5	IDV is a freely available, multiplatform visualization and analysis tool for interdisciplinary geoscience data. The IDV brings together the ability to display and work with a wide range of data including satellite imagery, gridded data, observations and radar data	list available here	Unidata – IDV
GEMPAK	7.4.5	The GEneral Meteorology PAcKage, is an analysis, display, and product generation package for meteorological data.		Unidata – GEMPAK
AWIPS-II	18.1.1	AWIPS II is a weather forecasting, display and analysis package currently being developed by the National Weather Service. AWIPS II is a Java application consisting of a data rendering client and a backend data server, and incorporates open source technologies such as HDF5, Qpid,Postgres and PyPIES..		Unidata – AWIPS-II
AAPP	8.3	AAPP is intended for the ingestion and the pre-processing tasks of direct readout HRPT data up to the pre-processed, calibrated and navigated data. The AAPP distribution includes scripts for visualization with GMT. Additionally, the NWP SAF can provide IDL and PV-Wave code for visualization of AAPP output (not an official NWP SAF deliverable)	Following satellites are supported: TIROS-N to NOAA-14 (HIRS/2, MSU, AVHRR/2), NOAA-KLM (HIRS/3, AMSU-A, AMSU-B, AVHRR/3), NOAA-NN’ (HIRS/4, AMSU-A, MHS, AVHRR/3), METOP (HIRS/4, AMSU-A, MHS, AVHRR/3, IASI), NPP/JPSS (ATMS, CrIS, VIIRS), FY3 (MWTS, MWHS, IRAS)	NWP SAF – AAPP – Visualisation
SeaDAS	7.3.1	The SeaWiFS Data Analysis System (SeaDAS) is a comprehensive image analysis package for the processing, display, analysis, and quality control of ocean color data	supported sensors: MODIS, SeaWiFS, MERIS, OCTS, and CZCS, VIIRS. HICO, Aquarius, OCM, OCM-2, OSMI, MOS	NASA – SeaDAS
CDO (Climate Data Operators)	1.7.2	CDO is a collection of command line operators to manipulate and analyze climate and forecast model data from different sources. There are more than 600 different operators available in this software package (e.g. for file operations, simple statistics, arithmetics, interpolation, etc)	GRIB 1/2, netCDF 3/4, SERVICE, EXTRA and IEG, interface for binary files, CM SAF hdf5-files	ZMAW – CDO or CM SAF – Products – Tools – CDO
Giovanni – Interactive Visualization and Analysis	Giovanni-3 (retired August 2015), Giovanni-4 (v4.28)	web-based application for visualization, analysis and accesss if vasta mouts if remote sensing data without having to download the data.	List of paramters and used satellite data available via web page	NASA – Giovanni
GDAL		translator library for raster geospatial formats, comes with variety of useful commandline utilities for data translation and processing	GeoTIFF, Erdas Imagine, SDTS, ECW, mrSID, JPEG2000, GRIB, etc, full list available for raster formats and vector formats	GDAL
Panoply	4.10.3	cross-platform application which plots geo-gridded arrays from netcdf, HDF and GRIB datasets	netcdf, HDF, GRIB	NASA – Panoply
hdfview	3.0	visual tool for browsing and editing HDF-files	HDF4, HDF5	HDF Group
ncview	2.1.7	visual browser for netCDF files	netcdf	UCSD
R	3.5.2	R is a free software environment for statistical computing (e.g., linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphics	Several data formats. For some, additional packages need to be installed (e.g. netcdf, hdf)	R Project
pytroll	several modules: pyresample, mipp, mpop, python-bufr, py-coast, pyorbital, + several other modules under development	pytroll provides different free and open source python modules for the reading, interpretation, and writing of weather satellite data.	Meteosat series (tested with 7, 8, 9, 10) GOES series, in HRIT/LRIT format (tested with 11, 12, 13, 15) MTSAT series, in HRIT/LRIT format (tested with 1R, 2) Electro L, in HRIT/LRIT format (tested with N1) NOAA series, in AAPP, GAC and LAC format (tested with 15, 16, 17, 18, 19) Metop-A/B, in EPS 1a and 1b format Aqua and Terra, in hdf-eos format Suomi NPP and JPSS series, in SDR hdf5 format TerraSAR-X Radarsat-2 SAR COSMO-SkyMed SAR Sentinel-1 SAR	Pytroll
MeteoExplorer (ME)	1.3, 1.4 (beta)	Cross-platform software for analyzing and rendering atmospheric science and geoscience data.	WMO GRIB1/GRIB2, NetCDF, HDF and MICAPS, and basic GIS functionalities	MeteoExplorer by EastModelSoft
IRIS and Cartopy	2.2 and 0.17	Iris and Cartopy are open source Python libraries initially created by the Met Office to enable the visualisation of weather and climate data.		SciTools
WMO link for data processing tools tools				WMO – Data Processing Tools
NASA link for data tools				NASA – Data tools
ESA tools		Miscellaneous tools for manipulating satellite data		ESA – command line tools

Table last updated: 30 January 2019

Original authors: Nathalie Selbach and Nigel Atkinson

Appendix A: Modification to file ProcUtils.py in ocssw_7.5

This is the modified version of function httpinit that is used at the Met Office to set up a tunnel through a proxy server, defined in environment variable https_proxy …

def httpinit(url, timeout=10, urlConn=None):
    """
    initialize HTTP network connection
    """
    import httplib
    import os
    from urlparse import urlparse

    proxy = None
    proxy_set = os.environ.get('https_proxy')
    if proxy_set is None:
        proxy_set = os.environ.get('http_proxy')
    if proxy_set:
        proxy = urlparse(proxy_set)

    if urlConn is None:
        #if proxy is None:
        #    urlConn = httplib.HTTPSConnection(url, timeout=timeout)
        #elif proxy.scheme == 'https':
        #    urlConn = httplib.HTTPSConnection(proxy.hostname,
        #                                      proxy.port, timeout=timeout)
        #else:
        #    urlConn = httplib.HTTPConnection(proxy.hostname,
        #                                     proxy.port, timeout=timeout)

        urlConn = httplib.HTTPSConnection(proxy.hostname,
                                              proxy.port, timeout=timeout)
        urlConn.set_tunnel(url,443)

    return urlConn, proxy

Appendix B: CADU manipulation

Raw data from direct broadcast are typically in CADU (Channel Access Data Unit) format. As an example, see Figure 2.1 of the FY-3A Satellite to Ground Interface Control
Document. The frame length is typically 1024 bytes, but you can verify this by looking for the first few occurrences of the frame sync:

od -t x1 -A d $infile | grep "1a cf fc 1d" | head -5

Common to all missions is a randomization, or scrambling, step – see Figure 2-3 of the FY-3A document. This, in effect, does an exclusive or of the input stream with a 255-bit pseudo-random sequence. The randomization and de-randomization processes are identical. For DB stations that deliver CADU data, de-randomization may or may not have been applied in the station processing; similarly, some level 0 software packages expect to receive scrambled data (e.g. metopizer tool cadu_to_ccsds) while others expect unscrambled data (e.g. FY3CL0pp).

If you have difficulty processing your CADU data, try converting your files. One option is to use the ESA tool CADU-convert, while another option is to build the short C program listed below.

/* cadu_scramble.c
   Scramble or unscramble a CADU file (e.g. Metop or FY-3) 
   Frame length normally 1024 bytes but MEOS files are 1072 bytes. Check your frame length before running.
   
   Note that the Metopizer tool cadu_to_ccsds ingests scrambled (randomized)
   files but fy3cl0db ingests unscrambled files.

   Build: cc -o cadu_scramble cadu_scramble.c
   Usage: ./cadu_scramble infile outfile [framelength]
   
   Reference: Fig 2-3 of http://mdkenny.customer.netspace.net.au/FY3_HRPT.pdf

   04/01/2018 Nigel Atkinson
*/


#include <stdio.h>
#include <stdlib.h>

main(int argc, char *argv[]) {

#define MAXFRAME 1072
#define MAXMASK MAXFRAME-4

  int i,j,k,framelength,masklength;
  unsigned char sequence[8] = { 1,1,1,1,1,1,1,1 };
  unsigned char mask[MAXMASK] = { 0 };
  unsigned char tmp1, tmp2, tmp3;
  
  unsigned char frame[MAXFRAME];
  FILE *fpin, *fpout;

/* open the files */

  if (argc < 3) {
    printf("usage: cadu_scramble infile outfile [framelength]\n");
    exit (0);
  }
  fpin = fopen(argv[1],"r");
  fpout = fopen(argv[2],"w+");
  if (argc > 3) sscanf(argv[3],"%d",&framelength); else framelength=1024;
  masklength = framelength - 4;
  
/* set up the scrambling mask */
  
  for (i=0; i<masklength; i++) {
    for (j=0; j<8; j++) {
      mask[i] =  (mask[i]<<1) + sequence[0];
      tmp1 = sequence[3] ^ sequence[0];       /* exclusive or */
      tmp2 = sequence[5] ^ tmp1;
      tmp3 = sequence[7] ^ tmp2;
      for (k=0; k<7; k++) sequence[k] = sequence[k+1];
      sequence[7] = tmp3;
    }
  }

/* read, modify and write, framelength bytes at a time */

  while (fread(frame,1,framelength,fpin) == framelength){
    for (i=0; i<masklength; i++) {
      frame[i+4] = frame[i+4] ^ mask[i];   /* leave first 4 bytes unchanged */
    }
    fwrite(frame,1,framelength,fpout);
  }
  
  fclose(fpin);
  fclose(fpout);  
}