Directory of software packages and tools for Direct Broadcast Level 0, 1, and 2 processing, visualization, and analysis

Contents

Direct Broadcast packages: Level 0 and Level 1
Other Level 1 processing packages (not DB)
Level 2 processing packages
Visualization and Analysis tools
Appendix
- Appendix A: Modification to file ProcUtils.py in ocssw_7.5
- Appendix B: CADU manipulation

Original authors: Nathalie Selbach and Nigel Atkinson
Page last updated by: Liam Gumley

Please email us (with the subject “PSWG – package update request”) to request page updates.

Direct Broadcast packages: Level 0 and Level 1

Name	Capabilities	Output format	Reference
RT-STPS	CCSDS decoding software. Outputs CCSDS Level 0 or Raw Data Records (RDRs). Supports Aqua, Terra, NPP, JPSS series.	CCSDS Level 0, HDF5	CIMSS – CSPP
Metopizer	Collection of tools for manipulating CCSDS Instrument Source Packets and other related data types (CADU, t-VCDU packets, Metop L0 products).	EPS Level 0	EUMETSAT – support software
IMAPP: Aqua AIRS/AMSU/HSB Level 1 Processing Package for Direct Broadcast	Level 1 processing for Aqua AIRS and AMSU.	HDF4, BUFR	CIMSS – IMAPP
IMAPP: MODIS Level 1 Processing Package for Direct Broadcast	Level 1 processing for Aqua and Terra MODIS.	HDF4	CIMSS – IMAPP
CSPP: SDR	Level 1 processing for Suomi NPP and JPSS series: ATMS, CrIS, VIIRS, OMPS. Based on Raytheon’s ADL software, with python and bash wrappers.	HDF5	CIMSS – CSPP
ATOVS and AVHRR Pre-processing Package (AAPP)	Level 1 processing for NOAA POES and Metop: AMSU, MHS, HIRS, AVHRR, IASI. Also provides re-mapping and pre-processing for S-NPP/JPSS and FY-3 sounders, VIIRS/AVHRR cloud masks; and conversion to BUFR.	Binary fixed length records (1b formats are similar to NOAA’s) or BUFR or HDF5	NWP SAF – AAPP
FY3: Level 0 Processor	Level 0 processing for FY-3 MWTS, MWHS, IRAS, MWRI, VIRR, MERSI. Note: assumes that the incoming CADU data have been de-randomised. See Appendix B below.	HDF5	Contact iusc_nsmc@cma.gov.cn for application form
FY3: Level 1 Processor	Level 1 processing for FY-3 MWTS, MWHS, IRAS, MWRI, VIRR, MERSI	HDF5	Contact iusc_nsmc@cma.gov.cn for application form

Table last updated: 2 May 2025

Name	Capabilities	Supported formats and sensors	Reference
McIDAS (Man computer Interactive Data Access System)	McIDAS (McIDAS-V and McIDAS-X) displays weather satellite (including hyperspectral) and other geophysical data in 2- and 3-dimensions, and can be used to analyze and manipulate the data with its powerful mathematical functions	List of supported data formats and products in the User Guide : section Data Formats and Sources	SSEC – McIDAS
CSPP: Hydra	HYDRA2 Multispectral Data Analysis Toolkit for the visualization and analysis of Suomi-NPP, JPSS and NASA EOS Aqua and Terra instrument data. Note: no longer supported.	S-NPP/NOAA-20: VIIRS, CrIS and ATMS Aqua: MODIS and AIRS Terra: MODIS Metop-A and B: IASI	CIMSS – CSPP
CSPP: Polar2grid	Python-based command-line software for transforming Level 1 and Level 2 products in swath projection to high quality images in GeoTIFF format.	VIIRS, MODIS, AVHRR, MERSI, and many Level 2 products	CIMSS – CSPP
CSPP: Sounder QuickLook	Plots for level 2 products of the following CSPP packages: NUCAPS, HSRTV, MIRS, IAPP	CrIS, ATMS, IASI, AMSU, AIRS	CIMSS – CSPP
IDV (Integrated Data Viewer)	IDV is a freely available, multiplatform visualization and analysis tool for interdisciplinary geoscience data. The IDV brings together the ability to display and work with a wide range of data including satellite imagery, gridded data, observations and radar data	See website	Unidata – IDV
GEMPAK	The GEneral Meteorology PAcKage, is an analysis, display, and product generation package for meteorological data.	See website	Unidata – GEMPAK
AWIPS-II	AWIPS II is a weather forecasting, display and analysis package currently being developed by the National Weather Service. AWIPS II is a Java application consisting of a data rendering client and a backend data server, and incorporates open source technologies such as HDF5, Qpid,Postgres and PyPIES..	See website	Unidata – AWIPS-II
AAPP	AAPP is intended for the ingestion and the pre-processing tasks of direct readout HRPT data up to the pre-processed, calibrated and navigated data. The AAPP distribution includes scripts for visualization with GMT. Additionally, the NWP SAF can provide IDL and PV-Wave code for visualization of AAPP output (not an official NWP SAF deliverable)	Following satellites are supported: TIROS-N to NOAA-14 (HIRS/2, MSU, AVHRR/2), NOAA-KLM (HIRS/3, AMSU-A, AMSU-B, AVHRR/3), NOAA-NN’ (HIRS/4, AMSU-A, MHS, AVHRR/3), METOP (HIRS/4, AMSU-A, MHS, AVHRR/3, IASI), NPP/JPSS (ATMS, CrIS, VIIRS), FY3 (MWTS, MWHS, IRAS)	NWP SAF – AAPP – Visualisation
SeaDAS	The SeaWiFS Data Analysis System (SeaDAS) is a comprehensive image analysis package for the processing, display, analysis, and quality control of ocean color data	supported sensors: MODIS, SeaWiFS, MERIS, OCTS, and CZCS, VIIRS. HICO, Aquarius, OCM, OCM-2, OSMI, MOS	NASA – SeaDAS
CDO (Climate Data Operators)	CDO is a collection of command line operators to manipulate and analyze climate and forecast model data from different sources. There are more than 600 different operators available in this software package (e.g. for file operations, simple statistics, arithmetics, interpolation, etc)	GRIB 1/2, netCDF 3/4, SERVICE, EXTRA and IEG, interface for binary files, CM SAF hdf5-files	ZMAW – CDO or CM SAF – Products – Tools – CDO
Giovanni – Interactive Visualization and Analysis	web-based application for visualization, analysis and accesss if vasta mouts if remote sensing data without having to download the data.	List of paramters and used satellite data available via web page	NASA – Giovanni
GDAL	translator library for raster geospatial formats, comes with variety of useful commandline utilities for data translation and processing	GeoTIFF, Erdas Imagine, SDTS, ECW, mrSID, JPEG2000, GRIB, etc, full list available for raster formats and vector formats	GDAL
Panoply	cross-platform application which plots geo-gridded arrays from netcdf, HDF and GRIB datasets	netcdf, HDF, GRIB	NASA – Panoply
hdfview	visual tool for browsing and editing HDF-files	HDF4, HDF5	HDF Group
ncview	visual browser for netCDF files	netcdf	UCSD
R	R is a free software environment for statistical computing (e.g., linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphics	Several data formats. For some, additional packages need to be installed (e.g. netcdf, hdf)	R Project
pytroll	pytroll provides different free and open source python modules for the reading, interpretation, and writing of weather satellite data. (includes several modules: pyresample, mipp, mpop, python-bufr, py-coast, pyorbital, + several other modules under development)	Meteosat series (tested with 7, 8, 9, 10) GOES series, in HRIT/LRIT format (tested with 11, 12, 13, 15) MTSAT series, in HRIT/LRIT format (tested with 1R, 2) Electro L, in HRIT/LRIT format (tested with N1) NOAA series, in AAPP, GAC and LAC format (tested with 15, 16, 17, 18, 19) Metop-A/B, in EPS 1a and 1b format Aqua and Terra, in hdf-eos format Suomi NPP and JPSS series, in SDR hdf5 format TerraSAR-X Radarsat-2 SAR COSMO-SkyMed SAR Sentinel-1 SAR	Pytroll
MeteoExplorer (ME)	Cross-platform software for analyzing and rendering atmospheric science and geoscience data.	WMO GRIB1/GRIB2, NetCDF, HDF and MICAPS, and basic GIS functionalities	MeteoExplorer by EastModelSoft
IRIS and Cartopy	Iris and Cartopy are open source Python libraries initially created by the Met Office to enable the visualisation of weather and climate data.	See website	SciTools
ESA tools	Miscellaneous tools for manipulating satellite data	See website	ESA – command line tools

Table last updated: 2 May 2025

Name	Capabilities	Output format	Reference
SSMIS Unified Pre-processor (UPP)	SSMIS re-mapping to common field of view, and removal of instrument biases due to solar intrusions and antenna emmisivity. Joint Met Office/NRL development. Run operationally by NRL, Monterey.	BUFR
SSMIS_PP (deprecated, see MWIPP)	The original Met Office processor for SSMIS. Supported DMSP F16 only. Now largely superseded by the UPP.	BUFR	NWP SAF – SSMIS_PP
SSMIS_PP UPP averaging module	Super-obbing module for SSMIS. Supports DMSP F16, 17, 18, 19. Intended to be run by users of the unaveraged UPP data	BUFR	NWP SAF – SSMIS_PP
MWIPP	Microwave Imager Pre-Processor from the NWP SAF, currently supporting SSMIS, AMSR-2, GMI, MWRI. With averaging and format conversion capabilities. Initial release in early 2019.	BUFR, hdf5	NWP SAF – MWIPP
IRSPP	Infrared Sounder Pre-Processor from NWP SAF, currently supporting MTG-S1, MTG-S2 instruments. With conversion to reconstructed radiances using sets of eigenvectors that accompany the spectra, and apodising specta.	BUFR, netCDF	NWP SAF – IRSPP

Table last updated: 02 July 2024

Level 2 processing packages

Name	Capabilities	Reference
CSPP: International ATOVS Processing Package (IAPP)	Temperature and humidity retrievals from ATOVS. Uses HIRS level 1d (from AAPP) as input. Released as part of CSPP.	CIMSS – CSPP
CSPP: CrIS, AIRS and IASI Hyperspectral Retrieval Software	Temperature and humidity retrieval algorithm developed at CIMSS. Released as part of CSPP.	CIMSS – CSPP
CSPP: NOAA/NESDIS/STAR Microwave Integrated Retrieval System (MiRS)	Retrievals from AMSU, MHS and ATMS. Released as part of CSPP.	CIMSS – CSPP
CSPP: NOAA Unique CrIS ATMS Processing System (NUCAPS)	Retrievals from CrIS/ATMS, IASI/AMSU, AIRS. Released as part of CSPP.	CIMSS – CSPP
CSPP: VIIRS Atmosphere EDRs	Retrieves cloud, aerosol, smoke, and volcanic ash environmental data records (EDRs) from VIIRS SDR inputs. Released as part of CSPP.	CIMSS – CSPP
CSPP: NOAA Clouds from AVHRR Extended (CLAVR-x) Retrieval Software Package in Support of VIIRS, MODIS and AHVRR	Single Field-of-View (FOV) cloud retrievals from AVHRR, MODIS, and VIIRS Imager calibrated/geolocated data files. Released as part of CSPP.	CIMSS – CSPP
CSPP: NOAA/STAR Advanced Clear-Sky Processor for Oceans (ACSPO)	Retrieves sea surface temperature from direct broadcast (DB) AVHRR, MODIS and VIIRS sensor data records. Released as part of CSPP.	CIMSS – CSPP
CSPP: NOAA JPSS VIIRS Active Fire Software	NOAA operational version of the Active Fire detection from VIIRS SDR inputs. Released as part of CSPP.	CIMSS – CSPP
CSPP: VIIRS Flood Detection Software	Automatic flood detection using VIIRS Sensor Data Record (SDR) inputs.	CIMSS – CSPP
IMAPP: MODIS and AIRS Level 2	Produces a range of MODIS, AIRS and AMSU Level 2 geophysical products.	CIMSS – IMAPP
Scatterometer wind processors: AWDP, PenWP	Wind processors for ASCAT, OSCAT, HSCAT, RapidScat and SeaWinds. Input is the level 1b Normalized Radar Cross Section.	NWP SAF – Scatterometer
1D-Var schemes	The NWPSAF 1D-Var package is a stand-alone system for doing 1D-Var retrievals with user-provided files of observation and background error covariances. The package is configured to support ATOVS, AIRS, IASI, ATMS, CrIS and SSMIS sensors, and is capable of supporting any nadir-sounding sensor for which a suitable radiative transfer model is available.	NWP SAF – 1DVAR
Inversion Coupled with Imager (ICI)	Temperature and humidity retrievals, making use of AVHRR cloud mask. Package was developed by Meteo-France, and was distributed by EUMETSAT prior to 2004, but no longer supported.	See ITWG description
Improved Initialization Inversion system (3I)	HIRS and MSU retrieval system, developed by LMD. The 3I Algorithm is available as a freeware product, for academic use or scientific research.	LMD – 3I

Table last updated: 2 May 2025

Visualization and analysis tools

Appendix

Appendix A: Modification to file ProcUtils.py in ocssw_7.5

This is the modified version of function httpinit that is used at the Met Office to set up a tunnel through a proxy server, defined in environment variable https_proxy …

def httpinit(url, timeout=10, urlConn=None):
    """
    initialize HTTP network connection
    """
    import httplib
    import os
    from urlparse import urlparse

    proxy = None
    proxy_set = os.environ.get('https_proxy')
    if proxy_set is None:
        proxy_set = os.environ.get('http_proxy')
    if proxy_set:
        proxy = urlparse(proxy_set)

    if urlConn is None:
        #if proxy is None:
        #    urlConn = httplib.HTTPSConnection(url, timeout=timeout)
        #elif proxy.scheme == 'https':
        #    urlConn = httplib.HTTPSConnection(proxy.hostname,
        #                                      proxy.port, timeout=timeout)
        #else:
        #    urlConn = httplib.HTTPConnection(proxy.hostname,
        #                                     proxy.port, timeout=timeout)

        urlConn = httplib.HTTPSConnection(proxy.hostname,
                                              proxy.port, timeout=timeout)
        urlConn.set_tunnel(url,443)

    return urlConn, proxy

Appendix B: CADU manipulation

Raw data from direct broadcast are typically in CADU (Channel Access Data Unit) format. As an example, see Figure 2.1 of the FY-3A Satellite to Ground Interface Control
Document. The frame length is typically 1024 bytes, but you can verify this by looking for the first few occurrences of the frame sync:

od -t x1 -A d $infile | grep "1a cf fc 1d" | head -5

Common to all missions is a randomization, or scrambling, step – see Figure 2-3 of the FY-3A document. This, in effect, does an exclusive or of the input stream with a 255-bit pseudo-random sequence. The randomization and de-randomization processes are identical. For DB stations that deliver CADU data, de-randomization may or may not have been applied in the station processing; similarly, some level 0 software packages expect to receive scrambled data (e.g. metopizer tool cadu_to_ccsds) while others expect unscrambled data (e.g. FY3CL0pp).

If you have difficulty processing your CADU data, try converting your files. One option is to use the ESA tool CADU-convert, while another option is to build the short C program listed below.

/* cadu_scramble.c
   Scramble or unscramble a CADU file (e.g. Metop or FY-3) 
   Frame length normally 1024 bytes but MEOS files are 1072 bytes. Check your frame length before running.
   
   Note that the Metopizer tool cadu_to_ccsds ingests scrambled (randomized)
   files but fy3cl0db ingests unscrambled files.

   Build: cc -o cadu_scramble cadu_scramble.c
   Usage: ./cadu_scramble infile outfile [framelength]
   
   Reference: Fig 2-3 of http://mdkenny.customer.netspace.net.au/FY3_HRPT.pdf

   04/01/2018 Nigel Atkinson
*/


#include <stdio.h>
#include <stdlib.h>

main(int argc, char *argv[]) {

#define MAXFRAME 1072
#define MAXMASK MAXFRAME-4

  int i,j,k,framelength,masklength;
  unsigned char sequence[8] = { 1,1,1,1,1,1,1,1 };
  unsigned char mask[MAXMASK] = { 0 };
  unsigned char tmp1, tmp2, tmp3;
  
  unsigned char frame[MAXFRAME];
  FILE *fpin, *fpout;

/* open the files */

  if (argc < 3) {
    printf("usage: cadu_scramble infile outfile [framelength]\n");
    exit (0);
  }
  fpin = fopen(argv[1],"r");
  fpout = fopen(argv[2],"w+");
  if (argc > 3) sscanf(argv[3],"%d",&framelength); else framelength=1024;
  masklength = framelength - 4;
  
/* set up the scrambling mask */
  
  for (i=0; i<masklength; i++) {
    for (j=0; j<8; j++) {
      mask[i] =  (mask[i]<<1) + sequence[0];
      tmp1 = sequence[3] ^ sequence[0];       /* exclusive or */
      tmp2 = sequence[5] ^ tmp1;
      tmp3 = sequence[7] ^ tmp2;
      for (k=0; k<7; k++) sequence[k] = sequence[k+1];
      sequence[7] = tmp3;
    }
  }

/* read, modify and write, framelength bytes at a time */

  while (fread(frame,1,framelength,fpin) == framelength){
    for (i=0; i<masklength; i++) {
      frame[i+4] = frame[i+4] ^ mask[i];   /* leave first 4 bytes unchanged */
    }
    fwrite(frame,1,framelength,fpout);
  }
  
  fclose(fpin);
  fclose(fpout);  
}