News
Data
User Resources
Provider Resources
Support
  MADIS NOAA Profiler Network Quality Control

MADIS NOAA Profiler Network Quality Control


This page provides a detailed description of the quality control (QC) processing and data structures for the NOAA Profiler Network dataset. For a general overview of MADIS quality control, click here.

Automated Quality Control

Level 1 QC checks are considered the least sophisticated, level 2 the most sophisticated checks. The following table lists the profiler variables* that are QC'ed, and the checks that are used:


  MADIS NOAA Profiler Network (NPN) Variables with QC
  ------------------------------------------------------------------------------------------
  Code       Name                      Max     Level | Level 1: |         Level 2:          
                                     Possible        |          |                           
                                     QC Level        | Validity |   Internal    Time-Height 
                                                     |          | Consistency   Consistency 
  ------------------------------------------------------------------------------------------
  
  Integrated Profile
  ------------------
  
  DD        wind direction             2                  X            X             X
  FF        wind speed                 2                  X            X             X
  U         u wind component           2                  X            X             X
  V         v wind component           2                  X            X             X
  
  Low Mode Levels
  ---------------
  
  DDL       wind direction             2                  X            X             X
  FFL       wind speed                 2                  X            X             X
  UL        u wind component           2                  X            X             X
  VL        v wind component           2                  X            X             X
  
  High Mode Levels
  ----------------

  DDH       wind direction             2                  X                          X
  FFH       wind speed                 2                  X                          X
  UH        u wind component           2                  X                          X
  VH        v wind component           2                  X                          X


  Internal Consistency Checks
  ---------------------------

    1   Bird contamination check

        (spectral width check on data below 5000 m MSL to locate the
         presence of migrating birds)

The level 1 validity checks restrict observations to falling within a TSP-specified set of tolerance limits. Wind speeds not falling within the limits are flagged as failing the QC check. The tolerances are specified as a function of pressure level, and the profiler heights are converted into pressure using the U.S. standard atmosphere calculation. The following table lists the tolerance limits:


  ---------------------------------------------------
  Validity Checks

                    Max Wind
  Level(mb)         Speed(Kts)
  ---------------------------------------------------
  1000                  70
   850                  90
   700                 120
   500                 200
   400                 250
   300                 300
   250                 300
   200                 300
   150                 200
   100                 200
    70                 200
    50                 200
    30                 200
    20                 200
    10                 200       
   <10                 200       

The level 2 time-height continuity check (Weber et al. 1993; Miller et al. 1994; Barth et al. 1994) uses pattern recognition techniques to identify winds which do not satisfy mathematical definitions of continuity in the time and height dimensions. The current profile is quality controlled by using past data in a 6-h sliding window. The algorithm is run separately on the U and V components. There are two stages to the algorithm: a pattern recognition step that is used to determine the order in which the individual points are examined, and a second step that checks the individual points for quality. The components are grouped into patterns in such a way that, from point to point within a given pattern, the values change smoothly. Bad points are then most likely to be found near pattern boundaries where, by definition, discontinuities occur. The data are checked by comparing each component with the values interpolated from neighboring winds using a least-squares linear interpolation. The neighborhood is defined as two heights above and below the target point, and two hours back in time. If a component differs more than a predetermined threshold from the interpolated value, the wind is flagged as bad. After these bad values have been eliminated, any remaining winds that are found in a pattern of smaller than eight points are also flagged as bad.

The level 2 internal consistency check is a check for contamination from migrating birds (Miller et al. 1997). The bird contamination check is a simple, 6-step test that is primarily based on identifying the large velocity variance signature that is thought to accompany the backscattered radar return from birds in profiler spectral moment data. The other steps in the test are meant to reduce the likelihood of false alarms, as other factors besides bird contamination can cause large velocity variance. The check flags an hourly wind observation as bad if all of the following criteria are met:

*It should be noted that while the QC checks discussed here are generally applied to the form of the variable stored in the database, the QC results will also be applied to any forms of the variable that are requested by the user and are derived from the primary variable. For example, wind speed and direction will get the QC results from the checks applied to the U and V wind components.

Subjective Intervention

Two text files, a "reject" and an "accept" list provide the capability to subjectively override the results of the automated QC checks. The reject list is a list of stations and associated input observations that will be labeled as bad, regardless of the outcome of the QC checks; the accept list is the corresponding list of stations that will be labeled as good, regardless of the outcome of the QC checks. In both cases, observations associated with the stations in the lists can be individually flagged. For example, wind observations at a particular station may be added to the reject list, but not the temperature observations.

Here are the current subjective intervention lists in use at ESRL/GSD:

QC Data Structures

The MADIS QC information available for each variable includes the following QC structures: a single-character "data descriptor", intended to define an overall opinion of the quality of each observation by combining the information from the various QC checks, and for users desiring detailed information, a "QC applied" bitmap indicating which QC checks were applied to each observation, and a "QC results" bitmap indicating the results of the various QC checks.

The following table provides a complete list of the data descriptors and the bits used in the bitmaps:

  ---------------------------------
  MADIS QC Information - Profiler
  ---------------------------------

  QC Data Descriptor Values
  -------------------------

  No QC available:

   Z - Preliminary, no QC

  Automated QC checks:

   C - Coarse pass, passed level 1
   S - Screened, passed levels 1 and 2
   V - Verified, passed levels 1, 2, and 3
   X - Rejected/erroneous, failed level 1
   Q - Questioned, passed level 1, failed 2 or 3

       where level 1 = validity
             level 2 = internal and time-height consistency checks
             level 3 = N/A

  Subjective intervention:

   G - Subjective good
   B - Subjective bad

  Bitmask for QC Applied and QC Results
  -------------------------------------

   Bit       QC Check                      Decimal Value
   ---       --------                      -------------
    1        Master Check                        1
    2        Validity Check                      2
    3        Reserved                            4
    4        Internal Consistency Check          8
    5        Reserved                           16
    6        Reserved                           32
    7        Reserved                           64
    8        Reserved                          128
    9        Reserved                          256
   10        Time-Height Consistency Check     512

The QC bitmask is used in the QC applied and QC result "words" returned along with the QC data descriptor. By examining the individual bits, the user can determine which checks were actually applied, and the pass/fail status of each check that was applied.

In the QC applied word, a bit value of 1 means the corresponding check was applied, a bit value of 0 indicates the check wasn't applied.

In the QC results word, a bit value of 1 means the corresponding check was applied and failed, a bit value of 0 indicates the check passed (given that the check was applied).

The "Master Check" is used to summarize all of the checks in a single bit. If any check at all was applied, this bit will be set in the QC applied word. If the observation failed any QC check, it will be set in the QC results word.

When read as decimal numbers, the different bits that are set in the bitmask are summed together. For example, a QC applied value of 515 should be interpreted as 1 + 2 + 512, meaning the validity and time-height continuity checks were applied.



References

Barth, M.F, R.B. Chadwick, and D.W. van de Kamp, 1994: Data processing algorithms used by NOAA's Wind Profiler Demonstration Network. Ann. Geophysicae, 12, 518-528.

Miller, P.A., M.F. Barth, J.R. Smart and L.A. Benjamin, 1997: The extent of bird contamination in the hourly winds measured by the NOAA Profiler Network: Results before and after implementation of the new bird contamination quality control check. 13th International Conference on Interactive Information and Processing Systems, Long Beach, CA., Amer. Meteor. Soc., 138-144.

Miller, P.A., M.F. Barth, D.W. van de Kamp, T.W. Schlatter, B.L. Weber, D.B. Wuertz, and K.A. Brewster, 1994: An evaluation of two automated quality methods designed for use with hourly wind profiler data. Ann. Geophysicae, 12, 711-724.

Technique Specification Package 88-21-R2 For AWIPS-90 RFP Appendix G Requirements Numbers: Quality Control Incoming Data, 1994. AWIPS Document Number TSP-032-1992R2, NOAA, National Weather Service, Office of Systems Development.

Weber, B.L., D.B. Wuertz, D.C. Welsh, and R. McPeek, 1993: Quality controls for profiler measurements of winds and RASS temperatures. J. Atmos. Oceanic Technol., 10, 452-464.


Last updated 16 June 2016