Gap-filling Upper Air Quality Control Checks
SHOUT
The level 1 validity checks restrict each observation to falling within a TSP-specified set of tolerance limits. Observations not falling within the limits are flagged as failing the QC check. The following table lists the basic tolerance limits:
--------------------------------------------- Validity Checks --------------------------------------------- Latitude -90 - 90 deg N Longitude -180 - 180 deg E Air temperature -100 - 60 C Dewpoint temperature -100 - 60 C Altitude 100 - 1042 mb Wind Direction 0 - 360 deg Wind Speed 0 - 300 kts Relative Humidity 0 - 100 %
The level 1 position consistency check compares location and time to previous reports to insure that an dropsonde's position is consistent with the reported movement. An inconsistency is identified as an unrealistic speed. Speed is calculated from the distance traveled and the time elapsed since the previous report. Distance, in miles on a spherical Earth, is calculated from the reported latitude and longitude coordinates using the Haversine formula expressed in terms of a two-argument inverse tangent function (Sinnot, 1984).
The level 2 internal consistency check enforces reasonable, meteorological relationships among observations measured in a single report. For dropsonde data, the dewpoint temperature observation must not exceed the corresponding temperature observation. If it does, both the dewpoint and temperature observation are flagged as failing their internal consistency check.
The level 2 temporal consistency check applied to temperature restricts the temporal rate of change of each observation to calculated tolerance limits that model the expected rate of change for descent. To examine the rate of change for the current observation, comparisons are made with the preceeding and succeeding observation from the same dropsonde.
The departure value (or observation difference) for the report at time j is calculated over the time interval i-k:
Then if | Odiff | > a threshold, the observation at point j fails the check.Odiff = Oj - ( Oi * DELTjk / DELTik ) + ( Ok * DELTij / DELTik )
where,
Odiff : departure value (deg C) j : index of observation being checked i : index of previous observation (in time) k : index of successive observation (in time) Oi : observation at point i (or j, k) DELTij : change in time between points ij (or ik, jk) (seconds)
The basic idea is to weight the previous and successive observations by how close they are in time to the observation being checked. If the three points are equidistant in time, and the observed value hasn't changed, the estimated value will equal the observed value, and the observation difference will be 0.
The temperature threshold (Tdiff) is calculated as:
Tdiff = ( XD * D ) + ( XA * 6.5 * | DELA | / 1000 )
where,
Tdiff : maximum allowable difference between observed and estimated temperature (deg C) D : distance over the three observations used in the QC (report being checked and preceeding & succeeding reports) (miles) DELA : altitude change over the three observations used in the QC -- the quantity 6.5 * |DELA|/1000 is the standard atmosphere lapse rate for the altitude range (m) XD : coefficient to weight the expected difference resulting from change in distance (expected to dominate the calculation at flight level) [ XD = 0.25 ] XA : coefficient to weight the expected difference resulting from change in altitude (expected to dominate the calculation for ascents/descents) [ XA = 1.97 ]
*It should be noted that while the QC checks discussed here are generally applied to the form of the variable stored in the database, the QC results will also be applied to any forms of the variable that are requested by the user and are derived from the primary variable. For example, U and V wind components will get the QC results from the checks applied to the wind speed and direction.
Subjective Intervention
Two text files, a "reject" and an "accept" list provide the capability to subjectively override the results of the automated QC checks. The reject list is a list of balloons and associated input observations that will be labeled as bad, regardless of the outcome of the QC checks; the accept list is the corresponding list of balloons that will be labeled as good, regardless of the outcome of the QC checks.
Here are the current subjective intervention lists in use:
- Reject List
- Accept List
QC Data Structures
The MADIS QC information available for each variable includes the following QC structures: a single-character "data descriptor", intended to define an overall opinion of the quality of each observation by combining the information from the various QC checks, and for users desiring detailed information, a "QC applied" bitmap indicating which QC checks were applied to each observation, and a "QC results" bitmap indicating the results of the various QC checks.
The following table provides a complete list of the data descriptors and the bits used in the bitmaps:
-----------------------------
MADIS QC Information - SHOUT AVAPS
-----------------------------
QC Data Descriptor Values
-------------------------
No QC available:
Z - Preliminary, no QC
Automated QC checks:
C - Coarse pass, passed level 1
S - Screened, passed levels 1 and 2
V - Verified, passed levels 1, 2, and 3
X - Rejected/erroneous, failed level 1
Q - Questioned, passed level 1, failed 2 or 3
where level 1 = validity and position consistency QC checks
level 2 = internal and temporal consistency checks
level 3 = N/A
Subjective intervention:
G - Subjective good
B - Subjective bad
Bitmask for QC Applied and QC Results
-------------------------------------
Bit QC Check Decimal Value
--- -------- -------------
1 Master Check 1
2 Validity Check 2
3 Position Consistency Check 4
4 Reserved 8
5 Reserved 16
6 Reserved 32
7 Reserved 64
8 Reserved 128
9 Reserved 256
10 Reserved 512
11 Reserved 1024
12 Reserved 2048
The QC bitmask is used in the QC applied and QC result "words" returned along with the QC data descriptor. By examining the individual bits, the user can determine which checks were actually applied, and the pass/fail status of each check that was applied.
In the QC applied word, a bit value of 1 means the corresponding check was applied, a bit value of 0 indicates the check wasn't applied.
In the QC results word, a bit value of 1 means the corresponding check was applied and failed, a bit value of 0 indicates the check passed (given that the check was applied).
The "Master Check" is used to summarize all of the checks in a single bit. If any check at all was applied, this bit will be set in the QC applied word. If the observation failed any QC check, it will be set in the QC results word.
When read as decimal numbers, the different bits that are set in the bitmask are summed together. For example, a QC applied value of 7 should be interpreted as 1 + 2 + 4, meaning the validity and position consistency checks were applied.
References
Sinnott, R.W., 1984: Virtues of the Haversine, Sky and Telescope, 68, 159-170.
Technique Specification Package 88-21-R2 For AWIPS-90 RFP Appendix G Requirements Numbers: Quality Control Incoming Data, 1994. AWIPS Document Number TSP-032-1992R2, NOAA, National Weather Service, Office of Systems Development.
WISDOM Balloon
The level 1 validity checks restrict each observation to falling within a TSP-specified set of tolerance limits. Observations not falling within the limits are flagged as failing the QC check. The tolerances for wind speed are specified as a function of pressure level, and the heights are converted into pressure using the U.S. standard atmosphere calculation. The following table lists the tolerance limits used for the wind speed validity check:
---------------------------------------------------
Validity Checks
Max Wind
Level(mb) Speed(Kts)
---------------------------------------------------
1000 70
850 90
700 120
500 200
400 250
300 300
250 300
200 300
150 200
100 200
70 200
50 200
30 200
20 200
10 200
<10 200
The following table lists the basic tolerance limits for the
other observations:
--------------------------------------------- Validity Checks --------------------------------------------- Latitude -90 - 90 deg N Longitude -180 - 180 deg E Wind Direction 0 - 360 deg
The level 1 position consistency check compares location and time to previous reports to insure that a balloon's position is consistent with the reported movement. An inconsistency is identified as an unrealistic speed. Speed is calculated from the distance traveled and the time elapsed since the previous report. Distance, in miles on a spherical Earth, is calculated from the reported latitude and longitude coordinates using the Haversine formula expressed in terms of a two-argument inverse tangent function (Sinnot, 1984).
The minimum allowable speed must be greater than 0 m/s and less than or equal to 250 m/s.
*It should be noted that while the QC checks discussed here are generally applied to the form of the variable stored in the database, the QC results will also be applied to any forms of the variable that are requested by the user and are derived from the primary variable. For example, U and V wind components will get the QC results from the checks applied to the wind speed and direction.
Subjective Intervention
Two text files, a "reject" and an "accept" list provide the capability to subjectively override the results of the automated QC checks. The reject list is a list of balloons and associated input observations that will be labeled as bad, regardless of the outcome of the QC checks; the accept list is the corresponding list of balloons that will be labeled as good, regardless of the outcome of the QC checks.
Here are the current subjective intervention lists in use:
QC Data Structures
The MADIS QC information available for each variable includes the following QC structures: a single-character "data descriptor", intended to define an overall opinion of the quality of each observation by combining the information from the various QC checks, and for users desiring detailed information, a "QC applied" bitmap indicating which QC checks were applied to each observation, and a "QC results" bitmap indicating the results of the various QC checks.
The following table provides a complete list of the data descriptors and the bits used in the bitmaps:
-----------------------------
MADIS QC Information - WISDOM
-----------------------------
QC Data Descriptor Values
-------------------------
No QC available:
Z - Preliminary, no QC
Automated QC checks:
C - Coarse pass, passed level 1
S - Screened, passed levels 1 and 2
V - Verified, passed levels 1, 2, and 3
X - Rejected/erroneous, failed level 1
Q - Questioned, passed level 1, failed 2 or 3
where level 1 = validity and position consistency QC checks
level 2 = N/A
level 3 = N/A
Subjective intervention:
G - Subjective good
B - Subjective bad
Bitmask for QC Applied and QC Results
-------------------------------------
Bit QC Check Decimal Value
--- -------- -------------
1 Master Check 1
2 Validity Check 2
3 Position Consistency Check 4
4 Reserved 8
5 Reserved 16
6 Reserved 32
7 Reserved 64
8 Reserved 128
9 Reserved 256
10 Reserved 512
11 Reserved 1024
12 Reserved 2048
The QC bitmask is used in the QC applied and QC result "words" returned along with the QC data descriptor. By examining the individual bits, the user can determine which checks were actually applied, and the pass/fail status of each check that was applied.
In the QC applied word, a bit value of 1 means the corresponding check was applied, a bit value of 0 indicates the check wasn't applied.
In the QC results word, a bit value of 1 means the corresponding check was applied and failed, a bit value of 0 indicates the check passed (given that the check was applied).
The "Master Check" is used to summarize all of the checks in a single bit. If any check at all was applied, this bit will be set in the QC applied word. If the observation failed any QC check, it will be set in the QC results word.
When read as decimal numbers, the different bits that are set in the bitmask are summed together. For example, a QC applied value of 7 should be interpreted as 1 + 2 + 4, meaning the validity and position consistency checks were applied.
References
Sinnott, R.W., 1984: Virtues of the Haversine, Sky and Telescope, 68, 159-170.
Technique Specification Package 88-21-R2 For AWIPS-90 RFP Appendix G Requirements Numbers: Quality Control Incoming Data, 1994. AWIPS Document Number TSP-032-1992R2, NOAA, National Weather Service, Office of Systems Development.
Last updated 16 March 2017
