From CedarWiki

Resolute Bay Fabry-Perot Interferometer (rfp) Ingest

Updated 19 April 2008 by emery@ucar.edu

1. Files and programs are in /aurora/d/emery/rfp
   1. Took 47 hours to add 2005-2008 and revise 2003-2005 red-line data to the CEDAR DB
2. Qian Wu of HAO/NCAR asks me to update his red-line (630.0 nm) FPI data because Jiuhou Lei of HAO/NCAR wants it for Nov-Dec 2006.
   1. I ask Qian to put the netcdf (.ncdf) files into /dione/d/qwu/resolute6300/
   2. Qian sends me the ~2 MB per year ascii file of meteorological data for Resolute from Environment Canada that he has to buy from them. This file includes cloud cover in tenths of the sky (0=clear, 10=overcast) which is merged with the FPI data analysis. Typical names are Resolute_HLY01_2005to2007_Data.txt
3. ls /dione/d/qwu/resolute6300/* > list.ncdf05_08 (314 files 2005263-2008116).
   1. Find later that have 5577 [O]and 8920 [OH] data for 2006300 because I find very low temperatures for that date since I am actually using the [OH] data file.
   2. Find later that only have 309 UT days because of 2 extra files above and 3 nights that start observations after UT midnight (which is 6 hours later than LT) which are also part of the same days that have observations that start before midnight UT.
   3. Revise testrd.pro to read this list of files and to print them out to be read in wrncdf.pro as 'allfile', which is a single long line.
4. Revise and run wrncdf.pro 11 times to create 29 files per time of rfp.yeardaynum ascii files (more than 29 files is too much for IDL code)
   1. cat rfp.2005* rfp.2006* rfp.2007* rfp.2008* > rfp.datanam05_08
   2. Delete 2005001-092 (old files) and revise and run strip.f to eliminate the first line with the number of observations per 'night' to get rfp.data05_08 with 23095 lines , which is the number of observations
      1. Remove **** and NaN lines or blocks of 5 directions of bad data in rfp.data05_08 to rfp.bad05_08. This removes 218 lines from rfp.data05_08
5. Revise and run rdenvca.f to make envca82.05_07 from Resolute_HLY01_2005to2007_Data.txt of only code 82 (cloud cover in tenths) from Environment Canada
6. Revise and run cldetc.f to read envca82.05_07 and rfp.data05_08 to make envca82.05_07.ut (6 hours later than LT) and rfp.data05_08r,nit,removed
   1. cldetc.f removed 373 points (first time around) in rfp.removed05_08 for large errors, neg Tn, or lo brightness
   2. cldetc.f found median line-of-sight (los) zero wind each night in ref.nit05_08
      1. The change-over ~local noon at 18.33 UT was not good with the smaller solar zenith angles, so revised cldetc.f for a better nightly break-point routine.
7. Look at cloud cover data to put in /instruments/xtradata/rfp for users with links to rfp.html
   1. See what differences there are in old and new envca82.94_05.ut from 2006 and the new rfp.data05_07.ut
      1. Found several -999 (missing values) were filled in from May-Oct 2005.
      2. Added new file to make envca82.94_07.ut.
      3. Made 4 winter season files from Sep-Apr (242 or 243 leap year days) of cloud cover for /instruments/xtradata/rfp on-line for users to take. This is an increase from Oct-Apr winter seasons from before because Qian started to take data at 95 degrees solar zenith angle (sza) starting in September 2005. From 2003-2008, the earliest FPI data are on 6 Sep 2007 and the latest data are on 2 April 2004 and 2005.
      4. Revised tenths.pro to plot envca82.res94_07.ut for 2 plots of 80 days each of cloud clover in tenths for xtradata/rfp. With the addition of the month of September , made tenths13.pro from tenths.pro to plot 13 days per row instead of 10 days per row (8 rows total for a total of 208 days in 2 plots.
8. Revise and run nit.f reading rfp.data05_08r and rfp.nit05_08 to create rfp.data05_08r2, rfp.nit05_08b, rfp.removed_a05_08 and rfp.ascnit05_08. Remove points if los zero wind in a block of 5 directions is >175 m/s from the median nightly los zero wind. Found the median los zero wind changed for 101 of 309 nights, or nit.f removed 324 pts. Did not add any special cases such as was done for the 2003-2005 period in nit.f. The rfp.ascnit05_08 contained in DB format the prologue, 1-D codes and values, and 2-D codes, but no 2-D values.
9. Revise and run mkasc.f to read rfp.data05_08r2 and rfp.ascnit05_08 to create rfp.data05_08r, an ascii file of the FPI data, and rfp05_08.asc for the CEDAR DB
   1. Found the min/max values are way too small/large for los Vn, Tn, etc!
      1. 2006 300 Oct 27 Tn 150-170K?? - was 20063008920.ncdf (OH) file!
      2. Tn>3000K for 3 times when sza~97
      3. Found 2 nights (2005317 and 2006012) had 0 Awind (los zero wind) in rfp.nit05_08. The rfp.2005317 and rfp.2006012 files also mostly had zero Hwind (horizontal or vertical wind).
      4. Found cloud cover zero when should have been missing after April 2007
10. In consultation with Qian, he re-ran his analysis to make new files 20053176300.ncdf and 20060126300.ncdf that he put on /dione/d/qwu/resolute6300/. He also found 20063005577.ncdf and 20063008290.ncdf here and deleted them. We decided to check on the magnitude of the Awind (los zero wind) vs the los wind, and the magnitude of the Hwind (horizontal or vertical wind) in cldetc.f
    1. Returned to step #4 to re-run wrncdf.pro for 20053176300.ncdf, 20060126300.ncdf and 20063006300.ncdf. The resulting rfp.yearday files were put into rfp.data05_08 to replace the other bad files.
    2. Returned to step #6 and revised removal criteria in cldetc.f. Added median nightly Tn and Wn (vertical wind) values to output file rfp.nit05_08, and fixed missing cloud cover values after 2007107 and re-ran.
    3. Returned to step #8 and revised nit.f to delete based on the difference of Tn or Wn from their median nightly values as well as the los zero wind removals. Most deletions were for sza<100, cloud cover (cc)>6, etc.
    4. Returned to step #9 and re-ran mkasc.f (better Wn,Tn limits, worse Un,Vn limits) 252 pts vs 271 removed
11. Check out DB format files and make summary plots for further checks
    1. Revised the header rfp.head2006 to rfp.head2008.
    2. cat rfp.head2008 rfp03_05.asc (from 2006) rfp05_08.asc > rfp03_08.asc
    3. The los error bar (code -800) is the same as the error bar for the los before los zero wind is subtracted (code -4173). Decided NOT to eliminate one of the error bars for the sake of clarity.
    4. chr2is.exe rfp03_08.asc rfp03_08.cbf
    5. cdbckf rfp03_08.cbf > cdbckf.rfp03_08
    6. in /cedar/d/emery/dbplots/rfp, revise and run rfp.gscr and rfp.pro 11 times for plots
       1. ran togif to go from .ps to .gif files
       2. Looked at plots for wind and Tn problems.
          1. W/E or N/S winds mostly divergent with bad Awind (los zero wind)
          2. Low Tn (<500 relative K) in 2006-2008 mostly from low brightness (~<35,000)
12. In consultation with Qian, I decided to eliminate on the basis of approximately 3 times the standard deviation from the mean for Awind, Tn and Wn. Also decided to re-run the 2003-2005 data because the removal criteria in cldetc.f and nit.f changed from before.
    1. Returned to step #6 and calculated the mean and standard deviation of Awind, Tn and Wn in cldetc.f to be printed out in rfp.nit03_05. Deleted the magnitude checks on Awind and Hwind in cldetc.f.
       1. Made list when std awind>90m/s (normally std ~25m/s, if >800m/s, have 1 0 Awind). Found other data generally poor for bad Awind if sza<100, but usually OK for sza>100.
       2. Found anomalously hi Tn 2005024 and 2005054 for anomalously high brier (brightness error) >2000
          1. Qian re-ran these and put them on /dione/d/qwu/resolute6300/. I ran wrncdf.pro and put the revised rfp.2005024 and rfp.2005054 into the old rfp.data03_05 file.
13. In consultation with Qian, I will replace 0 or poor Awinds with the median nightly values if sza>100, and eliminate all data if sza<100. I will also eliminate relative Tn<550K if bri<30,000, but the winds seem OK.
    1. After multiple runs of nit.f with various brightness and error brightness levels, found that data were bad for brightness<15,000 (instead of 10,000), and for error bars >5000 (instead of 10,000). Also found most relative Tn for bri<30,000 were >500K.
       1. Many of the low Tn were in the zenith (Z) or upward direction because the brightness is less in the zenith compared to other directions because of the Van Rijn effect when the weather is clear. This is a way of estimating if the sky is clear of not.
       2. Revised brightness magnitude checks in cldetc.f and re-ran.
14. Split nit.f into nit1.f and nit2.f.
    1. nit1.f only revised the los zero wind (Awind) for sza>100 or deleted all the data if sza<100. The revised Awind was set to the median nightly value and the los and Hwind values were re-calculated from the new Awind. The median, mean and standard deviation Awind, Tn, and Wn, not including the removed or changed blocks of directions, was then recalculated for each night of data. Output files are rfp.data05_08r2, rfp.nit05_08b, rfp.removed_1a05_08, rfp.changed_1a05_08 using input files rfp.data05_08r and rfp.nit05_08.
       1. When the Awind direction blocks were checked for consistency in the horizontal E and W winds and the N and S winds in rfp.changed_1a05_08, the difference between the nightly median Awind and individual Awind blocks was raised from 175 m/s to 200 m/s.
    2. nit2.f revised or deleted (based on sza >100 or <100) the Awind and subsequent los and Hwind values if it was more than 2.9 times the standard deviation from the revised nightly mean calculated in nit1.f. It also deleted data where the relative Tn was 400K different from the nightly median or more than 3 times the standard deviation of the nightly mean. Zenith times were removed if the vertical wind was more than 100 m/s from the nightly median or more than 3 times the standard deviation of the nightly mean. Output files are rfp.data05_08r2b, rfp.nit05_08b2, rfp.removed_atw05_08, rfp.changed_aw05_08, and rfp.ascnit05_08 using input files rfp.data05_08r2 and rfp.nit05_08b.
       1. The figure of 2.9 times the Awind standard deviation was found after multiple runs of nit2.f. Exceptions were found where the revised W/E and N/S winds were worse for differences greater than 2.9 (6 cases), and also where the revised W/E and N/S winds were better for differences smaller than 2.9 (5 cases). In consultation with Qian, I finally added these 11 individual exceptions to nit2.f.
       2. Added one more parameter code (parcod), 4174, for the case where either nit1.f or nit2.f changed the los zero wind to the median value and recalculated the los and Hwind values.
          1. Changed in ~bozo/is/docs parcods.updat, parcods.file, parcods.list. Sent new parcod to Jose Garcia.
15. Revised and re-ran mkasc.f with extra parcod and for a new kindat 17011. Input files are rfp.ascnit05_08 and rfp.data05_08r2b and output files remain rfp05_08.asc and rfp.data05_08.r3.
    1. Revised rfp.head2008 and returned to step #11 to check out the summary plots. (The summary plots were actually done more frequently, but not listed except for each major change.)
       1. The plots were improved with 1.8% of the data deleted or changed in 2003-2005 and 4.4% of the data deleted or changed in 2005-2008 (more because of the push to get lower solar zenith angles).
          1. Still had 14 cases of divergent winds that were checked out to see if changing the los zero wind (Awind) would help. In 10 cases, helping NS would wreck EW or vice versa. In 4 cases, changing the Awind helped or was about the same, but all 4 were for cloud cover conditions (close to zero winds), 3 of the 4 were for sza<100, and all had Awind differences from the nightly mean Awind of 2*standard deviation or less (not much). The conclusion was that there were good reasons for the winds to be anomalously divergent so no further processing was necessary.
16. Official ingest to CEDAR Database
    1. Revised in /instruments/: rfp.html, replace/added files to /xtradata/rfp on cloud cover in /aurora/d/emery/rfp and summary plots to /GIF/rfp from /cedar/d/emery/dbplots/rfp.
       1. Found this was irrelevant since it was not linked to on the wiki
    2. Revised rfp.gscr for new kindat 17011 and to make wiki.store. Re-ran rfp.gscr for all periods to make wiki.store. Used text of rfp.html and wiki store to update the wiki Instruments:rfp. Found some typos in the header and catalog records, but these were not fixed in the DB copy since they were found after the farch step below. However, the typos were fixed in rfp.head2008 for the next update of this data set.
    3. Ran: barch rfp03_08.asc > barch.rfp08 for rfp0301012b (need to retire a)
    4. Revised ~bozo/is/docs/akavsns.mgr for not public (np) a and new b files
    5. farch -r rfp031012a > farch.apr08
    6. In /project/cedar/data rm rfp*a.cbf (thus no data from DB until new update)
    7. In /project/cedar/src/emery/mysql updated txt.words, Macro.Tn, xmacro; revised Tn,Vn,Em.txt for 17011 kindat for kinst 5535 (rfp) instead of 17001.
       1. Ran combine.ob - still missing Em for 5535? - added it to txt.allwords and moved it to txt.allwords.17apr08. Asked Jose Garcia to update the DB with this mysql txt.
17. Clean up
    1. Clipped relevant hard copy plots etc together and put into DB stack on desk and tossed the rest
    2. cat rfp.data03_05r3 rfp.data03_05r3 > rfp.data03_08.r3 (rm middle text)
    3. Wrote rdasc.f for ~emery/cedar and for /aurora/d/emery/rfp using rfp03_08.asc
    4. Sent email to Jiuhou Lei and Qian Wu about new rfp data (rfp.data03_08.r3 and rfp03_08.asc)
    5. Wrote wiki Data Services:Ingest with rfp steps in ingest