IMBOT: report of initial testing phase #5
Description
IMBOT: report of initial testing phase
IMBOT, an automatic data checking and reporting routine for INTERMAGNET definitive data products, is currently tested for one-second and one-minute data products. As techniques and reporting are different for both data products, we will deal with both resolution types separately.
IMBOT one-second
Contents and jobs
- testing any newly uploaded one-second data for 2018,2019 and 2020
- checking data content and meta information
- checking agreement with definitive one-minute products
- converting data to newest IMAGCDF standard for archiving (monthly files in a separate GIN storage)
- reporting to data submitter (since July 2021) and definitive data committee
- as submitted data products are transferred to newest IMAGCDF monthly archive files and any additionally provided metainformation is incorporated. Converted files are uploaded to a separate storage at Paris GIN (currently only for a few selected observatories.).
Checking and levels
- IMBOT distinguishes three quality levels
- level 0: missing or corrupted data files, "significant" deviations between one-minute and one-second data products
- level 1: data widely complete but obligatory meta information eventually missing, minor discrepancies between meta-information and tested quantities, ...
- level 2: data and meta information complete, no significant differences to one-minute data products
Chronology:
- implementation in the beginning of 2020
- introduced firstly at IM online workshop in March 2021
- reporting to DD committee only until June 2021
- reporting to data provider and DD committee since July 2021
- minor bug fixes in July 2021(see IMBOT on GITHUB)
IMBOT one-minute
Contents and jobs
- testing any newly uploaded one-minute data for 2020
- basic read test by MagPy
- automatic application of check1min
- reporting
Chronology:
- implementation in May 2021
- reporting to data provider and data checker since June 2021
Next steps
- IMBOT one-minute could already be moved from test-phase into a productive state
- IMBOT one-second: decision necessary on how to proceed from automatically tested data sets
- IMBOT one second: all incoming data sets are transferred into monthly archive files of identical format, containing all originally provided meta information. Ideally one should proceed here for archiving and publishing, avoiding multiple formats, packing processes and data coverage.
- Both packages are modular: thus it is very easy to extend the reporting features and add additional testing routines if data checkers are interested.
(Among those extensions could be graphs on basevalues and adopted baselines, frequency analyses, comparisons with other data sets, etc)