Checking one-second data

[leonro]

Checking one-second data

1. Data checking principle

A data-check routine for one second data should make use of one-minute (preferably checked data) and one-second step_1 data. Contents of these data sets are compared, and, for this purpose, partly recalculated based on INTERMAGNET and IAGA recommended techniques. A quick test will investigate data from a randomly chosen month and is sufficient to check the principal correctness underlying routines, formats and meta information. A full test will perform a detailed data check for all data. A report with all check details should be obtained and made accessible to the data submitter.

2. Issues to be checked

• Submitted files and formats: Are all requested files available and are they submitted in correct and readable formats (IAGA-2002, IAF, IMCDF).

• Meta information: Do all files contain the requested meta information and is this meta information consistent between all different files.

• Data content: Are the IAF one-minute, one-hour, and daily averages consistent and obtained using IM filter procedures. Are F values provided for one-minute and one-second, and are they independent measures of the field. Are delta F variations within acceptable limits. IMCDF: Is the leap second table up-to-date.

• Data consistency: Can one-minute data be reproduced by filtering one-second data using IM recommended procedures on filtering and outlier treatment.

• Baseline variation: Are periodic measurements available. Are residuals and amplitudes in an acceptable range.

• Eventually: Is the spectral data content as expected for a natural geomagnetic signal or are there any significant technical contributions. Noise level.

3. Step wise performance of the data check

(as performed with MagPy 0.9.6)

1. directories and existence of files

2. file names, access and basic header information

3. data content and consistency of minute data

4. second data and consistency with minute data (IM recommendations)

5. basevalues and adopted baseline variation

6. yearly means, meta information

7. activity - "not included"

The results of each step are summarized and rated with a grade from 1 to 6. Grades of 5 and above indicate "significant" issues regarding IM recommendations to be considered. A detailed description on how to use MagPy for these tests can be found in the manual (http://cobs.zamg.ac.at/data/index.php/en/downloads/category/1-magnetism).

4. Issues and suggestions

• Currently one second data is uploaded in two different formats using, however, various different packing routines. Should we suggest a common submission format? For IAGA2002 I would recommend daily zipped IAGA files. For IMCDF monthly files are already requested.

• threshold for differences between minute data and IM filtered one-second data: how much difference do we tolerate?

5. Recommendations for analysis

• To avoid downloading all files, one could mount the ftp directories on the local system. However, most files are uploaded as various different archives.
• Run only quick tests first. Only for data passing quick test, run full test.
• When using MagPy eventually skip step 5 (baseline) and 6 (meta, yearmean) for quick test, if minute data has been accepted
• Some issues are easy to be solved, others give indications for underlying procedures.
  Example:
  - if filtered second data differs from minute data only for few days, this indicates that most likely the outlier treatment differs from the IAGA 90% rule
  - if the difference of filtered second data and minute data is evenly distributed with values exceeding the numerical uncertainty then most likely a different filter procedure (window, type) was used
  - if the difference is not evenly distributed and some "function" is observable, then eventually different baseline treatments might be the cause

6. Example analysis:

Please note, the following table contains only a test analysis and is not related to any decision from the data committee. All test are also used as a run-time test for MagPy. Error messages related to certain files structures (e.g. step 5 failure related to BLV files or step 6 related to yearmean files) do not necessary indicate that the file structure is wrong. MagPy eventually just cannot interpret the format correctly yet. With Python3 MagPy >= 0.9.6 is required for the analysis.

2016

Obs	Format (second)	QuickTest	Observed issues	QuickTest result	FullTest	Issues	Summary
WIC	IMCDF	1,1,1,1,1,3,1	Location and AltitudeDiff	second data OK
UPS	IMCDF daily	1,5,1,4,	second data in daily cdfs, minor filter differences for two days, step5 failed	Monthly files obligatory?
TUC	IAGA (year zip)	1,	iaga second data not readable
SJG	IAGA (year zip)	1,1,5,1,3,3,2	inconsistency in IAF daily mean 2016-11-21, yearly means differ between yearmean and blv files	second data OK
SIT	IAGA (monthly zip)	1,1,3,4,3,3,2	hourly IAF complete?, filtered one-second differs slightly from one-minute	second data OK, check z comp
SHU	IAGA (year zip)	1,1,3,4,3,3,2	hourly IAF complete?, filtered one-second differs slightly from one-minute	second data OK, check z comp
NEW
MMB	IMCDF daily	1,5,	second data in daily cdfs, IAF data not readable	Monthly files obligatory?
MCQ
MAW	IMCDF	1	no leap second information
LYC
LRM	IMCDF	1	no leap second information
KNY
KDU
KAK
HRN
HON
HLP
HER
GNG
FRN
FRD
EBR	IMCDF	1,3,1,1,	file names not as expected for ImagCDF, step5 failed	second data OK
DED
CTA
CSY
CNB
CMO
CKI
BSL
BRW
BOU
BEL
BDV
ASP
ABK

[Khomutov-SY]

1. Data checking principle
   A data-check routine for one second data should make use of one-minute (preferably checked data) and one-second step_1 data. Contents of these data sets are compared, and, for this purpose, partly recalculated based on INTERMAGNET and IAGA recommended techniques.

1. main task is to check 1-sec data, not 1-min data. So, checker should focus on exactly 1-sec data, using 1-min as auxiliary data, similar as data from other IMOs or IGRF model

therefore

2. checker must first analyze 1-sec data itself because

• 1-min data are calculated from 1-sec data using a few transformations. These transformations are the source of possible errors. As result, these 1-min date can be incorrect. For example, single 60 nT spike in 1 sec data gives only 1 nT anomaly in 1-min data. IMO will remove 1-sec spike due to its evidence, but 1-min value can be calculated with this spike
• 1-min data can be obtained using raw data, which differ from checked 1-sec data. For example, for calculation of 1-min data IMO can use the results from main variometer, but 1-sec data can be obtained from other magnetometer

2. Issues to be checked
   Eventually: Is the spectral data content as expected for a natural geomagnetic signal or are there any significant technical contributions. Noise level.

1-sec data is need to study magnetic variations which more quick than 1-min data presents. So, main attention must be to the frequency band about 1 – 1/60 Hz. For example – see two plots


Is this acceptable?

4. Issues and suggestions
   threshold for differences between minute data and IM filtered one-second data: how much difference do we tolerate?

This question (i.e. large differences) suggest that minute values from files are incorrect. But our task is to check just 1-sec data, not 1-min data. Should we inform the IMO and/or the checker about this?

[leonro]

Dear Sergey, dear all,

I perfectly agree with your suggestions. The primary aim of one-second data checking is to test the validity of one second data and not 1 min. Thus, the "principle" sections need to be reformulated.

My basic reasoning behind the first sentence is: one-minute data has usually been published and accepted before one-second data is submitted. One-minute data has been termed "definitive" by the submitting observatory, indicating that all test have been already performed, possible error sources are eliminated and this data set is representative for yearly geomagnetic variations of this specific observatory. Thus, it is legitimate to use this data as a reference to judge other data products from the same observatory.

In reality, however, I complete agree with you. Observed differences might not be related to one-second data but be related to instrumental issues or unrecognized/unrecognizable issues while producing one-minute data. At the end, the one question remains: how to deal with differences?

Regarding the spectral content: I personally believe that this is a powerful tool to test the quality of one-second data without auxiliary assumptions. A big problem however will be to find a quantitative measure of validity threshold. Noise level amplitude and the period at which we reach a white-noise level gives some indication whether one-second data is suitable at all. But these parameters mainly depend on the instruments. Besides, the spectral content and its signal/noise ratio is also related to the geomagnetic latitude. The spectrograms from MMB show distinctive technical peaks at periods of 3.5 sec and 11 sec (horizontal) and some more in the vertical component, characterized by very narrow frequency bands. I am preety sure, that similar spikes will be observed in most data sets. I can include a spectral analysis within my test analysis of 2016, just to get an overview what we can expect in one-second data products. However, I cannot finish that until next week. Eventually I can add some examples here already so that we have a better basis for discussions what might be acceptable or not.

[leonro]

Update 0.1

1. Data checking principle

The primary aim of one-second data check is to examine and validate such submitted data sets. Various aspects regarding file structure, contained meta information, data in time and frequency domain are investigated for this purpose. Conclusions about the validity of the data set are drawn solely from inherent one-second data set characteristics. Comparison with "auxiliary" one-minute data products are also performed. The data checkers inform the submitting organization about any significant differences in both definitve data sets, yet (if below a certain treshold?) these differences are not used for acceptance/rejection of one second data products.

2. Aspects to be checked

1. Submitted files and formats: Are all requested files available and are they submitted in correct and readable formats (IAGA-2002, IMCDF).

   • 12 monthly IMCDF files with sec-data
   • (or 365/366 daily IAGA 2002 files)
   • correct file names

2. Meta information: Do all files contain the requested meta information and is this meta information consistent between all different files.

   • required meta information should is described in the IM format descriptions

3. Data content

   • Correct data coverage in all files.
   • If F values are provided, they should be independent measures of the field (S), or they need to be correctly denoted in the file structure.
   • Delta F variations should be within acceptable limits.
   • For IMCDF: the leap second table needs to be up-to-date.

4. Data quality
   Of particular importance should be the frequency range from 0.5 Hz to periods of about 90 sec

   • Noise level (in frequnecy domain)
     (too be clarified: in which frequency range, on which data sets, critical threshold value)
     my suggestions:
     calculate average spectrum for 10 quiet days for x,y,and z (too be defined, at least 6 of these days need to be available)
     IM instrumet requirement is <10pT. I would suggest a critical threshold of <30pT for acceptance of data
   • White noise range (in frequnecy domain)
     my suggestions:
     testing the noise level is usually sufficient
   • Narrow-band "technical" disturbances should not be present
     (too be clarified)
   • Broad-band disturbances should not be present
     (too be clarified)

5. Data consistency

   • Is meta information consistent with one-minute data products, can one-minute data be reproduced by filtering one-second data using IM recommended procedures on filtering and outlier treatment.

3. Currently available software solutions:

Aspects 1,2,3,5 can be checked using MagPy's data checking routine. Aspect 4 requires additional software. Power spectra of single days can be calculated and visually inspected using MagPy. Average spectra, noise level determinations are not yet possible.

The following table was reorganized with one-second data focus, based on random one-month analysis. I did not yet include a summary of meta information tests, but they are included automatically in the MagPy quick test.

2016

Quick Test column contains MagPy data check results (XMagPy->Extra->DataCheck) from steps 1,2, and 4 which primarly focus on one-second data.
Noise Level and white noise are approximate values so far, estimated from the "flat" HF part from the X component (XMagPy->Analysis->Power). Frequnecy characteristics are qualitative descriptions. For all frequency related tests below a power spectrum has been calculated from a randomly picked day and therefore is not characteristic for the full data set.

Obs	Format (second)	QuickTest (one-second focus)	Observed issues (one-second only)	NoiseLevel	White-noise flattening	Frequency characteristics	Summary
WIC	IMCDF	1,1,1	None	~8 pT	None	Nothing special	data OK, format OK
UPS	IMCDF daily	1,5,4	second data in daily cdfs, minor differences to one-minute data for individual days	~20pT	below ~10 sec	Nothing special	data OK, format Update
TUC	IAGA (year zip)	1	iaga second data not readable in data check (DataCheck issue), single files no problem	~20 pT	broad-band signatures	Broad-band signature at periods below minutes	Open
SJG	IAGA (year zip)	1,1,1	None	~20 pT	below ~10 sec	Nothing special	data OK, format OK
SIT	IAGA (monthly zip)	1,1,4	filtered one-second differs slightly from one-minute, particularly check z-component	~10 pT	None	Nothing special	data OK, format OK
SHU	IAGA (year zip)	1,1,4	filtered one-second differs slightly from one-minute, particularly check z-component	>100 pT	below ~25 sec	Nothing special	Noise Level too high, format OK
NEW
MMB	IMCDF daily	1,5,	second data in daily cdfs	~ 11 pT	below ~15 sec	individual "technical" spikes	technical spikes, format Update
MCQ
MAW	IMCDF	1,3,4	no leap second information, file names not as expected, significant differences between min and sec (>5nT)	~400 pT	below ~20sec	individual "technical" spikes	Noise Level too high, format Update
LYC
LRM	IMCDF	1	no leap second information, file names not as expected
KNY
KDU
KAK	IMCDF daily	1,	second data in daily cdfs, single file analysis no problem, imcdf not readable in data check (DataCheck issue)	~10pT	None	Nothing special	data OK, format Update
HRN
HON
HLP
HER
GNG
FRN
FRD
EBR	IMCDF	1,3,1	file names not as expected for ImagCDF	100 pT	below 10 sec	minor technical peaks	noise level, format Update
DED
CTA
CSY
CNB
CMO
CKI
BSL
BRW
BOU
BEL	IMCDF	1,3,3	file names not as expected for ImagCDF, minor differences between minute and sec (1nT)	~ 8pT	None	Nothing Special	data OK, format Update
BDV
ASP
ABK

[Khomutov-SY]

Dear Roman,

4. Data quality
   Of particular importance should be the frequency range from 0.5 Hz to periods of about 90 sec
   Noise level (in frequnecy domain)
   (too be clarified: in which frequency range, on which data sets, critical threshold value)
   my suggestions:
   calculate average spectrum for 10 quiet days for x,y,and z (too be defined, at least 6 of these days need to be available)
   IM instrumet requirement is <10pT. I would suggest a critical threshold of <30pT for acceptance of data
   White noise range (in frequnecy domain)
   my suggestions:
   testing the noise level is usually sufficient
   Narrow-band "technical" disturbances should not be present
   (too be clarified)
   Broad-band disturbances should not be present
   (too be clarified)

This item is very closely related to the questions
(1) what should the verifier check?
(2) what level of authority does it have?

Magnetic field at some IMOs, which I check, has significant noise, a pick-to-pick value of 1-min data reaches 5 nT. In general, this is "uniform" noise and spikes and jumps due to different artificial sources, not from magnetometers or processing. So, there no any possibilities to improve situations by simple way. Should the checker accept this data? Can the checker accept this data?

Other example. I see spike in IMO 1-min DD, i.e. up to 3 nT. Background field is very quiet. Spike presences in variometer and scalar data, so there is no visible spike in deltaF. The field at IMO at distance of 1000 km is clear. My question – what is it? Answer – it is natural field variation. But when I use 1-sec data, I see (1) delay between variation and scalar records up to 5-10 sec and (2) spike is not "spike", really it is short bay-like smooth variation. Similar signal can be the results of car, moved at some distance from magnetometers. My question – may be car? Answer – we haven't seen the car, so we can't consider it's a noise and remove it.

Where are the limits of my rights as checker? I think that IMO has priority in similar issue. So I have to accept the data as is. ???

This is also the case with second data.

Narrow-band "technical" disturbances should not be present
Broad-band disturbances should not be present

But if the data is similar to the ones in my plots in the previous comment, what should we do?

[leonro]

Magnetic field at some IMOs, which I check, has significant noise, a pick-to-pick value of 1-min data reaches 5 nT. In general, this is "uniform" noise and spikes and jumps due to different artificial sources, not from magnetometers or processing. So, there no any possibilities to improve situations by simple way. Should the checker accept this data? Can the checker accept this data?

I personally would say no. Nevertheless we need to setup a transparent ruleset: a noise level threshold would be sufficient for that case. To my understanding: INTERMAGNET comprises high quality observatories. Thus each submitted data set need to satisfy some criteria. For one-second data these criteria should be relatively high. Noisy data below 1 min periods does not bear any useful information content exceeding the 1 minute data products.
Same with too many "technical" narrow band frequency spikes which predominantly are related to local disturbances in systems, grounding, power, etc. One could cover this issue also with the noiselevel criteria, however, there might be better ideas as well.

Individual spikes in the timeseries won't show up in the frequency diagrams as they have no distinct periodicity. Such spikes need to be checked there using e.g. delta F etc. similar as done for one-miunte data.

The rights of the data checker are extensively discussed at the moment regarding one-minute data. I see the position of the data checker a bit like a referee for a journal (also to be applied at one-second data) with one benefit: We can establish a clear ruleset. Based on this set most data can be judged (I like the idea of a simple checklist). Remaining questionable data or rejected data is send back to the IMO, with the option to resubmit after corrections (to the same checker) or resubmit requesting help and/or cross-check. Such resubmissions could end up in a pool with general access of all data checkers so that the individual checker is not left alone with particularly difficult decisions. Generally, I would say that new products like one-second data sets need strict rules. And this ruieset has to be established by us...

[leonro]

Another issue when looking at the one-second data currently submitted. Most ImagCDFdata sets are not following the IM naming and coverage rule and sometimes the files do not contain the expected time range. They start e.g. one second earlier in the previous month. And often, meta information is missing as well (especially contents which are not covered in the IAGA2002 header, like reference). It is possible to read this data and export it in a proper way. I even could write a job doing that automatically... However, is this something we should do or is this the duty of the IMO?`We might convert existing submission until e.g. 2018... Anyway it is a significant piece of work..

[benoitheumez]

Just a thought. The same problems arises in the discussion about data checking of 1-min data.
I wonder if judging the quality of data is only the data-checker's job (INTERMAGNET job).
Can IMOs give extra information about data quality with data error statistics?
An indication of error stats would give the data users a good information and may be decisive about whether using the data or not for a given application. That could also be useful to IMOs to realise their data, absolutes are good or not.

[JanReda]

For me it seems that we should have different expectation for 1-min definitive against for 1-sec data. 1-min definitive data are nice for analysis of not fast changes of magnetic field e.g. secular variations. May be I am wrong, but I suppose that 1-sec definitive will not be used for such investigations.
It seems to me that there is necessary I bit redefine status “definitive” for 1-sec definitive.
Because these data are named “definitive” their absolute level should of course agree with -1-min definitive.
However much important for 1-sec definitive are such requirements as:

• time accuracy (also 10 msec as for 1-min, but higher priority concerning this requirement; in case of 1-min some filtration causes shifts even 30 sec)
• statistically is needed lower level of artificial disturbances than in case of 1-min
• low noise of sensors
  Because of a large amount of data more tests should be realized in frequency domain

[leonro]

Yes, i think that this is rather important: one-second data should not be judged solely by comparing it to one-minute products. If there are significant differences between one-minute and one second data sets, then this should be pointed out to the IMO but only if theses diffs are getting large, then the data set should be rejected ("significant" and "large" however need some definition). I can put up some examples here later today to get an idea what we can expect.
Some essential parameters such as timing accuracy, however, cannot really be checked independently for HF data or am I wrong? Such checks would require some onsite test with timed field signatures. Thus, such data needs to be an extra info from the IMO.