miniSEED 3 SimpleHeader & FDSN Identifiers DRAFT 20170718

[chad-earthscope]

In the attached drafts I have tried to incorporate the changes relative to the 20170708 versions that appeared to be consensus (at least more than one supporter).

The biggest change is a retro-grade to simple header followed by encoded data. After the discussion in #26 and some internal discussion at the IRIS DMC I think we need a target specification for evaluation that does not try to cater to very low latency transmission. If very low latency transmission is deemed a high priority goal by the FDSN worth the complexity needed to support it, then we have the other draft specifications to look at. Otherwise this becomes a relatively simple record.

This is the target I will be implementing for evaluation. Presuming CBOR does not turn out to be awful I will probably also try a variant that is all CBOR.

Here are high lights of the changes to FDSN Identifiers:

• Include a convention for temporary network codes.
• Add warning about SOH and LOG codes in the "O" instrument.
• Place holder for convention for location codes.

Here are high lights of the changes to miniSEED3:

• Simple header followed by payload, variable length
• Nanosecond resolution
• Increase sample rate resolution to 64-bit float
• Clarify that opaque encoding is "only for use in special scenarios, not intended for long term archiving"
• Clarify data publication version to be relative to data center.

I did not combine the record header indicator and version together as I though I would earlier because keeping them separate is easier to document as one being particularly ASCII values (MS) and a binary value that can go up to 255.

I also gave up on trying to have binary fields aligned to 2-byte words, I don't think that's as important as it used to be on older architectures.

CBOR could be swapped out should a consensus emerge that something else is better.

Things that still need treatment:

• Location identifier conventions
• General compression, which one?
• Reserved header structure

miniSEED3 - SimpleHeader DRAFT 20170718.pdf
FDSN Identifiers - DRAFT 20170718.pdf

[chad-earthscope]

Also, I did not add the channel codes for synthetics discussed in #7. I do think the FDSN should consider a category of codes for synthetics but that it is not critical to the next generation miniSEED discussion.

[andres-h]

On Wednesday 2017-07-19 01:38, Chad Trabant wrote: The biggest change is a retro-grade to simple header followed by encoded data. After the discussion in #26 and some internal discussion at the IRIS DMC I think we need a target specification for evaluation that does not try to cater to very low latency transmission. If very low latency transmission is deemed a high priority goal by the FDSN worth the complexity needed to support it, then we have the other draft specifications to look at. Otherwise this becomes a relatively simple record.

Dropping low-latency support, presumably because DMC (currently) cannot handle partial records is IMO not a future proof solution for the whole FDSN, but let the community decide. I don't agree that SimpleHeader is really simpler that Protobuf. If you program in machine code or assembler, yes, but nobody does that nowadays. For SimpleHeader you must implement your own parser (incl. CBOR, which is *much* more complex than Protobuf), whereas for Protobuf you can use one of the existing software tools to access the record just as a normal C++/Python/Javascript/Go/etc. object. Note that the CBOR RFC takes 53 pages, whereas Protobuf's RFC (draft) takes only 10 pages, of which the encoding specification is 3 pages. When using a library, such as libmseed/libslink, there is no difference for a user at all. The library can optionally hold data until a record is complete, in which case a user will never see partial records.

[chad-earthscope]

@crotwell @krischer @djeastonca @kaestli
I would like to know how all the others feel about this draft. Assuming high efficiency with very low latency is not a primary design goal for miniSEED3, how much consensus do we have? At this point with the simple header approach we are not discussing much beyond small details like sample rate resolution, encoding of extras and things that can easily be tweaked before being finalized. Thanks.

[crotwell]

👍 From me.

I feel we are very close and I like what is in this draft. The only issues I see are:

• choosing structure of extra headers (json, cbor, etc)
• define fdsn reserved extra header keys (basically mseed2 to extra heading mapping)
• sample rate 32 vs 64
• num samples 16 vs 32
• new data compression (ie Brotli)

with only the first one seeming like it is still unsettled enough to require significant time and effort.

[crotwell]

Small typo in the table in section 2. It has both field 12 and 13 at offset 36, but should be 36 and 38. Fields 13 and 14 should also be 38+... instead of 36+....

Total length in the paragraph abouve the tabel should also be "sum of 38..."