File : ?/SpecLab/html/Seven_Bit_Packets_for_the_serial_port.htm
Online: www.qsl.net/dl4yhf/speclab/Seven_Bit_Packets_for_the_serial_port.htm
Date : 2023-08-06
Author: Wolfgang Buescher, DL4YHF .

Contents
  1. Introduction
    1. Glossary and Document Conventions
  2. Reason for the 'seven-bit-per-byte' payload
  3. The Packet Header Byte
  4. Optional / "Extra" Header Bytes
  5. Packets with 'densely packed' payload (example: multi-channel audio packet)
    1. Example: Densely packed 24-bit audio samples
    2. Example: Densely packed audio format description
    3. Fixed-point numbers (in 'densely packed' packets)
  6. Packets with '7-bit ASCII strings' (example describing the audio packet structure)
  7. Packet content types
  8. Reference implementation (written in plain "C")

See also (other parts of the Spectrum Lab manual): Audio-via-COM-port (or UDP), Keyword Index ("A to Z").

Introduction

This file describes a 'packet format' for the serial port, sufficiently robust to allow the receiver to recover quickly and reliably (after single- or multi-bit errors), without too much protocol overhead, suitable to be sent from simple microcontrollers.
It was first used in Spectrum Lab, to connect external A/D converters driven by simple Microcontrollers with UART interfaces. On the PC side, a 'COM' port, or USB-to-RS-232 converter may be used.
Details about receiving 'Audio via COM port' in Spectrum Lab are in an extra file.
A reference implementation is available in "C" - see last chapter of this document.

Glossary and Document Conventions

Frame
On a serial port like RS232, a frame consists of start bit, data bits, optional parity bit (not used here), at least one stop bit.
Here, we use eight data bits per 'serial frame', numbered from zero (LSBit) to seven (MSBit).

Packet
A block of bytes on the serial port. In the principle described here, the first byte in the block (header byte) has the most significant bit (bit 7) set, all other bytes of the same packet have bit seven cleared.

LSBit
Least Significant Bit in a byte, bitfield, or integer value.

MSBit
Most Significant Bit in a byte, bitfield, or integer value.

CRC
Cyclic Reduncancy Check. In the packet structure described in this document, a CRC is optional, and generally not used (at least not for audio samples travelling over a 'reliable' link like RS-232 or RS-485).

(Audio) Sample Point
In this context, a group of 'audio channels' all sampled at the same time, and wrapped in a single (short) packet for transmission over a serial port.

UART, USART
"Universal [Synchronous/] Asynchronous Reveiver / Transmitter".
The kind of 'on-chip peripheral' found in almost any modern microcontroller. If it's a USART, to communicate with a PC's "COM" port, we only need the UART part.

0xNNNNNNNN
Throughout this document, the prefix '0x' indicates a hexadecimal number, short: 'hex'.
Digits 'N' range from 0 to 9, and A to F.
An 8-digit hex number implies 32 bits, etc.

0bNNNNNNNN
Here, the prefix '0b' indicates a binary number.
Digits 'N' may only be 0 (zero) or 1 (one).
Each digit represents a bit. Thus, an 8-digit binary number (not including the '0b' prefix) may represent a 'BYTE' (8 bits).
In this document, if a binary number consists of multiple bitgroups (as in the packet header byte), those bitgroups may be separated by space characters, e.g.:
0b1 00 00111 (PACKET_HEADER_FLAG, PACKET_HDR_TYPE_AUDIO, SHORT_PAYLOAD_LENGTH=7).
In this documentation, asterisks as placeholders in a binary (or hexadecimal) value represent bits (or digits) with variable content. For example,
0b1 00 0****   may be a packet header byte for 'Audio', with a payload length of up to 15 bytes.


Reason for the 'seven-bit-per-byte' payload

For a simple error recovery, after the receiver 'lost track' about the packet borders in the received stream, it discards anything until the detection of a header byte.
Sidenote..
If the receiver wasn't a Windows PC but a microcontroller with the usual versatile UART / USART on-chip peripheral, serial frames with nine bits, or the precisely timed detection or transmission of a 'break' signal to indicate the begin of a packet would be more elegant and less wasteful. But Windows can neither accurately time the transmission of bytes or the 'break' signal (not to fractions of a millisecond), nor does it allow to 'measure times' during the reception of bytes with the standard 'Serial port API' - at least not sufficiently reliable - so all these methods are impossible, or terribly complex to implement on a windows PC.

So, all we have are 8-bit frames to construct our 'packets' from. We want the packets to be easy to assemble, easy to recognize on reception (even for a simple microcontroller), and we want to detect the 'next' packet even after any number of bits or bytes have been missing, or noise has been 'injected' into the serial signal (which may not only be electric, but also an infrared link or a simple wireless link without forward error correction).

Each packet begins with a header byte. Only in the packet header byte, bit 7 (the most significant bit) is set. The remaining seven bits (numbered six to zero) in the header byte indicate the type of packet, and (for 'small packets') the size of the packet. The next subchapters describe the structure of the packet header byte in detail, and the zero to three bytes (with only seven bits used) that follow immediately after the header byte, depending on the bitfields in the header byte.

The Packet Header Byte

As already mentioned in the introduction, only the packet header byte has its MSBit (bit seven) set, to distinguish it from the payload and optional bytes for larger packets, and packets protected with CRC, etc.


Packet Byte[0] = PACKET_HEADER_BYTE :
Bit 7 = PACKET_HEADER_FLAG : Always SET (0b1, or bitmask 0x80 in the C sourcecode)
Bits 6..5 in the header byte determines the basic packet type (PACKET_HEADER_TYPE):
  0b00 : PACKET_HDR_TYPE_AUDIO.
         This packet contains a "short" audio sample point, with the structure 
         specified in an extra packet (PACKET_CONTENT_TYPE_AUDIO_SAMPLE_FORMAT,
         optional but recommended if the microcontroller firmware permits).
  0b01 : PACKET_HDR_TYPE_OTHER.
         This packet contains something that is specified by an extra byte of the packet 
         (example later, see PACKET_CONTENT_TYPE).
  0b10 : PACKET_HDR_TYPE_ASCII.
         This packet contains an ASCII STRING, with seven bits per character, easily parsed (example later).
  0b11 : PACKET_HDR_TYPE_RESERVED.
         Reserved for future use (packet flow control, CRC-protected payload, etc).
         Also for this header type, the packet content type is specified
         in an extra byte.
 
Bits 4..0 = SHORT_PAYLOAD_LENGTH : Determines the number bytes that follow
                                  in the payload. The header byte itself,
                                  as well as the 'optional header bytes'
                                  listed in the next chapter,
                                  are not counted (included) in this bitfield.
       SHORT_PAYLOAD_LENGTH = 1...30 indicate "the number of bytes that follow,
                                              with SEVEN usable bits each".
       SHORT_PAYLOAD_LENGTH = 0 : Length of the payload not indicated. Just keep reading
                                  until an end-of-string marker (0x00 in PACKET_HDR_TYPE_ASCII),
                                  or until the next packet header byte arrives. (*)
       SHORT_PAYLOAD_LENGTH = 31: Larger packet, 
                                  with the actual packet length following in 
                                  the next two bytes, directly after the 
                                  PACKET_HEADER_BYTE . 
                                  See next chapter, LONG_PAYLOAD_LENGTH.

Anything after the packet header byte must have bit 7 cleared, thus there are only 7 bits of "payload per byte". This also applies to the optional 'extra header bytes' listed in the next chapter.
Integers larger than 7 bits are sent in subsequent bytes with the LEAST significant 7 bits first (aka 'little endian' byte order).
The next higher-valued 7 bits follow in the next packet-byte, etc. Depending on the payload, integers may be sent gap-less as in the example for 2 * 24 bit integers in a 'densely packed' audio packet.

That's all a receiver needs to know to extract packets from a stream of frames. Even without knowing the actual payload, the receiver can decide if the packet is correct or not, or if he wants to skip it in the byte stream and just wait for the next packet header byte.


(*) About SHORT_PAYLOAD_LENGTH = 0 :
Packets with no length indication at all were allowed with the following in mind:
Very small microcontrollers (like certain 8-bit PICs) only have a few bytes of internal RAM, and thus cannot assemble an entire, variable-sized packet in RAM before sending it (and counting the number of frames to send within the packet in advance).
Instead, using PACKET_HDR_TYPE_ASCII, the uC may simply 'print' directly into the UART's transmit buffer (or 8-bit transmit register), as if it was connected to an 'ordinary' terminal program.
By chance, the packet header byte will be 0b1 10 00000 = 0xC0, which appears like a 'Latin capital letter A with grave', if your terminal program interprets 0xC0 as 'eight-bit extended ASCII'. This makes debugging such 'seven-bit-ASCII packets' very simple.


Optional / "Extra" Header Bytes

Even though these bytes belong to the 'header' (at least logically), they all have their MSBit cleared, and thus to a simple receiver look like ordinary payload bytes (with seven bits of usable payload per 8-bit frame).
Like the packet header byte, none of these optional extra header bytes is counted in the SHORT_PAYLOAD_LENGTH or LONG_PAYLOAD_LENGTH, last not least because their presence is indicated by the packet header byte itself.

When to send (or 'expect') these 'extra' bytes, immediately after the Packet Header Byte ?
  1. LONG_PAYLOAD_LENGTH: If the SHORT_PAYLOAD_LENGTH (five-bit field in the packet header byte) contains 31, the actual packet payload size isn't 31 frames, but specified in two bytes following immediately after the header-byte.
    Since both of these bytes must have their MSBits cleared (to tell them from the packet header byte), the maximum packet length encoded this way, as a 14-bit unsigned integer, is 2^14 - 1 = 16383 [frames], each serial frame with a seven-bit payload, thus approximately 16383 * 7 / 8 = 14335 'fully usable' bytes of payload when densely packed. If the payload is a 7-bit ASCII string (with one character per frame), this 14-bit LONG_PAYLOAD_LENGTH is simply the number of characters in the string (including the optional trailing zero-byte, which is recommended to make parsing 'C friendly').

  2. PACKET_CONTENT_TYPE: If the PACKET_HEADER_TYPE is PACKET_HDR_TYPE_OTHER,
    the Packet Header Byte (and the optional 14-bit PACKET_SIZE) are followed by another 7-bit value that indicates the packet's content (usually including a header structure define in future versions of this document). Allowed values follow in a later chapter, e.g. PACKET_CONTENT_TYPE_AUDIO_SAMPLE_FORMAT.
    Some (future) packet content types also indicate the presence, type, and width of a CRC for the packet's payload. Besides that (but this outside the scope of this document), certain packets may have additional fields indicating the content type in the payload, available after re-arranging the 7-bit payload into a nicer, easier-to-parse 8-bit data block, containing e.g. a structured "C" data type.

Example for a 'long' packet, including packet header byte and three 'extra' bytes to indicate the length and content:

  -Packet Byte [0]- -Packet Byte [1]- -Packet Byte [2]- -Packet Byte [3]- -Packet Byte [4]- ... -Packet Byte [259]-
  [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0]     [7 6 5 4 3 2 1 0]
   : : : : : : : :   : : : : : : : :   : : : : : : : :   : : : : : : : :   : : : : : : : :       : : : : : : : :  
   1 0 1 1 1 1 1 1   0 1 1 1 1 1 1 1   0 0 0 0 0 0 0 1   0 ? ? ? ? ? ? ?   0 x x x x x x x       0 x x x x x x x
   | |_| |_______|     |___________|     |___________|     |___________|     |___________|         |___________|
   |  |      |                  |                  |            |             user payload  ...     user payload
   |  |  SHORT_PAYLOAD_LENGTH    |                  |   PACKET_CONTENT_TYPE,   |________________________________|
   |  |  = 31: "Long packet"  LONG_PAYLOAD_LENGTH,  |    | not counted in the     255 * 7 bits of 'user payload'
   |  PACKET_HDR_TYPE_OTHER   lower seven bits     |    | LONG_PAYLOAD_LENGTH",
 PACKET_HEADER_FLAG                LONG_PAYLOAD_LENGTH,  | presence indicated by PACKET_HDR_TYPE_OTHER' !
  |______________|                 upper seven bits.    '-----------------------------------------------    
  PACKET_HEADER_BYTE            0b0000001 1111111 = 0x00FF = 255 'frames' with SEVEN bits of payload per 'frame'
                       |_____________________________|
                        presence of this field (LONG_PAYLOAD_LENGTH)
                        indicated by SHORT_PAYLOAD_LENGTH=31 .

Packets with 'densely packed' payload

Principle for a packet with 'densely' packed payload (supported by Spectrum Lab for multi-channel sample streams since 2023-07):
  • There shall be no 'unused' bits in the payload, except (maybe) for the very last byte
  • At the begin of the payload, begin with the first bitgroup (integer, or whatever) at BIT ZERO (LSBit),
  • For DENSE PACKING, if space remains in bits 6..1 of a "payload byte", fill them with the NEXT value, again at BIT ZERO of the next integer (or whatever).
  • As for binary numbers, they are only DISPLAYED beginning with bit 7 (from left to right). In memory, consider a "byte begins at bit zero", then things become easier. Just because the CPU may use Little Endian BYTE ORDER, don't reverse "bits in a byte" !
With PACKET_LENGTH (in the PACKET_HEADER_BYTE) up to 30 times 7 "usable" bits, the netto payload of short packets is 30 * 7 = 210 bits.
Even for a hypothetical A/D converter with 32 bits per sample in a single channel, a 'densely packed' audio sampling point may contain up to six channels. With 24 bits per sample in a single channel a sample point could have up to int(210 / 24) = 8 channels, which should be sufficient even for very demanding applications.

In the reference implementation, SBP_WriteBitgroup() is used to append any bitgroup (value) with a length of 2 to 32 bits to the 'packet buffer' (seven bit payload) in RAM, taking care to leave the gaps for bit seven.

Example with "densely packed" 24-bit samples, two channels

The following example shows a stream of samples, acquired by from a 24-bit A/D converter with two analog inputs (actually, an ADS131). Both "ASCII-arts" with green background actually show the same content, the only difference is the display of bits in each byte. With the conventional binary format (MSBit on the left, LSBit on the right), the mapping of bits in the 24-bit samples from channels 'A' and 'B' look confusing, but when displayed with the LSBbit first, the principle of this "dense packing" becomes clearer. In any case, we use LITTLE ENDIAN, aka "Intel Byte Order". Of course, we don't mirror bits in a byte (been there, seen that in a CAN-bus network, ouch...) !



 Example for the "Bit layout" in the payload (best viewed as preformatted text with a monospaced font):

  -Packet Byte [1]- -Packet Byte [2]- -Packet Byte [3]- -Packet Byte [4]- -Packet Byte [5]- -Packet Byte [6]- -Packet Byte [7]-
  [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0] [7 6 5 4 3 2 1 0]
   0(bit 7 cleared)  0 : : : : : : :   0 : : : : : : :   0 : : : : : : :   0 : : : : : : :   0 : : : : : : :   0 : : : : : : :
     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : : 
Bit numbers in channels 'A' and 'B':     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :
     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :  
     A A A A A A A     A A A A A A A     A A A A A A A     B B B B A A A     B B B B B B B     B B B B B B B     - B B B B B B 
     0 0 0 0 0 0 0     1 1 1 1 0 0 0     2 1 1 1 1 1 1     0 0 0 0 2 2 2     1 0 0 0 0 0 0     1 1 1 1 1 1 1     - 2 2 2 2 1 1
     6 5 4 3 2 1 0     3 2 1 0 9 8 7     0 9 8 7 6 5 4     3 2 1 0 3 2 1     0 9 8 7 6 5 4     7 6 5 4 3 2 1     - 3 2 1 0 9 8
     |<------------------ Channel A -----------------|     |Ch B | |Ch A|     |<-------- Channel B ------------- ^ --------->|
                                                                                                                /|\
   ^                 ^                 ^                 ^                   One unused ("remaining") bit HERE --'
  /|\               /|\               /|\               /|\                ^                 ^                 ^
   |                 |                 |                 |                /|\               /|\               /|\
 ,-'-----------------'-----------------'-----------------'-----------------'-----------------'-----------------'
 '-- Bit seven in each 'payload frame' cleared to tell them from the packet header byte (not shown here)

If, only for the display, we show the bits in each byte in reverse order (beginning with the LSBit = bit zero on the left, and the MSBit = bit seven on the right), the 'dense filling' principle gets clearer:


 The same "Bit layout" in the payload as shown above, but bits in a byte shown in reversed order:

  -Packet Byte [1]- -Packet Byte [2]- -Packet Byte [3]- -Packet Byte [4]- -Packet Byte [5]- -Packet Byte [6]- -Packet Byte [7]-
  [0 1 2 3 4 5 6 7] [0 1 2 3 4 5 6 7] [0 1 2 3 4 5 6 7] [0 1 2 3 4 5 6 7] [0 1 2 3 4 5 6 7] [0 1 2 3 4 5 6 7] [0 1 2 3 4 5 6 7]
   : : : : : : : 0(bit 7 cleared)  0   : : : : : : : 0   : : : : : : : 0   : : : : : : : 0   : : : : : : : 0   : : : : : : : 0
Bit numbers in channels 'A' and 'B'    : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :/|\
   : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : :     : : : : : : : |
   A A A A A A A     A A A A A A A     A A A A A A A     A A A B B B B     B B B B B B B     B B B B B B B     B B B B B B - |
   0 0 0 0 0 0 0     0 0 0 1 1 1 1     1 1 1 1 1 1 2     2 2 2 0 0 0 0     0 0 0 0 0 0 0     1 1 1 1 1 1 1     1 1 2 2 2 2 - |
   0 1 2 3 4 5 6     7 8 9 0 1 2 3     4 5 6 7 8 9 0     1 2 3 0 1 2 3     4 5 6 7 8 9 0     1 2 3 4 5 6 7     8 9 0 1 2 3 - |
   |<------------------ Channel A -------------------------->| |<------------------- Channel B ------------------------->| ^ |
                 ^                 ^                 ^                 ^                                                  /|\|
                /|\               /|\               /|\               /|\             One unused ("remaining") bit HERE ---' |
   ,-------------'-----------------'-----------------'-----------------'-----------------------------------------------------'
   '-- Bit seven in each 'payload frame' cleared to tell them from the packet header byte (not shown here)


The actual format of the samples doesn't need to be specified in each sample.
It can be either taken from SL's configuration, or sent once every few hundred or thousand samples (like other types of packets that don't need to be sent frequently, like timestamps). Remember, the sender (microcontroller with ADC) starts emitting samples at any time, and the receiver (SL) may start at any time later, and thus may miss the initial packets. In fact, the receiver may have have to restart at any time, and (with a unidirectional connection) it cannot 'ask' the sender to repeat a certain packet.
Thus the need for a periodic (but not necessarily 'very frequent') transmission of the stream parameters like..
  • number of channels in each sample point,
  • number of bits per "single sample",
  • data type (floating point ? integer ? signed or unsigned ?),
  • sampling rate (precisely: number of sample points per second),
  • and maybe an occasionally transmitted timestamp.
We could either specify binary packet formats for all those parameters (awful due to the payload's seven-bit chunks), as in the following example. As a much simpler alternative, the stream parameters can be specified as an easy-to-parse ASCII string (example in a later chapter). But first, the format description of any audio-packet in binary form.

Example: Densely packed audio format description


The following example shows the packet structure, with values to describe the two-channel, 24-bit, signed integer audio packets from the previous subchapter. It does not contain audio samples itself, but provides everything the receiver needs to know for properly unpacking audio packets.
Note that only byte[0] (=packet header byte) to byte[3] ('number of channels per sample point') are mandatory. Beginning at byte[4], fields may be left out. The receiver detects this by the SHORT_PAYLOAD_LENGTH in the header byte.
For example, the sender may omit the sampling rate, if this never changes, and the user (of Spectrum Lab) is careful enough to enter the correct sampling rate manually.
If other fields are omitted (like byte[4] with the data type and endianness), and there is a 'default' value specified for that field, the default is used.


Byte[0] = PACKET_HEADER_BYTE : 0b1 01 0**** (PACKET_HEADER_FLAG, PACKET_HDR_TYPE_OTHER, SHORT_PAYLOAD_LENGTH=2 to 9)
Byte[1] = PACKET_CONTENT_TYPE : 0x01 = PACKET_CONTENT_TYPE_AUDIO_SAMPLE_FORMAT
          (again, since the presence of this 'optional header byte'
           is indicated in the packet header byte itself, it's not
           counted as part of the 'payload'. Instead, it belongs to the header,
           which in this case consists of Byte[0] and Byte [1] .
           Here, the variable-sized payload begins with the next byte,
           because this packet only needs a SHORT_PAYLOAD_LENGTH.)
Byte[2] : Number of bits per single sample, 1 .. 127, usually 1..32 (supported by Spectrum Lab)
Byte[3] : Number of channels per sample point, 1..127, usually 1..4 (supported by Spectrum Lab)
Byte[4] : Data type and endianness. 
            0 = Default. Signed integer and Little Endian. Highly recommended.
                Densely packed, with "bits 6..0 first", "bits 13..7 next", etc .
                In fact, the first reference implementation in Spectrum Lab
                didn't support anything else (no unsigned int, and no Big Endian).
            1 = Unsigned integer and Little Endian. For simple unipolar A/D converters.
            2 = Reserved for some BIG endian format (signed).
            3 = Reserved for another BIG endian format (unsigned).
            4 = IEEE-754 32-bit floating point ("single precision"), Little Endian as on Intel CPUs,
                each 32-bit value split into FIVE subsequent seven-bit slots with in the frame
                (to avoid too much bit-fiddling in the sender and receiver)
            Data types 5..127 reserved for "future use" / compressed sample / u-Law, etc.
Byte[5] : least significant SEVEN bits of the integer sampling rate
Byte[6] : Next SEVEN bits (bits 13..7) of the integer sampling rate
Byte[7] : Next SEVEN bits (bits 20..14) of the integer sampling rate (ok for up to ~ 1 MSample / second).
            If the sampling rate is omitted, Spectrum Lab will use whatever
            the user has entered in the Audio Settings tab.
Byte[8..10] : Reserved for a future 'fractional part', if the sampling rate isn't an integer.
Byte[11..?] : Reserved for future use. From the PACKET_LENGTH, the receiver knows if these bytes are sent.

As expected (and obvious from the above example), 'binary data' chopped into 7-bit chunks for the packet' payload tend to get messy. That's why easily parsable TEXT (consisting of 7-bit ASCII characters) for any kind of data may be better suited. Thus, if the serial port's bandwidth allows specifying the audio sample format as a string, and it can 'queue up' a few audio samples in memory (while sending such a string), the PACKET_CONTENT_TYPE_AUDIO_SAMPLE_FORMAT may be replaced by an 'easily parseable PACKET_HDR_TYPE_ASCII, like the one shown in the Seven-bit ASCII packet example.

Fixed-point numbers (in 'densely packed' packets)

Some of the pre-defined packet types (e.g. PACKET_CONTENT_TYPE_TIME_OF_DAY) contain integer fields with an optional fractional part. For example, a data acquisiton system (microcontroller-driven ADC with GPSDO and GPS receiver) may be able to deliver accurately timestamped samples, with a timing resolution of a few nanoseconds. The TIME_OF_DAY is measured in "seconds after midnight, in UTC", but for this hypothetical system, the "number of seconds" isn't just an integer, but a "fixed point number" with an integer part and a fractional part. The integer part is just a signed integer, and the fractional part is another signed integer, so the receiver (at least Spectrum Lab) will divide the fractional part by
  2^(21-1) = 1048576, because if sent at all,
the fractional part has been sent as a 21-bit signed integer with that scaling factor.
Details about this particular fixed-point format is in the reference implementation, SBP_DoubleToFixedPoint21() and SBP_FixedPoint21ToDouble(). The same fixed-point format with optional fractional part is also used in packets with PACKET_CONTENT_TYPE_AUDIO_SAMPLE_FORMAT, to report the sampling rate (if for some reason the microcontroller cannot use an integer sampling rate).
Integers with an optional fractional part can only be placed at the end of a packet (payload).
If (any only if) the fractional part is the last field in a packet, the packet's SHORT_PAYLOAD_LENGTH allows the receiver to find out if that bitgroup in the payload exits. Thus the fractional part can only be optional at the end of a packet, as in the following example:

Byte[0] = PACKET_HEADER_BYTE : 0b1 01 00*** (PACKET_HEADER_FLAG, PACKET_HDR_TYPE_OTHER, SHORT_PAYLOAD_LENGTH=3..6)
Byte[1] = PACKET_CONTENT_TYPE : 0x02 = PACKET_CONTENT_TYPE_TIME_OF_DAY
Byte[2] : Bits  6.. 0 of the integer part, mandatory (number of seconds in UTC)
Byte[3] : Bits 13.. 7 of the integer part, mandatory ..
Byte[4] : Bits 21..14 of the integer part, mandatory ...
Byte[5] : Bits  6.. 0 of the fractional part \ 
Byte[6] : Bits 13.. 7 of the fractional part |-- optional !
Byte[7] : Bits 21..14 of the fractional part /


Packets with '7-bit ASCII strings' (example describing the audio packet structure)

Example with PACKET_HDR_TYPE_ASCII to describe the sample format: (again, with 7 bits "netto payload" per byte, no need for any bit-fiddling, and no need to worry about endianness and data types):

AudioSampleFormat: BitsPerSample=24 Channels=4 SampRate=123456.789 UnixDate=2023-07-13_21:15:00.000
|________________| |______________________________________________________________________________|
Keyword/Label for    Audio-sample-specific Key=value pairs (all of them optional, in any sequence)
 the 'String Parser'

After recognizing the keyword (or 'label') at the begin of the ASCII string, the parser identifies each parameter in the list of Key=value pairs, modify only those parameters specified in the string.

If, as suggested in chapter 3, the packet header doesn't indicate the size (number of 'frames', here identical to the number of characters in the string), the string must be terminated by a trailing zero-byte, as commonly used in the "C" programming language.

Packet content types

Because the packets described in this document were primarily intended to exchange digitized audio (or similar streams for scientific purposes, e.g. ELF/VLF, see www.vlf.it), anything that doesn't contain audio samples (identified by two-bit-group 'PACKET_HDR_TYPE_AUDIO in the packet header byte) or ASCII strings must have it's PACKET_HEADER_TYPE set to PACKET_HDR_TYPE_OTHER.
The 'real' type of the packet's payload then follows in the PACKET_CONTENT_TYPE which, at the time of this writing, should be set to the values listed below. A receiver that doesn't recognize any of these types can simply skip the packet, and wait for the next one that is 'interesting' for him.

Note that all of the packet types listed in this chapter are optional
- for a basic exchange of audio samples via the serial port, PACKET_HDR_TYPE_AUDIO is sufficient.

PACKET_CONTENT_TYPE_UNKNOWN = 0x00
Dummy from the development phase. Use 'zero' for any kind of packet where the payload type cannot be indicated by any of the constants below.

PACKET_CONTENT_TYPE_AUDIO_SAMPLE_FORMAT = 0x01
This packet describes the format of any audio sample (or group of simultaneously sampled analog channels).
The 'binary' structure of this packet is shown in the chapter about densely packed payload (because that's what it is, densely packed, to keep the payload as short as possible).
As long as the receiver hasn't received at least one packet of this type, he doen't know how to process the bare-bone 'AUDIO' packets. Thus, if the microcontroller firmware permits (with a few bytes for a UART transmit buffer), send this packet periodically (say once a second, or after every few thousand 'AUDIO' packets). The reference implementation (in Spectrum Lab) sends it before a chunk of samples (typically 8192 sample points are converted into a huge block of 'AUDIO' packets, before passing that block to the windows FileWrite() function to send it via the 'COM port').

PACKET_CONTENT_TYPE_TIME_OF_DAY = 0x02
This optional packet contains a precise timestamp, typically from a GPS receiver (or even better, from a GPSDO that also delivers the sampling clock). In the latter case, the timestamp may have a resolution, or even an accuracy in the region of a few nanoseconds.
The mandatory part of this packet is a 21-bit signed integer, little endian, specifying the number of UTC seconds into the current day, densely packed Little Endian in three serial frames with 7 bit payload each.
With 24 hours per day, there is some headroom in the 21-bit integer, which allows sending the UNIX_DAY (see next packet type) only once in a week:
The receiver (i.e. Spectrum Lab) simply adds the TIMESTAMP value (in seconds) to the UNIX_DATE (multiplied by 86400 to convert into seconds). The result (sum) is then injected into Spectrum Lab's signal processing chain.
The optional fractional part is another 21-bit signed integer, that (together with the mandatory integer part) may form the same fixed-point format as used in some other packets.
The receiver detects the presence of the fractional part from the packet's SHORT_PAYLOAD_LENGTH. See specification of 'fixed-point numbers' here.

PACKET_CONTENT_TYPE_UNIX_DATE = 0x03
This optional packet contains an absolute calendar date, inspired by the Unix time (see Wikipedia on 'Unix time'). But to keep the packet shorter, it doesn't use date-and-time in seconds, but just the "integer number of days elapsed since the Unix birthdate, 1970-01-01 00:00:00 UTC" (in glorious ISO 8601 date-and-time format).
Similar as the TIME_OF_DAY, this packet (if sent at all) contains a densely packet 21-bit integer, thus the payload is only three serial frames long.
If the analog acquisition sytem provides this information, it should send the UNIX_DATE at least once per day (before the TIME_OF_DAY packet, which may be sent more frequently if the sampling rate isn't GPS-locked).

PACKET_CONTENT_TYPE_GPS_NMEA = 0x04
If your microcontroller firmware can afford it, and the serial port's bandwidth permits, a packet of this type may contain a GPS "NMEA 0183 sentence", as a plain 7-bit ASCII string. The payload should begin with the '$' character (which NMEA0183 calls the 'Start delimiter'), or with the '!' character (which NMEA0183 suggests as 'start of encapsulation sentence delimiter'), and end with the <CR><LF> (Carriage Return and LineFeed characters) that mark the end of the message (only followed by a two-digit hex checksum).
Since there is an NMEA parser in Spectrum Lab, anything seen as payload in this packet type is simply passed on to SL's "GPS (NMEA) Decoder", using a callback (function pointer) in module SevenBitPackets.c .


Reference implementation (written in plain "C")

A reference implementation (SevenBitPackets.c/.h) to assemble and parse packets compatible with this document is available in 'plain C'. The files (C source and header file) are available on request, or may exist on the author's website (dl4yhf/speclab/SevenBitPackets.zip).

The reference implementation contains..
  • "bit fiddling" functions to extract bitgroups with 1 to 32 bits from densely packed payload in RAM, as signed or unsiged integers, e.g. SBP_ReadBitgroup( instance, uBitgroupSize ).
    These bitgroups may begin at any bit-index in the packet (except for bit SEVEN, which is skipped in each byte), and may straddle byte-boundaries anywhere. Thus the need for "bit-fiddling" here.)
  • "bit fiddling" functions to append bitgroups with 1 to 32 bits to densely packed payload in RAM (with bit SEVEN in each byte cleared), e.g. SBP_WriteBitgroup( instance, u32Value, uBitgroupSize ).
  • functions to a assemble and disassembly complete packets, including header bytes and variable-sized payload, using a simple 'packet buffer' in RAM, e.g SBP_WritePacket_AudioSampleFormat(), SBP_ProcessReceivedPacket(), etc.
  • an optional unit test that covers the most important (and the most tricky) functions implemented in this module.
    Primarily used during the development phase, and so far only used in Spectrum Lab itself (which uses the reference implementation to send and receive audio samples via serial port or 'virtual COM port' for USB, in almost any possible format).

For a real implementation in a microcontroller firmware, some of the functions in the reference implementation are unneccesary (for example, to connect an ADC via microcontroller to the PC, the uC firmware doesn't need to receive or parse packets at all), but using the low-level "bit fiddling" functions avoids reinventing the wheel.

When used for Spectrum Lab, fragments in the code are compiled 'conditionally', controlled by the following macro constants in the header file (SevenBitPackets.h) :
  (informal - this code fragment may be outdated. Use the Source ...) :

// Compiling Spectrum Lab or a microcontroller firmware ?
#ifndef SBP_COMPILING_SPECLAB // if not defined elsewhere, make a guess:
# ifdef __BORLANDC__  // obviously compiling Spectrum Lab, because the compiler is Borland C
#  define SBP_COMPILING_SPECLAB 1 // 1 : "Compiling for Spectrum Lab", 0 : no
# else   // not compiling Spectrum Lab, guess we're compiling a microcontroller firmware..
#  define SBP_COMPILING_SPECLAB 0 // 0 because we're definitely NOT compiling Spectrum Lab.
# endif // compiling with Borland C ?
#endif // ndef SBP_COMPILING_SPECLAB ?

#if( SBP_COMPILING_SPECLAB ) // "Compiling Spectrum Lab" ?
#  define SBP_BUILT_IN_UNIT_TESTS 1 // 1 : implement the Unit Tests,        0 : don't
#  define SBP_USE_FLOATING_POINT  1 // 1 : may use 'float' and 'double',    0 : don't
#  define SBP_SUPPORT_PACKET_TX   1 // 1 : support TRANSMISSION of packets, 0 : don't
#  define SBP_SUPPORT_PACKET_RX   1 // 1 : support RECEPTION  of packets,   0 : don't
#  define SBP_MAX_AUDIO_CHANNELS  4 // 4 channels PER SAMPLE POINT, only used for internal buffering
#  define SBP_USE_CALLBACK_FOR_RECEIVED_AUDIO 1 // 1 : use callbacks via FUNCTION POINTER,
                                                // 0 : use SBP_AppCB_ProcessReceivedAudio()
#else   // not compiling Spectrum Lab, so guess we're compiling a microcontroller firmware,
   // which "only" drives an ADC (or read samples from its on-chip analog inputs),
   // but doesn't drive a DAC, thus it only needs to SEND but not RECEIVE packets
   // (all these are just guesses, so modify the following, or move these defs
   //  into a separate, "project specific" include file) .
#  define SBP_BUILT_IN_UNIT_TESTS 0 // 0 : don't implement the Unit Tests (save precious ROM..)
#  define SBP_USE_FLOATING_POINT  0 // 0 : neither use 'float' nor 'double' ("too expensive")
#  define SBP_SUPPORT_PACKET_TX   1 // 1 : support TRANSMISSION of packets, 0 : don't
#  define SBP_SUPPORT_PACKET_RX   0 // 0 : support RECEPTION  of packets,   0 : don't
#  define SBP_MAX_AUDIO_CHANNELS  2 // set this to the number of ADC- or DAC channels supported
#  define SBP_USE_CALLBACK_FOR_RECEIVED_AUDIO 0 // 1 : use callbacks via FUNCTION POINTER,
                                                // 0 : use SBP_AppCB_ProcessReceivedAudio()
#endif // SBP_COMPILING_SPECLAB ?


For the transmission of densely packed payload, SBP_WriteBitgroup() is used to append any bitgroup (e.g. integer value) with 1 .. 32 bits to a 'packet buffer' im RAM. This function is in fact the 'core' for most higher-level packet assembling subroutines. A stripped-down, and possibly outdated version in shown below for reference.

//---------------------------------------------------------------------------
void SBP_WriteBitgroup( // Appends a 1- to 32-bit value to the packet buffer, "densely packed".
   T_SBP_Instance *pInst, // [out] pInst->pbDst (always points to the "first payload byte")
                      // [in,out] pInst->iWriteBitIndex (INCREMENTED here)
   uint32_t u32Value, // [in] signed or unsigned integer with anything from 1 to 32 bits .
                      //      Will occupy up to ceil(32/7) = FIVE bytes in pbDst[],
                      //      but the remaining "unused" bits in the last byte
                      //      will be filled in the NEXT call of this function .
   int iBitgroupSize) // [in] Number of bits REALLY USED in u32Value. Range: 1..32 [bits].
{
   int iBitIndex  = pInst->iWriteBitIndex; // "bit index" into a packet buffer", 0 .. 8*SBP_PACKET_BUFFER_SIZE - 1
   int iBitOfByte;
   int iByteIndex;  // the "byte index" (into pbDst[]) is the "bit index" / 8 (or bitwise shifted right 3 times)
   int nBitsInByte;
   BYTE b;

   while( ((iByteIndex=(iBitIndex>>3))<SBP_PACKET_BUFFER_SIZE) && (iBitgroupSize>0) ) // more bits to "emit" ?
    {
      // How many "new bits" can be moved into pbDst[iByteIndex] before that byte is 'full' ?
      iBitOfByte = iBitIndex & 7;   // -> 0..7, where 7 would be the PACKET_HEADER_FLAG ("taboo" here)
      nBitsInByte = 7 - iBitOfByte; // -> number of bits that can be moved into pbDst[iByteIndex]
      if( iBitOfByte==0 ) // beginning a 'new byte', no bit-wise OR required now, may emit up to SEVEN bits:
       { b = (BYTE)u32Value;
         nBitsInByte = (iBitgroupSize >= 7) ? 7 : iBitgroupSize; // -> 1..7 bits to emit in THIS loop
         b &= (0xFF >> (8-nBitsInByte) );  // for example, with nBitsInByte=7, bitwise-AND with 0x7F
       }
      else  // nBitsInByte != 0 -> not a 'new byte', so read back some already-set-bits from the packet buffer:
       { b = pInst->pbDst[iByteIndex]; // <- already contains <iBitOfByte> bits that MUST NOT BE MODIFIED below.
         if( nBitsInByte > iBitgroupSize )
          {  nBitsInByte = iBitgroupSize;   // cannot emit more than what's left in u32Value !
          }
         // The bits already in 'b' must not be shifted but kept 'in place'.
         // Only <nBitsInByte> from u32Value must be shifted LEFT,
         //                    before bitwise ORing them into the 8-bit 'b'.
         b |= ( ( (BYTE)u32Value & (0xFF >> (8-nBitsInByte) ) ) << iBitOfByte );
         //       |_____________|  |________________________|   |_____________|
         //        next 1..7 (?)    isolate 7..1                 e.g. bitwise shift
         //        "new" bits       "new bits",                  LEFT 3 times,
         //      from the source    e.g bitwise AND              because iBitOfByte=3 [bits]
         //                         with 0x0F if only            were alreay in 'b'
         //                         4 bits are emitted
         //                         in this loop.
         //   |________________________________________________________________|
         //     After this bit-fiddling ("new" and "old" bits in byte 'b'),
         //     the maximum value in b cannot exceed 0x7F = 0b01111111 .
       }
      pInst->pbDst[iByteIndex] = b;  // write the 'new' byte, or write back the 'modified' byte
      iBitgroupSize -= nBitsInByte;  // decrement the REMAINING number of bits by 1..7 (not 8)
      u32Value >>= nBitsInByte;      // shift the next bits in place, for the next loop
      iBitIndex += nBitsInByte;      // increment the BIT INDEX into the packet buffer ...
      // If iBitIndex just reached bit SEVEN in a byte of the payload,
      // increment it by one more step (bit) to skip that bit:
      if( (iBitIndex & 7) == 7 )
       { ++iBitIndex; // skip bit SEVEN ( avoid PACKET_HEADER_FLAG occurring in the payload)
       }
    } // end while < more bits or bitgroups to "emit" into the packet buffer >

  pInst->iWriteBitIndex = iBitIndex; // write back the INCREMENTED packet-buffer-bit-index
} // end SBP_WriteBitgroup()