Remote CW Keyer - Implementation Details

Remote CW Keyer - Implementation Details
by DL4YHF, 2023-12-20

Introduction
TCP Client / Server model
1. Network Audio Routing
2. Network Morse Signal Routing
Sourcecode overview

1. Introduction

This document only contains the technical details about the implementation; mainly about the network protocol between client and server.
It it not required to "just use" the application (either as client or server). All you need to know about using the program, and what kind of simple hardware is used to connect a real Morse key to the PC on the client side, is explained in the Remote CW Keyer manual.

2. TCP-based Client / Server communication

Most of the communication between client and server happes in module CwNet.c. This module contains one thread for the client side [ClientThread()] and one thread for the server side [ServerThread()]. At the moment, the 'Remote CW Keyer' application only runs as either client or server (but not both simultaneously in a single instance of the program). However, to test and develop the client/server communication on a single machine, at least on Windows, multiple instances of the program can run side-by-side without a "real network" between them, using the 'localhost' aka dummy IP 127.0.0.1 .
The server can handle connections from multiple clients. This is usually the program that runs on the 'remote radio' site (either on a windows PC, or maybe Linux/WINE, or maybe one fine day on a power-saving microcontroller with an embedded TCP/IP network stack like LwIP).
A client can only connect to a single server (if he needs to connect to a server at all; because the 'Remote CW Keyer' can be used completely locally, using only USB- or serial port connections to the Morse key and the radio).


         ,----------------------------------------,
         |                  Server                |<----- USB ---> Radio
         |                                        |
         |  TCP/IP         TCP/IP         TCP/IP  |
         '----------------------------------------'    __ "Network" (TCP/IP):
            /|\  |         /|\  |         /|\  |         | Audio signals,
             |   |          |   |          |   |         | CW keying,
             |  \|/         |  \|/         |  \|/      __| and other data
         ,-----------,  ,-----------,  ,-----------,
         | Client #1 |  | Client #2 |  | Client #3 |
         '-----------'  '-----------'  '-----------'
          /|\ /|\  |     /|\ /|\  |     /|\ /|\  |
           |   |   |      |   |   |      |   |   |
           |   |  \|/     |   |  \|/     |   |  \|/
          Key Mic Spkr   Key Mic Spkr   Key Mic Spkr

Not shown above for simplicity:: The 'Server' may also have a Morse key and loudspeakers attached.
In fact, most of the times the server will be the usual 'shack PC',
temporarily running 'as server' to allow remote operators to listen to their own signal (on the server operator's radio), or even use the server operator's transmitter remotely - if the operator, and the authorities permit (see suggestions for German radio operators here).

To keep the 'network administration' overhead down, there will only be a single TCP socket per client-server connection. The binary data streams traveling from from client to server and back are multiplexed internally (in CwNet.c) to carry ...

Morse code for transmission (grep for CWNET_CMD_MORSE),
Audio samples for reception (grep for CWNET_CMD_AUDIO),
Icom CI-V packets for remote control (.. CWNET_CMD_CI_V),
and a few other types of 'commands', all defined in CwNet.h (CWNET_CMD_ ...) .

2.1 Network Audio Routing

Audio routing on the client side is trivial:

When configured with an active audio input device,
the client down-samples the audio to 8 kSamples/second, compresses it into an 8-bit-per-sample bytestream with simple A-Law compression, and merges it to whatever else needs to be sent to the remote server.
When configured with an active audio output device,
the client decompresses the A-Law compressed byte stream received from the remote server, resamples (interpolates) the stream from 8 to 48 kSamples/seconds, adds the result to his own, optional, 'locally generated' CW sidetone, and sends the resulting stream to his audio device (e.g. loudspeakers or headphone).

Audio routing on the server side is a bit more elaborate, because the server may be connected by multiple clients simultaneously:

Audio received from the server's radio (connected via built-in or external 'USB audio device') is down-sampled to 8 kSamples/second, and A-Law compressed. The result is buffered in a classic, lock-free, circular FIFO from where each of the server's "client instances" (in CwNet.h : T_CwNet.RemoteClient[]) can retrieve the next block of samples, which typically happens every 20 milliseconds in CwNet.c : ServerThread() -> CwNet_OnPoll()).
Thus (important for a possible implementation in a Microcontroller), the radio receiver's audio signal only needs to be processed once, regardless of how many remote clients are currently connected.
Audio received by the server from any of the currently connected clients is either ...
- ignored (if the client isn't allowed to talk / transmit or 'chat' with the other users),
- or sent to the other currently-logged-in users (helpful if the 'Remote CW Keyer' is used in an off-air round table or in an 'online CW lesson', when the instructor / teacher runs the server on his PC, possibly without using a real transceiver at all)
- or (future plan, not materialized yet because we have enough applications to remotely control radios for voice transmissions) pass on the audio signal to the transmitter (if the particular client currently "has the microphone").

Due to the 8-bit A-Law compression, and 8 kSamples/second, each client's audio stream adds approximately 64 kBit / second (plus some overhead for the internal multiplexing) to the server's up-stream bandwidth. This may be a problem when serving multiple clients from a really 'remote' site, without a fast internet connection.

2.2 Network Morse Signal Routing

On the client side, the input from the Morse key adapter is first converted into a stream of 'key-down' and 'key-up' events, each with a seven-bit timestamp (in milliseconds, relative to the previous event).

3. Sourcecode overview

After unzipping the program (including files and subdirectories) into a directory of your choice (not neccessarily "Programme", "Program File (x86)", "Program Files" or wherever your OS wants you to 'install' everything), that directory should contain at least this:


   Remote_CW_Keyer
     |--- manual
     |        |-- Remote_CW_Keyer.htm : The manual in HTML format 
     |        '-- *.png, *.jpg        : Images loaded by your browser
     |                                  when opening the manual   
     |--- sources : The 'most interesing sourcecodes' (written in C).
     |        |-- KeyerThread.c :
     |        |     This is where the program polls the operator's
     |        |     'local' Morse key, connected to a serial port.
     |        |-- Elbug.c : Emulates an electronic keyer
     |        |             ('dot' and 'dash' contact states in,
     |        |              Morse code pattern out, in real-time).
     |        |-- CwGen.c : Converts Text (ASCII) into Morse code.
     |        |-- CwDSP.c : Creates audio waveforms from the
     |        |             on/off keyer Morse code pattern, etc.
     |        |-- CwNet.c : Socket-based network server and client.
     |        |             Multiplexes and demultiplexes different
     |        |             kind of 'bytestreams' into the same
     |        |             network socket (TCP/IP connection 
     |        |             between client and server).
     |        |-- CwStreamEnc.c : 'CW Stream Encoder / Decoder' .
     |        |             Converts the raw 'CW keying signal'
     |        |             into a low-bandwidth bytestream,
     |        |             with an efficiently encoded 7-bit
     |        |             'timestamp' (actually the number of
     |        |             milliseconds to wait before emitting
     |        |             the next 'key up / key down'-flag),
     |        |             and the 'key up / key down'-flag
     |        |             in the most significant bit of each
     |        |             byte in the CW keying bytestream.
     |        |-- dsound_wrapper.c : Interface between the application
     |        |             and one of the many Windows 'Audio APIs'.
     |        |-- Timers.c, Utilities.c : Ugly stuff for Windows.
     |        '-- Keyer_Main.cpp :
     |              This is just the GUI (User Interface), written
     |              in C++ Builder, thus not sooo interesting.
     |    You can safely delete the entire 'sources' folder.
     |    It is not required to run the executable.
     |        
     '--- Remote_CW_Keyer.exe : This is the executable file .
          Lauch it via double-click, command shell, or create
          a link on the desktop to it.

Links in the above sourcecode overview only work after unpacking the zipped archive. They don't work when viewed 'online' on the website, and there is no public repository (GitXYZ) for it. The sources contained in the archive are considered 'open source', but take care - some of them may be 'available as source' but not really 'open', like the now-discarded ASIO support.

Contents

1. Introduction

2. TCP-based Client / Server communication

2.1 Network Audio Routing

2.2 Network Morse Signal Routing

3. Sourcecode overview