[GSoC2021] CalibrateSDR GSM Support

Overview

CalibrateSDR developed by Andreas Hornig, is a tool developed to determine the frequency drift of Software Defined Radios precisely using sync pulses of various Signal Standards.  

Cheaper SDRs use a low-quality crystal oscillator which usually has a large offset from the ideal frequency. Furthermore, that frequency offset will change as the dongle warms up or as the ambient temperature changes. The result is that any signals received will not be at the correct frequency, and they would drift as the temperature changes. CalibrateSDR can be used with almost any SDR to determine the frequency offset.

The work on GSM (2G) has been done by Jayaraj J, mentored by Andreas Hornig, as part of Google Summer of Code 2021, the working directory for the same can be found at Github.

proposed working of CalibrateSDR – made GSM signal compatible (the below guide shows how to get started)

Project Description

GSM uses time division to share a frequency channel among users. Each frequency is divided into blocks of time that are known as time-slots. 8 time-slots are numbers TS0 – TS7. Each time slot lasts about 576.9  μs. The bit rate is 270.833 kb/s, a total of 156.25 bits can be transmitted in each slot.

Each slot allocates 8.25 of its „bits time“ as guard-time, split between the beginning and the end of each time slot. Data transmitted within each time slot is called a burst. There are several types of bursts.

Frequency correction burst is a burst of a pure frequency tone at 1/4th the bitrate of GSM or (1625000 / 6) / 4 = 67708.3 Hz. By searching a channel for this pure tone, we can determine its clock offset by determining how far away from 67708.3Hz the received frequency is.

How is it working?
  1. Scanning of GSM Channels is based on the ARFCN frequency bands. For tuning into GSM Frequency, the ARFCN script can be used. We give the input of band to scan, the result will be the frequency. If we input the „scan all“ option, it will scan the whole GSM Frequencies in the given band and return the offset calculated from each frequencies. The code for the same can be found at: https://github.com/aerospaceresearch/CalibrateSDR/blob/jyrj_dev/calibratesdr/gsm/arfcn_freq.py
  2. After scanning channels in specific bands using ARFCN, the program will record the sample in the given sample rate. Make sure the SDR is connected with the device, or we can record it and give the input as a wave file.
  3. The determination of the position of FCCH bursts that we receive in SDRs is done by the code in fcch_ossfet.py. We measure it by how much shifted is the FCCH burst, than what we expect, that is at 67708.3 Hz from the frequency centre. Simply, if no offset is there, we could see these tone bursts at 67708.3 Hz offset concerning the centre frequency of the channel.
  4. The final output we receive include
    • Frequency drift from the expected FCCH position
    • The Offset of SDR in PPM
  5. The whole code works from the main cali.py and inturns gsm.py files
To test with the IQ_file.wave:

After cloning the repo locally, run the setup to install the requirements using the command: python setup.py install

After that, to run the visualization plots, run:

python cali.py -m gsm -f <location of wav file> -rs <sampling rate> -fc <frequency center>

First, we will get the plot of the average power spectrum plot. Play with the code to increase the N value, and you can see the sharpness of the line.

Further plots generated includes TDMA frames, the position of FCCH bursts visualisation as given below.

Screenshot-spectrogram_hann-Zoomed.png

We can see the pure tone FCCH bursts are occurring at specific intervals and can be visualized as small blue dots at a range of 0.25 from the centre.

Thus implementation of a filter bank and calculating the positions of these FCCH bursts will give us the offset frequency since we know these FCCH bursts occur at a distance of 67708.3 Hz from the frequency centre.

The average power spectrum of Discrete Fourier Transforms

Usage

Test files for GSM, recorded by Andreas can be found here.

  • Setup the environment, make sure the requirements are installed (preferably in a virtualenv). Use the setup.py to install necessary requirements.
  • To view the parameters for input run ~$ python cali.py -h

usage: cali.py [-h] [-f F] [-m {dab,dvbt,gsm}] [-s {rtlsdr}] [-c C] [-rs RS] [-rg RG] [-rd RD] [-nsec NSEC] [-gr] [-v] [-fc FC]

optional arguments:
-h, –help show this help message and exit
-f F select path to input file
-m {dab,dvbt,gsm} select mode
-s {rtlsdr} scan with rtlsdr
-c C scan by „all“ channels, by channel number „0,1,…n“ or by block name
-rs RS file/scan sample rate
-rg RG scan with gain
-rd RD scan with the device
-nsec NSEC scan for n-seconds
-gr, –graph activate graphs
-v, –verbose an optional argument
-fc FC frequency centreSetup the environment, make sure the requirements are installed (preferably in a virtualenv). Use the setup.py to install necessary requirements.

  • If testing with a recorded wav file, enter the parameters as:
~$ python cali.py -m gsm -f <location of wav file> -rs <sampling rate> -fc <frequency center>
  • If testing with an SDR stick, specify the ARFCN band, or specific frequency centre to scan for the GSM channel.

The ARFCN Bands include: GSM_850, GSM_R_900, GSM_900, GSM_E_900, DCS_1800, PCS_1900. For more information about the arfcn, checkout here.

Example usage:

~$ python cali.py -m gsm -s rtlsdr -c 900

The above parameters can be changed according to user needs. -rs can be specified with sample rate. The exact sample rate will be shown with the result. Make sure the SDR is connected when running the code with -s rtlsdr argument. Specify the -fc frequency argument, if the scan is to be done with a single frequency.

The expected output would look like this:

{'f': None, 'm': 'gsm', 's': 'rtlsdr', 'c': '900', 'rs': 2048000, 'rg': 20, 'rd': 0, 'nsec': 10, 'graph': False, 'verbose': False, 'fc': None}
let's find your SDR's oscillator precision
scanning…
starting mode: gsm
Found Rafael Micro R820T/2 tuner
Exact sample rate is: 270833.002142 Hz
Scanning all GSM frequencies in band: GSM_900
Offset Frequency: 31031.66666666667
Offset in PPM: 33.179448397684716

The Offset calculated from the frequency drift between fcch positions can be precisely derived and can be used to correct the oscillator.

Potential further improvements:
  1. LTE Signal support need to be included (currently in focus), and much more standards need to be made compatible for a wide usage of the tool.
  2. Making a platform-neutral API to communicate with more SDR devices.
  3. Optimising the user interface (command-line tool can be made more user friendly).

References:

Find out the project updates in my branch and do give a star for the project in AerospaceResearch org:

https://github.com/aerospaceresearch/CalibrateSDR/tree/jyrj_dev

[GSoC’21|Calibrate-SDR|Ayush] DVB-T for calibration

Hi everyone. I am writing this blog to share my work progress with everyone out there.

I have been working on extending the limitations in the Calibrate-SDR tool by adding support of DVB-T and DVB-T2 (terrestrial) signals. Because these signals are broadly used in Europe, Africa, Australia, and Asia region. So can be used here to provide calibration to more SDR users.

Visit here – http://www.dtvstatus.net/map/map.html

So the question arises What exactly I’m I doing?

As many of you would have worked on some sort of SDRs, might have faced errors due to the Frequency Offset of the Device(due to Crystal oscillators heating).

So here we have a tool named Calibrate-SDR to save you from correcting frequency offset repetitively. Calibrate -SDR is based on the idea of synchronization of devices by a constant frequency part present in the signal. This tool currently uses DAB+ signals to calculate the PPM shifting in frequency. I am enhancing it by using the DVB-T signal for this purpose and try to help more people out there.

Further reading about initial Calibrate-SDR refer to this blog.

Some words of wisdom about DVB-T signal

DVB-T, short for Digital Video Broadcasting — Terrestrial, is the DVB European-based consortium standard for the broadcast transmission of digital terrestrial television that was first published in 1997[1] and first broadcast in Singapore in February 1998. This system transmits compressed digital audio, digital video, and other data in a MPEG transport stream, using coded orthogonal frequency-division multiplexing (COFDM or OFDM) modulation. It is also the format widely used worldwide (including North America) for Electronic News Gathering for transmission of video and audio from a mobile newsgathering vehicle to a central receive point.

Wikipedia

Thanks to Wikipedia for providing historical details about this signal.

Some Technical Details about signal.

Would suggest reading the technical standard for more detailed idea about it.

https://www.etsi.org/deliver/etsi_en/300700_300799/300744/01.06.02_60/en_300744v010602p.pdf

I would cover only the part that was of value for me. Going through this paper and some research. I found out that DVB-T signals have a constant part called pilot inside the ODFM frame structure of DVB-T.

Visit etsi.org for better image.

So in addition to the transmitted data an OFDM frame contains:

– scattered pilot cells;

– continual pilot carriers;

– TPS carriers.

The modulation of all data cells is normalized so that E[c × c∗]= 1.

All cells which are continual or scattered pilots are transmitted at “boosted power level” so that for these E[c ×c∗] = 16/9.

The pilots can be used for frame synchronization, frequency synchronization, time synchronization, channel estimation, transmission mode identification and can also be used to follow the phase noise.

The carriers are determined by Kmin = 0 and Kmax = 1 704 in2K mode and 6 816 in 8K mode respectively. The spacing between adjacent carriers is 1/TU while the spacing between carriers Kmin and Kmax are determined by (K-1)/TU.

The numerical values for the OFDM parameters for the 8K and2K modes are given in tables for 8 MHz channels, for 6 MHz and 7 MHz channels.

We would collect some continual pilots and average them to get an overall current frequency. We would create an array of all the indexes of the continual pilot and use it.

Then we would subtract them with the known frequency of DVB-T. Hence, we would have the PPM shift. So that’s much of what we are doing for our tool.

It’s more interesting to work on once done with the boring Research work.

Anonymous Developer

Links –

Link to working repository is – https://github.com/AerospaceResearch/CalibrateSDR/tree/dvbt

I am attaching a test file too for a better understanding of this tool. Test file

Some more paper –

https://ca.rstenpresser.de/~cpresser/tmp/dvbt_7_paper.pdf

https://www.ese.wustl.edu/~nehorai/paper/Radar_Harms.pdf

http://ntur.lib.ntu.edu.tw/bitstream/246246/200704191002918/1/01258670.pdf

More Chat on-

https://www.linkedin.com/in/ayush-singh-101/

https://aerospaceresearch.zulipchat.com/#narrow/stream/281823-CalibrateSDR/topic/Signal.3A.20DVB-T

[GSoC2021] CalibrateSDR GSM Support – first coding period

Introduction

CalibrateSDR developed by Andreas Hornig is working perfectly with signals DAB+. We can use the python program to calibrate SDR devices. As part of the Google Summer of Code, I have been working around GSM Signal Standard to make CalibrateSDR compatible with it.

Before moving on, please read the initial blog on using CalibrateSDR, written by our mentor. The primary focus of this project is to extend the applicability towards more signal standards, so as to make it helpful for the SDR community. As DAB+ is mainly used in Europe, Signal standards like GSM, LTE, NOAA-Weather Satellites (use their sync pulses within the data) can be used.

Currently, it uses the pyrtlsdr package, which makes it work with RTL-SDR. Piping the API to work with other SDRs will also make the project have a wide range of applications in the SDR Community.

My first weeks of coding relied generally on implementing GSM signals. Working with the GSM frequency correction channel to calculate the offset is my primary task.

Proposed final working of CalibrateSDR

Working of GSM to calibrate the SDRs

GSM uses time division to share a frequency channel among users. Each frequency is divided into blocks of time that are known as time-slots. There are 8 time-slots that are numbers TS0 – TS7. Each time slot lasts about 576.9  μs. The bit rate is 270.833 kb/s, a total of 156.25 bits can be transmitted in each slot.

Each slot allocates 8.25 of its „bits time“ as guard-time, split between the beginning and the end of each time slot. Data transmitted within each time slot is called a burst. There are several types of bursts.

„Frequency correction“ burst, which is a burst of a pure frequency tone at 1/4th the bitrate of GSM or (1625000 / 6) / 4 = 67708.3 Hz. By searching a channel for this pure tone, we can determine its clock offset by determining how far away from 67708.3Hz the received frequency is.

How is it working?

A more robust way is to implement a hybrid of the FFT and filter methods. We could use the adaptive filter as described in the paper: G. Narendra Varma, Usha Sahu, G. Prabhu Charan, Robust Frequency Burst Detection Algorithm for GSM/GPRS (https://ieeexplore.ieee.org/document/1404796)

After finding the position of FCCH bursts that we receive in SDRs, it will be easy to calculate the offset. We measure it by how much shifted is the FCCH burst, than what we expect, that is at 67708.3 Hz from the frequency centre. Simply, if no offset is there, we could see these tone bursts at 67708.3 Hz offset with respect to the centre frequency of the channel.

I have completed the program to output the channel frequencies from the given ARFCN numbers of GSM. Check the code for the same here.

Detect and visualize the FCCH bursts!


Currently, the program has been tested only with IQ Wav file recordings. Even though the code has been designed to work with RTLSDR sticks, it is not tested yet with a live connection. Find the link to test files here.

After cloning the repo locally, run the setup to install the requirements using the command: python setup.py install

To test with GSM files, run:

python cali.py -m gsm -f <location of wav file> -rs <sampling rate> -fc <frequency center>

First, we will get the plot of the average power spectrum plot. Play with the code to increase the N value, and you can see the sharpness of the line.

The figure above shows the TDMA frames generated by the GSM Signal.

To determine the FCCH bursts from the signal, plot the spectrogram, using the function present in gsm.py. The spectrogram_plot function will do the fft and outputs the figure.

The generated FCCH bursts detection can be visualized as shown below:

We can see the pure tone FCCH bursts are occurring at specific intervals and can be visualized as small blue dots at a range of 0.25 from the centre.

Thus implementation of a filter bank and calculating the positions of these FCCH bursts will give us the offset frequency since we know these FCCH bursts occur at a distance of 67708.3 Hz from the frequency centre.

Will update the code after testing with gsm channels, and the function to calculate the final frequency offset will be committed to the repo.

The workarounds for the second coding period are implementation of LTE and NWS as well as the bridging of a more generalised SDR API, SoapySDR API, which has Python bindings to use as well.

Find out the project updates in my branch here: https://github.com/aerospaceresearch/CalibrateSDR/tree/jyrj_dev

References: