[GSoC2021] CalibrateSDR GSM Support – first coding period

Introduction

CalibrateSDR developed by Andreas Hornig is working perfectly with signals DAB+. We can use the python program to calibrate SDR devices. As part of the Google Summer of Code, I have been working around GSM Signal Standard to make CalibrateSDR compatible with it.

Before moving on, please read the initial blog on using CalibrateSDR, written by our mentor. The primary focus of this project is to extend the applicability towards more signal standards, so as to make it helpful for the SDR community. As DAB+ is mainly used in Europe, Signal standards like GSM, LTE, NOAA-Weather Satellites (use their sync pulses within the data) can be used.

Currently, it uses the pyrtlsdr package, which makes it work with RTL-SDR. Piping the API to work with other SDRs will also make the project have a wide range of applications in the SDR Community.

My first weeks of coding relied generally on implementing GSM signals. Working with the GSM frequency correction channel to calculate the offset is my primary task.

Proposed final working of CalibrateSDR

Working of GSM to calibrate the SDRs

GSM uses time division to share a frequency channel among users. Each frequency is divided into blocks of time that are known as time-slots. There are 8 time-slots that are numbers TS0 – TS7. Each time slot lasts about 576.9  μs. The bit rate is 270.833 kb/s, a total of 156.25 bits can be transmitted in each slot.

Each slot allocates 8.25 of its „bits time“ as guard-time, split between the beginning and the end of each time slot. Data transmitted within each time slot is called a burst. There are several types of bursts.

„Frequency correction“ burst, which is a burst of a pure frequency tone at 1/4th the bitrate of GSM or (1625000 / 6) / 4 = 67708.3 Hz. By searching a channel for this pure tone, we can determine its clock offset by determining how far away from 67708.3Hz the received frequency is.

How is it working?

A more robust way is to implement a hybrid of the FFT and filter methods. We could use the adaptive filter as described in the paper: G. Narendra Varma, Usha Sahu, G. Prabhu Charan, Robust Frequency Burst Detection Algorithm for GSM/GPRS (https://ieeexplore.ieee.org/document/1404796)

After finding the position of FCCH bursts that we receive in SDRs, it will be easy to calculate the offset. We measure it by how much shifted is the FCCH burst, than what we expect, that is at 67708.3 Hz from the frequency centre. Simply, if no offset is there, we could see these tone bursts at 67708.3 Hz offset with respect to the centre frequency of the channel.

I have completed the program to output the channel frequencies from the given ARFCN numbers of GSM. Check the code for the same here.

Detect and visualize the FCCH bursts!


Currently, the program has been tested only with IQ Wav file recordings. Even though the code has been designed to work with RTLSDR sticks, it is not tested yet with a live connection. Find the link to test files here.

After cloning the repo locally, run the setup to install the requirements using the command: python setup.py install

To test with GSM files, run:

python cali.py -m gsm -f <location of wav file> -rs <sampling rate> -fc <frequency center>

First, we will get the plot of the average power spectrum plot. Play with the code to increase the N value, and you can see the sharpness of the line.

The figure above shows the TDMA frames generated by the GSM Signal.

To determine the FCCH bursts from the signal, plot the spectrogram, using the function present in gsm.py. The spectrogram_plot function will do the fft and outputs the figure.

The generated FCCH bursts detection can be visualized as shown below:

We can see the pure tone FCCH bursts are occurring at specific intervals and can be visualized as small blue dots at a range of 0.25 from the centre.

Thus implementation of a filter bank and calculating the positions of these FCCH bursts will give us the offset frequency since we know these FCCH bursts occur at a distance of 67708.3 Hz from the frequency centre.

Will update the code after testing with gsm channels, and the function to calculate the final frequency offset will be committed to the repo.

The workarounds for the second coding period are implementation of LTE and NWS as well as the bridging of a more generalised SDR API, SoapySDR API, which has Python bindings to use as well.

Find out the project updates in my branch here: https://github.com/aerospaceresearch/CalibrateSDR/tree/jyrj_dev

References:

[GSoC 2020 | MOLTO | Brandon] Refactorization MOLTO

Introduction

Hi, my name is Brandon Escamilla, I am an Aerospace Engineer by Universidad Marista de Guadalajara (Guadalajara, México). This is the second time I got selected for the GSoC. I am really grateful for this opportunity. This time, I came along with a proposal to improve the actual MOLTO project. The same I did work the last year. MOLTO is a big project, which has tons of work before going to production. Last year, I worked in a way to connect MOLTO with a user interface, it was not an easy work since MOLTO is a Matlab tool which requires special connections and is not as easy as consuming a normal API, so I did create an API with Python/Flask to communicate my requests from the Frontend to Matlab directly. It did work, we did communicate successfully with the Matlab tool. But there were some problems we need to resolve in order to have a „production environment“. At the end of the GSoC, we had a UI created in React.js, an API using Flask, and a local database using SQLite. This was enough to prove the architecture I wrote in my proposal, easy architecture to get to the web those projects created in Matlab which can’t go to production because Matlab closed license.

Last year blog: [GSoC 19′ |UC3M ] MOLTO – Mission Designer]

So, here you have the new flow of MOLTO from a route-based perspective, and from a logical flow.

The main issues that needed to be solved before to go to production were the ones I will list below:

  1. Error communication between Matlab and Python.
  2. Delete Real-time communication between UI – Python – Matlab. (Sockets)
  3. Create a new service based on Codes to retrieve the mission.

But there were also some improvements needed to be added with lower priority:

  1. Friendly user interface for new users.
  2. New Design
  3. Create a database in production to save missions configuration and results.
  4. Create an email service to send mission codes.
  5. Create CMS to add information in an easy way for maintainers.
  6. New view for the service based on codes.
  7. Improve deployment of Frontend.
  8. Response optimization from Matlab Genetic Algorithm.
  9. Toy problem for new users.
  10. Optimize responsive views of the site.
  11. Improve Celery implementation for Background tasks.

Before continuing, I would like to add that after GSoC, I continued working on MOLTO, adding small features, and improving user experience as fas as I could. In our constant communication between my last mentor David and I, he did invite me to start a research stay in the University Carlos III where he was doing his Doctorate. This led to another experience where David has been my bachelor’s degree thesis advisor. (A good story thanks to GSoC! 😃)

Hands-On!

Once accepted, I started working on my proposal which did look something like this:

1. CMS Implementation

For this purpose, I did use an open-source application called Strapi, which allows you to develop a CMS locally and in production in an easy way. It helps you to develop the database, API, endpoints, CDN requests, models, emails, and more…

I did install Strapi locally and started creating the services were needed for MOLTO which can be divided in this way:

  1. Collaborators
  2. Missions
  3. Users
  4. Email Service
  5. Motors

Once I create all the services described before, I just started to launch the CMS to production. The easiest way to do this is using Heroku, which allows you to have an app in production with very few steps and configuration. Finally, you can find the Admin in this URL: https://molto-admin.herokuapp.com/admin, Of course, it has a login and just the maintainers of MOLTO can access. But I leave you a few screenshots, so you can see the interface.

I am using a PostgreSQL database which actually is a plugin from Heroku app. ✅

And this is the documentation of the endpoints:

https://molto-admin.herokuapp.com/documentation/v1.0.0

2. New design MOLTO

Before going to production we needed a new design because the first one was more like an MVP, finally, I added a link to the old site, the designs, and the newly implemented design. Almost all the components of the website did change, from the home to MOLTO-IT, and also new views were added.

Here I will leave you some screenshots of the old design and the new design. (I will include the links in case you want to check it out )

New site: https://molto-it-ui-old.vercel.app/

New site: https://molto-it-ui.vercel.app/

Another new feature is in the motors section of MOLTO-IT, where you can actually see the motor configuration clicking in more information.

Another feature is the mobile version of MOLTO and the new menu, at this moment is really important to have a good mobile version of websites, since most users will visit your site from their cellphones. So, I did refactor the mobile version and right now is good to work from cellphones.

3. Tour MOLTO

In order to have a better experience in MOLTO, I did try a lot of ways to do it. I started using a library called React tour, and after another one called React joyride, but I did notice that those were very intrusive with the user experience and that actually the performance of the application was really bad when using both libraries. So I did prefer to create my own component which shows an information icon and if you hover on desktop or click it in mobile, display a box with useful information in order to know what to add in the inputs or what are the inputs for.

I found this way less intrusive and useful in my opinion. Here you have one screenshot of this component.

4. New service for creating a mission or search for a mission created.

One of the requirements of the last year was to create a UI with the possibility to see how the genetic algorithm evolves in the time. This was possible, but also a bad idea from a user perspective. Once they select their configuration, they needed to be waiting for the response of the API, this time could go from 3 minutes to 10 minutes – more generations, more population, more time-, which is the time Matlab started the genetic algorithm, in this point Flask opened a socket to start consuming the files were being created in real-time in a directory of the server where MOLTO lives. This was working well but just in one situation: A mission with really low generations and population, since this kind of missions will return results fast. So, once you started a mission with more than 30 generations and more than 50 population, you needed to wait a lot of time before the sockets could return the first generation, and this leads to another problem if the user wanted to see the final generation, the user needed to wait from minutes to hours, without closing the browsers – once you closed the browser, the socket connection finished-. So in the meantime between GSoC and GSoC, I created a new architecture based on codes where you create your mission and the website returns you a code, and also the possibility to send the code to your email, so you could return, in 1 hour, 1 day or 1 week, and all your results will be there stored in the database. Of course, this was a lot of work, I need to almost change most of the logical code was created to connect with Python via sockets. And also new views needed to be created to retrieve the mission, send the code to email, etc.

Almost in the middle of the second evaluation I start working on this, and after days of coding, the new service was available, here I will add few screenshots, but of course, you could to the MOLTO website.

The view which looks for missions, has an input where you need to put the code MOLTO gives you when you finished the configuration of your mission, this input has the ability to detect invalid codes, and also returns the current status of the mission. This is thanks to Celery, which is a tool that before was just running tasks in the background, but with the proper configuration, you can check the status of the mission in real-time. 🛰

4. Celery and Matlab Errors

As I said at the top of this post, one big problem was that I had issues trying to connect Matlab errors with python, due to this once a mission failed, I didn’t know the real reason why it was failing. At the start, it was not a problem, because I was using always the same JSON for creating missions and testing. But once you put different configurations, it was randomly working, sometimes it works, sometimes it just crashes, and I didn’t know what was happening. So, this year, I decided to solve this issue, as said before, using Celery properly, and also the Matlab Engine For Python.

The big issue was that I was not adding some configurations to check the tasks in the UI of Celery called flower, and I was also not using some configuration to read logs from Matlab in python, It was a hard task, but finally, it is working, so I will put here one screenshot of the logs I am receiving in the server where I can know exactly why Matlab is not working.

I can also see all the missions in real-time in a dashboard, all the missions that failed, all the successful missions, and also the tasks that are running.

5. New host for Frontend

There are a lot of ways to host your frontend applications, we can host our frontend in the MOLTO server, or maybe in another service like AWS, etc. But I recently started using Vercel to host other projects, and it was a great experience since you can have multiple environments for testing, production, development, etc. All of this in one place, connected to your repository in Github or to your CLI. It makes easier the development and that’s why it is the platform MOLTO will be using for frontend hosting.

We have right now two environments dev, and production, all the changes that will be applied to the UI of MOLTO will pass first by dev, after approval all these changes could be applied to production.

6. Toy problem

One problem I faced when I was demonstrating MOLTO was that I was the only person who knows how to use it. That was a problem because you can’t deploy an application to production if it is not intuitive.

In order to improve this situation, I used the based architecture of data management called Redux to pre-load a problem, so every time you enter to MOLTO-IT you can change just the name and go to the last tab, click send, and here you have a useful mission, from Earth to Jupyter. So you can test this mission, and actually see the Pareto front without any problem.

7. New flow with code – Pareto Front

I’ve been talking about the new flow but I didn’t show you how it looks after you put a code that has a finished state. Well, actually, this view has some improvements also.

The first one is that you are able to see all the results from generation 1 to the last generation with its respective results. So you can test any value to plot the orbit. You can also see in real-time how the chart changes once you select another Pareto point.

Conclusion

It was an honor to work again in the Google Summer of Code 2020, finally, I would say I finished what I proposed at the start of this project. I also want to thank all the persons who make this possible, Dr. Manuel Sanjurjo, Dr. David Morante for guiding me, and helping me every time I have issues or problems to resolve. I also want to thank you for the research stay at UC3M, I hope I can continue working along with both in this and other projects.

I also want to thank Andreas Hornig for being there for any question and always provide the necessary stuff to keep working. Also for always remember me the deadlines 😅, and pushing me to give the best of me.

As far as I know, this is the last GSoC in which I can contribute as a student ☹️, but my next goal is to keep contributing to open-source and why not contribute also as a mentor if possible in the next GSoC’s. I would really like to share all that I’ve learned during these 2 years. 🛰

Thank you for reading.

Brandon Escamilla

Useful resources:

  1. Production website: molto-it-ui.vercel.app
  2. Repository: https://github.com/uc3m-aerospace/MOLTO-IT
  3. Email: brandon.escamilla@outlook.com | brandon.escamilla@aerospaceresearch.net