Introduction
Hi, my name is Brandon Escamilla, I am an Aerospace Engineer by Universidad Marista de Guadalajara (Guadalajara, México). This is the second time I got selected for the GSoC. I am really grateful for this opportunity. This time, I came along with a proposal to improve the actual MOLTO project. The same I did work the last year. MOLTO is a big project, which has tons of work before going to production. Last year, I worked in a way to connect MOLTO with a user interface, it was not an easy work since MOLTO is a Matlab tool which requires special connections and is not as easy as consuming a normal API, so I did create an API with Python/Flask to communicate my requests from the Frontend to Matlab directly. It did work, we did communicate successfully with the Matlab tool. But there were some problems we need to resolve in order to have a „production environment“. At the end of the GSoC, we had a UI created in React.js, an API using Flask, and a local database using SQLite. This was enough to prove the architecture I wrote in my proposal, easy architecture to get to the web those projects created in Matlab which can’t go to production because Matlab closed license.
Last year blog: [GSoC 19′ |UC3M ] MOLTO – Mission Designer]
So, here you have the new flow of MOLTO from a route-based perspective, and from a logical flow.
The main issues that needed to be solved before to go to production were the ones I will list below:
- Error communication between Matlab and Python.
- Delete Real-time communication between UI – Python – Matlab. (Sockets)
- Create a new service based on Codes to retrieve the mission.
But there were also some improvements needed to be added with lower priority:
- Friendly user interface for new users.
- New Design
- Create a database in production to save missions configuration and results.
- Create an email service to send mission codes.
- Create CMS to add information in an easy way for maintainers.
- New view for the service based on codes.
- Improve deployment of Frontend.
- Response optimization from Matlab Genetic Algorithm.
- Toy problem for new users.
- Optimize responsive views of the site.
- Improve Celery implementation for Background tasks.
Before continuing, I would like to add that after GSoC, I continued working on MOLTO, adding small features, and improving user experience as fas as I could. In our constant communication between my last mentor David and I, he did invite me to start a research stay in the University Carlos III where he was doing his Doctorate. This led to another experience where David has been my bachelor’s degree thesis advisor. (A good story thanks to GSoC! ?)
Hands-On!
Once accepted, I started working on my proposal which did look something like this:
1. CMS Implementation
For this purpose, I did use an open-source application called Strapi, which allows you to develop a CMS locally and in production in an easy way. It helps you to develop the database, API, endpoints, CDN requests, models, emails, and more…
I did install Strapi locally and started creating the services were needed for MOLTO which can be divided in this way:
- Collaborators
- Missions
- Users
- Email Service
- Motors
Once I create all the services described before, I just started to launch the CMS to production. The easiest way to do this is using Heroku, which allows you to have an app in production with very few steps and configuration. Finally, you can find the Admin in this URL: https://molto-admin.herokuapp.com/admin, Of course, it has a login and just the maintainers of MOLTO can access. But I leave you a few screenshots, so you can see the interface.
I am using a PostgreSQL database which actually is a plugin from Heroku app. ✅
And this is the documentation of the endpoints:
https://molto-admin.herokuapp.com/documentation/v1.0.0
2. New design MOLTO
Before going to production we needed a new design because the first one was more like an MVP, finally, I added a link to the old site, the designs, and the newly implemented design. Almost all the components of the website did change, from the home to MOLTO-IT, and also new views were added.
Here I will leave you some screenshots of the old design and the new design. (I will include the links in case you want to check it out )
New site: https://molto-it-ui-old.vercel.app/
New site: https://molto-it-ui.vercel.app/
Another new feature is in the motors section of MOLTO-IT, where you can actually see the motor configuration clicking in more information.
Another feature is the mobile version of MOLTO and the new menu, at this moment is really important to have a good mobile version of websites, since most users will visit your site from their cellphones. So, I did refactor the mobile version and right now is good to work from cellphones.
3. Tour MOLTO
In order to have a better experience in MOLTO, I did try a lot of ways to do it. I started using a library called React tour, and after another one called React joyride, but I did notice that those were very intrusive with the user experience and that actually the performance of the application was really bad when using both libraries. So I did prefer to create my own component which shows an information icon and if you hover on desktop or click it in mobile, display a box with useful information in order to know what to add in the inputs or what are the inputs for.
I found this way less intrusive and useful in my opinion. Here you have one screenshot of this component.
4. New service for creating a mission or search for a mission created.
One of the requirements of the last year was to create a UI with the possibility to see how the genetic algorithm evolves in the time. This was possible, but also a bad idea from a user perspective. Once they select their configuration, they needed to be waiting for the response of the API, this time could go from 3 minutes to 10 minutes – more generations, more population, more time-, which is the time Matlab started the genetic algorithm, in this point Flask opened a socket to start consuming the files were being created in real-time in a directory of the server where MOLTO lives. This was working well but just in one situation: A mission with really low generations and population, since this kind of missions will return results fast. So, once you started a mission with more than 30 generations and more than 50 population, you needed to wait a lot of time before the sockets could return the first generation, and this leads to another problem if the user wanted to see the final generation, the user needed to wait from minutes to hours, without closing the browsers – once you closed the browser, the socket connection finished-. So in the meantime between GSoC and GSoC, I created a new architecture based on codes where you create your mission and the website returns you a code, and also the possibility to send the code to your email, so you could return, in 1 hour, 1 day or 1 week, and all your results will be there stored in the database. Of course, this was a lot of work, I need to almost change most of the logical code was created to connect with Python via sockets. And also new views needed to be created to retrieve the mission, send the code to email, etc.
Almost in the middle of the second evaluation I start working on this, and after days of coding, the new service was available, here I will add few screenshots, but of course, you could to the MOLTO website.
The view which looks for missions, has an input where you need to put the code MOLTO gives you when you finished the configuration of your mission, this input has the ability to detect invalid codes, and also returns the current status of the mission. This is thanks to Celery, which is a tool that before was just running tasks in the background, but with the proper configuration, you can check the status of the mission in real-time. ?
4. Celery and Matlab Errors
As I said at the top of this post, one big problem was that I had issues trying to connect Matlab errors with python, due to this once a mission failed, I didn’t know the real reason why it was failing. At the start, it was not a problem, because I was using always the same JSON for creating missions and testing. But once you put different configurations, it was randomly working, sometimes it works, sometimes it just crashes, and I didn’t know what was happening. So, this year, I decided to solve this issue, as said before, using Celery properly, and also the Matlab Engine For Python.
The big issue was that I was not adding some configurations to check the tasks in the UI of Celery called flower, and I was also not using some configuration to read logs from Matlab in python, It was a hard task, but finally, it is working, so I will put here one screenshot of the logs I am receiving in the server where I can know exactly why Matlab is not working.
I can also see all the missions in real-time in a dashboard, all the missions that failed, all the successful missions, and also the tasks that are running.
5. New host for Frontend
There are a lot of ways to host your frontend applications, we can host our frontend in the MOLTO server, or maybe in another service like AWS, etc. But I recently started using Vercel to host other projects, and it was a great experience since you can have multiple environments for testing, production, development, etc. All of this in one place, connected to your repository in Github or to your CLI. It makes easier the development and that’s why it is the platform MOLTO will be using for frontend hosting.
We have right now two environments dev, and production, all the changes that will be applied to the UI of MOLTO will pass first by dev, after approval all these changes could be applied to production.
6. Toy problem
One problem I faced when I was demonstrating MOLTO was that I was the only person who knows how to use it. That was a problem because you can’t deploy an application to production if it is not intuitive.
In order to improve this situation, I used the based architecture of data management called Redux to pre-load a problem, so every time you enter to MOLTO-IT you can change just the name and go to the last tab, click send, and here you have a useful mission, from Earth to Jupyter. So you can test this mission, and actually see the Pareto front without any problem.
7. New flow with code – Pareto Front
I’ve been talking about the new flow but I didn’t show you how it looks after you put a code that has a finished state. Well, actually, this view has some improvements also.
The first one is that you are able to see all the results from generation 1 to the last generation with its respective results. So you can test any value to plot the orbit. You can also see in real-time how the chart changes once you select another Pareto point.
Conclusion
It was an honor to work again in the Google Summer of Code 2020, finally, I would say I finished what I proposed at the start of this project. I also want to thank all the persons who make this possible, Dr. Manuel Sanjurjo, Dr. David Morante for guiding me, and helping me every time I have issues or problems to resolve. I also want to thank you for the research stay at UC3M, I hope I can continue working along with both in this and other projects.
I also want to thank Andreas Hornig for being there for any question and always provide the necessary stuff to keep working. Also for always remember me the deadlines ?, and pushing me to give the best of me.
As far as I know, this is the last GSoC in which I can contribute as a student ☹️, but my next goal is to keep contributing to open-source and why not contribute also as a mentor if possible in the next GSoC’s. I would really like to share all that I’ve learned during these 2 years. ?
Thank you for reading.
Brandon Escamilla
Useful resources:
- Production website: molto-it-ui.vercel.app
- Repository: https://github.com/uc3m-aerospace/MOLTO-IT
- Email: brandon.escamilla@outlook.com | brandon.escamilla@aerospaceresearch.net