Distribution Proposal

[KotzaBoss]

Distribution Proposal

Due to inexperience with development and distribution of cross platform applications it is important to use as many widely-used, universal tools available. Inherently C++ is not a machine independent language. This means that to be sure that the program will work on all machines every user should build the program from source when they acquire it. This document will propose a way to ensure that the program will require as little architecture/os specific steps in order to work. The distribution environment can later evolve and become more "sophisticated" if needed. Primarily, git will be used as the main means of distribution.

Overview

All users will acquire an installer/wizard from the openPilotLog site which will perform all the necessary checks and operations in order for the program to work. The program acquisition has as a whole two phases. Building and Maintaining. A graphical representation of the distribution is as follows:

Building

For a Qt application the minimum requirements are:

• Qt framework
• Compiler

The installer will clone the stable branch of the openPilotLog repository on the user's computer. After ensuring that the environment is sane and all requirements are met (Qt, compiler, git, c/qmake) it will build the program. If any prerequisite is not met the user will be prompted to resolve the issue if it is not possible. A successful installation means that the user has the following components on their machine.

• Qt Framework
• Compiler
• Git
• c/qmake
• openPilotLog stable repo

Maintaining

Now that everything is setup the user can enjoy the application. Unless not already clear, Git will show its use in this section. Should any update be done to the stable branch of the application the user will be prompted through the app. From their perspective its as simple as clicking a button. On the background the repo is pulled, bringing the update to the user's computer and the updated modules are rebuilt. Depending on the scope of the update rebuilding can vary in time.

Roundup

The proposed distribution of openPilotLog will ensure that all users independant of OS or CPU architecture can acquire and use the app. Any problems that may arise stem from the development side of the application. From idiomatic (compiler specific) code to Qt version incompatibilities, these can be caught and fixed before they even get pushed to the stable branch. CI/CD can ensure that anything the users acquire is going to work on their machines.

[fiffty-50]

While this is a very interesting approach to deployment, I believe it comes with significant downsides. I will try to elaborate on what I mean.

I see 2 major challenges with this approach: resource usage and support/usability.

Resource usage

Let's quickly sum up a few items:
Git installer: ~45Mb
Cmake installer ~25Mb
qt framework: anywhere from 800Mb - 3Gb
MinGw ~ 20Mb

So, in order to install one app, the user would have to download upwards of 1Gb, install all these unneeded files, set them up and understand how to use them. This would be a very tough pill to swallow for most. And it doesn't stop there, which leads to the next point.

Usability / Support

While for you as a seasoned IT specialist compiling a program might seem benign, think back to the last time you helped your parents to install a printer. Downloading and installing qt, a framework aimed at developers, using git, a tool for developers, and last but not least, compiling a program is outside of the scope of an end user. We cannot expect users to carry out this task. Users need to download an installer and call it a day. Don't underestimate the complexity of compiling a program for someone not involved in programming.

Apart from that, imagine the complexity of designing an installer that installs and configures a complete development toolchain, as well as builds an application. For each of these applications there are hour long guides explaining how to set up each single one of them, let alone make them work together.

The combination of these two factors lead to almost all software distribution happening based on binary packages. Think of downloading a windows virus.exe, of sudo apt install chromium or pacman -S gnome or hitting install in the App-Store of your phone. What all these have in common is that they are binary packages, that is packages that have been compiled by the developers, and bundled by either themselves or package maintainers.

Distributing binaries has some key advantages:

• The end-user does not have to maintain a toolchain, something that would overwhelm the vast majority of users.

• They are much more space-efficient, because the user only gets what he needs. No git, no compiler, no development framework.

• Compiling takes a long time, especially on older or less powerful hardware. For example, installing gnome via pacman takes a minute or two on a modern system. Compiling it from source takes several hours.

Due to inexperience with development and distribution of cross platform applications it is important to use as many widely-used, universal tools available.

I completely agree with you there. But I believe it is very important to make one key distinction

End user != Developer

We need to seperate these two domains. Git, cmake, Qt, compiling code, debugging, testing - these are developer responsibilites, our responsibilities.
When your friend, who knows nothing about technology, sends an email, he just wants to write it and click the send button. It is not his job to set up a LAMP stack, get a SSL certificate and configure a SMTP server. These kinds of jobs are done for him.

So what's the big idea then?

We don't have the capacities to support all platforms. Luckily we can reach around 93 % of potential users by providing packages for Windows and MacOS. I am personally a big fan of Linux, so I would like to include a release for that as well. Which means we need 3 builds. Luckily, we are not the first ones who want to deploy a Qt application to these 3 environments, so we can use well-established tools:

The idea behind these three different tools is fundamentally the same:

1. Build the program and put in in a directory. Cross-compile when necessary, but local builds are preferable
2. Bundle the dependencies. Figure out which libraries the program needs to run and add them to the directory
3. Package the directory into a format that the target system understands.

Step 2 can be very complex in the linux world, since every distribution has different versions of libraries, sometimes in different folders and sometimes under different names. For this reason, there have been a couple of projects aimed at making linux apps more distribution-agnostic. The 3 major players are snaps, flatpaks and appimages. I am currently still evaluating whether flatpaks or appimages are the better solution, I have had bad experiences with snaps myself so I am least interested in that option.

I am currently leaning towards appimage as a suitable format. Please have a look at linuxdeployqt, which does a great job at explaining the process in not too many words.

TL;DR

The target audience for this program is normal people. Managing a development toolchain and compiling the program themselves would hopelessly overwhelm them. Distributing binaries is the standard for software distribution and has many key benefits. While there is a lot to be said about how to get there, I believe a binary package should be the end goal.

[KotzaBoss]

Axiom

Let me get this out of the way and i will assume it is implied with everything i say.
The user only clicks a button.

Counter Args

Resource Usage

This is a valid argument and the primary downside of the proposal with Qt being the obvious problem. The counter argument is the conclusion of this comment.

[...] set them up and understand how to use them. This would be a very tough pill to swallow for most. [...]

See the above axiom

Usability / Support

Every download and setup of tools can be done with a python script since languages like bash can be daunting.
Assuming we are completely understand what words like "cross compile", "bundle dependencies" and "package directory" mean (which we are not), your counter proposal will make us spend more time figuring out 3times the technologies in order to achieve the same thing git and our current tools will.

The end-user does not have to maintain a toolchain, something that would overwhelm the vast majority of users

As mentioned above they wont because every operation is handled by us in through scripts.

Compiling takes a long time, especially on older or less powerful hardware. For example, installing gnome via pacman takes a minute or two on a modern system. Compiling it from source takes several hours.

Gnome is a desktop environment, ours is more or less just a database interface. Its important to know the scope of the program when comparing. Building can take time especially the first time its built. In the current state building takes 1minute and 30ish seconds. People dont really care how long it takes to download and install (within reason, one day is obviously alot) as long as it is lightning fast at runtime.

Apart from that, imagine the complexity of designing an installer that installs and configures a complete development toolchain, as well as builds an application. For each of these applications there are hour long guides explaining how to set up each single one of them, let alone make them work together.

The toolchain proposed is simpler than the counter-proposed. It isnt easy, but then again nothing is that is a truism. What we are using right now (git qmake) is the only external tool required. For starters the git commands required really are git clone to get the repo and git fetch/pull to see if any changes have occured. After that its just qmake. The .pro file (qmake) has all the necessary commands in order to build our sources. What we do need to figure out is all the commands to actually install the tools on the users machine. Light OS specific stuff can be bundled with the installer. Python can handle pretty much any file download from the net.

Counter Conclusion

Im not disregarding the tools mentioned but for someones first non trivial application especially with the purpose of being cross platform, working in a bulky but understood toolchain is better than working well with an more complex toolchain.

The proposed toolchain is bulky and simple. The counter proposed is sophisticated and complex, something it shouldnt be for neither the experience level of its devs nor the project stage.

[fiffty-50]

The user only clicks a button.

While being a great concept there is a problem with this. When installing several pieces of software you are dealing with several licenses, each of which have to be accepted by the user. Qt itself for example has a rather complicated licensing structure, with LGPL, GPL and commercial options. I am not even sure about the legalities of distributing all these programs.

The user only clicks a button.

Let me emphasise again. I agree with this point. But I believe we need to separate the responsibilities here. Going back to your original proposal, this is how we should look at it: The users responsibility is clicking a button. That's it.

Python can handle pretty much any file download from the net.

Python is an interpreted language, which means the user has to install it first as well, this would add even more complexity to the process. We would have to write a script to:

• download an installer for python
• Install an interpreter
• launch a python script to install a compiler, framework and tools
• compile the program.
• on 3 different platforms

At any of these steps, errors can occur, which would then be ours to troubleshoot. This seems like a much taller order than learning how to create an application bundle.

In the current state building takes 1minute and 30ish seconds. [...] People don't really care how long it takes to download and install

This is because you are only compiling the program itself, the libraries it uses are already installed on your computer. Running the script as you are imagining it, installing python, git, cmake, a compiler, the qt framework, then compiling the app would take at least 10 minutes, probably longer. Significantly longer if you're not behind a fibre connection. Installing an app shouldn't take longer than installing an operating system. I could get a full desktop up and running in a VM before a script like that would finish.

Reinventing the wheel

If everybody is doing something one way, it does not automatically mean it's the right way. But it normally is that way for a reason.

Almost all software is distributed in binary packages.

In the Linux world, software is distributed as source sometimes. However, always under the prerogative 'Here's the code. You do the compiling', making it the users responsibility to get the toolchain working. And even then, there is almost always a binary available as well (if the program is directed at an audience of normal users). On Windows I have not once seen a program even being distributed as source, on MacOs I have too little experience to say.

Let me go out on a limb and rephrase this:

All software not aimed at developers is distributed in binary packages.

Allow me to challenge you to this:

Name one cross-platform application that is being distributed the way you are suggesting.

If you manage to find an example, I would be more than happy to take a look at it. Otherwise I feel like trying to reinvent the wheel is more trouble than going out and learning how to drive a car.

[fiffty-50]

Proof of Concept

I created a package for windows and Linux as a proof-of-concept. I used windeployqt and tools provided by appimage for the Linux release. While the process is certainly not trivial, it is also nothing too daunting.

I tested the packages on a clean install of windows and on live environments for Ubuntu 14.04, Xubuntu 1604, Ubuntu 18.04, Manjaro 20.01, openSuse Leap 15.2 and Fedora 33. None of these had any additional libraries, tools or other dependencies installed.

They all worked effortlessly and out of the box in different distributions and desktop environments, even different OS's (with MacOs still to be tested), so I am confident that this would be a viable option for distribution. The only problem I encountered was on Ubuntu 14.04, but it has been end-of-life for almost a year at this point so I am not too concerned about that.

The total archive is around 90Mb for the App Image and around 30Mb for the Windows package. While that's pretty OK I am sure this can still be trimmed.