Added script to export list of private jets and improved methodology (#…

…14)

README.md

@@ -26,6 +26,9 @@ and has anonymous and public read permissions.
 
 ## How to use
 
+This repository contains both a Rust library and a set of [`examples/`](./examples) used
+to perform actual calculations. To use one of such examples:
+
 1. Install Rust
 2. run `cargo run --example single_day -- --tail-number "OY-GFS" --date "2023-10-20"`
 3. open `OY-GFS_2023-10-20_0.md`
@@ -47,106 +50,13 @@ As of today, the flag `--azure-sas-token` is only available when the code is exe
 from `main`, as writing to the blob storage must be done through a controlled code base
 that preserves data integrity.
 
-## Assumptions
-
-* Aircrafts are uniquely identified by a tail number (aka registration number), e.g.
-  `OY-EUR`, by the owner of the aircraft.
-* Civil aviation in Europe is mandated to have an ADS-B transponder turned on in-flight.
-* Every aircraft flying has a unique transponder identifier (hereby denoted the ICAO number),
-  e.g. `4596B2`.
-* At any given point in time, there is a one-to-one relationship between the ICAO number and a tail number (`OY-EUR -> 4596B2`)
-
-## Functional specification
-
-### FS-1 - Behaviour
-
-This solution is a CLI executed in a terminal on Windows, Linux or Mac OS.
-
-It receives two arguments, a tail number and a date, and writes a
-markdown file with a description of:
-* the owner of said tail number
-* the legs that tail number flew on that date
-* how many emissions (CO2e) were emitted
-* how many emissions (CO2e) would have been emitted if a commercial flight would
-  have been taken instead.
-* how many emissions per year (CO2e/y) a Dane emits
-* The source of each of the claims.
-
-templated based on [`src/template.md`](./src/template.md).
-
-### FS-2 - Methodology
-
-The methodology used to support this solution is the follow:
-
-#### 1. Identify aircraft types whose primary use is private jet flying
-
-This was performed by a human, and consisted in going through different aircraft
-manufacturers' websites and identifying the aircrafts that were advertised as used
-for private flying.
-
-For example, `Dassault Falcon 2000` (`F2TH` in https://www.icao.int) is advertised as a
-private jet on https://www.dassaultfalcon.com/aircraft/overview-of-the-fleet/.
-
-This is stored in [`./src/types.csv`](./src/types.csv).
-
-#### 2. Identify all aircrafts, ICAO number tail number and type
-
-This is performed automatically by the computer program and consists
-in extracting the database of all aircrafts in https://globe.adsbexchange.com.
-
-Details are available in the source code, [src/aircraft_db.rs](./src/aircraft_db.rs).
-
-#### 3. Identify aircraft owner in denmark
-
-This was performed by a human, and consisted in extracting the ownership of the active
-tail number from website https://www.danishaircraft.dk.
-
-For example `OY-CKK` results in 3 records, whose most recent, `OY-CKK(3)`, is registered
-to owned by `Kirkbi Invest A/S`.
-
-This is stored in [`./src/owners.csv`](./src/owners.csv).
-
-It also consisted in extracting statements or slogans from these owners from their websites
-to illustrate the incompatibility between owning a private jet and their sustainability goals.
-
-This is stored in [`./src/owners.json`](./src/owners.json).
-
-#### 4. Identify ICAO number's route in a day
-
-This is performed automatically by the computer program and consists in looking for
-the historical route of the ICAO number in https://globe.adsbexchange.com.
-This contains the sequence of `(latitude, longitude)` and other information.
-
-Details are available in the source code, [src/legs.rs](./src/legs.rs).
-
-#### 5. Identify legs of a route
-
-This is performed automatically by the computer program and consists in identifying
-points during the flight that the aircraft is in mode "ground", and computing the leg
-between two ground situations.
-
-Since some aircrafts only turn on the transponder while in flight, we set that below 1000 feet
-the aircraft is considered on the ground.
-
-Details are available in the source code, [src/legs.rs](./src/legs.rs).
-
-#### 8. Compute emissions of leg
-
-This is performed automatically by the computer program and consists in using the same
-metholodogy as used by myclimate.org, available [here](https://www.myclimate.org/en/information/about-myclimate/downloads/flight-emission-calculator/), to compute the emissions of a commercial
-flight in first class.
-
-Details are available in the source code, [src/emissions.rs](./src/emissions.rs).
-
-#### 9. Write output
+## Methodology
 
-This is performed automatically by the computer program and consists in a template, available
-in [`src/template.md`](./src/template.md), to produce a complete document.
+The methodology used to extract information is available at [`methodology.md`](./methodology.md).
 
-Details are available in the source code, [src/main.rs](./src/main.rs).
+## Generated datasets
 
-## Design
+### Set of worldwide aicrafts whose primary use is to be a private jet:
 
-* Information can only be obtained from trustworthy publicly available sources that can
-be easily verified.
-* Main statements must be referenced against these sources
+* [Data](https://privatejets.blob.core.windows.net/data/database/private_jets/2023/11/06/data.csv)
+* [Description](https://privatejets.blob.core.windows.net/data/database/private_jets/2023/11/06/description.md)

examples/dk_jets.rs

@@ -67,19 +67,20 @@ async fn legs(
 
     let tasks = dates.map(|date| async move {
         Result::<_, Box<dyn Error>>::Ok(
-            flights::positions(&aircraft.icao_number, date, 1000.0, client)
+            flights::positions(&aircraft.icao_number, date, client)
                 .await?
                 .collect::<Vec<_>>(),
         )
     });
 
     let positions = futures::stream::iter(tasks)
         // limit concurrent tasks
-        .buffered(50)
+        .buffered(5)
         .try_collect::<Vec<_>>()
         .await?;
 
-    Ok(flights::real_legs(positions.into_iter().flatten()))
+    log::info!("Computing legs {}", aircraft.icao_number);
+    Ok(flights::legs(positions.into_iter().flatten()))
 }
 
 #[tokio::main]

examples/export_private_jets.rs

@@ -0,0 +1,96 @@
+use std::error::Error;
+
+use clap::Parser;
+use simple_logger::SimpleLogger;
+
+use flights::BlobStorageProvider;
+use flights::{load_aircraft_types, load_aircrafts};
+
+#[derive(clap::ValueEnum, Debug, Clone)]
+enum Backend {
+    Disk,
+    Azure,
+}
+
+const ABOUT: &'static str = r#"Exports the database of all worldwide aircrafts whose primary use is to be a private jet to "data.csv"
+and its description at `description.md` (in disk).
+If `azure_sas_token` is provided, data is written to the public blob storage instead.
+"#;
+
+#[derive(Parser, Debug)]
+#[command(author, version, about = ABOUT)]
+struct Cli {
+    /// The Azure token
+    #[arg(short, long)]
+    azure_sas_token: Option<String>,
+    #[arg(short, long, value_enum, default_value_t=Backend::Azure)]
+    backend: Backend,
+}
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn Error>> {
+    SimpleLogger::new()
+        .with_level(log::LevelFilter::Info)
+        .init()
+        .unwrap();
+
+    let cli = Cli::parse();
+
+    // optionally initialize Azure client
+    let client = match (cli.backend, cli.azure_sas_token.clone()) {
+        (Backend::Disk, None) => None,
+        (Backend::Azure, None) => Some(flights::fs_azure::initialize_anonymous(
+            "privatejets",
+            "data",
+        )),
+        (_, Some(token)) => Some(flights::fs_azure::initialize_sas(
+            &token,
+            "privatejets",
+            "data",
+        )?),
+    };
+
+    // load datasets to memory
+    let aircrafts = load_aircrafts(client.as_ref()).await?;
+    let types = load_aircraft_types()?;
+
+    let private_jets = aircrafts
+        .values()
+        // its primary use is to be a private jet
+        .filter(|a| types.contains_key(&a.model))
+        .collect::<Vec<_>>();
+
+    let mut wtr = csv::Writer::from_writer(vec![]);
+    for jet in private_jets {
+        wtr.serialize(jet).unwrap()
+    }
+    let data_csv = wtr.into_inner().unwrap();
+    let specification_md = r#"This dataset was created according to
+[this methodology](https://github.com/jorgecardleitao/private-jets/methdology.md).
+
+It contains 3 columns:
+* `icao_number`: The transponder identifier
+* `tail_number`: The tail number of the aircraft
+* `model`: The icao number of the aircraft type. It is only one of the ones
+  identified as private jet according to the methodology.
+
+Both `icao_number` and `tail_number` are unique keys (independently).
+"#;
+
+    if cli.azure_sas_token.is_some() {
+        let client = client.unwrap();
+        client
+            .put("database/private_jets/2023/11/06/data.csv", data_csv)
+            .await?;
+        client
+            .put(
+                "database/private_jets/2023/11/06/description.md",
+                specification_md.as_bytes().to_vec(),
+            )
+            .await?;
+    } else {
+        std::fs::write("data.csv", data_csv)?;
+        std::fs::write("description.md", specification_md.as_bytes())?;
+    }
+    Ok(())
+}

examples/period.rs

@@ -112,13 +112,13 @@ async fn main() -> Result<(), Box<dyn Error>> {
         increment: time::Duration::days(1),
     };
 
-    let iter = iter.map(|date| flights::positions(icao, date, 1000.0, client.as_ref()));
+    let iter = iter.map(|date| flights::positions(icao, date, client.as_ref()));
 
     let positions = futures::future::try_join_all(iter).await?;
     let mut positions = positions.into_iter().flatten().collect::<Vec<_>>();
     positions.sort_unstable_by_key(|x| x.datetime());
 
-    let legs = flights::real_legs(positions.into_iter());
+    let legs = flights::legs(positions.into_iter());
     log::info!("number_of_legs: {}", legs.len());
     for leg in &legs {
         log::info!(

examples/single_day.rs

@@ -35,19 +35,29 @@ enum Backend {
     Azure,
 }
 
-/// Simple program to greet a person
+const ABOUT: &'static str = r#"Writes a markdown file per leg (named `{tail-number}_{date}_{leg}.md`) on disk with a description of:
+* the owner of said tail number
+* the from and to
+* how many emissions (CO2e) were emitted
+* how many emissions (CO2e) would have been emitted if a commercial flight would
+  have been taken instead.
+* how many emissions per year (CO2e/y) a Dane emits
+* The source of each of the claims
+"#;
+
 #[derive(Parser, Debug)]
-#[command(author, version, about, long_about = None)]
+#[command(author, version, about = ABOUT)]
 struct Cli {
     /// The tail number
     #[arg(short, long)]
     tail_number: String,
     /// The date in format `yyyy-mm-dd`
     #[arg(short, long, value_parser = parse_date)]
     date: time::Date,
-    /// The Azure token
+    /// Optional azure token to write any new data to the blob storage
     #[arg(short, long)]
     azure_sas_token: Option<String>,
+    /// The backend to read cached data from.
     #[arg(short, long, value_enum, default_value_t=Backend::Azure)]
     backend: Backend,
 }
@@ -88,7 +98,7 @@ async fn flight_date(
     let icao = &aircraft.icao_number;
     log::info!("ICAO number: {}", icao);
 
-    let positions = positions(icao, date, 1000.0, client).await?;
+    let positions = positions(icao, date, client).await?;
     let legs = legs(positions);
 
     log::info!("Number of legs: {}", legs.len());

methodology.md

@@ -0,0 +1,90 @@
+# Methodology
+
+This document describes the general methodology used by this solution.
+
+## Assumptions
+
+* Aircrafts are uniquely identified by a tail number (aka registration number), e.g.
+  `OY-EUR`, by the owner of the aircraft.
+* Civil aviation in most of the world is mandated to have an ADS-B transponder turned on in-flight.
+* Every aircraft flying has a unique transponder identifier (hereby denoted the ICAO number),
+  e.g. `4596B2`.
+* At any given point in time, there is a one-to-one relationship between the ICAO number and a tail number (`OY-EUR -> 4596B2`)
+
+## Design
+
+* Information can only be obtained from trustworthy publicly available sources that can
+be easily verified.
+* Statements must be referenced against either existing sources or this methodology.
+
+## Methodology
+
+The methodology used to support this solution is the follow:
+
+### M-1: Identify all aircrafts, ICAO number tail number and type
+
+This is performed automatically by the solution and consists
+in extracting the database of all aircrafts in https://globe.adsbexchange.com.
+
+Details are available in the source code, [src/aircraft_db.rs](./src/aircraft_db.rs).
+
+### M-2: Identify aircraft types whose primary use is to be a private flying
+
+This was performed by a human, and consisted in going through different aircraft
+manufacturers' websites and identifying the aircrafts that were advertised as used
+for private flying.
+
+For example, `Dassault Falcon 2000` (`F2TH` in https://www.icao.int) is advertised as a
+private jet on https://www.dassaultfalcon.com/aircraft/overview-of-the-fleet/.
+
+This is stored in [`./src/types.csv`](./src/types.csv).
+
+**NOTE**: not all uses of a model whose primary use is to be a private jet is
+private jet. For example, private jets are often used for emergency services.
+
+### M-3: Identify ICAO number's route in a day
+
+This is performed automatically by the computer program and consists in looking for
+the historical route of the ICAO number in https://globe.adsbexchange.com.
+This contains the sequence of `(latitude, longitude)` and other information.
+
+Each position is assigned the state `Grounded` whether
+the transponder returns "grounded" or the (barometric) altitude is lower than 1000 feet,
+else it is assigned the state `Flying`.
+
+Source code is available at [src/icao_to_trace.rs](./src/icao_to_trace.rs).
+
+### M-4: Identify legs of a route
+
+This is performed automatically by the computer program and consists in identifying
+legs: contiguous sequence of positions that start and end on the state grounded.
+
+Furthermore, only legs fullfilling the below conditions are considered:
+
+* Its distance is higher than 3km
+* Its duration is longer than 5m
+
+Source code is available at [src/legs.rs](./src/legs.rs).
+
+### M-5: Compute emissions of leg in a commercial flight
+
+This is performed automatically by the computer program and consists in using the same
+metholodogy as used by myclimate.org, available [here](https://www.myclimate.org/en/information/about-myclimate/downloads/flight-emission-calculator/), to compute the emissions of a commercial
+flight in first class.
+
+Details are available in the source code, [src/emissions.rs](./src/emissions.rs).
+
+### M-6: Identify aircraft owner in Denmark
+
+This was performed by a human, and consisted in extracting the ownership of the active
+tail number from website https://www.danishaircraft.dk.
+
+For example `OY-CKK` results in 3 records, whose most recent, `OY-CKK(3)`, is registered
+to owned by `Kirkbi Invest A/S`.
+
+This is stored in [`./src/owners.csv`](./src/owners.csv).
+
+It also consisted in extracting statements or slogans from these owners from their websites
+to illustrate the incompatibility between owning a private jet and their sustainability goals.
+
+This is stored in [`./src/owners.json`](./src/owners.json).