Population-Density: Only list countries for download where data is available

[mattigrthr]

So far, we have listed all countries using the i18n-iso-countries npm package. However, there is no data available for some countries, e.g., due to bad satellite imagery. Facebook constantly improves its approach and data so that they might add new countries in the future. Therefore, we should use the AWS CLI to only list countries where data is available to avoid an error during the processing.

The command for that is:

aws s3 ls --no-sign-request s3://dataforgood-fb-data/csv/month=2019-06/

This has to adapt to the month partition dynamically.

[tjukanovt]

Hi @mattigrthr! Have you thought about an option to user Kontur poulation grid instead? It contains GHSL and more and it is already aggregated to H3 grids.

Link to the data: https://data.humdata.org/dataset/kontur-population-dataset

[mattigrthr]

Hi @tjukanovt, no, we didn't know about that. That seems mind-blowing. 🤯 Do I understand it correctly that they basically incorporate the Facebook population density datasets we're currently processing and combine them with other population sources? So you would not lose the level of granularity?

[tjukanovt]

Well the Facebook data is supposed to be 30 meter resolution, but as it is an estimate and derived from satellite images e.g. it never really is that detailed. The Kontur data is 400 meters so you would lose some granularity, but in return you would get global coverage. And yes: they have combined the FB data with other data sources.

[floriankuwala]

Do you have an insight on what for kind of features are provided? Could not find it on their website, HD or in Desaster Ninja. It would be super interesting if they have also implemented a temporal variable like year. Also, did you do already the parsing of the Kontur file into a different file format? It would be awesome to integrate that data source as well :-)

[mattigrthr]

I created a separate issue for including a new data source to keep this one here purely about the download UX.

Let's continue the feature discussions, etc. over here 👉🏽 #35

[mattigrthr]

This issue has been resolved by switching the file download to the hdx Python package. The file downloader can be found in kuwala/common/python_utils/src/FileSelector.py.