Merge branch 'development' into feat/GDPR-compliance

Signed-off-by: Anuj Gupta <84966248+Anuj-Gupta4@users.noreply.github.com>

.pre-commit-config.yaml

@@ -9,7 +9,7 @@ repos:
   # Lint / autoformat: Python code
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: "v0.9.1"
+    rev: "v0.9.2"
     hooks:
       # Run the linter
       - id: ruff
@@ -21,7 +21,7 @@ repos:
 
   # Deps: ensure Python uv lockfile is up to date
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.5.18
+    rev: 0.5.21
     hooks:
       - id: uv-lock
         files: src/backend/pyproject.toml

CHANGELOG.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## 2025.1.0 (2025-01-24)
+
+### Feat
+
+- **backend**: add `integrations` router with API key functionality (external apps) (#2110)
+- entity pulse effect on rejected submissions (#2018)
+- **mapper**: distance constraint add on frontend (#2084)
+- **frontend**: submission table date range filter (#2091)
+- **backend**: get api for project's geometry log (#2090)
+- capability to draw new polygon and linestring geoms (#2082)
+- **backend**: add filters for submission date in submission table and downloads (#2077)
+- **backend**: osm-fieldwork --> 0.18.0 (submission filter param + config new feat geom type)
+- **geolocation**: comments add
+- add submission ids in entities statuses endpoint (#2038)
+- consider every additional entities without clipping them with AOI (#2017)
+- update the version of osm-fieldwork (#2029)
+- **backend**: send org approval message to creator (#2008)
+- add submission ids as a dataset property for the entities (#2007)
+- update the version of fmtm-splitter 2.0.0 (#1996)
+- **mapper**: project details section add to bottom sheet (#1994)
+- **mapper**: prompt user to download custom ODK Collect on first load (#1989)
+
+### Fix
+
+- **backend**: delete submission photos while deleting project to avoid foreignkey constraint (#2112)
+- **backend**: review state for received on submission table (#2101)
+- **+page**: remove irrelevant subscribeToEntityStatusUpdates call in onMount
+- use task index instead of id in task boundary geojson properties (#2095)
+- **mapper**: get entities after page load to speed up first paint (#2051)
+- **backend**: get total_tasks count on single project response
+- **backend**: various fixes based on sentry error reports (#2053)
+- default odk creds when organisation do not have their own during project creation (#2070)
+- change geom to geojson in db model
+- **generateBasemap**: update tile source option value (#2050)
+- **geolocation**: fetch routing api on every 10 seconds
+- **dialog-entities-actions**: show alert instead of turning on location on navigation
+- **createProjecSlice**: clear additionalFeatureGeojson state after successful project creation (#2041)
+- compose file name for backend test stage
+- **mapper**: replace task id with index on activities panel (#2002)
+- **mapper**: task id with task index on mapper frontend  (#1997)
+- **backend**: include organisation name in minimal project query results (#1993)
+- **frontend**: small fix to reset frontend login if cookie refresh fails
+- **mapper**: offline mode button visibility & basemap component TS type error (#1990)
+- parse geojson to featcol in generate data extract (#1983)
+
+### Refactor
+
+- **mapper**: feature legend & layer-switcher (#2107)
+- **mapper**: relocate entity sync button (#2100)
+- **frontend**: organization management pages (#2097)
+- **frontend**: JS to TS conversion: update useDispatch to useAppDispatch  (#2076)
+- **frontend**: update TS types on actions (#2054)
+- replace incorrect osm libya logo with official osm logo
+- **frontend**: terminologies and wording update for users (#1978)
+- **mapper**: ts errors on frontend (#2006)
+- **mapper**: add mapping guide to sidebar + update links
+
 ## 2024.5.0 (2024-12-11)
 
 ### Feat

README.md

@@ -20,7 +20,7 @@
 
 | **CI/CD** | | [![Build and Deploy](https://github.com/hotosm/fmtm/actions/workflows/build_and_deploy.yml/badge.svg?branch=main)](https://github.com/hotosm/fmtm/actions/workflows/build_and_deploy.yml?query=branch%3Amain) [![Build CI Img](https://github.com/hotosm/fmtm/actions/workflows/build_ci_img.yml/badge.svg?branch=development)](https://github.com/hotosm/fmtm/actions/workflows/build_ci_img.yml) [![Build ODK Images](https://github.com/hotosm/fmtm/actions/workflows/build_odk_imgs.yml/badge.svg?branch=development)](https://github.com/hotosm/fmtm/actions/workflows/build_odk_imgs.yml) <br> [![🔧 Build Proxy Images](https://github.com/hotosm/fmtm/actions/workflows/build_proxy_imgs.yml/badge.svg?branch=development)](https://github.com/hotosm/fmtm/actions/workflows/build_proxy_imgs.yml) [![Publish Docs](https://github.com/hotosm/fmtm/actions/workflows/docs.yml/badge.svg?branch=development)](https://github.com/hotosm/fmtm/actions/workflows/docs.yml) [![pre-commit.ci](https://results.pre-commit.ci/badge/github/hotosm/fmtm/development.svg)](https://results.pre-commit.ci/latest/github/hotosm/fmtm/development) |
 | :--- | :--- | :--- |
-| **Tech Stack** | | ![FastAPI](https://img.shields.io/badge/FastAPI-005571?style=for-the-badge&logo=fastapi) ![React](https://img.shields.io/badge/react-%2320232a.svg?style=for-the-badge&logo=react&logoColor=%2361DAFB) ![Postgres](https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white) ![Kubernetes](https://img.shields.io/badge/kubernetes-%23326ce5.svg?style=for-the-badge&logo=kubernetes&logoColor=white) ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white) |
+| **Tech Stack** | | ![FastAPI](https://img.shields.io/badge/FastAPI-005571?style=for-the-badge&logo=fastapi) ![React](https://img.shields.io/badge/react-%2320232a.svg?style=for-the-badge&logo=react&logoColor=%2361DAFB) ![Svelte](https://img.shields.io/badge/Svelte-4A4A55?style=for-the-badge&logo=svelte) ![Postgres](https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white) ![Kubernetes](https://img.shields.io/badge/kubernetes-%23326ce5.svg?style=for-the-badge&logo=kubernetes&logoColor=white) ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white) |
 | **Code Style** | | [![Backend Style](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/format.json&labelColor=202235)](https://github.com/astral-sh/ruff) [![Frontend Style](https://img.shields.io/badge/code%20style-prettier-F7B93E?logo=Prettier)](https://github.com/prettier/prettier) [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://pre-commit.com) |
 | **Quality** | | [![Coverage](https://docs.fmtm.dev/coverage.svg)](https://docs.fmtm.dev/coverage.html) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/9218/badge)](https://www.bestpractices.dev/projects/9218) |
 | **Community** | | [![Slack](https://img.shields.io/badge/Slack-Join%20the%20community!-d63f3f?style=for-the-badge&logo=slack&logoColor=d63f3f)](https://slack.hotosm.org) [![All Contributors](https://img.shields.io/github/all-contributors/hotosm/fmtm?color=ee8449&style=flat-square)](#contributors-) |
@@ -121,13 +121,14 @@ Alternatively see the [docs](https://docs.fmtm.dev) for various deployment guide
 |✅| 📱 open ODK Collect with feature already selected |
 |✅| 📱 live updates during mapping (if online) |
 |✅| 📱 features turn green once mapped |
-|⚙️| 📱 better support for mapping **new** points, lines, polygons |
-|⚙️| 📱 navigation and capability for routing to map features |
-| | 📱 integrate ODK Web Forms (to avoid switching apps) |
+|✅| 📱 better support for mapping **new** points, lines, polygons |
+|✅| 📱 navigation and capability for routing to map features |
+|⚙️| 📱 integrate ODK Web Forms (to avoid switching apps) |
+|⚙️| 🖥️ multiple approaches to task splitting algorithm |
+|⚙️| 🖥️ user role management per project |
 | | 📱 fully offline field mapping |
 | | 🖥️ organization creation and management |
 | | 🖥️ simplify project creation with basic / advanced workflows |
-| | 🖥️ refinements to task splitting algorithm |
 | | 🖥️ improvements to the validation criteria and workflow |
 | | 🖥️ export (+merge) the final data to OpenStreetMap |
 | | 🖥️ better data visualisation and export options |

chart/Chart.yaml

@@ -3,7 +3,7 @@ type: application
 name: fmtm
 description: Field Mapping Tasking Manager - coordinated field mapping.
 version: "0.1.0"
-appVersion: "2024.5.0"
+appVersion: "2025.1.0"
 maintainers:
 - email: sam.woodcock@hotosm.org
   name: Sam Woodcock

chart/values.yaml

@@ -114,7 +114,7 @@ s3:
   replicas: 1
   image:
     repository: quay.io/minio/minio
-    tag: RELEASE.2024-12-18T13-15-44Z
+    tag: RELEASE.2025-01-20T14-49-07Z
   environment:
     MINIO_BROWSER: "off"
   mountPath: /mnt/data

contrib/just/start/Justfile

@@ -32,7 +32,7 @@ backend:
   cd {{justfile_directory()}}
   docker compose up -d api
 
-# Start backend API only
+# Start backend API without docker
 [no-cd]
 backend-no-docker:
   #!/usr/bin/env sh
@@ -43,11 +43,34 @@ backend-no-docker:
   OSM_SECRET_KEY="" ENCRYPTION_KEY="" \
     uv run uvicorn app.main:api --host 0.0.0.0 --port 8000
 
-# Start frontend UI only
+# Start frontend UI (also starts backend)
 [no-cd]
 frontend:
   docker compose up -d ui
 
+# Start frontend UI without docker, connected to staging
+[no-cd]
+frontend-dev:
+  #!/usr/bin/env sh
+
+  cd {{justfile_directory()}}/src/frontend
+
+  pnpm install
+  VITE_API_URL=https://api.stage.fmtm.hotosm.org \
+  pnpm run dev
+
+# Start mapper frontend UI without docker, connected to staging
+[no-cd]
+mapper-frontend-dev:
+  #!/usr/bin/env sh
+
+  cd {{justfile_directory()}}/src/mapper
+
+  pnpm install
+  VITE_API_URL=https://api.stage.fmtm.hotosm.org \
+  VITE_SYNC_URL=https://sync.stage.fmtm.hotosm.org \
+  pnpm run dev
+
 # Start FMTM without ODK Central
 [no-cd]
 without-central:

docker-compose.development.yml

@@ -178,7 +178,7 @@ services:
     restart: "on-failure:2"
 
   s3:
-    image: "docker.io/minio/minio:${MINIO_TAG:-RELEASE.2024-12-18T13-15-44Z}"
+    image: "docker.io/minio/minio:${MINIO_TAG:-RELEASE.2025-01-20T14-49-07Z}"
     environment:
       MINIO_ROOT_USER: ${S3_ACCESS_KEY}
       MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY}

docker-compose.main.yml

@@ -119,7 +119,7 @@ services:
     restart: "on-failure:2"
 
   s3:
-    image: "docker.io/minio/minio:${MINIO_TAG:-RELEASE.2024-12-18T13-15-44Z}"
+    image: "docker.io/minio/minio:${MINIO_TAG:-RELEASE.2025-01-20T14-49-07Z}"
     environment:
       MINIO_ROOT_USER: ${S3_ACCESS_KEY}
       MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY}

docker-compose.yml

@@ -225,7 +225,7 @@ services:
     restart: "on-failure:2"
 
   s3:
-    image: "docker.io/minio/minio:${MINIO_TAG:-RELEASE.2024-12-18T13-15-44Z}"
+    image: "docker.io/minio/minio:${MINIO_TAG:-RELEASE.2025-01-20T14-49-07Z}"
     environment:
       MINIO_ROOT_USER: ${S3_ACCESS_KEY:-fmtm}
       MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY:-somelongpassword}

docs/dev/Backend.md

@@ -25,7 +25,7 @@ URLs defined in the docker-compose file and your env file.
    and ensure that it is running on your local machine.
 2. From the command line: navigate to the top level directory
    of the FMTM project.
-3. From the command line run: `docker-compose pull`.
+3. From the command line run: `docker compose pull`.
    This will pull the latest container builds from **main** branch.
 4. Make sure you have a `.env` file with all required variables, see
    [here](../INSTALL.md#2-create-an-env-file).

docs/dev/Frontend.md

@@ -30,19 +30,17 @@ For details on how to run the API first, please see:
 
 ## 2. Start the Frontend locally
 
-### 2A: Navigate to the frontend subdirectory
+To run the frontend locally, connected to the staging server as a backend:
 
-`cd src/frontend`
-
-### 2B: Install dependencies
-
-`npm install`
-
-### 2C. Run the project
+```bash
+just --unstable start frontend-dev
+```
 
-Run the frontend with hot-reloading: `npm run dev`
+The mapper frontend can be started with a similar command:
 
-The frontend should now be accessible at: `http://127.0.0.1:<PORT_NUMBER>`
+```bash
+just --unstable start mapper-frontend-dev
+```
 
 ## Frontend Tips
 

src/Dockerfile.ui.debug

@@ -1,4 +1,4 @@
-FROM docker.io/node:20-slim
+FROM docker.io/node:22-slim
 RUN set -ex \
     && apt-get update \
     && DEBIAN_FRONTEND=noninteractive apt-get install \
@@ -9,6 +9,6 @@ WORKDIR /app
 COPY --from=code ./package.json ./pnpm-lock.yaml ./
 ENV PNPM_HOME="/pnpm"
 ENV PATH="$PNPM_HOME:$PATH"
-RUN corepack enable && corepack use pnpm@9.3.0
+RUN corepack enable && corepack use pnpm@9.15.4
 RUN pnpm install
 ENTRYPOINT ["pnpm", "run", "dev"]

src/Dockerfile.ui.prod

@@ -1,11 +1,11 @@
-FROM docker.io/node:20-slim AS base
+FROM docker.io/node:22-slim AS base
 ARG VITE_API_URL
 ARG VITE_SYNC_URL
 ENV VITE_API_URL=${VITE_API_URL} \
     VITE_SYNC_URL=${VITE_SYNC_URL} \
     PNPM_HOME="/pnpm" \
     PATH="$PATH:/pnpm"
-RUN corepack enable && corepack use pnpm@9.3.0
+RUN corepack enable && corepack use pnpm@9.15.4
 WORKDIR /app
 
 

src/backend/Dockerfile

@@ -16,7 +16,7 @@
 #
 ARG PYTHON_IMG_TAG=3.12
 ARG UV_IMG_TAG=0.5.2
-ARG MINIO_TAG=${MINIO_TAG:-RELEASE.2024-12-18T13-15-44Z}
+ARG MINIO_TAG=${MINIO_TAG:-RELEASE.2025-01-20T14-49-07Z}
 FROM ghcr.io/astral-sh/uv:${UV_IMG_TAG} AS uv
 FROM docker.io/minio/minio:${MINIO_TAG} AS minio
 

src/backend/app/__version__.py

@@ -1 +1 @@
-__version__ = "2024.5.0"
+__version__ = "2025.1.0"

src/backend/app/auth/auth_deps.py

@@ -19,7 +19,7 @@
 """Auth dependencies, for restricted routes and cookie handling."""
 
 from time import time
-from typing import Optional
+from typing import Annotated, Optional
 
 import jwt
 from fastapi import Header, HTTPException, Request, Response
@@ -262,7 +262,7 @@ async def refresh_cookies(
 
 
 async def login_required(
-    request: Request, access_token: str = Header(None)
+    request: Request, access_token: Annotated[Optional[str], Header()] = None
 ) -> AuthUser:
     """Dependency for endpoints requiring login."""
     if settings.DEBUG:
@@ -277,7 +277,7 @@ async def login_required(
 
 
 async def mapper_login_required(
-    request: Request, access_token: str = Header(None)
+    request: Request, access_token: Annotated[Optional[str], Header()] = None
 ) -> AuthUser:
     """Dependency for mapper frontend login."""
     if settings.DEBUG:

src/backend/app/db/models.py

@@ -159,6 +159,7 @@ class DbUser(BaseModel):
     tasks_validated: Optional[int] = None
     tasks_invalidated: Optional[int] = None
     projects_mapped: Optional[list[int]] = None
+    api_key: Optional[str] = None
     registered_at: Optional[AwareDatetime] = None
     last_login_at: Optional[AwareDatetime] = None
 
@@ -1358,6 +1359,12 @@ async def update(
     async def delete(cls, db: Connection, project_id: int) -> bool:
         """Delete a project and its related data."""
         async with db.cursor() as cur:
+            await cur.execute(
+                """
+                DELETE FROM submission_photos WHERE project_id = %(project_id)s;
+            """,
+                {"project_id": project_id},
+            )
             await cur.execute(
                 """
                 DELETE FROM background_tasks WHERE project_id = %(project_id)s;
@@ -1795,6 +1802,7 @@ def slugify(name: Optional[str]) -> Optional[str]:
 class DbGeometryLog(BaseModel):
     """Table geometry log."""
 
+    id: Optional[UUID] = None
     geojson: dict
     status: GeomStatus
     project_id: Optional[int] = None
@@ -1834,6 +1842,25 @@ async def create(
             new_geomlog = await cur.fetchone()
         return new_geomlog
 
+    @classmethod
+    async def all(cls, db: Connection, project_id: int) -> Optional[list[Self]]:
+        """Retrieve geometry logs from a project."""
+        async with db.cursor(row_factory=class_row(cls)) as cur:
+            await cur.execute(
+                """
+                SELECT * FROM geometrylog WHERE project_id=%(project_id)s;
+            """,
+                {"project_id": project_id},
+            )
+            if cur.rowcount == 0:
+                raise HTTPException(
+                    status_code=HTTPStatus.NOT_FOUND,
+                    detail=f"""
+                    No geometry log with project_id {project_id}
+                    """,
+                )
+            return await cur.fetchall()
+
     @classmethod
     async def delete(
         cls,

src/backend/app/integrations/__init__.py

@@ -0,0 +1 @@
+"""FMTM integrations API, for linking to external services."""