updates 2025-02-08 - v0.1.2 changes

README.md

@@ -68,3 +68,54 @@ This equates to a lot of data and processing it can be slow and painful if you'r
 - Automate your development and deployment so you can focus on analysis and delivering value from the data.
 - Utilise the power of cloud compute to process data faster.
 - Utilise elements of the Modern Data Stack that allow for slick reporting and BI.
+
+## Impact Scores Model
+
+I currently use this pipeline to generate a monthly street works impact score for each USRN in England
+
+## Overview
+This model calculates and normalises impact scores for road works across England's highway network.
+
+It combines permit data with traffic and infrastructure metrics to produce weighted impact scores that reflect both the direct impact of works and the broader network.
+
+## Input Data Sources
+1. **Permit Data**
+   - In-progress works (`in_progress_list_england`)
+   - Completed works (`completed_list_england`)
+   - Key fields: USRN, street name, highway authority, work category, TTRO requirements, traffic sensitivity, traffic management type
+
+2. **Infrastructure Data**
+   - UPRN-USRN mapping (`uprn_usrn_count`)
+   - DFT Local Authority data (`dft_la_data_latest`) contains road length and traffic flow information
+
+## Impact Score Calculation
+
+### Base Impact Factors
+- **Work Category Impact** (0-5 points)
+  - Major works: 5 points
+  - Immediate works: 4 points
+  - Standard works: 2 points
+  - Minor works: 1 point
+  - etc
+
+- **Additional Impact Factors**
+  - TTRO Required: +0.5 points
+  - Traffic Sensitive: +0.5 points
+  - Traffic Management Impact: +0-2 points based on severity
+  - UPRN Density Impact: +0.2-1.6 points based on UPRN point density on a USRN
+
+### Network Context Adjustment
+The model applies a network importance factor based on:
+- Total road length in the authority
+- Traffic flow data (2023)
+- Traffic density per km (length/flow)
+- Normalised network importance factor (0-1 scale)
+
+## Output
+The final model produces a table with:
+- Location identifiers (USRN, street name, highway authority)
+- Raw impact scores
+- Network metrics (road length, traffic flow, density)
+- Final weighted impact scores that account for both direct works impact and network importance
+
+This model helps identify high-impact works areas by considering both the immediate disruption of works and their context within the broader road network.

makefile

@@ -5,15 +5,16 @@
 include .env
 
 # Docker and AWS section
-.PHONY: docker-login docker-build docker-tag docker-push docker-verify docker-all
+# AWS ECR Docker deployment configuration
+.PHONY: docker-all docker-login docker-build docker-tag docker-push docker-verify
 
 docker-all: docker-login docker-build docker-tag docker-push docker-verify
 
 docker-login:
 	aws ecr get-login-password --region $(REGION) | docker login --username AWS --password-stdin $(ACCOUNT_ID).dkr.ecr.$(REGION).amazonaws.com
 
 docker-build:
-	docker build -t $(REPO_NAME) .
+	docker buildx build --platform=linux/amd64 -t $(REPO_NAME) .
 
 docker-tag:
 	docker tag $(REPO_NAME):latest $(ACCOUNT_ID).dkr.ecr.$(REGION).amazonaws.com/$(REPO_NAME):latest

next_steps.md

@@ -2,48 +2,53 @@
 
 Model for impact scores
 
-Completing v0.1.1
-
-Impact scoring is created by combining several factors:
-
-Street Manager Data
-1. Base Impact Score (from work category):
-- Major works: 5 points
-- Immediate works: 3 points
-- Standard/HS2: 2 points
-- Minor works: 1 point
-
-2. Additional Impact Factors:
-- TTRO (Temporary Traffic Regulation Order) required: +0.5
-- Traffic sensitive street: +0.5
-- Traffic Management Impact:
-  - High impact (road closure, contra flow etc.): +2.0
-  - Medium impact (give and take, stop/go): +1.0
-  - Low impact (some carriageway use): +0.5
-  - No impact: 0
-  - Unknown: +0.5
-
-UPRNs
-3. UPRN (Property) Density Impact:
-Adds 0.3-1.5 points based on number of properties affected:
-- ≤5 properties: +0.3
-- ≤10: +0.4
-- ≤25: +0.5
-- ≤50: +0.7
-- ≤100: +0.9
-- ≤200: +1.1
-- ≤500: +1.3
-- >500: +1.5
-
-The model:
-1. Combines data from both in-progress and completed works
-2. Joins with UPRN counts per street (using USRNs for the join)
-3. Calculates individual impact scores
-4. Groups by street (USRN) to get total impact
-5. Includes street metadata and geometry
-
-Final output gives each street's total impact score based on
-- works activity
-- traffic sensitivity
-- property density
-- traffic management measures in place.
+Completing v0.1.2
+
+I'll create a clear summary of this dbt model that calculates impact scores for road works in England.
+
+## Impact Scores Model
+
+## Overview
+This model calculates and normalizes impact scores for road works across England's highway network. It combines permit data with traffic and infrastructure metrics to produce weighted impact scores that reflect both the direct impact of works and the broader network context.
+
+## Input Data Sources
+1. **Permit Data**
+   - In-progress works (`in_progress_list_england`)
+   - Completed works (`completed_list_england`)
+   - Key fields: USRN, street name, highway authority, work category, TTRO requirements, traffic sensitivity, traffic management type
+
+2. **Infrastructure Data**
+   - UPRN-USRN mapping (`uprn_usrn_count`)
+   - DFT Local Authority data (`dft_la_data_latest`) contains road length and traffic flow information
+
+## Impact Score Calculation
+
+### Base Impact Factors
+- **Work Category Impact** (0-5 points)
+  - Major works: 5 points
+  - Immediate works: 4 points
+  - Standard works: 2 points
+  - Minor works: 1 point
+  - etc
+
+- **Additional Impact Factors**
+  - TTRO Required: +0.5 points
+  - Traffic Sensitive: +0.5 points
+  - Traffic Management Impact: +0-2 points based on severity
+  - UPRN Density Impact: +0.2-1.6 points based on UPRN point density on a USRN
+
+### Network Context Adjustment
+The model applies a network importance factor based on:
+- Total road length in the authority
+- Traffic flow data (2023)
+- Traffic density per km (length/flow)
+- Normalised network importance factor (0-1 scale)
+
+## Output
+The final model produces a table with:
+- Location identifiers (USRN, street name, highway authority)
+- Raw impact scores
+- Network metrics (road length, traffic flow, density)
+- Final weighted impact scores that account for both direct works impact and network importance
+
+This model helps identify high-impact works areas by considering both the immediate disruption of works and their context within the broader road network.