Release Notes (2.4)

App Changes

• Date range chooser in analytics to choose what time period to graph data over - mainly useful for the hourly queries.
• Day/Hour choice in analytics.
• Filter by Region uses the title not the id of a region when autocompleting.
• Filter by Region orders the list by the level of the region type, so countries before counties.
• NUTS1, ISO Countries and UK Countries are now supported as region types.
• Retrospective tweets can now be displayed previously determined by the existence of html field - no longer required. NB: if the profile picture is broken nothing is shown.
• Lots of small changes to help support retrospective data display made to support the demo environment.
• Filter by Region gets its filter list from ref_geo_regions now. ref_geo_regions is the authoritative source of region data from now onwards.
• Numerous performance fixes in the app to reduce latency in the application itself, which became noticeable now the DB optimisations are in place.
• Potentially sensitive (i.e. porn, violence etc.) tweets are now made transparent (making them de-emphasised) and no media will be shown for those tweets.

Server (Lambda) Changes

• The server call for the new stats calculation uses the same REST request, the old one is now .../complex-stats ready for the addition of more complex criteria.
• All data from the server over a specified limit is now GZIP'ed.
• Geography is loaded one region at a time as the size of the data broken API Gateway's limit.

Database Changes

• Three different stored procedures are called, one every 5 mins, one every hour and one every day. They update the various materialized view tables so that we have quick current data, hourly 4 day window updates (for Twitter compliance) and daily complete refreshes (for new regions being added).
• Event Scheduling is also now turned on (see above)
• mat_view_regions is now calculated using geographic function to compare the location in live_text with the boundary in ref_geo_regions.
• The boundaries have been fixed so that MySQL considers them correct. In some cases this required adding a small (metres) extension to their boundary to remove artefacts.
• Invalid X,Y co-ordinates (outside of UK bounding box) are swapped around each night, temporary fix until the data is consistent.
• mat_view_days and mat_view_hours are additional tables which help query performance they contain all days and all hours respectively for which there exists data.
• mat_view_timeseries_date and mat_view_timeseries_hour are now periodically updated to contain data to be used by the timeseries analytics graphs.
• Scheduled task also rebuilds the tables with OPTIMIZE TABLE every day
• Internal_debug table stores the debug logging from stored procedures
• The materialised views for the analytics time series now have a prefix in source_text which is the MD5 hash of source + ":" + hazard + ":" + region, boolean searches are now prefixed with something like +(5675675675de65ed76e cf45f765765f6f567f) which means that the text search should also match one of the hashes. This effectively creates a combined free text and field index 😄
• The possible enumerated and boolean criteria (which I refer to as the common criteria) are listed in the table mat_view_map_criteria. This is to support faster exceedance calculations.
• The table mat_data_days has been added, this contains the total number of days for the common criteria that have data. Used for exceedance.
• The table mat_view_text count contains the daily totals for the common criteria. Also used for exceedance.
• The most common use case of exceedance has been optimised

Test Changes

• All the tests have been reworked for a new end date (16th Sept) which has a little more data than the previous one.
• All tests have been fixed and several timing issues (especially in analytics) have been fixed.
• New test 13_url_state_through_components_spec.js added to deal with issues moving between the application parts and maintaining/shedding state as required.

Issues resolved

• Region selection box on Analytics #310
• Empty Drop Downs #311
• Graph Menu #309
• Loading page #306
• PWS - 3.3 Add export function to save tweet data to local file. #248
• Create test for the EOC value, selected county and layer being passed between parts of the app. #337
• PWS - 3.2a - Display in local time t #251
• PWS - 3.3 Add export function to save tweet data to local file. #248