-
Notifications
You must be signed in to change notification settings - Fork 2
Release Notes (2.5)
Most of this release is dedicated to the new (public) display mode. A read only, animated version of the map specifically for displaying on overhead (public) displays. This has not required any corresponding database changes, it has required the addition of a single new REST path of /map/:map/region-type/:regionType/text-for-public-display, this is paged with the page and pageSize values in the request body.
A new option has been added to the top menu to provide a display mode, which is reachable from the map. After clicking on display the user is presented with a dialog box with configuration options the first of which chooses preset or current map view. If a preset is used only the speed of animation speed can be selected by the user. If the current map view is used then additionally the start date for the animation and the visible window step & size can be configured. Sensible defaults have been provided.
Upon clicking 'Open' the user is presented with a new screen, this screen is not intended to be interactive, most interactions have been removed and the intention is to remove them all possible ones. The reason for the lack of interactions is that the display is animated and user interactions will clash with the animations.
The display screen has a selection of tweets, with the better tweets given priority. These scroll at a rate specified in the preferences. In the 'all' mode (see 'publicDisplayTweetScroll') it is a selection of the best tweets from the entire time window. In the 'window' mode it is a sorted set of tweets from the current displayed (rolling) time window.
To close the display screen and return to the 'map' view, the user should click on the top right X button.
Note the display screen is fully responsive and can be viewed at any resolution (down to mobile phones) and either portrait or landscape aspect.
exceedanceThreshold: 100,
countThreshold: 0,
publicDisplayTweetScroll: "all",
publicDisplayTweetScrollRate: 3000,
publicDisplayMaxTweets: 60,
publicDisplayMaxTweetsRetrieved: 200,
The exceedanceThreshold is used for noise removal, all regions with an exceedance % exceeding this will be ignored, likewise all regions with a count that's less than the countThreshold will also be ignored.
The next four parameters are for public displays.
The publicDisplayTweetScroll is how tweets should be scrolled through all will show a sample of the tweets from the entire time range (e.g. 3 days) for the display. Whereas window will show tweets for the current time window being displayed (e.g. 6 hours). The rate at which these tweets are scrolled in milliseconds is publicDisplayTweetScrollRate.
The maximum number of tweets to display in the all mode is determined by publicDisplayMaxTweets. These tweets are the best tweets from the first publicDisplayMaxTweetsRetrieved from the time window. So with the above settings 200 tweets are retrieved (max) and the best 60 are displayed. Increasing the retrieval size has a performance impact on the initial load. Note that if the number of actual tweets retrieved is less than publicDisplayMaxTweets then all but the blacklisted and 'potentially sensitive' tweets will be displayed.
The selection of the best tweets is done by reducing the tweets from the retrieved (publicDisplayMaxTweetsRetrieved) to the displayed (publicDisplayMaxTweets). The algorithm below is used to reduce the tweets :
private cleanTweetsAndLimit(tweets: Tweet[]): Tweet[] {
// remove blacklisted
tweets = tweets.filter(i => this.filterTweet(i));
// remove duplicates
const map = {};
tweets.forEach(i => {
return map[JSON.stringify(i.tokens)] = i;
});
tweets = Object.values(map);
// if there are spare tweets, clean out the rubbish ones
// this is done at each stage until we get to `publicDisplayMaxTweets` tweets
// so these filters are conditional and not always applied
// the order is important. the earlier the rule the more important it is
// filter out greylisted (contains low quality words)
if (tweets.length > this.pref.combined.publicDisplayMaxTweets) {
tweets = tweets.filter(i => !i.greylisted);
}
// filter out more spammy users
if (tweets.length > this.pref.combined.publicDisplayMaxTweets) {
tweets = tweets.filter(i => i.json.user.followers_count / (i.json.user.friends_count || 1) > 1);
}
// filter out all tweets without a photo
if (tweets.length > this.pref.combined.publicDisplayMaxTweets && tweets.filter(
i => i.mediaCount !== 0).length > this.pref.combined.publicDisplayMaxTweets / 2) {
tweets = tweets.filter(i => i.mediaCount !== 0);
}
// filter out tweets with urls but don't contain images (usually promotional tweets)
if (tweets.length > this.pref.combined.publicDisplayMaxTweets) {
tweets = tweets.filter(i => !(i.mediaCount === 0 && i.json.entities?.urls?.length > 0));
}
// filter out tweets with oo many mentions (3+) usually promotional/activism tweets
if (tweets.length > this.pref.combined.publicDisplayMaxTweets) {
tweets = tweets.filter(i => !(i.json.entities?.user_mentions?.length > 2));
}
// filter out tweets with too many hashtags (3+) usually promotional/activism tweets
if (tweets.length > this.pref.combined.publicDisplayMaxTweets) {
tweets = tweets.filter(i => !(i.json.entities?.hashtags?.length > 2));
}
// filter out tweets from verified users, usually news/broadcast tweets
// they rarely contain direct eyewitness reporting
if (tweets.length > this.pref.combined.publicDisplayMaxTweets) {
tweets = tweets.filter(i => !i.json.user.verified);
}
// sort tweets so that when we slice them we slice off the least relevant
tweets.sort((i, j) => {
return this.allTweetSortOrderForTweet(i) - this.allTweetSortOrderForTweet(j);
});
// okay now slice off any remaining excess tweets
return tweets.slice(0, this.pref.combined.publicDisplayMaxTweets);
}A new test for CSV downloading has been added, an extensive clear up of all tests has been performed. Tests are once again required for builds on the release branches. A bug had stopped failures from failing a build.
At present there are no specific tests for the public display, awaiting feature stability before starting on them.
Create a version suitable for use in a display mode. customer enhancement #249
Put the hazard name in the default csv filename enhancement internal #428
Add "today" and "yesterday" as options in the time window menu? customer enhancement #376
Can the map show the time/date of last update somewhere customer enhancement #375
CSV Download Test internal testing #422
Support Snow Hazard customer enhancement #294
Also included in this release are enhancements to the integration tests, including the testing of CSV download files.
Full Changelog: https://github.com/socialsensingbot/frontend/compare/release/2.4.4...release/2.5