Analysis: Do we need a Webanalytics tool for Altinn Apps (and if so - what tool)?

[GGunnar]

The need for webanalytics in Studio is moved to a separate issue Altinn/altinn-studio#4040

Description

To be able to give app owners a better insight on how our services and applications are used, there is a need to look at a webanalytics tool we could use in the running Apps.

Example tools that can be used:

• We should see if Matomo which is used in many other web applications and has integrations with React can be used. So similar to task Monitoring of Altinn Studio for sysadmins (generation 1) altinn-studio#321 we would like to look at Matomo.
• We can also look into hotjar which is also used by other organisations for getting feedback on end user behaviour. This is used on applications / services similar to the applications altinn will have.

https://matomo.org/
https://www.hotjar.com/tour

It is expected that we would create more issues after completing this issue.

User needs we should look into:

• As an App Developer I want deep user behaviour insights for editions used in portal
• As an App Developer I want deep insights for edition API usage
• As an application designer i want insight into user behaviour to understand how / what works and does not work, so i can easily identify usability issues and issues they encounter.

Technical considerations

• We need to understand what and where the data / information will be stored.
• How we can keep the end-user privacy while using analytics tools.

Acceptance criterea

• What is allowed/not allowed
• Validations
• Error messages and warnings
• ...

Tasks

• Analyse pros and cons for using matomo
• Analyse pros and cons for using hotjar
• Look into if there are other tools that can be better suited for our needs.

[pckofstad]

In Altinn 3, Application Insight is registered and used for all backend components. It would be nice to have this registered in the frontend app as well. This would give a more complete picture of what the flow of the app is and where the user experience bad response-times and warnings/crashes.

[olemartinorg]

Closing as duplicate
This thing went a bit belly-up, as the original issue got stuck during transferal to another repo, and 3 duplicates were created here in app-frontend-react. When GitHub helped us solve the issue, the original issue was transferred to a private repo where it is inaccessible for outside parties. I'm re-opening this issue as the new master, and manually pasting in missing comments.

The other duplicate issues:

• Analysis: Do we need a Webanalytics tool for Altinn Apps (and if so - what tool)? #876
• Analysis: Do we need a Webanalytics tool for Altinn Apps (and if so - what tool)? #880

[olemartinorg]

@ToveiToll on Jun 9, 2021 (comment that disappeared during transfer)

Lars Vegard - does this mean the tool (or tools) will be integrated into Altinn studio by 2021-Q3? Tolletaten would very much like the option to use for instance Hotjar.

[olemartinorg]

@RonnyB71 on Feb 11, 2022 (comment that disappeared during transfer)

I would argue that this should be something that the app owners should be able to decide themselves ie. what tool(s) to use. We should be able to inject any web analytics tool and perhaps provide a default that you can choose to disable or replace. We should also not limit the implementation to one tool, in many cases it gives meaning to support multiple as they might cover different things.

[olemartinorg]

@indiamaydesign on Dec 19, 2022 (comment that disappeared during transfer)

Insight work on this was undertaken by myself from January to March 2022, consulting stakeholders across Architecture, Communications, Legal and Design. After competitive analysis and consultations, it was decided to investigate the use of Matomo as a Google Analytics replacement for Altinn Studio Docs, and potentially Digdir as a whole.

A tooling proposal document was drafted by myself awaiting legal advice, with the proposal covering the following aspects for intended application with Altinn Studio Docs: (1) rationale for use, (2) proposed use and data collection, (3) data retention, (4) legal basis for use and (5) out-of-scope concerns for future consideration. At the same time, I was drawing up plans for a trial of Matomo on docs.altinn.studio for us to gain early feedback from stakeholders and further refine our needs and requirements. However, on 18/03 replacing Google Analytics officially became a portfolio item and all work on analytics has since been led by Jørgen Ferkingstad for Digdir.

In my current role assisting Team Portal (led by Jørgen), I have continued to work on a Google Analytics replacement that will aim to address the needs of Altinn Studio Docs and other Digdir products and services in future, and we are working with a longer term plan to help Altinn deliver analytics to its service owners. In June I prepared a presentation for Jørgen comparing six platforms that are Google Analytics alternatives including Amplitude, Hotjar, Matomo, Piwik Pro, Siteimprove and Webtrends. This report took into account factors including user satisfaction, current companies using it (with focus on Nordic companies and institutions where possible), cost, GDPR compliance, PII statements and more.

Further, a cross-disciplinary workshop for Analytics was developed and held by myself and Dorte on 26 October with representation from our customer service teams, developers, architects, designers and data analysts, roles spanning across Digdir. This workshop covered users needs and perspectives, priorities, desired metrics, ecosystems of analytics, desired use of analytics and sharing and collaboration strategies for Analytics at Digdir moving forward. The results from this workshop and plans moving forward will be shared within the company early next year from Team Portal.

[olemartinorg]

@pckofstad on Feb 1, 2023 (comment that disappeared during transfer)

The Udir PBU project expect to have heavy loads in planned periods. Results from our load testing shows us that our plattform (both the parts built in Altinn Studio and outside Altinn Studio) is not always behaving like expected when pushed hard. We would like all the insight we can get to ensure that we follow up on bottlenecks and parts of the code/flow that can be made more robust.

In Altinn 3, Application Insight is registered and used for all backend components. It would be nice to have this registered in the frontend app as well. This would give a more complete picture of what the flow of the app is and where the user experience bad response-times and warnings/crashes.

I don't have enough experience to say for sure that it will provide the expected value for all users en scenarios. My concern is that the endpoints and page-names might be so generic that the tracking will not differentiate enough go give value.

Making i optional to turn on and off would be a good feature regardless of what service is selected.

[benteSSB]

@indiamaydesign: er det mulig å få tilgang til presentasjonen du laga (i fjor) der du samanliknar seks ulike alternativ til Google Analytics? Og/eller ev. anna tilsvarande dokumentasjon. Me har ein tilsvarande prosess på gang i SSB og er slik sett interesserte i å utveksla erfaringar på dette området.