This repository has been archived by the owner on Aug 15, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6
/
postmortem.html
274 lines (245 loc) · 24.9 KB
/
postmortem.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-144225776-4"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-144225776-4');
</script>
<!-- Browser/Platform Settings -->
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="skype_toolbar" content="skype_toolbar_parser_compatible">
<meta name="google" content="notranslate">
<meta name="mobile-web-app-capable" content="yes">
<meta name="format-detection" content="telephone=no">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="default">
<meta name="renderer" content="webkit">
<meta name="x5-fullscreen" content="true">
<meta name="x5-page-mode" content="app">
<meta name="full-screen" content="yes">
<meta name="imagemode" content="force">
<meta name="browsermode" content="application">
<meta name="nightmode" content="disable">
<meta name="layoutmode" content="fitscreen">
<!-- Primary Meta Tags -->
<title>DiscoverSUTD 2020 - Pillar Talks Postmortem</title>
<meta name="title" content="DiscoverSUTD 2020 - Pillar Talks Postmortem">
<meta name="description" content="Postmortem for Pillar Talks 2020 Livestreams.">
<meta name="author" content="OpenSUTD">
<meta name="keywords" content="discover, sutd, discoversutd, freshmore, freshman, freshmen, introductory, introduction, new, experience">
<meta name="robots" content="follow">
<!-- Open Graph / Facebook -->
<meta property="og:type" content="website">
<meta property="og:url" content="https://opensutd.org/DiscoverSUTD-2020/postmortem.html">
<meta property="og:title" content="DiscoverSUTD 2020 - Pillar Talks Postmortem">
<meta property="og:description" content="Postmortem for Pillar Talks 2020 Livestreams.">
<meta property="og:image" content="assets/branding/discover.png">
<meta property="og:site_name" content="SUTD">
<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image">
<meta property="twitter:url" content="https://opensutd.org/DiscoverSUTD-2020/postmortem.html">
<meta property="twitter:title" content="DiscoverSUTD 2020 - Pillar Talks Postmortem">
<meta property="twitter:description" content="Postmortem for Pillar Talks 2020 Livestreams.">
<meta property="twitter:image" content="assets/branding/discover.png">
<!-- Favicon Package -->
<link rel="apple-touch-icon" sizes="180x180" href="assets/media/favicons/apple-touch-icon.png">
<link rel="apple-touch-icon-precomposed" sizes="180x180" href="assets/media/favicons/apple-touch-icon-precomposed.png">
<link rel="icon" type="image/png" sizes="32x32" href="assets/media/favicons/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="194x194" href="assets/media/favicons/favicon-194x194.png">
<link rel="icon" type="image/png" sizes="192x192" href="assets/media/favicons/android-chrome-192x192.png">
<link rel="icon" type="image/png" sizes="16x16" href="assets/media/favicons/favicon-16x16.png">
<link rel="manifest" href="assets/media/favicons/site.webmanifest">
<link rel="mask-icon" href="assets/media/favicons/safari-pinned-tab.svg" color="#cc0000">
<link rel="shortcut icon" href="assets/media/favicons/favicon.ico">
<meta name="apple-mobile-web-app-title" content="DiscoverSUTD">
<meta name="application-name" content="DiscoverSUTD">
<meta name="msapplication-TileColor" content="#cc0000">
<meta name="msapplication-TileImage" content="assets/media/favicons/mstile-144x144.png">
<meta name="msapplication-config" content="assets/media/favicons/browserconfig.xml">
<meta name="theme-color" content="#ffffff">
<!-- iOS Splashscreens Package -->
<link href="assets/media/splashscreens/iphone5_splash.png" media="(device-width: 320px) and (device-height: 568px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/iphone6_splash.png" media="(device-width: 375px) and (device-height: 667px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/iphoneplus_splash.png" media="(device-width: 621px) and (device-height: 1104px) and (-webkit-device-pixel-ratio: 3)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/iphonex_splash.png" media="(device-width: 375px) and (device-height: 812px) and (-webkit-device-pixel-ratio: 3)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/iphonexr_splash.png" media="(device-width: 414px) and (device-height: 896px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/iphonexsmax_splash.png" media="(device-width: 414px) and (device-height: 896px) and (-webkit-device-pixel-ratio: 3)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/ipad_splash.png" media="(device-width: 768px) and (device-height: 1024px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/ipadpro1_splash.png" media="(device-width: 834px) and (device-height: 1112px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/ipadpro3_splash.png" media="(device-width: 834px) and (device-height: 1194px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link href="assets/media/splashscreens/ipadpro2_splash.png" media="(device-width: 1024px) and (device-height: 1366px) and (-webkit-device-pixel-ratio: 2)" rel="apple-touch-startup-image">
<link rel="stylesheet" type="text/css" href="assets/css/main.css">
<noscript><link rel="stylesheet" type="text/css" href="assets/css/noscript.css"></noscript>
</head>
<body class="is-preload">
<div id="wrapper">
<header id="header">
<nav>
<a href="#menu">Menu</a>
</nav>
</header>
<nav id="menu">
<ul class="links">
<li><a href="index.html">Home</a></li>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#timeline">Timeline</a></li>
<li><a href="#possible-causes">Possible Causes</a></li>
<li><a href="#mitigating-actions">Mitigating Actions</a></li>
<li><a href="#resolution">Resolution</a></li>
<li><a href="#lessons">Lessons</a></li>
<li><a href="#appendix">Appendix</a></li>
</ul>
</nav>
<div id="main" class="alt padded-top">
<div class="inner">
<header class="major">
<h3 class="section-sep" id="postmortem">Pillar Talks Post-Mortem</h3>
</header>
</div>
<!--Section Start-->
<section>
<div class="inner">
<header class="major">
<h1 id="introduction">Introduction</h1>
</header>
<p>Dear respected staff, faculty members and Class of 2024 undergraduate students,<br><br>
I am James from the DiscoverSUTD 2020 Organizing Committee, and we would like to sincerely apologise for the technical difficulties that were experienced during the Pillar Talks for DiscoverSUTD 2020 on 17 September 2020. Pillar selection is an important decision for Freshmores and hence we think that an adequate post-mortem is necessary and to highlight next steps. This is also to ensure that future talks do not suffer in the same way.
</p>
<p><em><b>NOTE</b>: ESD is unaffected by this incident since they utilized their own Zoom Meeting for their Pillar Talk. This is due to the need to facilitate discussion during their logistics games in a video-call style.</em></p>
<header class="major">
<h1 id="timeline">Timeline</h1>
</header>
<table style="width:100%;" id="timeline">
<tr>
<th>Time (hrs)</th>
<th>Activities</th>
<th>Situation</th>
</tr>
<tr>
<td>1300</td>
<td>Kenneth and James set up the main laptop for Microsoft Stream for ASD, EPD and ISTD. 3 laptops were borrowed from IT Service Desk with Microsoft Teams and Streamlabs OBS. A desktop and a mobile are used to monitor the stream from a Freshmore’s perspective.</td>
<td><em>Not applicable.</em></td>
</tr>
<tr>
<td>1350</td>
<td>ASD and EPD livestreams began. ISTD livestream was not started due to audio issue for one faculty member.</td>
<td>ASD and EPD livestreams were received intermittently.</td>
</tr>
<tr>
<td>1410</td>
<td>Decision was made to switch to recording instead. Pillar Heads and Faculty Members involved in the Pillar Talks were informed. EPD’s <em>sli.do</em> code was received.</td>
<td>Freshmores were informed via the Telegram AMA. However not all Freshmores were in the group. They were informed via email as well.</td>
</tr>
<tr>
<td>1422</td>
<td>EPD’s <em>sli.do</em> code was included in the page which was hosted on GitHub (<a href="https://github.com/OpenSUTD/DiscoverSUTD-2020/commit/34cc3ce566299cfd7cec431e3056b3e4095505f2" target="_blank" rel="noopener noreferrer">34cc3ce</a>).</td>
<td>It took some time for the page to update and thus <em>sli.do</em> was not immediately available.</td>
</tr>
</table>
<header class="major">
<h1 id="possible-causes">Possible Causes</h1>
</header>
<p>While we are still unable to exactly pinpoint the root cause due to the sheer scale of the livestreams (and all the different parts that were involved), we have identified 3 possible points of failure ordered from the most likely to the least:</p>
<ol type="1" start="1">
<li>The 3 laptops borrowed from IT Service Desk. We have been granted administrator access to all of the loaned laptops, but even then, it's evident that some of the laptops are partially faulty. Some networking misconfiguration might have caused local connection issues. As evident from a post-incident network investigation by Registrar and SUTD's IT team, no issues related to the school network were identified (yet?). This is also further supported by the fact that we were able to successfully conduct the Hostel Briefing (albeit with the unavoidable 45-seconds lag, only 1 concurrent livestream and some lagging network connection) using our own personal laptop and SUTD's Wi-Fi (with a sizable viewcount of 450 out of the 475 Freshmore students, even if that 450 includes duplicates and non-Freshmore views).</li>
<li>SUTD_Wifi. Most of the Freshmores were staying at SUTD Hostel during the Pillar Talks (17 September 2020, 1400hrs – 1600hrs). Having 400+ Freshmore students watching 3 concurrent livestreams on the same website page might have squeezed out the bandwidth of SUTD’s Wi-Fi service. Case in point, people have reported slow network access during Hostel Briefing and also during the Pillar Talks. I was also unable to view the Hostel Briefing live on my mobile device using SUTD’s Wi-Fi. When I switched over to use my cellular data instead, I was able to view the Hostel Briefing live (albeit with the inevitable 45-seconds lag, but with no error).</li>
<li>Microsoft Stream’s server(s) (and to an extent, my Microsoft Stream account). Again, due to the scale of the livestreams (400+ students, 3 concurrent livestreams), my account could have potentially been rate-limited heavily, and hence data might not have been received in the smoothest way possible. However, this is quite unlikely since I have requested permission from IT Service Desk for the “Live Event” feature to be enabled for my account. Secondly, based on <a href="https://docs.microsoft.com/en-us/stream/live-event-overview" target="_blank" rel="noopener noreferrer">this documentation page</a>, the limit is much higher than what we had (10000 concurrent viewers, 15 concurrent livestreams and 4 hours maximum length per broadcast). In fact, as outlined <a href="https://techcommunity.microsoft.com/t5/microsoft-teams-blog/what-s-new-in-microsoft-teams-april-2020/ba-p/1347929" target="_blank" rel="noopener noreferrer">here</a>, the limit was actually temporarily raised to accommodate 20000 concurrent viewers, 50 concurrent livestreams and 16 hours maximum length per broadcast, due to the COVID-19 situation.</li>
</ol>
<header class="major">
<h1 id="mitigating-actions">Mitigating Actions</h1>
</header>
<p style="text-decoration: underline;">Deciding on the streaming provider</p>
<ol type="1" start="1">
<li>Livestreams (and uploading the Fifth Row Showcase videos as well) need to be conducted in a closed manner. This would mean that a login wall is required. An unlisted video on YouTube, for example, would not suffice since anyone in the public would still be able to view the embedded version on the DiscoverSUTD website. Microsoft Stream provides an easy integration to this since SUTD’s email accounts are managed on the Microsoft Outlook platform.</li>
<li>Video conferencing allow all users to speak. Microsoft Teams allow meeting organizers to mute all participants. However, participants are able to unmute themselves.</li>
<li>Teams meetings allow for people to join via invite links, including members of the public. Since the event is targeted at Freshmores, we do not want people who are not involved hopping in and out, disrupting the event. Although there are permissions allow people to either bypass the lobby or be required to wait to be admitted, it is logistically taxing to check whether the participant is invited or not.</li>
<li>Video conferencing requires sign-ups. This allows for the organizer to send out invite links. Since we are not limited by space in a virtual event, as compared to a physical event due to COVID-19 safe-distancing measures, an unlimited number of students can attend the talk. However, sign ups create an unnecessary hassle.</li>
<li>Unlike classes where participants are limited and are mandated to join the lesson, we think livestreams provide us with the control for events where the target audience is constrained but participation count is not pre-determined. Freshmores are able to join in if they decide to do so at the last minute and disruption to the event is minimized, especially for an online event. The login requirement for Microsoft Stream also prevents public viewing of the livestream.</li>
</ol>
<p><em>However, considering that Microsoft Stream was fairly untested for larger sizes, it was hard to tell if it would successfully execute the pillar talks. The features of integrating this with the Freshmores' emails would also aid in the management of the event.</em></p>
<p style="text-decoration: underline;">IT testing</p>
<p>This format was tested twice:</p>
<ul>
<li>7 September 2020, 1255hrs – 1700hrs: 4 livestreams ran concurrently from our respective homes and we streamed from our own personal laptops. No issues with streams.</li>
<li>14 September 2020, 2255hrs – 0030hrs (of the next day): We conducted this in SUTD Hostel’s Block 55 Quiet Study Room, utilizing both SUTD_Wifi and the loaned laptops from IT. We encountered some minor hiccups but we managed to livestream from 4 out of the 5 loaned laptops, 3 of which we used for the actual Pillar Talks.</li>
</ul>
<p><em>Therefore, the team evaluated that this format would probably work on the actual day. What was not accounted for was that traffic would be at least x50 times more. Yet, this cannot be adequately simulated.</em></p>
<p style="text-decoration: underline;">Recordings</p>
<p>In order to preserve the Pillar Talks, the livestreams were recorded.</p>
<ul>
<li>ASD: No file was created and laptop did not visually respond to pressing the stop recording button. After leaving it for about 1.5 hours post-event with the recording still running, a MP4 file was created and started filling with data. We left it to fill for about 30 minutes when it crashed and could no longer be switched on. See <a href="#appendix" target="_self" rel="noopener noreferrer"><em>Appendix</em></a> for images. Trying to navigate to boot device manually resulted in no hard disk found. Computer stayed non-functional until the next morning, showing the same screen. Miraculously at 1125hrs before returning the laptop, we tried to start it and it booted to Windows. Recording was not saved.</li>
<li>EPD: Recording was only saved 40 minutes into the talk and started to fill with data at about 1MB every 2 minutes. Left the laptop running overnight and file completed and saved at 0428hrs next day. Total duration to save was about 13 hours. Final file size was 503MB. Audio was fully intact, but only the top-left quarter of the video was intact.</li>
<li>ISTD: No issues, i.e. MP4 file was created at the start of recording and immediately finished processing upon pressing stop recording.</li>
</ul>
<p><em>There was too much reliance on a single source of the recording. However, considering that the stream did not work, another laptop could not have recorded concurrently either.</em></p>
<header class="major">
<h1 id="resolution">Resolution</h1>
</header>
<p>The following actions are done in consultation with the respective Pillars:</p>
<ul>
<li>ASD: We will be uploading a recording of ASD’s previous outreach event during Singapore Virtual University Fair (original source from <a href="https://youtu.be/LSTBfbOL9Dk" target="_blank" rel="noopener noreferrer">here</a>). Even though the target audience is different and while this is no real substitute for the Pillar Talk, the essentials of a more in-depth introduction of Architecture as a study and career are still there.</li>
<li>EPD: We will be uploading a reconstructed “simulated recording” of the actual talk. Since we have the audio intact, we will recreate the recording using the slides that EPD has prepared for the Pillar Talk. We would also provide a list of the presenters on the website under the EPD’s recording section, with hyperlinks directing to their respective profiles on SUTD’s website.</li>
</ul>
<header class="major">
<h1 id="lessons">Lessons</h1>
</header>
<ol type="1" start="1">
<li>Utilise Zoom as it has proven to be a much more reliable format. However, signups will have to be gated. The Zoom Webinar feature will suffice and can hold up to 100 participants but a subscription is required. Holding up to 500 participants would require a Zoom Enterprise account. Alternatively, we can also use Microsoft Teams Live Event or implement an <a href="https://docs.microsoft.com/en-us/stream/ecdn" target="_blank" rel="noopener noreferrer">eCDN</a> for the livestreams.</li>
<li>Staggering the timings of the Pillar Talks.</li>
<li>Better contingency planning in the event of stream failures could have been done.</li>
<li>All material and preparation should be done on the day before including the submission of <em>sli.do</em> link and presentation slides for Pillar Talks so that Freshmores can still get the information to be disseminated.</li>
</ol>
<p>Once again, we would like to deeply apologize for the technical difficulties that were faced and caused this disruption in your DiscoverSUTD experience. We are committed to providing you with the best experience possible. Thus, we seek your kind understanding on the matter and hope to grow from this experience together.<br><br>
Thank you.<br><br>
James Raphael Tiovalen<br>
<em>On behalf of the DiscoverSUTD 2020 Organizing Committee</em>
</p>
<header class="major">
<h1 id="appendix">Appendix</h1>
</header>
<br>
<p>These are some photos of the laptop used for the ASD livestream. A Blue Screen of Death (BSoD) was encountered before it manifested these corresponding displays (we did not manage to record the BSoD's error code). As seen in the pictures, it encountered a BootDevice error and a BOOTMGR Image error.</p><br>
<div id="appendix-container"><img id="asd-laptop" src="assets/media/images/bootdevice_error.jpg"></div>
<br>
<div id="appendix-container"><img id="asd-laptop" src="assets/media/images/bootmgr_error.jpg"></div>
</div>
</section>
<!--Section End-->
<hr>
<!--Section Start-->
<section id="footer">
<div class="inner">
<header class="major">
<h2>About DiscoverSUTD</h2>
</header>
<p>DiscoverSUTD is a 3-weeks long introduction to the SUTD community for the incoming Freshmores. It is organised by a team formed by the two SUTD Freshmore 2019 Reps, with assistance and guidance from the OpenSUTD team. It is a joint effort by more than a hundred SUTDents, faculty, researchers, staff and alumni, and collaboratively planned and executed on <b>OpenSUTD</b>, an open organisation owned by the SUTD community.</p>
<ul class="actions">
<li><a href="https://github.com/OpenSUTD/DiscoverSUTD-2020" target="_blank" rel="noopener noreferrer" class="button next"><i class="icon brands fa-github" aria-hidden="true"></i>The DiscoverSUTD Team</a></li>
</ul>
<br>
</div>
</section>
<!--Section End-->
</div>
</div>
<script src="assets/vendor/jquery/jquery-3.5.1.min.js"></script>
<script src="assets/vendor/jquery.scrolly.min.js"></script>
<script src="assets/vendor/jquery.scrollex.min.js"></script>
<script src="assets/vendor/browser.min.js"></script>
<script src="assets/vendor/breakpoints.min.js"></script>
<script src="assets/js/util.js"></script>
<script src="assets/js/main.js"></script>
</body>
<!--
Psst. Hey you. Yes, you.
Curious chap, aren't you? You might be a good fit to join us!
Join the OpenSUTD Telegram Group at https://t.me/joinchat/C8HJABJqfwYEPoT4EYre-A
We're growing our team so that we can continue to do things for the SUTD community!
-->
</html>