-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathreview-criteria.html
297 lines (261 loc) · 14.5 KB
/
review-criteria.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<title>HCOMP 2016: Review Criteria</title>
<!-- Bootstrap Core CSS -->
<link href="css/bootstrap.min.css" rel="stylesheet">
<!-- favicon link-->
<link rel="shortcut icon" type="image/png" href="img/favicon.png"/>
<!-- Custom CSS -->
<link href="css/landing-page.css" rel="stylesheet">
<!-- Custom Fonts -->
<link href="font-awesome/css/font-awesome.min.css" rel="stylesheet" type="text/css">
<link href="http://fonts.googleapis.com/css?family=Lato:300,400,700,300italic,400italic,700italic" rel="stylesheet" type="text/css">
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<!-- Navigation -->
<nav class="navbar navbar-default navbar-fixed-top topnav" role="navigation">
<div class="container topnav">
<!-- Brand and toggle get grouped for better mobile display -->
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand topnav" href="index.html">HCOMP 2016</a>
</div>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
<li>
<a class="hcomp-active" href="submit.html">Submit</a>
</li>
<li>
<a href="participate.html">Participate</a>
</li>
<li>
<a href="program.html">Program</a>
</li>
<li>
<a href="attend.html">Attend</a>
</li>
<li>
<a href="industry.html">Industry</a>
</li>
<li>
<a href="organizers.html">Organizers</a>
</li>
<li>
<a href="sponsors.html">Sponsors</a>
</li>
<li>
<a href="past%20meetings.html">Past Meetings</a>
</li>
</ul>
</div>
<!-- /.navbar-collapse -->
</div>
<!-- /.container -->
</nav>
<!-- Header -->
<div class="submit-header">
<div class="intro-message"></div>
</div>
<!-- Page Content -->
<div class="container">
<div class="row">
<h2 class="section-head"><span>HCOMP 2016 Review Criteria</span></h2>
</div>
</div>
<!-- Container a-->
<div class="container">
<div class="intro-text">
<div class="row">
<div class="col-md-12">
<div class="intro-paragraph">
<ul class="list-inline">
<li><i class="icon-font fa fa-desktop"></i></li>
<li><h2 class="section-heading">Overview</h2></li>
</ul>
<p class="lead">
Full papers submitted to HCOMP 2016 will be evaluated according to the following review criteria. Our intent in posting review criteria online is to further improve transparency of the conference's peer-review process and to provide additional guidance to authors in preparing their submissions (especially for young researchers, as well as researchers from diverse disciplines).
</p>
<p class="lead">
Feedback or suggestions for further improving these review criteria? Tweet us <a class="theme-link" href="https://twitter.com/hcomp_conf">@hcomp_conf</a> or email us at <a class="theme-link" href="mailto:hcomp16@gmail.com">hcomp16@gmail.com</a>.
</p>
<ul class="list-inline">
<li><i class="icon-font fa fa-check"></i></li>
<li><h2 class="section-heading">Relevance</h2></li>
</ul>
<ul class="lead">
<li>How relevant is the work to HCOMP? Are many conference attendees likely to be interested?
<li><mark>From the <a class="theme-link" href="submit.html">Call for Papers</a></mark></p>
<ul class="lead">
<li>"To ensure relevance, submissions are encouraged to include research questions and contributions of broad interest to crowdsourcing and human computation, as well as discuss relevant open problems and prior work in the field."
<li>"When evaluation is conducted entirely within a specific domain, authors are encouraged to discuss how findings might generalize to other communities and application areas using crowdsourcing and human computation."
</ul>
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-pencil"></i></li>
<li><h2 class="section-heading">Presentation & Writing</h2></li>
</ul>
<ul class="lead">
<li>Is the English writing correct and comprehensible?
<li>Are the ideas, methods, results and discussions well-presented?
<li>Are research questions and major findings clearly articulated?
<li>Is any domain-specific terminology and methodology explained for a diverse HCOMP audience?
<li><mark>HCOMP-specific Note</mark>: Is any language use criticizing the crowd balanced with praise when appropriate? Do the authors either avoid inflammatory language in characterizing crowd contributors (e.g., incompetent, cheaters, spammers, etc.) or provide rigorous evidence justifying use of such terms?
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-check"></i></li>
<li><h2 class="section-heading">Citations & Prior Work</h2></li>
</ul>
<ul class="lead">
<li>Do authors cite prior work most relevant to their own study?
<li>Is the review of prior work correct and sufficiently detailed?
<li>Do authors discuss all prior work needed to interpret and assess this paper?
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-check"></i></li>
<li><h2 class="section-heading">Novelty</h2></li>
</ul>
<ul class="lead">
<li>Does the paper establish a new problem?
<li>Are new methods or theorems proposed?
<li>Is evaluation conducted on novel data?
<li>Are new evaluation procedures described for establishing validity?
<li>Is the problem or approach so novel that it may be difficult for the authors to rigorously evaluate or for the reviewers to assess?
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-bolt"></i></li>
<li><h2 class="section-heading">Significance & Impact</h2></li>
</ul>
<ul class="lead">
<li>How significantly will this work change future research and practice in the field?
<li>Where is it on the spectrum from incremental to transformative?
<li>Does it pose important new problems, challenge accepted knowledge and practice, or otherwise prompt or enable new avenues of research?
<li>How likely is this paper to be cited?
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-code"></i></li>
<li><h2 class="section-heading">Data, Code, & Resources</h2></li>
</ul>
<ul class="lead">
<li>Do the authors commit to sharing any new datasets, sourcecode, and/or other resources for others to use?
<li>If so, are many people likely to use these new resources?
<li>How greatly would these new resources impact future research and practice?
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-bar-chart"></i></li>
<li><h2 class="section-heading">Soundness & Validity</h2></li>
</ul>
<ul class="lead">
<li>Are research methods appropriate, sufficient, and correctly employed?
<li>Is the experimental design sound, analyses thorough, proofs valid, and findings supported by evidence?
<li>Is the work brought to an appropriate state of completion?
<li><mark>HCOMP-specific Note</mark>: Do the authors consider the impact of their own task design in any evaluation of crowd reliability & quality?
</ul>
<ul class="list-inline">
<li><i class="icon-font fa fa-refresh"></i></li>
<li><h2 class="section-heading">Reproducibility</h2></li>
</ul>
<ul class="lead">
<li>Are method descriptions sufficiently detailed and clear?
<li>Are the resources used in the paper (e.g., data, code, computing infrastructure) already publicly available, committed to be shared by the authors, or easily substitutable with similar resources?
<li>Could someone else conduct similar experiments to verify the results?
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="container">
<div class="row">
<h2 class="section-head"><span>Camera-ready Acknowledgments</span></h2>
</div>
</div>
<!-- Container a-->
<div class="container">
<div class="intro-text">
<div class="row">
<div class="col-md-12">
<div class="intro-paragraph">
<ul class="list-inline">
<li><i class="icon-font fa fa-users"></i></li>
<li><h2 class="section-heading">Thank Your Crowd!</h2></li>
</ul>
<p class="lead">When writing acknowledgments in the final, camera-ready versions of accepted papers, we encourage authors to thank their crowd contributors. After all, we couldn't conduct our research or run our crowd-powered systems without the many individual contributors who choose to participate!</p>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Footer -->
<footer>
<div class="container">
<h2 class="section-head"><span><a class="theme-link-org" href="sponsors.html">Conference Sponsors</a></span></h2>
<div class="row">
<div class="intro-text">
<div class="intro-paragraph">
<div class="lead col-md-6">
<p style="text-align: center;"><font size="+3"><b>Platinum</b></font><br/>
<a class="theme-link" href="https://spare5.com"><img src="img/sponsors/spare5.png" alt="Spare 5" width="" height="50" style="border-style: none"/></a>
</p>
</div>
<div class="lead col-md-6">
<p style="text-align: center;"><font size="+3"><b>Gold</b></font><br/>
<a class="theme-link" href="http://www.microsoft.com"><img src="img/sponsors/microsoft.png" alt="Microsoft" width="" height="50" style="border-style: none"/></a>
<br/><br/>
<a class="theme-link" href="http://www.stitchfix.com"><img src="img/sponsors/stitchfix.png" alt="STITCH FIX - YOUR PARTNER IN PERSONAL STYLE" width="" height="50" style="border-style: none"/></a>
</p>
</div>
<div class="lead col-md-6">
<p style="text-align: center;"><font size="+3"><b>Silver</b></font><br/>
<br/>
<a class="theme-link" href="https://alegion.com"><img src="img/sponsors/alegion.png" alt="Alegion Crowdsourcing Automation Platform" width="" height="40" style="border-style: none"/></a>
<br/><br/>
<a class="theme-link" href="http://informationevolution.com"><img src="img/sponsors/ie.png" alt="INFORMATION EVOLUTION - RELIABLE, EXPERIENCED, AFFORDABLE HUMAN BANDWIDTH" width="250" height="" style="border-style: none"/></a>
<br/><br/>
<a class="theme-link" href="http://www.google.com"><img src="img/sponsors/google.png" alt="Google" width="190" height="" style="border-style: none"/></a>
<br/><br/>
<a class="theme-link" href="https://microworkers.com"><img src="img/sponsors/microworkers.png" alt="Microworkers" width="200" height="" style="border-style: none"/></a>
<br/><br/>
<a class="theme-link" href="http://www.ischool.utexas.edu"><img src="img/sponsors/ut-ischool.jpg" alt="The School of Information at the University of Texas at Austin" width="250" height="" style="border-style: none"/></a>
</p>
</div>
<div class="lead col-md-6">
<p style="text-align: center;"><font size="+3"><b>Bronze</b></font><br/>
<br/>
<a class="theme-link" href="https://www.asist.org/publications/jasist/"><img src="img/sponsors/jasist.png" alt="ournal of the Association for Information Science and Technology (JASIST)" width="200" height="" style="border-style: none"/></a>
</p>
</div>
</div>
</div>
</div>
<div class="row intro-text">
<div class="col-lg-12">
<p class="copyright text-muted">Copyright © HCOMP 2016. All Rights Reserved | Designed by <a class="theme-link-org" href="https://www.linkedin.com/in/yangzhangux">Yang Zhang</a></p>
</div>
</div>
</div>
</footer>
<!-- jQuery -->
<script src="js/jquery.js"></script>
<!-- Bootstrap Core JavaScript -->
<script src="js/bootstrap.min.js"></script>
</body>
</html>