| Metadata | |
|---|---|
| cEP | 0019 |
| Version | 1.0 |
| Title | Meta-review System |
| Authors | Boxuan Li mailto:liboxuan@connect.hku.hk |
| Status | Proposed |
| Type | Process |
This cEP describes the details of the process of implementing the meta-review system as a part of the GSoC'18 project.
People respond to review comments on pull requests by giving feedback. Emoji make it much easier to give and receive feedback without a long comment thread. Those emoji are called reactions on GitHub or award emoji on GitLab. They are used by people to publicly express their feelings about review comments, which provides feedback to comment authors and all other members.
GitLab is beyond the scope of this project, as GitHub and GitLab have quite different APIs and most core projects of coala are based on GitHub. This proposal will only discuss reactions on GitHub, but not award emoji on GitLab.
There are 6 types of reactions on GitHub:
| Name | Markdown | Unicode | Emoji |
|---|---|---|---|
| THUMBS_UP | :+1: |
U+1F44D | 👍 |
| THUMBS_DOWN | :-1: |
U+1F44E | 👎 |
| LAUGH | :smile: |
U+1F604 | 😄 |
| HOORAY | :tada: |
U+1F389 | 🎉 |
| CONFUSED | :confused |
U+1F615 | 😕 |
| HEART | :heart: |
U+FE0F | ❤️ |
There are three types of reviews on a pull request:
-
Issue Comment see sample
-
Review Summary see sample
-
Review Discussion see sample
Those reactions have been added to GitHub since 2016 March. However, they are often used in an adhoc way. They are spread over all PRs and are not collected and analyzed. Moreover, GitHub doesn't send notifications to reviewers who receive reactions (see related discussion). By tracking reactions, a nice feedback loop would be created for both senior and junior reviewers.
The meta-review project is about making use of reactions to build a meta-review system, similar to a meta-moderation system. These responses are to be collected, processed and displayed on gh-board, which is a nice kanban board. Compared to Huboard which requires server-side support, gh-board is serverless.
The scope of this project would exclude non-public meta-reviews. That is, there would not be any anonymous meta-reviews as that introduces too many additional complexities. The public only version would be used for a long time before adding support for non-public meta-reviews.
Note there is a technical limitation in that reactions can not be placed onto the "review summary", which means general review comments should better not be put into the "review summary" text box, as it will be not able to be meta-reviewed.
One goal of this project is to encourage people to do meta-reviews, systematically. Meta-reviews in need will be displayed on gh-board. http://coala.io/meta-review would be redirected to gh-board page which shows meta-reviews in need. The implementation details are discussed in the next section.
Also, to encourage people to do more meta-reviews, statistics of meta-reviews are to be collected and analyzed. People who do meta-reviews will get scores according to some metrics. A ranking list is to be displayed on the community site.
Another objective is that the newcomer process is able to be changed after this project. Newcomers will be required to finish at least one meta-review and receive at least one positive meta-review, once this project is ready for production.
To conclude, this project builds a meta-review system on the community site and gh-board to display score ranking statistics and meta-reviews in need, respectively, and encourage people to do meta-review.
As mentioned in the background section, GitHub provides 6 types of reactions. They have different meanings in the context of meta-review system.
THUMBS_UP and THUMBS_DOWN are most commonly used. They express strong feelings: THUMBS_UP means the review comment is well-written and helpful, while THUMBS_DOWN means the review comment is misleading, or even worse, violates coala Community Code of Conduct. They will contribute to the score which is explained in the Ranking list section.
HEART, HOORAY and LAUGH are reactions that express positive feelings, but not as clear as THUMBS_UP, and consequently will be excluded from the scoring until an analysis of how they should be used is done.
CONFUSED expresses negative feeling. CONFUSED are used when people feel the review comment is not descriptive enough. For example a common advice given to a newcomer PR is to follow coala guidelines for commit and amend the commit message. There are various ways in which reviewers can express this, some can be descriptive enough, some can be very descriptive with links and stuff, also some can be non descriptive comprising of our internal workflow lingo and hence non newcomer friendly. If the review comment is confusing, but maybe correct, then people should definitely give a CONFUSED reaction. It is not clear whether this should be neutral or negative. As a consequence, it will be excluded from the scoring until an analysis of how they should be used is done.
Meta-review info, namely reactions, can be fetched via GitHub API V4 GraphQL. A sample query is as follows:
{
repository(owner: "coala", name: "gh-board") {
pullRequest(number: 18) {
author {
login
url
}
reviews(first: 5) {
nodes {
bodyText
comments(first: 1) {
nodes {
bodyText
author {
login
url
}
reactions(first: 100) {
totalCount
nodes {
createdAt
content
user {
login
}
}
}
}
}
}
}
comments(first: 1) {
nodes {
bodyText
author {
login
url
}
reactions(first: 100) {
totalCount
nodes {
createdAt
content
user {
login
}
}
}
}
}
}
}
}The response is in JSON format as follows:
{
"data": {
"repository": {
"pullRequest": {
"author": {
"login": "li-boxuan",
"url": "https://github.com/li-boxuan"
},
"reviews": {
"nodes": [
{
"bodyText": "This is looking quite good, but closed issues/prs is empty:\nhttps://deploy-preview-18--coala-gh-board.netlify.com/#/r/coala:coala?s=closed\nmeta-review will need to be able to see closed PRs",
"comments": {
"nodes": []
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "Leftover?",
"author": {
"login": "blazeu",
"url": "https://github.com/blazeu"
},
"reactions": {
"totalCount": 0,
"nodes": []
}
}
]
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "Yup, thanks.",
"author": {
"login": "li-boxuan",
"url": "https://github.com/li-boxuan"
},
"reactions": {
"totalCount": 0,
"nodes": []
}
}
]
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "You made the your own getTime function below.",
"author": {
"login": "blazeu",
"url": "https://github.com/blazeu"
},
"reactions": {
"totalCount": 1,
"nodes": [
{
"createdAt": "2018-05-07T03:09:08Z",
"content": "LAUGH",
"user": {
"login": "li-boxuan"
}
}
]
}
}
]
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "Can be turned into arrow function and put at the very top.",
"author": {
"login": "blazeu",
"url": "https://github.com/blazeu"
},
"reactions": {
"totalCount": 1,
"nodes": [
{
"createdAt": "2018-05-05T02:19:08Z",
"content": "THUMBS_UP",
"user": {
"login": "li-boxuan"
}
}
]
}
}
]
}
}
]
},
"comments": {
"nodes": [
{
"bodyText": "To be honest, I don't understand why we need fetch-issues.js. From the relevant code, I can see fetch-issues.js does some pre-fetching and initial-data.js dumps it into the database. But what's the aim of doing that? It doesn't seem to save API hits. These pre-fetched issues in database are replaced later anyway.",
"author": {
"login": "li-boxuan",
"url": "https://github.com/li-boxuan"
},
"reactions": {
"totalCount": 0,
"nodes": []
}
}
]
}
}
}
}
}When a new issue/pull request is detected, gh-board will fetch that issue and loads it asynchronously. Meta-reviews also need to be fetched during this procedure.
This section explains how meta-review information can be utilized to generate a meaningful analysis.
A ranking list would be displayed on coala community website, according to points people get. Some metrics are to be collected and used to generate the ranking formula:
P1 = total number of THUMBS_UP 👍 a person gets for all reviews he did.
P2 = total number of THUMBS_UP 👍 a person gives to other people for their reviews.
N1 = total number of THUMBS_DOWN 👎 a person gets for all reviews he did.
N2 = total number of THUMBS_DOWN 👎 a person gives to other people for their reviews.
Note P2 and N2 are used to encourage people to get involved in this reactions feedback system.
An example formula for final score S is:
S = k * (P1 - N1) + c1 * P2 + c2 * N2
c1 = 0.2, c2 = 0.4, are coefficients. c2 is larger than c1 because people are usually reluctant to give negative reactions.
k is the impact factor. The higher score a person has, the more impacts he has, thus his meta-reviews are more valuable. k is based on the scores in the previous iteration and is normalized. For example, in a previous iteration, Alice got 2 marks, Bob got 0.8 marks and Charlie got 10 marks. The calculation procedure would be as follows:
>>> c = [2, 0.8, 10]
>>> max_score = max(c)
>>> min_score = min(c)
>>> result = [(i - min_score) / (max_score - min_score) for i in c]
>>> print(result)
[0.13043478260869565, 0.0, 1.0]
>>> result_adjust = [i + 0.1 for i in result] # add bias
>>> print(result_adjust)
[0.23043478260869565, 0.1, 1.1]
>>> result_rounded = [round(i, 3) for i in result_adjust]
>>> print(result_rounded)
[0.23, 0.1, 1.1]Therefore, the k factor for Alice, Bob and Charlie in the next iteration would be: 0.23, 0.1 and 1.1, respectively.
If a person has zeros for all factors, then he is regarded as an inactive reviewer and would not be displayed. If a person has negative score, he will still be displayed, but the meta-reviews he gives to other people will not count.
The ranking formula will reside in the community repo and can be modified over time in order to achieve different results.
When does a review need meta-review? There are two perspectives:
-
Meta-review before merge/close. Those review comments need meta-reviews by all people. They are sorted according to update time on gh-board. In particular, one will always see the review comment that is under his PR first.
-
Meta-review after merge/close. Review comments after merge/close need meta-reviews by maintainers. Those meta-reviews will reflect the accuracy of the review comments. Meta-reviews done by other developers after merge/close will not be counted.
Note that review comments by gitmate-bot are not to be meta-reviewed and would not be displayed on gh-board.
There is also concern for edited comments and deleted comments. Users are able to edit/delete their review comments on GitHub. If someone receives negative reactions, they might edit/delete their review comment. To ensure that meta-review data is not discarded even if it is edited/deleted on GitHub, raw information of comment contents and reactions are to be stored in community repo. If the author edits/deletes their comment after it has been meta-reviewed, they are destroying feedback from the community. This improper behaviour will lead to a warning and a negative score given to them.
Review comments that need meta-review will be displayed on a separate column on gh-board. There are two approaches:
-
A basic version is to store the identifier of the PR comments which do not have meta-reviews and gh-board links to the PR review comment in GitHub PR web app. Doing it on GitHub has the benefit of letting users have their usual tools, e.g. refined-github.
-
In the ultimate version, gh-board would show the review comments. In the future, people could do meta-reviews directly on gh-board. This also provides an approach to doing meta-reviews anonymously.
Currently, fetch-issues.js pre-fetches issues and pull requests to speed up gh-board performance and save users' API hits. Meta-reviews are also to be fetched during this procedure.
The GSoC project provides a blueprint for future enhancement. Additional features could be implemented to make the meta-review system more sophisticated:
-
A basic ranking list will be built first. In the best case, we should offer an overall ranking list as well as a recent ranking list.
-
Offer an option for users to do meta-reviews directly on gh-board. In the future users can do meta-reviews on gh-board anonymously (currently, there is no way for people to do anonymous meta-reviews on GitHub page). Note that people may need context to do meta-reviews. The diff hunk to which the comment applies needs to be displayed on gh-board so that people have a basic context to do meta-reviews directly on gh-board.
-
Display visitor’s own meta-reviews if the visitor logs in. People may want to see meta-reviews they have done before.
-
Provide notifications of meta-reviews to a user if they receive any meta-review. Suppose I ask for someone's agreement and they use 👍 to show their agreement, but I wouldn't receive any notification. I have to check that page regularly. gh-board is a nice place to provide notifications of meta-reviews.
-
There is a potential for reverted commits to be meta-reviewed. A reverted commit should trigger a new meta-review of the original PR, as time may have proved that the merge was incorrect, or maybe the merge was a self-aware time-sensitive workaround, and is no longer needed.
-
An important technical detail of meta-review in coala on GitHub is that the context disappears on an amend push, so meta-review by maintainers after the merge may be without the necessary context to be properly evaluated. However, if the commit SHA is mentioned, GitHub will keep the commit even if it is no longer in the PR. A gitmate plugin is needed in the future to mention any SHAs which have been reviewed, so that meta-reviews by maintainers will have more context and be more effective.
-
The UI of gh-board could have factual aggregated statistical information like how many of each emoji a person has given/received, again with a legend explaining their intended meanings, but without trying to aggregate across the different reaction types. Note that this information should not be displayed as a column, but as a manager-ish page like the Burnup Chart.