-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs(weave): Update Models page with example of pairwise eval #3739
Conversation
Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=d0f04fe4621973759c4d24fa4d14950462af1528 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🧹 Nitpick comments (3)
docs/docs/guides/core-types/models.md (3)
83-83
: Spelling Correction Needed.
There is a typographical error in the sentence: “createing” should be corrected to “creating.”
120-128
: Include Import for Dataset.
The code sample usesDataset
when constructing the evaluation dataset but does not include an import statement for it. To ensure the sample is fully self-contained and executable, consider adding an import (for example:from weave import Dataset
).
134-134
: Fenced Code Block Language Specification.
A markdown linter hint indicates that fenced code blocks should specify a language for better syntax highlighting and compliance with style guidelines. Please verify that all fenced code blocks in this document (or in the closing fence at this location) have an appropriate language specifier.🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
134-134: Fenced code blocks should have a language specified
null(MD040, fenced-code-language)
134-134: Code block style
Expected: indented; Actual: fenced(MD046, code-block-style)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
docs/docs/guides/core-types/models.md
(2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.{md,mdx}`: Focus on technical accuracy. Check for brok...
**/*.{md,mdx}
: Focus on technical accuracy.
Check for broken links.
Verify code examples are up-to-date.
Look for clarity and completeness.
Don't focus on grammar/spelling unless significant.
docs/docs/guides/core-types/models.md
🪛 markdownlint-cli2 (0.17.2)
docs/docs/guides/core-types/models.md
134-134: Fenced code blocks should have a language specified
null
(MD040, fenced-code-language)
134-134: Code block style
Expected: indented; Actual: fenced
(MD046, code-block-style)
⏰ Context from checks skipped due to timeout of 90000ms (41)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, scorers)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
- GitHub Check: Trace nox tests (3, 13, trace)
- GitHub Check: Trace nox tests (3, 12, trace)
- GitHub Check: Trace nox tests (3, 11, trace)
- GitHub Check: Trace nox tests (3, 10, trace)
🔇 Additional comments (2)
docs/docs/guides/core-types/models.md (2)
6-7
: Informative Model Description.
The added paragraph clearly explains what aModel
is in Weave and highlights the benefits of using this API. No changes are needed here.
79-80
: Clear Introduction to Pairwise Evaluation.
The new section on pairwise evaluation is well introduced. It concisely explains the rationale behind using relative metrics over absolute ones for subjective tasks such as text generation and summarization.
Description
Adds new subsection to https://weave-docs.wandb.ai/guides/core-types/models describing how to do pairwise evaluation of two models. Based on #3688 and https://github.com/wandb/weave/pull/3688/files
Testing
yarn start
on localSummary by CodeRabbit