This repo contains the frontend code for cBioPortal which uses React, MobX and TypeScript. Read more about the architecture of cBioPortal here.
main branch | upcoming release branch | later release candidate branch | |
---|---|---|---|
Branch name | master |
-- | rc |
Description | All bug fixes and features not requiring database migrations go here. This code is either already in production or will be released this week | Next release that requires database migrations. Manual product review often takes place for this branch before release | Later releases with features that require database migrations. This is useful to allow merging in new features without affecting the upcoming release. Could be seen as a development branch, but note that only high quality pull requests are merged. That is the feature should be pretty much ready for release after merge. |
Test Status | CircleCI master workflow | -- | CircleCI rc workflow |
Live instance frontend | https://frontend.cbioportal.org / https://master--cbioportalfrontend.netlify.app/ | -- | https://rc--cbioportalfrontend.netlify.app |
Live instance backend | https://www.cbioportal.org / https://master.cbioportal.org | -- | https://rc.cbioportal.org |
Note: you can check the frontend version of the live instance by checking window.FRONTEND_COMMIT
in the console.
Make sure you have installed the node version and yarn version specified in package.json.
Tip: We recommend that you use nvm: Node Version Manager and yvm: Yarn Version Manager to switch between versions more easily.
Windows Tip: If you are developing on Windows, we recommend that you use Ubuntu / Windows Subsystem for Linux.
Remove old compiled node_modules
if it exists
rm -rf node_modules
To install all app and dev dependencies
yarn install --frozen-lockfile
To build DLLs in common-dist folder (must be done prior to start of dev server)
yarn run buildDLL:dev
To start the dev server with hot reload enabled
# set the environment variables you want based on what branch you're branching
# from
export BRANCH_ENV=master # or rc if branching from rc
# export any custom external API URLs by editing env/custom.sh
yarn run start
Example pages:
To run unit/integration tests
yarn run test
Windows Tip: There is a known solved hiccup running the tests on Ubuntu via Windows Subsystem for Linux (WSL): #7096
To run unit/integration tests in watch mode
yarn run test:watch
To run unit/integration tests in watch mode (where specName is a fragment of the name of the spec file (before .spec.
))
yarn run test:watch -- --grep=#specName#
When you make a git commit, PrettierJS will automatically run in write mode on all the files you changed, and make formatting adjustments to those entire files as necessary, before passing them through to the commit (i.e. this is a "pre-commit git hook"). No action from you is necessary for this. You may observe that your changes don't look exactly the same as you wrote them due to formatting adjustments.
When you make a pull request, CircleCI will run PrettierJS in check mode on all of the files that have changed between
your pull request and the base branch of your pull request. If all of the files are formatted correctly, then the
CircleCI prettier
job will pass and you'll see a green check on Github. But if, for whatever reason, this check fails,
you must run the following command in your cbioportal home directory:
yarn run prettierFixLocal
This will make PrettierJS run through the same files that CircleCI checks (i.e. all files changed since the base branch)
but in write mode and thus adjust those files to have correct formatting. When you make this update, the CircleCI
prettier
job should pass. To check if it will pass, you can also run the same command that CircleCI will run:
yarn run prettierCheckCircleCI
If the version of the desired API URL is the same as the one used to generate
the typescipt client, one can change the API_ROOT
variable for development in
my-index.ejs. If the version is different, make sure the API
endpoint works with the checked in client by changing the API URL in
package.json and running:
# set the environment variables you want based on what branch you're branching
# from
export BRANCH_ENV=master # or rc if branching from rc
# export any custom external API URLs by editing env/custom.sh
yarn run updateAPI
yarn run test
Go to https://cbioportal.org (master
branch) or https://rc.cbioportal.org/ (rc
branch)
In your browser console set:
localStorage.setItem("localdev",true)
This will use whatever you are running on localhost:3000
to serve the JS (i.e. you need to have the frontend repo running on port 3000). To unset do:
localStorage.setItem("localdev",false)
or clear entire local storage
localStorage.clear()
You can also use a netlify deployed cbioportal-frontend pull request for serving the JS:
- Create the following bookmarklet:
javascript:(function()%7Bvar pr %3D prompt("Please enter PR%23")%3Bif (pr %26%26 Number(pr)) %7B localStorage.netlify %3D %60deploy-preview-%24%7Bpr%7D--cbioportalfrontend%60%3Bwindow.location.reload()%3B %7D%7D)()
- Navigate to the cBioPortal installation that you want to test.
- Click the bookmarklet and enter your pull request number.
End-to-end tests can be run against public cbioportal instances or against a local dockerized backend. These two e2e-tests types are referred to as remote
and local
types of e2e-tests.
First, install webdriver-manager, which manages standalone Selenium installation:
yarn global add webdriver-manager
Run updater to get necessary binaries
webdriver-manager update
// ALTERNATIVE
// Specify chrome version if your local chrome version is not up to date.
// This ensures that chromedriver is compatible with your local chrome
// If it's out of sync, tests will not run
webdriver-manager update --versions.chrome=[e.g. 86.0.4240.198]
Start the webdriver-manager
webdriver-manager start
Tip: To verify that your webdriver-manager is running, go to: http://127.0.0.1:4444/wd/hub/static/resource/hub.html.
Windows Tip: If you are developing on Ubuntu under Windows Subsystem for Linux (WSL), we recommend that you install and run webdriver-manager via Powershell/Windows. If you install and run webdriver-manager via Ubuntu, webdriver-manager has a hard time finding the Chrome browser, and this will cause all the end-to-end tests to fail.
In a second terminal, run the frontend
# set the environment variables for your branch
export BRANCH_ENV=master # or rc if branching from rc
# export any custom external API URLs by editing env/custom.sh
yarn run startSSL
Tip: Before going to the next step, verify that your front end code is running by going to: https://localhost:3000/.
In a third terminal, run the e2e tests
export SPEC_FILE_PATTERN=./remote/specs/**/*.spec.js
export SCREENSHOT_DIRECTORY=./remote/screenshots
export BRANCH_ENV=master # or rc if branching from rc
# export any custom external API URLs in env/custom.sh
cd end-to-end-test
yarn install
yarn run test-webdriver-manager
# use `test-webdriver-manager-debug` for debugging of tests
If all goes well, you will soon seen the end-to-end tests running via your Chrome browser.
Tip: If you want to run just one end-to-end test, you can modify
SPEC_FILE_PATTERN
accordingly. For example, if you want to only run thegenomicEvolution.spec.js
test, you can use:export SPEC_FILE_PATTERN=./remote/specs/core/genomicEvolution.spec.js
.
A custom frontend can be tested against any backend in the web browser using a local node server (command yarn run start
) and the localdev
flag passed to th e browser (see section 'Check in cBioPortal context'). For remote backends that communicate over a HTTP over SSL (https) connection (e.g., cbioportal.org or rc.cbioportal.org), the frontend has to be served over SSL as well. In this case run yarn run startSSL
in stead of yarn run start
.
To enable e2e-tests on for features that depend on data that are not included in studies served by the public cBioPortal instance, cbioportal-frontend provides the e2e local database
(refered to as e2e-localdb or local e2e in this text) facility that allows developers to load custom studies in any backend version used for e2e-tests. CircleCI runs the e2e-localdb
tests as a separate job.
Files for the local database e2e-tests are located in the ./end-to-end-test/local
directory of cbioportal-frontend. The directory structure of ./end-to-end-test/local
is comparable to that of the ./end-to-end-test/remote
directory used for e2e-tests against remote public cBioPortal instances.
- Start cBioPortal (including session service) using the cBioPortal docker compose solution. cBioPortal must be accessible on http://localhost:8080.
cd
git clone https://github.com/cBioPortal/cbioportal-docker-compose.git
cd cbioportal-docker-compose
./init.sh
docker-compose up -d
- Install gene panels and gene sets for study_es_0:
cd ~/cbioportal-docker-compose
docker-compose run --rm cbioportal sh -c '
cd /cbioportal/core/src/main/scripts/ \
&& ./importGenePanel.pl --data /cbioportal/core/src/test/scripts/test_data/study_es_0/data_gene_panel_testpanel1.txt \
&& ./importGenePanel.pl --data /cbioportal/core/src/test/scripts/test_data/study_es_0/data_gene_panel_testpanel2.txt \
&& ./importGenesetData.pl --data /cbioportal/core/src/test/resources/genesets/study_es_0_genesets.gmt --new-version msigdb_6.1 \
&& ./importGenesetHierarchy.pl --data /cbioportal/core/src/test/resources/genesets/study_es_0_tree.yaml'
Restart cBioPortal
docker-compose restart cbioportal
- Load study_es_0 of the deployed (!) backend version and all studies in end-to-end-test/local/studies. At the moment of this writing:
docker-compose run --rm -v <e2e_study_dir>:/studies cbioportal sh -c '
cd /cbioportal/core/src/main/scripts/importer \
&& ./cbioportalImporter.py -s /cbioportal/core/src/test/scripts/test_data/study_es_0 \
&& ./cbioportalImporter.py -s /studies/genepanel_test_study'
# add other studies if present
:info: The <e2e_study_dir>
entry refer to the absolute path to end-to-end-test/local/studies on the host system.
-
Add
export CBIOPORTAL_URL="http://localhost:8080"
to/env/custom.sh
. -
In a terminal, install webdriver-manager:
yarn global add webdriver-manager
export PATH=$PATH:$(yarn global dir)/node_modules/webdriver-manager/bin
and start webdriver-manager:
webdriver-manager update
webdriver-manager start
- If not already running at localhost:3000, open a second terminal and start the frontend dev server:
export BRANCH_ENV=custom
yarn install --frozen-lockfile
yarn buildDLL:dev
yarn start
- In a third terminal, run the tests:
export BRANCH_ENV=custom
eval "$(./scripts/env_vars.sh)"
export SPEC_FILE_PATTERN=./local/specs/**/*.spec.js
export SCREENSHOT_DIRECTORY=./local/screenshots
cd end-to-end-test
yarn install
yarn run test-webdriver-manager-debug
E2e-tests on CircleCI and CircleCI+PR context are triggered via hooks configured on GitHub. Configuration of hooks falls beyond the scope of this manual.
E2e-testing against a local database removes dependence on data provided by public cbioportal instances for testing. This makes it possible to test features for data types that are not provided by public cbioportal instances or test features that depend on a backend feature not yet integrated in public cbioportal instances. E2e-localdb tests make use of the BACKEND
environmental variable to test against a specific backend version. Depending on the running context (see section above) setting the BACKEND
environmental variable is required or optional (see table below).
Requirement for setting the BACKEND variable depends on the context of the job:
context | BACKEND var | comments |
---|---|---|
Local | mandatory | |
CircleCI | mandatory for feature branches | not for master or rc builds |
CircleCI+PR | optional | 1. When specified, GitHub PR must be of 'draft' state. 2. When not-specified, backend branch will mirror frontend branch (rc frontend vs. rc backend) |
The value of the BACKEND
variable must be formatted as <BACKEND_GITHUB_USER>:<BACKEND_BRANCH>
. For example, when the /env/custom.sh file contains export BACKEND=thehyve:uwe7872A
this is interpreted as a requirement for the commit uwe7872A
of the github.com/thehyve/cbioportal repository.
Using the BACKEND
variable e2e-localdb tests can be conducted against any backend version. This poses a risk when testing in CircleCI+PR context, i.e., tests show up as succesful but should have failed against the backend version that compatible with the target cbioportal-frontend branch. To guard against this and prevent merging of incompatible branches into cbioportal-frontend the e2e-localdb tests enforce the use of draft pull requests (see here for more info). When a cBioPortal backend version is specified (i.e., may require a not yet merged backend branch) and the branch is part of a pull request, the pull request must be in state draft. Only when the BACKEND
variable is not defined a (non-draft) e2e-localdb tests will be conducted on branches that are part of pull requests. Needles to say, pull request should for this and other reasons only be merged when the e2e-localdb tests succeed!
When the BACKEND
variable is not set, the backend version will be set to the target branch of the pull request, i.e. a pull request to 'rc' branch will be tested against the 'rc' branch of the backend.
Some random remarks on e2e-test development
- Screenshot tests and DOM-based tests are contained in files that end with *.screenshot.spec.js or *.spec.js, respectively.
- Screenshot tests should only be used to test components that cannot be accessed via the DOM.
- Screenshots should cover as little of the page possible to test behavior. Larger screenshots will make it more likely the screenshot will need to be updated when an unrelated feature is modified.
- For DOM selection webdriverio selectors are used. Although overlapping with jQuery selectors and both using the '$' notation these methods are not equivalent. See this link for more information on webdriverio selectors.
- At the moment of this writing webdriverio v4 is used. Selectors for this version are not fully compatible with webdriverio v5. For instance, selecting of a element with id test
$('id=test')
does not work; this should be$([id=test])
. I was not able to find documentation of v4 selectors. - e2e tests use the node.js assert library for assertions. It has an API that is different API from chai assertion library used in unit tests of cbioportal-frontend! See the assert documentation for information on assert API.
- Screenshots for failing tests are placed in the
screenshots/diff
andscreenshots/error
folders. These are a valuable asset to debug tests on when developing in Local context. - A great tool for test development is the ability of webdriverio to pause execution with
browser.debug()
. When placing this command in the test code and using therun_local_screenshot_test.sh
facility, a prompt becomes available on the command line that allows testing of DOM selectors in the webbrowser. In addition, the browser window is available on screen; opening of DevTools allows to explore the DOM and observe the effects of webdriverio commands on the command line. - Although webdriverio takes asynchronous behavor of webbrosers into account it does not defend against asynchronous behavior of specific web components (e.g., database access). Not taking this asynchronicity into account will result in
flaky
tests. Typically, flaky test run well on the local system used for development (that has plenty of free resources at moment of test), but fail often on a CI system. Often this is the result of longer times needed page/component update causing tests to fail because the test evaluates a condition before it is loaded. In webdriverio thewaitForExist()
,waitForVisible()
andwaitFor()
method should be used to pause test execution until the page has been updated. Sometimes it is needed to wait for the appearance of a DOM element which presence is tested.
browser.waitForExist('id=button');
assert($('id=button'));
- Reference screenshosts that are created on host system directly (not in dockerized process) differ from screenshots produced by the dockerized setup (e.g., on CircleCI) and cannot be used as references
Making e2e-tests follows the current procedure for the e2e-tests:
- Create junit test file and place in the
./end-to-end-test/local/specs
or./end-to-end-test/remote/specs
directory. - [Optional] Add a folder with an uncompressed custom study in the
./end-to-end-test/local/studies
directory.
- Study_es_0 is imported by default.
- Gene panel and gene set matrix data of custom studies must comply with gene panel/sets imported as part of study_es_0.
- Imports of custom seed data for gene panels and gene sets are not implemented at the moment of this writing.
- In order to minimize time of local database e2e-tests the size of custom studies should be kept as small as possible.
- When developing in Local context port 8081 can be used to access the cbioportal instance ('http://localhost:8081/cbioportal').
Here are some errors that have been encountered and are hard to debug.
This error occurs when an e2e test tries to take a screenshot of an element that doesn't exist.
This error occurs in CircleCI when the reference screenshot file is somehow corrupted. It can be fixed by deleting and updating the reference screenshot.
We are utilizing yarn workspaces
to maintain multiple packages in a single repo (monorepo). The monorepo approach is an
efficient way of working on libraries in the same project as the application that is their primary consumer.
The cbioportal-frontend
is the main web application workspace. It is used to build and deploy the cBioPortal frontend webapp.
Workspaces under packages
directory are separate modules (npm packages) designed to be imported by cbioportal-frontend
workspace as well as by external projects.
Please note: config
and typings
directories under the packages
directory are NOT workspaces or packages. They are intended to share common settings among all packages under the packages
directory.
To add a new workspace, create a new directory under packages
and add a package.json
file. (See cbioportal-frontend-commons
workspace for example configuration).
The recommended way to add a new dependency to an existing workspace is to run yarn workspace <workspace name> add <package name>
instead of just yarn add <package name>
. For example, run yarn workspace cbioportal-frontend add lodash
instead of yarn add lodash
.
Similarly, to remove a package, run yarn workspace <workspace name> remove <package name>
.
Please abide by the following rules for importing dependencies in a monorepo:
- If you are working on
cbioportal-frontend
repository, import modules from packages using the package's alias:
// CORRECT, uses alias:
import {something} from 'cbioportal-frontend-commons'
// INCORRECT, uses relative paths:
`import {something} from ../../packages/cbioportal-frontend-commons/src/lib/someLib`
-
When working on a package, never import custom code from outside that package unless you really intend for that package to be a dependency. For example, commons packages should not import from the main cbioportal project.
-
Avoid circular dependencies at all costs. For example, while it is okay to import a module from
cbioportal-frontend-commons
inreact-mutation-mapper
, there should not be any imports fromreact-mutation-mapper
incbioportal-frontend-commons
. If you happen to need some component from fromreact-mutation-mapper
incbioportal-frontend-commons
, consider moving that component intocbioportal-frontend-commons
package.
Remember that the packages are used by other projects and compatibility needs to be carefully managed.
When you update code under packages a new version of changed packages automatically published once the code is merged to master. However, in a rare case when you would like to set a custom package version, you can run
yarn run updatePackageVersion
Alternatively you can manually set a custom version. When updating manually you should update the version number in the corresponding package.json
as well as the dependencies of other packages depending on the package you update. For example if you update the cbioportal-frontend-commons
version from 0.1.1
to 0.1.2-beta.0
, corresponding cbioportal-frontend-commons
dependency in the package.json
for react-mutation-mapper
and cbioportal-frontend
should also be updated to the new version.
Note that when setting a custom version if you want the next published package version to be, for example, 1.0.6
, then you should set the new version to 1.0.6-beta.1
or a similar prerelease version. If you set the custom version to 1.0.6
, the next published version will be 1.0.7
not 1.0.6
. This is because the auto publish script runs in any case to detect changes in all packages including custom versioned packages.
yarn run updateAPI
Components under packages
should only depend on either external node modules or workspaces under packages
.
Please make sure to not introduce any dependencies from cbioportal-frontend
workspace when updating or adding new files under packages
.
cbioportal-frontend-commons is a separate public npm library which contains basic utility functions and components.
Mutation Mapper component has been moved to a separate GitHub repository: cBioPortal/react-mutation-mapper.
For more information about react-mutation-mapper
development please see react-mutation-mapper.md.
react-mutation-mapper is a separate public npm library that contains the Mutation Mapper and related components.
# You need to change pom.xml with the new version before you run the following code.
mvn -DskipTests clean install