web-monorepo

Table of Contents

• Prerequisites
• Quick Start
  • Install yarn dependencies
• Overview
• Common Tasks
  • Start a local dev site
  • Modifying code (and updating a running local dev site)
  • View workspace dependency graph
  • Execute a workspace’s yarn script using the nx task runner
  • Rebuild dependencies when running a development server
• Notes on individual packages
  • EDA dev server
  • VEuPathDB sites
• IDE hints
  • emacs tide
• Client Bundle Server
  • Browsers
  • Docker Image
  • Paths
  • Testing

Prerequisites

• Node ⇐16.17.1

• Either a global installation of yarn >=1.17.0, or volta >=1.1.0

  • With such a version of yarn globally installed, all yarn commands executed in this repo will be delegated to the "pinned" yarn@3.3.1 installation included in .yarn/releases.

  • With such a version of volta installed, the required version of yarn will automatically be installed.

It is recommended to install volta, as it will keep your local toolchain in sync with the repository.

Quick Start

Install yarn dependencies

From anywhere in the repo:

yarn

Overview

This repository is a "monorepo", using the nx build system and yarn@3 dependency manager. The source code is divided into one of three types of packages: "config", "lib", and "site".

• "config" packages (./packages/configs) include various build and development configurations and tools.

• "lib" packages (./packages/libs) include standalone library and webapp source code.

• "site" packages (./packages/sites) include complete website source code.

Many "lib" and "site" packages include development tools, such as development servers, test scripts, etc.

Many commands require a reference to the package name. All package names are currently prefixed with @veupathdb/. For example, the eda package name is @veupathdb/eda.

The repository is currently configured as a "package based repository". This is subject to change, in the future (see https://nx.dev/concepts/integrated-vs-package-based to read about the differences between package based and integrated repos).

Common Tasks

The following tasks are common performed by developers. This serves as a reference guide, and is not exhaustive in any way. If you feel something is missing, create an issue, or open a pull request.

All commands are expected to be run in the repository’s root directory, unless otherwise specified.

Start a local dev site

"Site" projects are stored in the packages/sites directory. Each one corresponds to a "cohort". For example, packages/sites/genomics-site contains the code used to build the client code for a genomics website.

Each cohort contains a .env.sample file. Copy this to a sibling .env file. You may need to modify some values, such as usernames, passwords, etc. You can also specify a website to use for various services. Typically a deployed QA site will suffice, but you can also run a "local backend" via various methods. (TODO: link to relevants docs/repos).

Once you have created a .env file, you can run the local dev site with the command:

yarn nx start <package name>

For example, if you want to run a local clinepi site, you would run the command:

yarn nx start @veupathdb/clinepi-site

Once the website has been compiled, the dev server will output some build statistics and automatically open a browser tab. You can kill the local dev server with CTRL-C in the terminal where you started it.

Modifying code (and updating a running local dev site)

When a local dev site is running, it will detect when build dependencies are updated and reload the active webpage.

Note: the following refers to code tracked by the monorepo, and not third-party dependencies from npm.

There are two types of souce code that can be updated:

1. Source code within the package being served.

2. Source code within a dependent package.

Changes to code within the package being served will be detected automatically. The local dev service should reompile the affected module and reload the website without intervention.

Changes to code within a dependent workspace package will require a build command for the local dev server to detect the change.

For example, if you change code in packages/libs/eda, you will need to run this command in a new terminal window, from the repository root:

yarn workspace @veupathdb/eda build-npm-modules

Once this command completed, the running dev server will see the updated build artifacts, recompile the website, and reload the webpage. There are some cases where this might not work as expected, such as if the recompile step fails. In those cases, you may need to restart the local dev server. You can monitor the progress of the recompilation step in the terminal where you started the dev server.

View workspace dependency graph

From the root package.json:

yarn nx graph

Execute a workspace’s yarn script using the nx task runner

From the root package.json:

yarn nx run <workspaceName>:<scriptName>

For example, you can start the MultiBLAST dev server by running

yarn nx run @veupathdb/multi-blast:start

Rebuild dependencies when running a development server

When running a development server (such as yarn nx start @veupathdb/eda or yarn nx start @veupathdb/clinepi-site), use the following command to rebuild changes made to dependencies, and to have the dev site reload with the changes:

cd packages/libs/<package>
yarn build-npm-modules

Note: You may need to manually reload your website to see the changes the first time.

Using the equivalent nx command (yarn nx build-npm-modules @veupathdb/<package>) has proven inadequate in this scenario.

Notes on individual packages

EDA dev server

Directory: packages/libs/eda

You will need to configure the server with a packages/libs/eda/.env.local file that sets various environment variables.

For more documentation see the package README and this sample file.

VEuPathDB sites

Directory: packages/sites/{site name}-site

Copy the packages/sites/{site name}-site/.env.sample file to packages/sites/{site name}-site/.env and configure the new file with passwords and the desired backend for the site.

Run yarn to update dependencies if necessary.

Run the command yarn nx start @veupathdb/{site name}-site. For example, to run the ortho site use yarn nx start @veupathdb/ortho-site.

IDE hints

emacs tide

If it is showing errors for tsx imports (especially in eda) and tide-verify-setup mentions tsserver version 3.x then it is time to upgrade emacs tide (to, at time of writing 4.5.4):

M-x package-reinstall <ret> tide <ret>

Client Bundle Server

The Client Bundle Server is a Docker image based on NGINX that is used to serve VEuPathDB client code over HTTP.

As the client code comes in 2 flavors (bundles), legacy and modern, this NGINX server has an internal path rewrite based on the requesting browser’s user agent string to the appropriate client bundle component on request.

This means using a modern browser, requesting the file genomics/site-client.bundle.js will cause the server to actually return modern/genomics/site-client.bundle.js whereas requesting that same file from an older or unsupported browser (such as CURL or Postman) the server will return legacy/genomics/site-client.bundle.js.

Browsers

Whether a browser is considered modern or legacy is dependent on the version of the browser compared to a RegEx constructed by the browserslist-useragent-regexp library using the input query constructed in the browserslist-config package of this repo. (See index.js for the raw queries)

Docker Image

The docker image is based on NGINX-Perl and includes NodeJS for executing a script based on browserslist-useragent-regexp that determines which path a specified file should be served from.

The image build is multi-staged with the first stage compiling primary contents of this repository, and the second stage setting up NGINX and the secondary JS script included in the docker directory (makeSupportedBrowsersScript.js).

Paths

Content is served from the following paths from the root path used to reach a running instance of the built Docker image:

{URL}/clinepi/{target-file}
{URL}/genomics/{target-file}
{URL}/mbio/{target-file}
{URL}/ortho/{target-file}

These paths correspond to the following container internal paths:

/var/www/legacy/clinepi/{target-file}
/var/www/modern/clinepi/{target-file}

/var/www/legacy/genomics/{target-file}
/var/www/modern/genomics/{target-file}

/var/www/legacy/mbio/{target-file}
/var/www/modern/mbio/{target-file}

/var/www/legacy/ortho/{target-file}
/var/www/modern/ortho/{target-file}

Testing

The Docker image may be tested locally by performing the following steps from the docker/ subdirectory:

1. Build and Start the image:

   make docker-build
   make docker-run

2. Using your favorite HTTP request making tool such as Postman, CURL, or a web browser, make a request to http://localhost/genomics/site-client.bundle.js.LICENSE.txt . If the service is working you should receive a LICENSE text file’s contents as the response with a 200 status code. If it is not working you will receive a 403 or 404 error.