This contains the source files for the "Omnipedia - Changes" Drupal module, which provides the wiki page changes functionality for Omnipedia.

⚠️ Why open source? / Spoiler warning

---

Description

This contains our infrastructure for generating the wiki page changes between two in-universe dates. We realized early on in development that the sheer amount of content that editors would have to deal with would become a nightmare to manage and keep track of if they also had to manually mark up the changes, and this problem would only become exponentially worse the more content and dates were added. A completely automated solution to generate these changes was a must.

Under the hood

One major problem when dealing with trying to generate a diff between two strings that are HTML is that a lot of libraries out there don't actually understand HTML elements and would mangle the HTML structure in expected and unexpected ways. We could render the HTML as plain text, without the HTML elements, but that would require somehow reconstructing the HTML on top of a diff, which did not seem remotely practical. What we needed was a library that understands HTML and where elements start and end. After a bit of research, we looked into what the Diff module uses, and found our solution: the caxy/php-htmldiff library.

After solving that initial problem, we needed to customize the output of the library, but it didn't offer any useful way to do this before it rendered its diffs. The solution we settled on, like many other things on Omnipedia, was to parse the rendered diffed HTML into a DOM and manipulate it using the Symfony DomCrawler component and PHP's DOM classes that it wraps.

The core code that orchestrates all of this is the wiki changes builder service, which configures the caxy/php-htmldiff instance, validates that a wiki page can have a diff (i.e. has a previous in-universe date to diff against), returns a cached copy if one is found, or renders the actual diff and dispatches events both before the diff is generated and after. Once that core system was in place, we wrote several event subscribers to alter the output to our requirements.

Asynchronicity

It's at this point that we ran into a serious problem: while uncached changes for most wiki pages would take a second or two to generate and be sent to the browser, a few outliers would consistently take far longer, up to 30 seconds or more. We were hitting the limits of what the library and PHP could handle, even after some excellent work by the library maintainer to improve performance.

The solution to this required significantly more engineering. The server was fully capable of generating the wiki page changes, so what we had to do was to render those changes independently of when they were requested; they would have to be rendered ahead of time, asynchronously, in a separate process. Drupal core has a queue system that allows for batch processing, which the Warmer module builds on top of to allow for performing cache warming tasks.

We wrote our own custom Warmer plug-in which is invoked via a cron job that runs multiple times an hour. The plug-in determines all possible variations a set of changes would need to be rendered in, specifically different sets of user permissions, and then renders them one by one. These are then cached to a Permanent Cache Bin so that they survive any Drupal cache clear that may be required when deploying updated code (though we try to minimize cache clears).

All together now

While all of this is happening in the background process, the changes route controller was rewritten to handle three possible states so that it always returns a fast response to the browser:

1. If no changes have been built between the requested couple of dates, it will show a placeholder message telling the user to check back in a few minutes; it doesn't risk trying to build the changes and potentially make the user wait a long time to see them.

2. If changes have been built, but one of the two wiki pages was updated and thus the changes cache item was invalidated and the cron job hasn't run yet, it will show the old (invalidated) cache item.

3. If changes have been built and the cache item is valid, it will show that item.

---

Requirements

• Drupal 10 or 11

• PHP 8.1

• Composer

Drupal dependencies

Before attempting to install this, you must add the Composer repositories as described in the installation instructions for these dependencies:

• The ambientimpact_core module.

• The following Omnipedia modules:

  • omnipedia_core

  • omnipedia_date

  • omnipedia_main_page

  • omnipedia_user

Front-end dependencies

To build front-end assets for this project, Node.js and Yarn are required.

---

Installation

Composer

Set up

Ensure that you have your Drupal installation set up with the correct Composer installer types such as those provided by the drupal/recommended-project template. If you're starting from scratch, simply requiring that template and following the Drupal.org Composer documentation should get you up and running.

Repository

In your root composer.json, add the following to the "repositories" section:

"drupal/omnipedia_changes": {
  "type": "vcs",
  "url": "https://github.com/neurocracy/drupal-omnipedia-changes.git"
}

Patching

This provides one or more patches. These can be applied automatically by the the cweagans/composer-patches Composer plug-in, but some set up is required before installing this module. Notably, you'll need to enable patching from dependencies (such as this module 🤓). At a minimum, you should have these values in your root composer.json (merge with existing keys as needed):

{
  "require": {
    "cweagans/composer-patches": "^1.7.0"
  },
  "config": {
    "allow-plugins": {
      "cweagans/composer-patches": true
    }
  },
  "extra": {
    "enable-patching": true,
    "patchLevel": {
      "drupal/core": "-p2"
    }
  }
}

Important: The 1.x version of the plug-in is currently required because it allows for applying patches from a dependency; this is not implemented nor planned for the 2.x branch of the plug-in.

Installing

Once you've completed all of the above, run composer require "drupal/omnipedia_changes:^7.0@dev" in the root of your project to have Composer install this and its required dependencies for you.

Front-end assets

To build front-end assets for this project, you'll need to install Node.js and Yarn.

This package makes use of Yarn Workspaces and references other local workspace dependencies. In the package.json in the root of your Drupal project, you'll need to add the following:

"workspaces": [
  "<web directory>/modules/custom/*"
],

where <web directory> is your public Drupal directory name, web by default. Once those are defined, add the following to the "dependencies" section of your top-level package.json:

"drupal-omnipedia-changes": "workspace:^7"

Then run yarn install and let Yarn do the rest.

Optional: install yarn.BUILD

While not required, yarn.BUILD is recommended to make building all of the front-end assets even easier.

---

Building front-end assets

This uses Webpack and Symfony Webpack Encore to automate most of the build process. These will have been installed for you if you followed the Yarn installation instructions above.

If you have yarn.BUILD installed, you can run:

yarn build

from the root of your Drupal site. If you want to build just this package, run:

yarn workspace drupal-omnipedia-changes run build

---

Major breaking changes

The following major version bumps indicate breaking changes:

• 4.x - Front-end package manager is now Yarn; front-end build process ported to Webpack.

• 5.x - Moved and refactored the omnipedia.wiki_node_changes_user service to multiple services in the omnipedia_user module.

• 6.x - Requires Drupal 9.5; includes backward compatible Drupal 10 deprecation fixes but is still not fully compatible.

• 7.x:

  • Requires Drupal 10 due to non-backwards compatible change to \Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher::dispatch().

  • Requires ambientimpact_core 2.x due to the event dispatcher change.