Skip to content

Commit

Permalink
Add updated routerlicious diagram and update routerlicious README.md (#…
Browse files Browse the repository at this point in the history
  • Loading branch information
curtisman authored Feb 3, 2021
1 parent 02524ab commit bae1be1
Show file tree
Hide file tree
Showing 2 changed files with 711 additions and 22 deletions.
41 changes: 19 additions & 22 deletions server/routerlicious/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ If you want to build API documentation locally, see [Building Documentation](htt
* Perf === Magic

## Architecture
![](./docs/Routerlicious-Architecture.svg)

### Microservices

Expand All @@ -126,41 +127,26 @@ have clear input and output characteristics. Many can be run as serverless lambd
[lambda framework](./src/kafka-service). We chose this path to have greater control over the throughput and latency
characteristics of our message processing. But could be also be run with Azure Functions, AWS Lambdas, Fission, etc...

#### [Alfred](./src/alfred)
#### [Alfred](./packages/routerlicious/src/alfred)

Alfred is the entry point to the system. Clients connect to Alfred to join the operation stream. Joining the stream
allows them to receive push notifications for new operations, retrieve old operations, as well as create new ones. We
make use of Redis for push notifications. New operations are placed inside of Apache Kafka for processing.

#### [Deli](./src/deli)
#### [Deli](./packages/routerlicious/src/deli)

Deli retrieves unsequenced messages from Kafka and then attaches a new sequence number to them. Sequence numbers
are per-document monotonically increasing numbers. Sequenced messages are placed back into Apache Kafka for processing.

#### [Scriptorium](./src/scriptorium)
The Deli microservice also runs the [Broadcaster](./packages/lambdas/src/broadcaster) lambda that directly put sequenced
message into redis so that alfred can listen and broadcast back to the clients.
#### [Scriptorium](./packages/routerlicious/src/scriptorium)

Scriptorium retrieves sequenced messages from Kafka. It then broadcasts the new message and writes the message
to a database for storage. We currently make use of Redis for broadcasting and MongoDB for storage.

#### [Paparazzi](./src/paparazzi)

The logical storage model for documents is an ordered sequence of operations. Rather than requiring clients to replay
all operations when loading a document we instead periodically create consolidated logs of the operations. These
consolidated logs, or snapshots, are designed for quick and efficient loading of the document at a particular
sequence number.

Paparazzi was initially charged with just creating snapshots of documents. But it has since evolved to run
intelligent agents. Paparazzi agents are designed to be isomorphic - that is they can be run on both the server
and the client. This enables a connected client join in with a pool of server Paparazzi instances to perform
snapshotting and intelligence on a document.

Paparazzi instances connect to Foreman to receive instructions on what operations to perform on the document.

#### [Foreman](./src/foreman)
#### [Scribe](./packages/routerlicious/src/scribe)

Foreman is in charge of managing a pool of Paparazzi instances. It listens to the same stream of Kafka messages as
Scriptorium but uses this to understand which documents are active. It then schedules and manages work to be run
across the pool of Paparazzi instances (snapshot, spell check, entity extraction, etc...).
Scribe is responsible for listening to inbound summary ops and then writing them to the public record in the Historian

#### [Historian](../historian)

Expand All @@ -173,6 +159,17 @@ More details on content-addressable file systems and Git can be found at
* https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
* http://stefan.saasen.me/articles/git-clone-in-haskell-from-the-bottom-up/

### Other Microservices

#### [Copier](./packages/routerlicious/src/copier)

Copier directly reads the raw (unticketed) operations and store it in the database. The data can later be retrieved
via alfred for testing and verification.
#### [Foreman](./packages/routerlicious/src/foreman)

Foreman is in charge of managing a pool of remote agent instances. It listens to the same stream of Kafka messages as
Scriptorium but uses this to understand which documents are active. It then schedules and manages work to be run
across the pool of remote agent instances (spell check, entity extraction, etc...).
## Distributed data structures

The API currently exposes four distributed data structures
Expand Down
Loading

0 comments on commit bae1be1

Please sign in to comment.