🎲 Arithrandom

A producer/consumer system, made from the scratch

---

Table of contents

• About
• Architecture
• Information flow
• Features
• Running the backend
  • Skaffold
  • Kubectl
  • Docker compose
  • Traditional way
• Running the frontend
• Roadmap
• Tests
• Author

---

About

The project aims to show a simple Producer/Consumer system where the producer generates random arithmetic expressions and sends them to the consumer. So far, only sum expressions are generated, but the consumer is already prepared to receive other operations.

In addition, no pre-existing message broker on the market was used. After researching, I decided to create an EventBus that centralizes message exchange between microservices. The operation is simple: the microservices send messages to EventBus, and it sends them to the microservices.

This communication could be done by socket.io or gRPC, but the simplest way was chosen to demonstrate the communication, with HTTP requests through the axios. At the same time, to facilitate the development of the application, feathersjs was adopted combined with Express.js.

Architecture

The project has been separated into a few folders, where each folder is a separate project.

|_ client # frontend (React)
|_ consumer # backend (featherjs + typescript)
|_ event-bus # backend (expressjs + typescript)
|_ infra # devops (kubernetes)
|_ producer # backend (featherjs + typescript)

Ps.: The producer was made with the Repository Pattern in mind, having an extra layer to separate the responsibilities

Information flow

The flow starts with an HTTP call to the producer on the POST /expressions route, which triggers the creation of the random expression and sending it to the EventBus. In turn, EventBus generates a unique key (eventId - uuidv4), sends the message to all subscribers (in principle, all microservices are subscribed), and returns the key to the producer.

After the consumer receives the message, the equation is solved and a message is sent to the EventBus with the result. As in the first part, the EventBus fires for everyone subscribed. In this case, the producer becomes a consumer and updates the result in its database.

Ps.: At first, the data was stored in memory, but in a future implementation, it is easy to adopt some database. As EventBus receives a lot of data and it can change according to the topic of the message, I believe it would be better to use a non-relational database such as MongoDB. Producer, on the other hand, has a better-defined structure, so it could use a relational database, such as Postgres or MySQL.

Features

• The Producer generating random addition expressions of two positive integers, e.g. "2+3="
• The Consumer computing and sending an event with the correct mathematical result for each expression it receives
• The consumer is ready to support more than simple addition (see: compute-expression.ts)
• If the consumer goes down, when it goes up again it will solve the pending equations
• Every service has winston configured to log the messages
• Unit tests in the core of the project (generation and computing of the expressions) with Jest
• The Producer and Consumer as separate services
• An event bus to handle the events between the services
• Docker configuration
• Kubernetes configuration (infra/k8s folder)
• Skaffold configuration

Running the backend

The first step is to set the environment variables. To do this, duplicate the .env.example files to .env, changing the variables as you wish.

After this step, there are some ways to run the entire application:

Skaffold

The easiest way to run the project is to have Skaffold installed and running the following command in the main project folder:

skaffold dev

And then, to know the server port use the command:

kubectl get services

Ps.: you can import the endpoints from the file postman_collection.json

Kubectl

The other way is to have kubectl (Kubernetes) installed and run:

cd producers/
docker build -t wenderpmachado/producers .

cd ../event-bus
docker build -t wenderpmachado/event-bus .

cd ../consumer
docker build -t wenderpmachado/consumers .

cd ..
kubectl apply -f infra/k8s/

Docker compose

If you are familiar with docker compose, each project has a file where you can run:

docker-compose -f "docker-compose.yml" up -d --build

Traditional way

Finally, if you intend to run all microservices on your terminal, each folder has a package.json that has the initialization command:

npm run dev
# or
yarn dev

Running the frontend

After start the backend application, you can open the client folder and run the command:

npm run dev

# or

yarn dev

Tests

To run the tests, the following functions were created in package.json:

yarn test # run all unit tests
test:watch # every change made to the code, runs the tests again
test:cov # generate coverage report ('/coverage')
test:clear # remove cache and the coverage from tests

Ps.: After running the coverage test, try to open /coverage/lcov-report/index.html in your browser

Roadmap 🔭

• Producer generates other random expressions
• Producer accept expression from frontend
• Finish implementing MongoDB in the event-bus microservice
• Finish implementing Postgres in the producer microservice
• Increase the unit test coverage
• Add CI/CD with GitHub Actions
• Add ESlint
• Add EventBus endpoint to Consumers subscribe / unsubscribe
• Add load testing
• Improve the frontend

---

Author

Made by Wender Machado 🚀