Skip to content

Commit

Permalink
Update readme for current progress
Browse files Browse the repository at this point in the history
  • Loading branch information
gautamp8 committed Jun 23, 2020
1 parent 4276873 commit 8b4f231
Showing 1 changed file with 17 additions and 10 deletions.
27 changes: 17 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,30 @@
# Celery-Kubernetes-Operator
A basic Celery operator to be written in Python. To be used to manage simple Celery applications on a Kubernetes cluster. To be built as a demo project to proposed EuroPython 2020 proposal. We'll be using open source [KOPF](https://github.com/zalando-incubator/kopf) framework to write this operator in Python.
A basic Celery operator written in Python. To be used to manage simple Celery applications on a Kubernetes cluster. It is being built as a demo project to proposed EuroPython 2020 proposal. With some minor tweaks, to be used in Grofers production as well in the future. This operator is being written with the help of [KOPF](https://github.com/zalando-incubator/kopf) framework open sourced by Zalando SE.

# Project Scope
This project aims to have following things in place-
1. A Custom Resource Definition(CRD) to spec out a Celery deployment having these attributes -
The general idea is to bridge the gap between infrastructure and application developers where application developers can just spec out a simple celery deployment yaml and have to do nothing more than `kubectl apply -f <file_name>`to spin up their own celery cluster.

It aims to have following things in place-
1. A Custom Resource Definition(CRD) to spec out a Celery deployment having (but not limited to) these attributes -
- Celery image
- Celery app module name
- Broker URL
- Worker Class(sync, gevent etc)
- Queue name
- Resource Constraints(lim_cpu, req_cpu, lim_mem, req_mem)
- Log Level
- Number of threads
- Flower setup configuration(if any)
- Number of workers
- Concurrency
- Basic Flower setup configuration for monitoring
2. A custom controller implementation that registers and manages self-healing capabilities of custom Celery resource for these operations
- CREATE - Creates the Celery CRD resource along with exposing a native Service object for Flower
- UPDATE - Reads the CRD modifications and updates the running Celery resource appropriately(delete and re-create)
- DELETE - Deletes the resource and flower service
- CREATE - Creates the worker and flower deployments along with exposing a native Service object for Flower
- UPDATE - Reads the CRD modifications and updates the running deployments using RollingUpdate strategy
- DELETE - Deletes the custom resource and all the child deployments
3. Keep a watch on CPU/Memory metrics of resource pod and autoscale/downscale the number of workers on given constraints

# Current Progress

1. `crd.yaml` to spec out a simple celery deployment available under `templates/` directory.
2. Create and update handlers are defined and act accordingly as custom resource is created and updated. Deletion works automagically, thanks to KOPF adopt functions.

# Directory Structure

# Inspiration
Expand Down

0 comments on commit 8b4f231

Please sign in to comment.