Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
gautamp8 authored Jun 27, 2020
1 parent 815a07c commit 0957609
Showing 1 changed file with 6 additions and 18 deletions.
24 changes: 6 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,17 @@
# Celery-Kubernetes-Operator(WIP)
A basic Celery operator written in Python. To be used to manage simple Celery applications on a Kubernetes cluster. It is being built as a demo project to proposed EuroPython 2020 proposal. With some minor tweaks, to be used in Grofers production as well in the future. This operator is being written with the help of [KOPF](https://github.com/zalando-incubator/kopf) framework open sourced by Zalando SE.
A basic Celery operator written in Python. To be used to manage Celery applications on a Kubernetes cluster. It is being built as a demo project to proposed EuroPython 2020 [proposal](https://ep2020.europython.eu/talks/BbvZjFa-advanced-infrastructure-management-in-kubernetes-using-python/).
Beyond the conference, it'll be pursued to be a production ready project. Feedback/Suggestions to improve are welcome. Please report an issue. This operator is being written with the help of [KOPF](https://github.com/zalando-incubator/kopf) framework open sourced by Zalando SE.

# Project Scope
The general idea is to bridge the gap between infrastructure and application developers where application developers can just spec out a simple celery deployment yaml and have to do nothing more than `kubectl apply -f <file_name>`to spin up their own celery cluster.

It aims to have following things in place-
1. A Custom Resource Definition(CRD) to spec out a Celery deployment having (but not limited to) these attributes -
- Celery image
- Celery app module name
- Queue name
- Resource Constraints(lim_cpu, req_cpu, lim_mem, req_mem)
- Log Level
- Number of workers
- Concurrency
- Basic Flower setup configuration for monitoring
2. A custom controller implementation that registers and manages self-healing capabilities of custom Celery resource for these operations
1. A Custom Resource Definition(CRD) to spec out a Celery and Flower deployment having all the options that they support.
2. A custom controller implementation that registers and manages self-healing capabilities of custom Celery object for these operations
- CREATE - Creates the worker and flower deployments along with exposing a native Service object for Flower
- UPDATE - Reads the CRD modifications and updates the running deployments using RollingUpdate strategy
- UPDATE - Reads the CRD modifications and updates the running deployments using specified strategy
- DELETE - Deletes the custom resource and all the child deployments
3. Keep a watch on CPU/Memory metrics of resource pod and autoscale/downscale the number of workers on given constraints

# Current Progress

1. `crd.yaml` to spec out a simple celery deployment available under `templates/` directory.
2. Create and update handlers are defined and act accordingly as custom resource is created and updated. Deletion works automagically, thanks to KOPF adopt functions.
3. Keep a watch on important metrics(queue length, cpu, memory etc) using flower(and native K8s solutions) to autoscale/downscale the number of workers on specified constraints.

# Directory Structure

Expand Down

0 comments on commit 0957609

Please sign in to comment.