v1.5.0-Beta

What's New

Support Task Dependency

In most mainstream computing platforms such as MPI and Tensorflow, different pods undertake different roles, for example, master/worker. It is necessary to start master or worker first due to the working principle for different platforms. This feature aims to provide the ability to make the start order correct. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/task-launch-order-within-job.md. (#1920, #1833, @hwdef @shinytang6 @Thor-wl )

Support Reserve Resource for Queue

This feature provides the ability to reserve resources for specified queues in order to make sure there is always guaranteed resources for urgent jobs instead of waiting for resource release or being preempted. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/queue-guarantee-resource-reservation-design.md (#1905, #1904, @qiankunli )

Support Specified Nodes for Volcano in Cluster

In some scenarios such as multiple schedulers, it is necessary for Volcano to be only responsible for part of nodes in the cluster. This feature enable users to configure the nodes that are responsible for the Volcano. More details can be referred to #1834 (#1821, @qiankunli )

Add Tendorflow Job Plugin

Volcano provides a unified object for job management which allows user to run AI training such as Tensorflow, Pytorch, Mxnet, MPI with Volcano Job and enjoy the enhanced lifecycle management. However it is a bit complex for some users. This features is to add Tensorflow plugin based on Volcano job plugin framework which reduces the complexity of running Tensorflow with Volcano and make it easy to use. More details can be referred to https://github.com/volcano-sh/volcano/blob/master/docs/design/distributed-framework-plugins.md (#1874, @LuBingtan )

Other Notable Changes

• update CRD version to v1(#1919, @Thor-wl )
• update golang to v1.17(#1912, @Thor-wl )
• optimize: reuse predicate error on same task group(#1906, @justadogistaken )
• default sort task by index(#1898, @xiaoanyunfei )
• update PriorityClass from v1beta1 to v1 for go-client(#1897, @lc2705 )
• support label volcano.sh/task-priority(#1896, @qiankunli )
• enhance the security of TLS client authentication for webhook(#1895, @huone1 )
• add healthz and metric switch for deploy controller and scheduler(#1888, @huone1 )
• add elastic scheduler design doc(#1887, @qiankunli )
• add eventhandler framework proposal(#1886, @sivanzcw )
• add a argument csi-storage to control the storage capacity resource(#1875, @huone1 )
• add rbac for csinode(#1871, @Thor-wl )
• feat: add imagelocality priority to nodeOrder(#1868, @justadogistaken )
• optimize the CA parse Process(#1862, @huone1 )
• ignore the update event if pod is allocated in cache but not present in NodeName(#1857, @xing0821)
• improve taintTolerationScore interPodAffinityScore throghput when failure occurs(#1856, @justadogistaken )
• switch the order of patch and existence check(#1852, @zzr93)
• support to set healthz address and metrics address(#1849, @huone1 )
• add nodeSelector design doc(#1834, @qiankunli )
• enhance the volcano topology framework(#1762, @huone1 )
• support preempt with priority plugin alone(#1757, @Thor-wl )
• support reserved node(#1821, @qiankunli )
• support multi-cluster scheduling in framework(#1521, @william-wang )
• cleanup scheduler cache informerFactory(#1831, @xiaoanyunfei )
• cleanup AddPriorityClass(#1828, @xiaoanyunfei )
• clean addNumaInfo(#1829, @xiaoanyunfei )
• clean up readAdmissionConf(#1823, @xiaoanyunfei )
• Remove default quota info in NewNamespaceCollection(#1817, @zen-xu )
• add multiple tasks support(#1820, @hwdef )
• refactor the cache to support batch bind api for better performance(#1796, @huone1 )
• optimize resource comparision functions for performance(#1769, @huone1 )
• optimize some logs in admission process(#1738, @huone1 )
• add setting MinResources to pg for normal pod(#1666, @huone1 )
• don't return err message when the pod isn't in the nodeinfo cache(#1478, @huone1 )
• update vendor for resource reservation(#1494, @huone1 )
• Proposal: Add Machine Learning Framework Plugins in Volcano(#1806, @LuBingtan )
• upgrade spf13/cobra version to 1.2.1(#1801, @marffin)
• refactor the volcano to support multi-scheduler with each job and node get conresponding scheduler based on hash.(#1795, @william-wang )
• support multi-scheduler for k8s workload deployment, etc(#1792, @huone1 )
• use root context(#1715, @lowang-bh )
• add design docs for task-leve advanced scheduling policy(#1630, @hwdef )
• Add livenessProbe and readinessProbe in Grafana Container(#1788, @dipanjank )
• enhance the admission conf check(#1799, @huone1 )
• Catch add pod out of sync error(#1783, @zhiyuone )
• add UT for elect action(#1780, @Thor-wl )
• Adding oidc import to enable vcctl work with oidc cluster(#1793, @igormishsky)
• Add job conditions (status&lastTransitionTime)(#1764, @HecarimV )
• add ut converage report for v1.4.0(#1766, @Thor-wl )
• refactor the Jobinfo functions to reduce redundant computing(#1745, @william-wang )

Bug Fixes

• fix scheduling process starts even if resource synchronization is not complete(#1916, @huone1 )
• fix: allocate ut for "two Jobs on one node"(#1913, @justadogistaken )
• fix the deep clone of JobInfo(#1883, @lc2705 )
• fix the security alert from Kubernetes(#1873, @Thor-wl )
• fix pod cannot be allocated with sufficient resource(#1851, @aidaizyy )
• fix: avoid chan block within taintTolerationScore(#1848, @justadogistaken )
• fix: scheduler crash fatal error: concurrent map writes(#1847, @Jason-Liu-Dream)
• fix syntax error in function Remove GPUIndexPatch(#1841, @Thor-wl )
• fix: All pods is existing when restart count exceed max retry(#1719, @LuBingtan )
• fix there is nil pointer access in function setNodeState(#1800, @huone1 )
• fix OOM will occur if pod info is sync before node info(#1662, @huone1 )
• fix controller panic when create A large number of pods(#1814, @huone1 )
• fix a problem about equivalence ecache feature (#1593, @huone1 )
• fix(scheduler) gang plugin task min avaliable check(#1732, @king-jingxiang)
• fix: fix possible panic when 'SetNode' is called(#1685, @eggiter )
• fix bug that vcjob is not compeleted when maxRetry is 1(#1746, @Thor-wl )
• fix the security alerts(#1770, @Thor-wl )
• fix(scheduler): improve job/task clone func(#1729, @shinytang6 )
• fix broken grafana dashboard configuration.(#1773, @dipanjank )

v1.4.0

Changes since v1.4.0-Beta

• fix bug about not record queue label in metric(#1722, @lowang-bh )
• fix: do not set taskInfo.NodeName to empty when nodeInfo.RemoveTask is called(#1716, @eggiter )
• fix(underused): Do not check overused when there is no UnderUsedResourceFn added(#1726, @eggiter )
• pass kubeClient to admission service(#1730, @hack-qian)
• upgrade k8s to v1.19.11 because of security notification(#1733, @Thor-wl )
• optimize some logs in admission process(#1738, @huone1 )
• change the Mutex to RWMutex in predicateCache(#1741, @william-wang )
• fix vcjob not work when mount volume(#1742, @Thor-wl )
• e2e cases about pod affinity skip cancel(#1743, @Thor-wl )
• fix bug that vcjob is not compeleted when maxRetry is 1(#1746, @Thor-wl )
• fix gen-admission-secret.sh(#1752, @yahaa )

v1.4.0-Beta

What's New

1. Support multi-scheduler

In Kubernetes cluster with multiple schedulers, different kinds of workloads should be mapped to certain scheduler sometimes. For example, K8s native workloads such as deployment in namespace kube-system are mapped to default-sheduler while AI and Big data jobs are mapped to Volcano. This feature aims to implements that automaticallty. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/multi-scheduler.md. (#1576, #1521, @huone1 @william-wang )

2. Support proportion of resources for GPU node

In order to make full use of scarce resources such as GPU, one solution is to bind them with other resources as shares. For example, it is common to see a lot of CPU-intensive workloads are scheduled to GPU nodes. When GPU-intensive workloads come, they cannot be scheduled because of lack of CPU or Memory in GPU nodes. If workloads requiring both GPU, CPU, Memory at certatin range can be scheduled to GPU nodes first, it is possible to make full use of GPUs. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/proportional.md. (#1527, @king-jingxiang )

3. Support CPU NUMA-Aware scheduling

As to CPU-intensive workloads especially in AI, Big Data and HPC fileds, It will result in a significant performance improvement if CPU NUMA is enabled. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md. (#1493, @huone1 )

4. Provide framework of stress test

In this release, A kind of framework for Volcano stress test is provided. (#1516, @rudeigerc )

Other Notable Changes

• update roadmap for v1.5(#1708, @william-wang )
• refine scheduler framework(#1705, @wpeng102 )
• fix daily release job fails(#1701, @Thor-wl )
• optimization register controller log(#1683, @hwdef )
• lower ssh key access permission(#1682, @hwdef )
• automatically set GOMAXPROCS for scheduler & controller & webhook-manager(#1681, @SataQiu )
• add revive linter as drop-in replacement of archived golint(#1678, @gy95 )
• modify default scheduler cfg(#1673, @shinytang6 )
• expose detailed scheduling reason of pending tasks(#1672, @eggiter)
• format env key(#1660, @wpeng102 )
• not delete failed pods on last try(#1657, @wpeng102 )
• optimize function setOversubscription(#1653, @huone1 )
• upgrade controller tools to v0.6.0(#1649, @shinytang6 )
• enable configuring volcano webhooks(#1645, @hacker-qian)
• add user-provided rsa key-pair and mount rsa key-pair in InitContainers since we want to clone private git repository before run training task(#1644, @python279 )
• refactor the preempt function in plugin gang(#1643, @huone1 )
• delete bindingTasks from NodeInfo structure(#1636, @huone1 )
• scheduler plugin framework support SharedInformerFactory(#1635, @wpeng102 )
• add task-spec in pod label(#1626, @python279 )
• rename resource comparision functions(#1624, @Thor-wl )
• update resource comparision doc(#1622, @Thor-wl )
• add Equal function(#1621, @Thor-wl )
• add LessEqualPartly function(#1613, @Thor-wl )
• add LessEqualInAllDimension function and remove LessEqual/LessEqualStrict functions(#1611, @thor)
• doc: add resource quota plugin doc(#1583, @merryzhou )
• improve: clean unready status when job ready(#1582, @lowang-bh )
• improve: reduce calculation of total nodes' resource in stead of storing it when snapshot(#1578, @lowang-bh )
• add resource comparison doc(#1573, @Thor-wl )
• update Less function(#1569, @Thor-wl )
• support oversubscription in volcano framework(#1566, @wpeng102 )
• add multi-scheduling design doc(#1565, @huone1 )
• improve: preempt break out if intersection is null(#1563, @lowang-bh )
• add arch node selectors to the installer arm&amd64(#1556, @holdenk )
• optimizing openSession(#1541, @hacker-qian)
• improve: remove nodename from taskinfo when RemoveTask(#1517, @lowang-bh )
• the member Others should be copied in func (ni *NodeInfo) Clone() (#1512, @huone1 )
• use struct NodeScoreList better than HostPriorityList for score(#1510, @huone1 )
• add min success design(#1505, @zen-xu )
• change min resource to 0.1 in resource_info(#1489, @wpeng102 )

Bug Fixes

• fix diff function(#1712, @Thor-wl )

• make 'existing pods anti-affinity rules' work(#1668, @eggiter)

• add setting MinResources to pg for normal pod(#1666, @huone1 )

• fix OOM will occur if pod info is sync before node info(#1662, @huone1 )

• fix addmission parsing bug(#1656, @hacker-qian)

• fix overused judgement when deal with allocate and proportion(#1637, @Thor-wl )

• fix bugs about resource comparison(#1628, @Thor-wl )

• reset task.NodeName after call DeallocateFunc(#1618, @merryzhou )

• adds the missing field to the Webhook(#1615, @hwdef )

• fix a problem about equivalence ecache feature (#1593, @huone1 )

• func FeasibleNodesToFind to use list with a centain order(#1574, @lowang-bh )

• fix static check warning in pkg folder(#1552, @hwdef )

• fix bug in predicates plugin(#1547, @hacker-qian)

• fix(scheduler): reclaim action minus and comparison bug(#1540, @shinytang6 )

• fix resource comparasion bug in task topology(#1546, @Thor-wl )

• fix select wrong queue when proportion is disable(#1497, @zen-xu )

• fix the error for make verify(#1508, @huone1 )

v1.3.0

What's New

1. Support minAvailable at task level

Just as the minAvailable at job level, minAvailable at task level will regard replicases at the same task as group and decide whether to schedule pods at the task. Only when the minAvailable is meet will the pods will be scheduled together. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/task-minavailable.md. (#1459, @shinytang6 )

2. Support minSuccess for Job

Support to configure the least number of pods belonging to the job. It's useful to mark the status of job when minsuccess reaches or not and accelerates the job status judgement. (#1384, @zen-xu )

3. Support task-topology

In big data processing jobs like Tensorflow & Spark, tasks transmitted a large amount of data between each other, causing transmission delay took a large proportion in job execution time. So task topology plugin was proposed to modify scheduling strategy according to transmission topology inside a job, so as to cut the data amount to be transmitted between nodes, decrease transmission delay proportion in job execution time, and improve resource utilization. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/task-topology-plugin.md. (#1353, @jiangkaihua )

4. Create new repository volcano.sh/apis

Separate apis from volcano.sh/volcanosh. Any downstream projects can introduce the CRD clientset/lister/informer with the K8s version it needs. (https://github.com/volcano-sh/apis, @Thor-wl )

Other Notable Changes

• fix the bug of CRD apiversion and installation volcano with kind(#1483, @Thor-wl )
• doc: add schedulerName in gpu sharing user guide(#1481, @ChAnYaNG97 )
• update scheduler default QPS and Burst(#1480, @Thor-wl )
• vcctl queue support get kubeconfig from env(#1477, @yahaa )
• add queue annotation in deployment example and add queue yaml(#1474, @nolimitkun )
• support K8s v1.19. (#1444, @Thor-wl )
• optimize yaml unmarshal logic(#1427, @sniperking1234 )
• doc: add multi schedulers design doc(#1403, @zen-xu )
• simplify unit-test(#1394, @zen-xu )
• add new target update-development-yaml in Makefile(#1386, @zen-xu )
• add additional printer columns to crd Job(#1385, @zen-xu )
• update ca.crt and server.csr validity period to 10 years(#1382, @zen-xu )
• Helm support crd v1(#1378, @zen-xu )
• feat(webhook): add podgroup admission(#1375, @shinytang6 )
• support auto updating crd manifests in helm templates when run make generate-yaml(#1374, @zen-xu )
• refactor(e2e): separate utils as a single package(#1362, @rudeigerc )
• support taint toleration preferNoScheduler in release-0.4(#1354, @huone1 )
• support taintToleration preferNoschdule(#1352, @huone1 )

Bug Fixes

• fix: lose preemptor when considering Preemption between Tasks within same Job (#1453, @lowang-bh )
• scheduler need configmap role to enable elect funtion(#1443, @wpeng102 )
• fix(scheduler): use nodeMap to fix anti-affinity problem(#1430. @shinytang6 )
• fix: use task.Name to make podName in admission(#1412, @merryzhou )
• add bindingTasks to judge whether adding node to the snapshot.(#1388, @zen-xu )
• fix reserving for deleted targetJob raise nil pointer(#1371, @zen-xu )
• fix sla jobOderFn when sla not set(#1365, @merryzhou )
• fix: it is possible to Occur OutOfCpu， when exist some pods including init container(#1364, @huone1 )
• fix wrong Pipeline in action allocate(#1360, @yzs981130 )
• fix: prevent SelectBestNode func arise panic(#1344, @yahaa )
• fix(scheduler): move JobInfo helper functions to method(#1343, @Thrimbda )

v1.2.0

What's New

1. Add TDM plugin

TDM(Time Division Multiplexing) plugin aims to provide a mechanism for nodes, which can be used for K8S and other cluster(such as Yarn) in separate time.(#1269, @yahaa )

2. Add SLA plugin

SLA(Service Level Agreement) plugin works for job resource reservation feature. Users can set SLA for jobs to ensure specified jobs to be scheduled in time. It provides an better design and implementation for job resource reservation. (#1303, @jiangkaihua )

Other Notable Changes

• improve addResourceList func in job_controller_util.go(#1332, @shinytang6 )
• update overcommit plugin(#1324, @jiangkaihua )
• add e2e for sla plugin(#1319, @jiangkaihua )
• make sure non-preemptable and revocable workload not preempt other tasks in tdm plugin(#1314, @wpeng102 )
• support only specify preemptable=true for revocable workload(#1313, @wpeng102 )
• support revocable-zone annotaion for workload(#1312, @wpeng102 )
• add fail event for annotation admission(#1308, @wpeng102 )
• support min pod alive for tdm plugin(#1300, @wpeng102 )
• update enqueue action, import overcommit plugin to limit pending jobs from inqueue.(#1298, @jiangkaihua )
• build cache for revocable nodes(#1293, @yahaa )
• separate JobPipelined into two semantics for preempt action(#1288, @wpeng102 )
• support minAlive and evictMaxNum for job(#1287, @wpeng102 )
• non preemptable deployment preempt resource(#1286, @wpeng102 )
• update job-resource-reservation-design doc(#1282, @Thor-wl )
• add tdm design doc(#1277, @wpeng102 )
• refine deployment.yaml example(#1274, @wpeng102 )
• tdm plugin add victimsFn(#1276, @wpeng102 )
• add Makefile flag SUPPORT_PLUGINS(#1266, @zen-xu )
• update ssh secret when job updated(#1263, @shinytang6 )
• add job plugin example(#1254, @shinytang6 )

Bug Fixes

• replace removed command of kind when getting kube config(#1315, @rudeigerc )
• fix log in job_controller_actions.go(#1305, @gaocegege )
• correct log info in cache.go(#1302, @juchaosong )
• optimize nodeorder plugin(#1292, @huone1 )
• enhance tdm max evict step(#1290, @yahaa )
• revert ssh subpath for ssh plugin(#1280, @shinytang6 )
• fix e2e helm install timeout(#1262, @huone1 )
• fix more pods are reclaimed than required(#1260, @huone1 )
• fix CI: add hacky retry mechanism(#1248, @shinytang6 )

v1.1.2

Changes since v1.1.1

• bug fix: Use musl-gcc build image, because vc-scheduler default image is alpine, which only has musl-libc(#1225, @zen-xu)

v1.1.1

What's New

1. support vc-scheduler loading custom plugins

Separate plugin implementation with scheduler. Support implement custom plugins and load to vc-scheduler dynamically.(#1218, @zen-xu)

2. add MaxRequeueNum as a controller-manager param

Support configure MaxRequeueNum in config file of vc-scheduler, default to 15 times.(#1087, @shinytang6)

3. add design documentation of CPU careful regulation

Give the design of CPU careful regulation in socket level.(#1051, @ProgramerGu)

Other Notable Changes

• add deployment example with volcano scheduling(#1222, @Thor-wl)
• add Queue & Namespace support for volcano monitoring(#1200, @alcorj-mizar)
• job event optimize (#1192, @mikechengwei)
• optimize local script(#1191, @mikechengwei)
• generate v1 crds. remove subresource status in pg crd(#1179, @stpabhi)
• mutate default of job spec(#1170, @shinytang6)
• CI switch to Github Action(#1160, @daixiang0)
• queue-resource design add(#1158, @hudson741)
• feature design for queue resource reservation(#1130, @Thor-wl)
• add podName and svcName length validate(#1127, @mikechengwei)
• update getting-started.md(#1126, @daixiang0)
• some improvements of scheduler(#1111, @shinytang6)

Bug Fixes

• fix allocate action: high priority queue should not block others(#1209, @yesterday)
• fix script: fix daily release err(#1208, @yesterday)
• fix prepare-for-development dead link(#1205, @naveensrinivasan)
• fix gox path err im Makefile(#1201, @shinytang6)
• fix proportion can not reclaim issue(#1194, @wpeng102)
• fix dose not delete network policy when job finished(#1186, @wpeng102)
• fix queue cannot use idle resource issue(#1176, @wpeng102)
• fix scheduling duration update after complete(#1167, @alcorj-mizar)
• fix queue unknown state bug(#1151, @wpeng102)
• fix terminate job and release resources when drop job out of queue(#1138, @merryzhou)
• fix helm upgrade bugs on namespace hardcode(#1132, @alcorj-mizar)

v1.1.0

What's New

1. Add monitor compontent

Monitor compontent added support display some base metrics about Volcano.(#1066, @alcorj-mizar)

2. Support resource reservation for big job automatically

Reserve resource for pending job which is at highest priority among pending jobs and waits for a long time. The big job is recognized by scheduler automatically.(#1044, @Thor-wl)

3. Support HDRF

Hierarchical dominant resource fairness is configured with a weighted tree, such that each node in the tree has a positive weight value.(#928, @ggaaooppeenngg)

Other Notable Changes

• add queue weight validation(#1092, @Thor-wl)
• add file watcher for scheduler config file(#1016, @hzxuzhonghu)
• add arm64 support(#1000, @Thor-wl)

Bug Fixes

• not skip lower prioprity job when job with high priority allocated failed(#1089, @merryzhou)
• remove creationTimestamp comparation in jobOrderFn of gang plugin(#1061, @zionwu)
• fix queue capability overuse when specify minAvailable less than task replicas(#1042, @zen-xu)
• fix duplicate preemptee in victims(#1023, @xiaoanyunfei)
• fix scheduler panic when volcano job use pvc(#1022, @wpeng102)
• fix scheduler panic if minResource not set(#1010, @hzxuzhonghu)
• remove controlled resource in OnJobDelete(#1005, @hzxuzhonghu)
• fix ssh authorize key and remove no-root flag(#996, @hzxuzhonghu)
• plugins/binpack: fix typo of BinpackMemory(#994, @aixeshunter)
• scheduler: avoid pushing back empty jobs in allocate action(#992, @lixiang233)
• fix unallocate(#984, @hzxuzhonghu)
• fix bug of queue capability lose efficacy(#974, @hzxuzhonghu)

v0.4.2

• Fix queue capability validation failed when some running jobs finished or deleted (#959, @Thor-wl)

v1.0.1

Changelog since v1.0.0

• Fix job's scheduler name does not take effect (#944, @hzxuzhonghu)
• Fix podgroup status and event (#951, @hzxuzhonghu)
• Support job scale down to zero (#945, @hzxuzhonghu)
• Fix queue capability validation failed when some running jobs finished or deleted (#959, @Thor-wl)