mirror of
https://github.com/optim-enterprises-bv/kubernetes.git
synced 2025-11-03 19:58:17 +00:00
Merge pull request #18827 from sdminonne/workflow_resource
Auto commit by PR queue bot
This commit is contained in:
k8s-merge-robot
295
docs/proposals/workflow.md
Normal file
295
docs/proposals/workflow.md
Normal file
@@ -0,0 +1,295 @@
|
|||||||
|
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
||||||
|
|
||||||
|
<!-- BEGIN STRIP_FOR_RELEASE -->
|
||||||
|
|
||||||
|
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
|
||||||
|
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
||||||
|
|
||||||
|
If you are using a released version of Kubernetes, you should
|
||||||
|
refer to the docs that go with that version.
|
||||||
|
|
||||||
|
Documentation for other releases can be found at
|
||||||
|
[releases.k8s.io](http://releases.k8s.io).
|
||||||
|
</strong>
|
||||||
|
--
|
||||||
|
|
||||||
|
<!-- END STRIP_FOR_RELEASE -->
|
||||||
|
|
||||||
|
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
||||||
|
|
||||||
|
|
||||||
|
## Abstract
|
||||||
|
|
||||||
|
This proposal introduces [workflow](https://en.wikipedia.org/wiki/Workflow_management_system)
|
||||||
|
functionality in kubernetes.
|
||||||
|
Workflows (aka [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) workflows
|
||||||
|
since _tasks_ are organized in a Directed Acyclic Graph) are ubiquitous
|
||||||
|
in modern [job schedulers](https://en.wikipedia.org/wiki/Job_scheduler), see for example:
|
||||||
|
|
||||||
|
* [luigi](https://github.com/spotify/luigi)
|
||||||
|
* [ozie](http://oozie.apache.org/)
|
||||||
|
* [azkaban](https://azkaban.github.io/)
|
||||||
|
|
||||||
|
Most of the [job schedulers](https://en.wikipedia.org/wiki/List_of_job_scheduler_software) offer
|
||||||
|
workflow functionality to some extent.
|
||||||
|
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
* As a user, I want to create a _JobB_ which depends upon _JobA_ running to completion.
|
||||||
|
* As a user, I want workflow composability. I want to create a _JobA_ which will be triggered
|
||||||
|
as soon as an already running workflow runs to completion.
|
||||||
|
* As a user, I want to delete a workflow (eventually cascading to running _tasks_).
|
||||||
|
* As a user, I want to debug a workflow (ability to track failure): in case a _task_
|
||||||
|
didn't run user should have a way to backtrack the reason of the failure, understanding which
|
||||||
|
dependency has not been satisified.
|
||||||
|
|
||||||
|
|
||||||
|
## Implementation
|
||||||
|
|
||||||
|
In this proposal a new REST resource `Workflow` is introduced. A `Workflow` is represented as a
|
||||||
|
[graph](https://en.wikipedia.org/wiki/Graph_(mathematics)), more specifically as a DAG.
|
||||||
|
Vertices of the graph represent steps of the workflow. The workflow steps are represented via a
|
||||||
|
`WorkflowStep`<sup>1</sup> resource.
|
||||||
|
The edges of the graph represent _dependecies_. To represent edges there is no explicit resource
|
||||||
|
- rather they are stored as predecessors in each `WorkflowStep` (i.e. each node).
|
||||||
|
The basic idea of this proposal consists in creation of each step postponing execution
|
||||||
|
until all predecessors' steps run to completion.
|
||||||
|
|
||||||
|
### Workflow
|
||||||
|
|
||||||
|
A new resource will be introduced in the API. A `Workflow` is a graph.
|
||||||
|
In the simplest case it's a graph of `Job`s but it can also
|
||||||
|
be a graph of other entities (for example cross-cluster objects or other `Workflow`s).
|
||||||
|
|
||||||
|
```go
|
||||||
|
|
||||||
|
// Workflow is a DAG workflow
|
||||||
|
type Workflow struct {
|
||||||
|
unversioned.TypeMeta `json:",inline"`
|
||||||
|
|
||||||
|
// Standard object's metadata.
|
||||||
|
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata.
|
||||||
|
api.ObjectMeta `json:"metadata,omitempty"`
|
||||||
|
|
||||||
|
// Spec defines the expected behavior of a Workflow. More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status.
|
||||||
|
Spec WorkflowSpec `json:"spec,omitempty"`
|
||||||
|
|
||||||
|
// Status represents the current status of the Workflow. More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status.
|
||||||
|
Status WorkflowStatus `json:"status,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WorkflowList implements list of Workflow.
|
||||||
|
type WorkflowList struct {
|
||||||
|
unversioned.TypeMeta `json:",inline"`
|
||||||
|
|
||||||
|
// Standard list metadata
|
||||||
|
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
|
||||||
|
unversioned.ListMeta `json:"metadata,omitempty"`
|
||||||
|
|
||||||
|
// Items is the list of Workflow
|
||||||
|
Items []Workflow `json:"items"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WorkflowSpec contains Workflow specification
|
||||||
|
type WorkflowSpec struct {
|
||||||
|
// Standard object's metadata.
|
||||||
|
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
|
||||||
|
api.ObjectMeta `json:"metadata,omitempty"`
|
||||||
|
|
||||||
|
//ActiveDealineSeconds contains
|
||||||
|
ActiveDeadlineSeconds *int64 `json:"activeDeadlineSeconds,omitempty"`
|
||||||
|
|
||||||
|
// Steps is a map containing the workflow steps. Key of the
|
||||||
|
// map is a string which uniquely identifies the workflow step
|
||||||
|
Steps map[string]WorkflowStep `json:"steps,omitempty"`
|
||||||
|
|
||||||
|
// Selector for created jobs (if any)
|
||||||
|
Selector *LabelSelector `json:"selector,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WorkflowStep contains necessary information to identifiy the node of the workflow graph
|
||||||
|
type WorkflowStep struct {
|
||||||
|
// JobTemplate contains the job specificaton that should be run in this Workflow.
|
||||||
|
// Only one between externalRef and jobTemplate can be set.
|
||||||
|
JobTemplate *JobTemplateSpec `json:"jobTemplate,omitempty"`
|
||||||
|
|
||||||
|
// External contains a reference to another schedulable resource.
|
||||||
|
// Only one between ExternalRef and JobTemplate can be set.
|
||||||
|
ExternalRef *api.ObjectReference `json:"externalRef,omitempty"`
|
||||||
|
|
||||||
|
// Dependecies represent dependecies of the current workflow step.
|
||||||
|
// Irrilevant for ExteranlRef step
|
||||||
|
Dependencies []string `json:"dependencies,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type WorkflowConditionType string
|
||||||
|
|
||||||
|
// These are valid conditions of a workflow.
|
||||||
|
const (
|
||||||
|
// WorkflowComplete means the workflow has completed its execution.
|
||||||
|
WorkflowComplete WorkflowConditionType = "Complete"
|
||||||
|
)
|
||||||
|
|
||||||
|
type WorkflowCondition struct {
|
||||||
|
// Type of workflow condition, currently only Complete.
|
||||||
|
Type WorkflowConditionType `json:"type"`
|
||||||
|
|
||||||
|
// Status of the condition, one of True, False, Unknown.
|
||||||
|
Status api.ConditionStatus `json:"status"`
|
||||||
|
|
||||||
|
// Last time the condition was checked.
|
||||||
|
LastProbeTime unversioned.Time `json:"lastProbeTime,omitempty"`
|
||||||
|
|
||||||
|
// Last time the condition transited from one status to another.
|
||||||
|
LastTransitionTime unversioned.Time `json:"lastTransitionTime,omitempty"`
|
||||||
|
|
||||||
|
// (brief) reason for the condition's last transition.
|
||||||
|
Reason string `json:"reason,omitempty"`
|
||||||
|
|
||||||
|
// Human readable message indicating details about last transition.
|
||||||
|
Message string `json:"message,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WorkflowStatus represents the
|
||||||
|
type WorkflowStatus struct {
|
||||||
|
// Conditions represent the latest available observations of an object's current state.
|
||||||
|
Conditions []WorkflowCondition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
|
||||||
|
|
||||||
|
// StartTime represents time when the job was acknowledged by the Workflow controller
|
||||||
|
// It is not guaranteed to be set in happens-before order across separate operations.
|
||||||
|
// It is represented in RFC3339 form and is in UTC.
|
||||||
|
// StartTime doesn't consider startime of `ExternalReference`
|
||||||
|
StartTime *unversioned.Time `json:"startTime,omitempty"`
|
||||||
|
|
||||||
|
// CompletionTime represents time when the workflow was completed. It is not guaranteed to
|
||||||
|
// be set in happens-before order across separate operations.
|
||||||
|
// It is represented in RFC3339 form and is in UTC.
|
||||||
|
CompletionTime *unversioned.Time `json:"completionTime,omitempty"`
|
||||||
|
|
||||||
|
// Statuses represent status of different steps
|
||||||
|
Statuses map[string]WorkflowStepStatus `json:statuses`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WorkflowStepStatus contains necessary information for the step status
|
||||||
|
type WorkflowStepStatus struct {
|
||||||
|
//Complete is set to true when resource run to complete
|
||||||
|
Complete bool `json:"complete"`
|
||||||
|
|
||||||
|
// Reference of the step resource
|
||||||
|
Reference api.ObjectReference `json:"reference"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`JobTemplateSpec` is already introduced by
|
||||||
|
ScheduledJob controller proposal](https://github.com/kubernetes/kubernete2s/pull/11980).
|
||||||
|
Reported for readability:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type JobTemplateSpec struct {
|
||||||
|
// Standard object's metadata.
|
||||||
|
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
|
||||||
|
api.ObjectMeta
|
||||||
|
|
||||||
|
// Spec is a structure defining the expected behavior of a job.
|
||||||
|
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status
|
||||||
|
Spec JobSpec
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Controller
|
||||||
|
|
||||||
|
Workflow controller will watch `Workflow` objects and any `Job` objects created by the workflow.
|
||||||
|
the `Job`s objects created in each step.
|
||||||
|
Each step can contain either another `Workflow` referenced via `workflowStep.ExternalRef`
|
||||||
|
or a `Job` created via `workflowStep.jobTemplate`.
|
||||||
|
For each non finished workflow (similarly to Job, Workflow completion is detected iterating
|
||||||
|
over all the `status.conditions` condition) we check if deadline is not expired.
|
||||||
|
If deadline expired the workfow is terminated.
|
||||||
|
If deadline didn't expires the workflow controller iterates over all workflow steps:
|
||||||
|
- If step has a status (retrieved via step name (map key) in the `status.statuses`
|
||||||
|
map check whether the step is already completed.
|
||||||
|
- If step is completed nothing is done.
|
||||||
|
- If step is not completed two sub-cases should be analyzed:
|
||||||
|
* Step containing workflow: check wether workflow terminated and eventually update
|
||||||
|
the `status.statuses[name].complete` entry if applicable
|
||||||
|
* Step containing job: check whether job needs to be started or is already started.
|
||||||
|
- A step/job is started if it has no dependecies or all its dependencies are already
|
||||||
|
terminated. Workflow controller adds some labels to the Job.
|
||||||
|
This will permit to obtain the workflow each job belongs to (via `spec.Selector`).
|
||||||
|
The step name is equally inserted as a label in each job.
|
||||||
|
- If the job is already running, a completion check is performed. If the job terminated
|
||||||
|
(checked via conditions `job.status`) the field `status.statusues[name].complete` is updated.
|
||||||
|
- When all steps are complete: `complete` condition is added to `status.condition` and the
|
||||||
|
`status.completionTime` is updated. At this point, workflow is finished.
|
||||||
|
|
||||||
|
|
||||||
|
## Changing a Workflow
|
||||||
|
|
||||||
|
### Updating
|
||||||
|
|
||||||
|
User can modify a workflow only if the `step`s under modification are not already running.
|
||||||
|
|
||||||
|
|
||||||
|
### Deleting
|
||||||
|
|
||||||
|
Users can cancel a workflow by deleting it before it's completed. All running jobs will be deleted.
|
||||||
|
Other workflows referenced in steps will not be deleted as they are not owned by the parent workflow.
|
||||||
|
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
The events associated to `Workflow`s will be:
|
||||||
|
|
||||||
|
* WorkflowCreated
|
||||||
|
* WorkflowStarted
|
||||||
|
* WorkflowEnded
|
||||||
|
* WorkflowDeleted
|
||||||
|
|
||||||
|
## Kubectl
|
||||||
|
|
||||||
|
Kubectl will be modified to display workflows. More particulary the `describe` command
|
||||||
|
will display all the steps with their status. Steps will be topologically sorted and
|
||||||
|
each dependency will be decorated with its status (wether or not step is waitin for
|
||||||
|
dependency).
|
||||||
|
|
||||||
|
## Future evolution
|
||||||
|
|
||||||
|
In the future we may want to extend _Workflow_ with other kinds of resources, modifying `WorkflowStep` to
|
||||||
|
support a more general template to create other resources.
|
||||||
|
One of the major functionalities missing here is the ability to set a recurring `Workflow` (cron-like),
|
||||||
|
similar to the `ScheduledJob` [#11980](https://github.com/kubernetes/kubernetes/pull/11980) for `Job`.
|
||||||
|
If the scheduled job is able to support
|
||||||
|
[various resources](https://github.com/kubernetes/kubernetes/pull/11980#discussion_r46729699)
|
||||||
|
`Workflow` will benefit from the _schedule_ functionality of `Job`.
|
||||||
|
|
||||||
|
### Relevant use cases out of scope of this proposal
|
||||||
|
|
||||||
|
* As an admin I want to set quota on workflow resources
|
||||||
|
[#13567](https://github.com/kubernetes/kubernetes/issues/13567).
|
||||||
|
* As an admin I want to re-assign a workflow resource to another namespace/user<sup>2</sup>.
|
||||||
|
* As a user I want to set an action when a workflow ends/start
|
||||||
|
[#3585](https://github.com/kubernetes/kubernetes/issues/3585)
|
||||||
|
|
||||||
|
|
||||||
|
<sup>1</sup>Something about naming: literature is full of different names, a commonly used
|
||||||
|
name is _task_, but since we plan to compose `Workflow`s (i.e. a task can execute
|
||||||
|
another whole `Workflow`) we have chosen the more generic word `Step`.
|
||||||
|
|
||||||
|
|
||||||
|
<sup>2</sup>A very common feature in industrial strength workflow tools.
|
||||||
|
|
||||||
|
|
||||||
|
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||||
|
[]()
|
||||||
|
<!-- END MUNGE: GENERATED_ANALYTICS -->
|
||||||
Reference in New Issue
Block a user