— layout: page title: Design countheads: true toc: true comments: true —

Remote Execution Technology

User Stories

Design

Although specific providers are mentioned in the design, it's used mainly for distinguishing different approaches to the remote execution than to choose specific technologies

Ssh Single Host Push

{% plantuml %} actor User participant “Foreman Server” as Foreman participant “Foreman Proxy” as Proxy participant “Host” as Host

autonumber User -> Foreman : JobInvocation Foreman -> Proxy : ProxyCommand Proxy -> Host : SshScript Activate Host Host –> Proxy : ProgressReport[1, Running] Host –> Proxy : ProgressReport[2, Running] Proxy –> Foreman : AccumulatedProgressReport[1, Running] Host –> Proxy : ProgressReport[3, Running] Host –> Proxy : ProgressReport[4, Finished] Deactivate Host Proxy –> Foreman : AccumulatedProgressReport[2, Finished] {% endplantuml %}

JobInvocation: see see scheduling

ProxyCommand:

SSHScript:

ProgressReport[1, Running]:

ProgressReport[2, Running]:

AccumulatedProgressReport[1, Running]:

ProgressReport[3, Running]:

ProgressReport[4, Finished]:

AccumulatedProgressReport[2, Finished]:

Ssh Single Host Check-in

This case allows to handle the case, when the host is offline by the time of job invocation: the list of jobs for the host is stored on the Foreman server side for running once the host is online.

This approach is not limited to the ssh provider only.

{% plantuml %} actor User participant “Foreman Server” as Foreman participant “Foreman Proxy” as Proxy participant “Host” as Host

autonumber User -> Foreman : JobInvocation Host -> Proxy : CheckIn Proxy -> Foreman : CheckIn Foreman -> Proxy : ProxyCommand Proxy -> Host : SshScript {% endplantuml %}

Ssh Multi Host

{% plantuml %} actor User participant “Foreman Server” as Foreman participant “Foreman Proxy” as Proxy participant “Host 1” as Host1 participant “Host 2” as Host2

autonumber User -> Foreman : JobInvocation Foreman -> Proxy : ProxyCommand Foreman -> Proxy : ProxyCommand Proxy -> Host1 : SSHScript Proxy -> Host2 : SSHScript {% endplantuml %}

ProxyCommand:

ProxyCommand:

{% info_block %} we might want to optimize the communication between server and the proxy (sending collection of ProxyCommands in bulk, as well as the AccumulatedProgerssReports). That would could also be utilized by the Ansible implementation, where there might be optimization on the invoking the ansible commands at once (the same might apply to mcollective). On the other hand, this is more an optimization, not required to be implemented from the day one: but it's good to have this in mind {% endinfo_block %}

MCollective Single Host

{% plantuml %} actor User participant “Foreman Server” as Foreman participant “Foreman Proxy” as Proxy participant “AMQP” as AMQP participant “Host” as Host

autonumber User -> Foreman : JobInvocation Foreman -> Proxy : ProxyCommand Proxy -> AMQP : MCOCommand AMQP -> Host : MCOCommand Activate Host Host –> AMQP : ProgressReport Deactivate Host AMQP –> Proxy : ProgressReport Proxy –> Foreman : AccumulatedProgressReport {% endplantuml %}

JobInvocation:

ProxyCommand:

MCOCommand:

ProgressReport:

AccumulatedProgressReport:

Ansible Single Host

{% plantuml %} actor User participant “Foreman Server” as Foreman participant “Foreman Proxy” as Proxy participant “Host” as Host

autonumber User -> Foreman : JobInvocation Foreman -> Proxy : ProxyCommand Proxy -> Host : AnsibleCommand Activate Host Host –> Proxy : ProgressReport Deactivate Host Proxy –> Foreman : AccumulatedProgressReport

{% endplantuml %}

JobInvocation:

ProxyCommand:

AnsibleCommand:

ProgressReport:

AccumulatedProgressReport:

Job Preparation

User Stories

Scenarios

Creating a job template

  1. given I'm on new template form

  2. I select from a list of existing job names or fill in a new job name

  3. I select some option to add an input

  4. Give the input a name

  5. Select the type 'user input'

  6. Give the input a description (space separated package list)

  7. I select from a list of known providers (ssh, mco, salt, ansible)

  8. I am shown an example of how to use the input in the template

  9. I am able to see some simple example for the selected provider??

  10. I fill in the template

  11. I select one or more organizations and locations (if enabled)

  12. I click save

Creating a smart variable based input

  1. given i am creating or editing a job template

  2. I select to add a new input

  3. Give the input a name

  4. Define a smart variable name

Design

{% plantuml %}

class JobTemplate { name:string job_name: string retry_count: integer retry_interval: integer splay: integer provider_type: string == has_and_belongs_to_many :taxonomies has_many :inputs has_many :audits }

class ConfigTemplateInput { name: string required: bool input_type: USER_INPUT | FACT | SMART_VARIABLE fact_name: string smart_variable_name: string description: string == has_one :job_template }

ConfigTemplate “1” – “N” ConfigTemplateInput {% endplantuml %}

Job Invocation

User Stories

Scenarios

Fill in target for a job

  1. when I'm on job invocation form

  2. then I can specify the target of the job using the scoped search syntax

  3. the target might influence the list of providers available for the invocation: although, in delayed execution and dynamic targeting the current list of providers based on the hosts might not be final and we should count on that.

Fill in template inputs for a job

  1. given I'm on job invocation form

  2. when I choose the job to execute

  3. then I'm given a list of providers that I have enabled and has a template available for the job

  4. and each provider allows to choose which template to use for this invocation (if more templates for the job and provider are available)

  5. and every template has input fields generated based on the input defined on the template (such as list of packages for install package job)

See the calculated template inputs for a job

  1. given I'm on job invocation form

  2. when I choose the job to execute

  3. and I'm using a template with inputs calculated base on fact data template available for the job

  4. then the preview of the current value for this input should be displayed

  5. but for the execution the value that the fact has by the time of execution will be used.

Fill in job description for the execution

  1. given I'm on job invocation form

  2. there should be a field for task description, that will be used for listing the jobs

  3. the description value should be pregenerated based on the job name and specified input (something like “Package install: zsh”)

Fill in execution properties of the job

  1. when I'm on job invocation form

  2. I can override the default values for number of tries, retry interval, splay time, timeout, effective user...

  3. the overrides are common for all the templates

Set the execution time into future (see scheduling for more scenarios)

  1. when I'm on a job invocation form

  2. then I can specify the time to start the execution at (now by default)

  3. and I can specify if the targeting should be calculated now or postponed to the execution time

Run a job from host detail

  1. given I'm on a host details page

  2. when I click “Run job”

  3. then a user dialog opens with job invocation form, with pre-filled targeting pointing to this particular host

Run a job from host index

  1. given I'm on a host index page

  2. when I click “Run job”

  3. then a user dialog opens with job invocation form, with prefiled targeting using the same search that was used in the host index page

Invoke a job with single remote execution provider

  1. given I have only one provider available in my installation

  2. and I'm on job invocation form

  3. when I choose the job to execute

  4. then only the template for this provider is available to run and asking for user inputs

Invoke a job with hammer

  1. given I'm using CLI

  2. then I can run a job with ability to specify:

  3. targeting with scoped search or bookmark_id

  4. job name to run

  5. templates to use for the job

  6. inputs on per-template basis

  7. execution properties as overrides for the defaults coming from the template

  8. start_at value for execution in future

  9. in case of the start_at value, if the targeting should be static vs. dynamic

  10. whether to wait for the job or exit after invocation (–async option)

Re-invoke a job

  1. given I'm in job details page

  2. when I choose re-run

  3. then a user dialog opens with job invocation form, with prefiled targeting parameters from the previous execution 1 and I can override all the values (including targeting, job, templates and inputs)

Re-invoke a job for failed hosts

  1. given I'm in job details page

  2. when I choose re-run

  3. then a user dialog opens with job invocation form, with prefiled targeting parameters from the previous execution 1 and I can override all the values (including targeting, job, templates and inputs)

  4. I can choose in the targeting to only run on hosts that failed with the job previously

Edit a bookmark referenced by pending job invocation

  1. given I have a pending execution task which targeting was created from a bookmark

  2. when I edit the bookmark

  3. then I should be notified about the existence of the pending tasks with ability to update the targeting (or cancel and recreate the invocation)

Email notification: opt in

  1. given I haven't configured to send email notifications about my executions

  2. then the job invocation should have the 'send email notification' turned off by default

Email notification: opt out

  1. given I haven't configured to send email notifications about my executions

  2. then the job invocation should have the 'send email notification' turned off by default

Design

Class diagram of Foreman classes

{% plantuml %}

class Bookmark { name:string query:string controller:string public:bool owner_id:integer owner_type:string }

class Targeting { query: string dynamic: bool }

class Host class User

class TemplateInvocation { inputs }

class JobInvocation { tries retry_interval splay concurrency effective_user email_notification: bool }

class JobTask { start_at: datetime }

Bookmark “1” - “N” Targeting Targeting “M” - “N” Host : (polymorphic) Targeting “N” – “1” User JobInvocation “1” – “1” Targeting JobInvocation “1” – “N” TemplateInvocation TemplateInvocation “N” – “1” JobTemplate JobInvocation “1” – “N” JobTask

{% endplantuml %}

Query is copied to Targeting, we don't want to propagate any later changes to Bookmark to already planned job executions.

We can store link to original bookmark to be able to compare changes later.

For JobInvocation we forbid later editing of Targeting.

Open questions

Job Execution

User Stories

Scenarios

Cancel pending bulk task: all at once

  1. given I've set a job to run in future on multiple hosts

  2. when I click 'cancel' on the corresponding bulk task

  3. then the whole task should be canceled (including all the sub-tasks on all the hosts)

Cancel pending bulk task: task on specific host

  1. given I've set a job to run in future on multiple hosts

  2. when I show the task representation on a host details page

  3. when I click 'cancel' on the task

  4. then I should be offered whether I should cancel just this instance or the whole bulk task on all hosts

Fail after timeout

  1. given I've invoked a job

  2. when the job fails to start in given specified timeout

  3. then the job should be marked as failed due to timeout

Retried task

  1. given I've invoked a job

  2. when the job fails to start at first attemt

  3. then the executor should wait for retry_timeout period

  4. and it should reiterate with the attempt based on the tries number

  5. and I should see the information about the number of retries

Design

Class diagram for jobs running on multiple hosts

{% plantuml %}

class Host { get_provider(type) }

class BulkJobTask { state: $TaskState start_at: datetime started_at: datetime ended_at datetime cancel() }

class JobTask { retry: integer retry_interval: integer timeout: integer splay: integer concurrency: integer type: string state: $TaskState start_at: datetime started_at: datetime tried_count: integer ended_at datetime {abstract} support_cancel?() {abstract} proxy_endpoint() cancel() }

abstract class ProxyCommand { }

class SSHProxyCommand { {static} support_cancel?() proxy_endpoint():string }

class MCollectiveProxyCommand { {static} support_cancel?() proxy_endpoint():string }

BulkJobTask “N” - “1” JobInvocation BulkJobTask “1” – “N” JobTask TemplateInvocation “N” - “1” JobInvocation TemplateInvocation “1” — “N” JobTask JobTask “1” – “1” ProxyCommand JobTask “N” – “1” Host

ProxyCommand <|– SSHProxyCommand ProxyCommand <|– MCollectiveProxyCommand

{% endplantuml %}

Class diagram for jobs running a single host

{% plantuml %}

class Host { get_provider(type) }

class JobTask { retry: integer retry_interval: integer timeout: integer splay: integer concurrency: integer type: string state: $TaskState started_at: datetime start_at: datetime tried_count: integer ended_at datetime cancel() {abstract} support_cancel?() {abstract} proxy_endpoint() plan() cancel() }

class ProxyCommand { }

JobTask “N” - “1” JobInvocation TemplateInvocation “N” - “1” JobInvocation TemplateInvocation “1” - “N” JobTask JobTask “1” – “1” ProxyCommand JobTask “1” – “1” Host {% endplantuml %}

Reporting

User Stories

Scenarios

Track the job running on a set of hosts

  1. given I've set a job to run in future on multiple hosts

  2. then I can watch the progress of the job (number of successful/failed/pending tasks)

  3. and I can get to the list of jobs per host

  4. and I'm able to filter on the host that it was run against and state

Track the job running on a single host

  1. given I've set a job to run on a specific host

  2. when I show the task representation page

  3. then I can watch the progress of the job (updated log), status

History of jobs run on a host

  1. given I'm on host jobs page

  2. when I can see all the jobs run against the host

  3. and I'm able to filter on the host that it was run against and state, owner etc.

History of invoked jobs

  1. given I'm on job invocation history page

  2. when I can see all the jobs invoked in the system

  3. scoped by a taxonomy (based on the hosts the jobs were run against)

  4. and I'm able to filter on the host that it was run against and state, owner etc.

Email notification: send after finish

  1. given I've invoked a job with email notification turned on

  2. when the job finishes

  3. then I should get the email with report from the job after it finishes

Design

Class diagram for jobs running on multiple hosts

{% plantuml %}

class Host { }

class JobInvocation { email_notification: bool }

class BulkJobTask { state: $TaskState start_at: datetime started_at: datetime ended_at datetime }

class JobTask { type: string state: $TaskState start_at: datetime started_at: datetime ended_at datetime tried_count: integer command: string output: string exit_code: string }

BulkJobTask “N” - “1” JobInvocation BulkJobTask “1” – “N” JobTask JobTask “1” – “1” Host

{% endplantuml %}

Scheduling

User Stories

Scenarios

Job set for the future

  1. given I've invoked a job at future time

  2. when the time comes

  3. the job gets executed

Creating reoccurring job

  1. given I'm in job invocation form

  2. when I check 'reoccurring job'

  3. then I can set the frequency and valid until date

Showing the tasks with reoccurring logic

  1. when I list the jobs

  2. I can see the information about the reoccurring logic at every job

  3. and I can filter the jobs for those with the reoccurring logic

Canceling the reoccurring job

  1. given I have reoccurring job configured

  2. when I cancel the next instance of the job

  3. then I'm offered to cancel the reoccurring of the job in the future

Design

{% plantuml %}

class Schedule { start_at: datetime end_at: datetime cronline: string }

class JobTask { }

JobTask “N” – “1” JobInvocation JobInvocation “1” – “1” Schedule

{% endplantuml %}

Developer API

User Stories

Scenarios

Defining a predefined job without provided inputs

  1. given I'm a Foreman developer

  2. and I want to expose 'puppet run' feature to the user

  3. then define the 'Puppet Run' as predefined job in the code

  4. and specify the default job name to be used for the mapping

Defining a predefined job with provided inputs

  1. given I'm a Katello developer

  2. and I want to expose 'package install' feature to the user

  3. then I define the 'Package Install' predefined job with list of packages as provided input in the code

  4. and I specify default job name to be used for the mapping

  5. and I specify default mapping of the provided inputs to template inputs

Preseeding the predefined jobs

  1. given I've defined the 'Package Install' predefined job

  2. when the seed script is run as part of the Foreman installation

  3. the systems tries to create the default mapping from the predefined job to the existing templates based on the developer-provided defaults

Configuring the predefined jobs mapping

  1. given I'm the administrator of the Foreman instance

  2. then I can see all the predefined jobs mapping

  3. when I edit existing mapping

  4. then I can choose job name, template and provided input -> template inputs mapping

Configuring the predefined jobs mapping with organizations

  1. given I'm the administrator of the Foreman instance

  2. then I can scope the mapping of the predefined job to a specific organization

  3. and the system doesn't let me to create two mappings for the same predefined job and provider visible in one organization

Using the predefined jobs without provided inputs

  1. given I'm a Foreman user

  2. when I'm on host details page

  3. and I press 'Puppet Run'

  4. the job is invoked on the host based on the predefined mapping

Using the predefined jobs with provided inputs

  1. given I'm a Katello user

  2. when I'm on host applicable errata list

  3. and I select a set of errata to install on the host

  4. and I click 'Install errata'

  5. the job will be invoked to install the packages belonging to this errata

Using the predefined jobs with customization

  1. given I'm a Katello user

  2. when I'm on host applicable errata list

  3. and I select a set of errata to install on the host

  4. and I click 'Install errata (customize)'

  5. then the job invocation form will be opened with pre-filled values based on the mapping

  6. and I can update the values, including setting the start_at time or reoccurring logic

Design

{% plantuml %}

class PredefinedJob { predefined_job_name: string == has_and_belongs_to_many :taxonomies }

class PredefinedJobInputMapping { provided_input_name: string }

PredefinedJob “1” – “N” PredefinedJobInputMapping PredefinedJobInputMapping “N” – “1” ConfigTemplateInput PredefinedJob “M” – “N” JobTemplate note on link red: 1:1 per organization and provider

{% endplantuml %}

Security

User Stories

Scenarios

Allow user A to invoke package installation on host B

  1. given user A can view all hosts and job templates

  2. when he invoke package installation job on host B

  3. then his job task fails because he does not have execution permission for such job task

Allow user A to run package installation on host B

  1. given I've permissions to assign other user permissions

  2. and user A can view all hosts and job templates

  3. and user A can create job invocations

  4. when I grant user A execution permission on resource JobTask

  5. and I set related filter condition to “host_name = B and job_name = package_install”

  6. and user A invokes package install execution on hosts B and C

  7. then the job gets executed successfully on host B

  8. and job execution will fail on host C

User can set effective user

  1. given the provider of job template supports changing effective user

  2. when user invokes a job

  3. then he can set effective user under which job is executed on target host

User can disallow running job as different effective user

  1. given I've permissions to assign other user permissions

  2. and user A can view all hosts and job templates

  3. and user A can create job invocations

  4. when I grant user A execution permission on resource JobTask

  5. and I set related filter condition to “effective_user = user_a”

  6. and user A invokes job execution with effective user set to different user (e.g. root)

  7. then the job execution fails

New permissions introduced

Design

{% plantuml %}

class JobTemplate { effective_user: string }

class JobInvocation { effective_user: string }

{% endplantuml %}

Katello Client Utilities

Design

katello-agent provides three main functions aside from remote management:

Orchestration

User Stories

Design

Design: the whole picture

{% plantuml %} class Host { get_provider(type) }

package “Job Preparation” { class JobTemplate { name: string job_name: string retry_count: integer retry_interval: integer splay: integer provider_type: string effective_user: string == has_and_belongs_to_many :taxonomies has_many :inputs has_many :audits }

class ConfigTemplateInput { name: string required: bool input_type: USER_INPUT | FACT | SMART_VARIABLE fact_name: string smart_variable_name: string description: string == has_one :job_template }

JobTemplate “1” – “N” ConfigTemplateInput }

package “Job Invocation” { class Bookmark { name:string query:string controller:string public:bool owner_id:integer owner_type:string }

class Targeting { query: string dynamic: bool }

class TemplateInvocation { inputs }

class JobInvocation { tries retry_interval splay concurrency email_notification: bool effective_user: string }

class User

Bookmark “1” -DOWN- “N” Targeting Targeting “M” -DOWN- “N” Host Targeting “N” -UP- “1” User JobInvocation “1” -LEFT- “1” Targeting JobInvocation “1” -DOWN- “N” TemplateInvocation TemplateInvocation “N” -LEFT- “1” JobTemplate

}

package “Scheduling” { class Schedule { start_at: datetime end_at: datetime cronline: string }

JobInvocation “1” -UP- “0..1” Schedule }

package “Execution” { class BulkJobTask { state: $TaskState start_at: datetime started_at: datetime ended_at datetime cancel() }

class JobTask { state: $TaskState start_at: datetime started_at: datetime tried_count: integer ended_at datetime retry: integer retry_interval: integer timeout: integer splay: integer concurrency: integer provider: string

command: string
output: string
exit_code: string

{abstract} support_cancel?()
{abstract} proxy_endpoint()
cancel()

}

abstract class ProxyCommand { }

class SSHProxyCommand { {static} support_cancel?() proxy_endpoint():string }

class MCollectiveProxyCommand { {static} support_cancel?() proxy_endpoint():string }

BulkJobTask “N” -LEFT- “1” JobInvocation BulkJobTask “1” – “N” JobTask TemplateInvocation “1” – “N” JobTask JobTask “1” -RIGHT- “1” ProxyCommand JobTask “N” -UP- “1” Host }

ProxyCommand <|– SSHProxyCommand ProxyCommand <|– MCollectiveProxyCommand

package “Developer API” { class PredefinedJob { predefined_job_name: string == has_and_belongs_to_many :taxonomies }

class PredefinedJobInputMapping { provided_input_name: string }

PredefinedJob “1” – “N” PredefinedJobInputMapping PredefinedJobInputMapping “N” -RIGHT- “1” ConfigTemplateInput PredefinedJob “M” -RIGHT- “N” JobTemplate }

{% endplantuml %}

Wireframes

Here are wireframes PDF from 2015-08-14 which we follow where underlaying backends allow us.