Overview
...
The user invoking the install script MUST have permission to the MyProjectNamespace, otherwise the install will fail. When installing a namespace scoped agent, the kubernetes role bindings for the agent will be limited to namespace scoped permissions. This ensures that the agent runs with limited capability inside the kubernetes cluster.
The absence of the --namespace flag indicates a desire to install the agent with cluster wide permissions, which as it is done prior to this design. The agent will have cluster wide permissions, and it will be installed into the openhorizon-agent namespace.
...
A new built-in node property called openhorizon.kubernetesNamespace is introduced, the value reflects the namespace in which the agent is installed. This property is read-only, it is always set by the OH runtime and is not settable by any user role. This property MAY be used in a deployment or policy constraint expression.
...
Deployment:
When publishing a service definition, the operator definition is introspected for namespace definitions. If one is found, the CLI user will receive a warning.
...Talk about the openhorizon.service.kubernetesNamespace built-in property.....if we find a need for it
Deployment:
When an edge cluster service is deployed, by default, it is deployed into the same namespace as the agent/node.
When deploying an edge cluster service, the service deployer MAY write a constraint expression referencing the built-in openhorizon.kubernetesNamespace property in order to limit the placement of the edge service to nodes in a specific namespace or set of namespaces.
When deploying an edge cluster service to cluster scoped nodes, the service deployer needs a way to indicate the target namespace. A new field is added to the service section of a deployment policy, indicating the target namespace for the service's deployment.
"service": { ...
"cluster_namespace": <string>
}
options:
This field is optional and ignored for services deployed to a device or a namespace scoped node <=== seems like it violates the principle of least astonishment.
an edge cluster service is deployed, by default, it is deployed into the same namespace as the agent/node.
When deploying an edge cluster service, the service deployer MAY write a constraint expression referencing the built-in openhorizon.kubernetesNamespace property in order to limit the placement of the edge service to nodes in a specific namespace or set of namespaces. This is the recommended way to target edge cluster services to nodes in a specific namespace (or namespaces).
However, OH currently allows an edge cluster service definition to contain a kubernetes namespace definition (yaml) embedded within the operator definition. The namespace definition indicates the target namespace into which the service should be deployed. There are two problems with this feature. First, it is the wrong placement of function because the namespace in which a service runs is a deployment concern, not an implementation concern. Second, it creates a semantic conflict when the deployer tries to deploy to a namespace scoped node in a different namespace. The use of this feature is not recommended and MAY be deprecated in a future version of OH.
To address both concerns, the deployer needs a way to explicitly indicate the target namespace of a deployment.
A new field is added to the service section of a deployment policy, indicating the target namespace for the deployment. This namespace overrides a namespace definition in the operator definition of an edge cluster service.
"service": { ...
"clusterNamespace": <string>
}
This field is optional and ignored for services deployed to a device. If a deployment policy constraint expression chooses a namespace scoped node as a deployment target
For cluster scoped nodes, this field defines the namespace where the deployment occurs, effectively overriding an embedded namespace definition in the service.
For namespace scoped nodes, this field acts as a built-in constraint that causes namespace scoped nodes in namespaces other than the one specified by this field to be eliminated as deployment targets. The deploycheck CLI MUST detect this case. <=== there is no way for this builtEffectively, this field acts as a built-in constraint to become an OR condition, it is essentially an AND tacked onto all ANDed to the user specified constraints, therefore this might force the user to create additional deployment policies.
The OH cluster scoped agent already allows an edge cluster service definition to contain a kubernetes namespace definition (yaml) embedded within the operator definition. The namespace definition indicates the target namespace into which the service should be deployed. There are three problems with this feature. First, it is the wrong placement of function because the namespace in which a service runs is a deployment concern, not an implementation concern. Second, it creates a semantic conflict when the deployer tries to deploy to a namespace scoped node in a different namespace. Third, it introduces a similar semantic conflict if the deployer specifies "cluster_namespace" in the deployment policy.
options:
The first problem is solved by the introduction of the "cluster_namespace" field in the deployment policy. This field allows deployers to have control of the target namespace, especially when the deployer is dealing primarily with cluster scoped nodes.
The second problem can be solved by having the embedded namespace definition act as a built-in constraint which eliminates namespace scoped nodes with a different namespace. The deploycheck CLI MUST detect this case.
The third problem is detected by the deployment publish CLI and returns an error to the user. <=== violates separation of concerns between service developer and deployer by not allowing the deployer to have control of a deployment concept, the deployer would have to ask the developer to change the operator definition in the service def before the service can be deployed.
The third problem is handled by treating the "cluster_namespace" as an override of the operator definition.
----------------------------------------------------------------------------------------------------------------------------------
A namespace specified in the deployment policy overrides any namespace defined in the operator definition.
The Agbot calculates the target namespace of a cluster based service as follows:
- If present, use the namespace in the deployment policy.
- If present, use the namespace in the service definition.
- Use openhorizon-agent namespace (this is the default namespace where the cluster scoped agent is installed).
Once the Agbot has calculated the target namespace it:
- Uses this namespace as a built-in constraint when searching for deployment targets (nodes) that are not in the openhorizon-agent namespace.
- Ignores this namespace for nodes in the openhorizon-agent namespace (these nodes are assumed to have cluster scope permissions and are therefore valid targets for services in any namespace. That is, there is no built-in constraint on deployments for nodes in the openhorizon-agent namespace.
- Includes this namespace as built-in service property (openhorizon.service.kubernetesNamespace), so that the node owner can create constraint expressions referring to the target namespace of a service.
Note: The node owner is always free to configure a deployment constraint expression that limits the namespace
Patterns:
A new field is added to the schema of a pattern (as a top level field in the schema), indicating the target namespace for the pattern's deployment.
"namespace": <string>The namespace field is optional and ignored for patterns deployed to a device.
A namespace specified in the pattern overrides any namespace defined in the operator definition of all services in the pattern.
A pattern is in error if it attempts to deploy services to a namespace scoped node where the collection of services in the pattern are NOT deployable to the same namespace. Clearly this can only happen when the namespace is NOT specified in the pattern definition but is contained within the operator definition.
Terminology:
Cluster scoped agent - An OH agent installed in an edge cluster node where the agent has permission to deploy services into any namespace.
Namespace scoped agent - An OH agent installed in an edge cluster where the agent has permission to deploy services into ONLY the namespace where it is installed.
DevOps user - a conflation of roles found in the practice of DevOps; e.g. service developer, or service deployer.
Usage scenarios:
As a DevOps user, I want to install the OH agent into one or more namespaces that I have permission to use for my project.
As a service deployer, I want to select the namespace into which a service is deployed, for both cluster scoped and namespace scoped agents.
As a service deployer, I want to use a node's namespace as (one of) the criteria for selecting deployment targets.
As a service deployer, I want to know if my deployment policy will deploy to a namespace or cluster scoped node.
As a node owner, I want OH ensure that DevOps teams using my edge cluster are isolated from each other, based on the namespace(s) I have given to each team.
There are no new commands or verbs introduced by this design, but the behavior of some existing CLI commands is changed to accommodate this feature.
constraint expression. The deploycheck CLI MUST detect this case.
In the absence of the "clusterNamespace" field, a namespace definition embedded in the operator definition acts as a built-in constraint that causes namespace scoped nodes in namespaces other than embedded namespace definition to be eliminated as deployment targets. Effectively, the embedded namespace definition acts as a built-in constraint ANDed to the user specified constraint expression. The deploycheck CLI MUST detect this case.
There could be a semantic conflict if the deployer specifies a "clusterNamespace" and uses openhorizon.kubernetesNamespace in a constraint expression. It is likely that such a policy will result in no deployments, which would be the technically correct behavior but it might also be surprising to the user. This is why the hzn deploycheck command exists, to help the user understand semantic mismatches in deployment policy. The deploycheck command MUST detect whether or not there is a real semantic conflict in this case.
Deployment Examples:
Here are some examples that illustrate deployment outcomes resulting from the behavior described above. The examples show that the design is compatible with existing OH behavior and also enables control of the target namespace for the deployer.
- An edge cluster node with cluster privileges is chosen as a deployment target by the policy's constraint expression. The deployment policy has no
"clusterNamespace"defined and no embedded namespace definition. The edge service is deployed in the openhorizon-agent namespace. - An edge cluster node with cluster privileges is chosen as a deployment target by the policy's constraint expression. The deployment policy has
"clusterNamespace":"ABC". The edge service is deployed in the ABC namespace. - An edge cluster node with cluster privileges is chosen as a deployment target. The deployment policy has no
"clusterNamespace"defined. The edge service has an embedded namespace definition for XYZ. The edge service is deployed in the XYZ namespace. - An edge cluster node with cluster privileges is chosen as a deployment target. The deployment policy has
"clusterNamespace":"ABC"defined. The edge service has an embedded namespace definition for XYZ. The edge service is deployed in the ABC namespace.
And finally, some additional examples for namespace scoped nodes.
- A node in namespace ABC. The deployment policy has
"clusterNamespace":"ABC". The edge service is deployed to the node. - A node in namespace ABC. The deployment policy has no
"clusterNamespace". The edge service is deployed to the node. - A node in namespace ABC. The deployment policy has no
"clusterNamespace". The deployment policy has a constraint expression openhorizon.kubernetesNamespace=XYZ. The edge service is NOT deployed to the node. - A node in namespace ABC. The deployment policy has
"clusterNamespace":"ABC". The deployment policy has a constraint expression openhorizon.kubernetesNamespace=XYZ. The edge service is NOT deployed to the node. The user received a warning when publishing the policy that the policy might result in no service deployments.
Pattern:
A new field is added to the schema of a pattern (as a top level field in the schema), indicating the target namespace for the pattern's deployment.
"clusterNamespace": <string>The "clusterNamespace" field is optional and ignored for patterns deployed to a device. Namespace scoped nodes not in the specified namespace are not eligible to deploy the pattern. Cluster scoped nodes only deploy patterns that have an empty "clusterNamespace" field. A pattern with an empty "clusterNamespace" MUST NOT be deployed to a namespace scoped node.
A namespace specified in the pattern overrides any namespace defined in the operator definition of all services in the pattern.
A pattern is in error if it attempts to deploy services to a namespace scoped node where the collection of services in the pattern are NOT deployable to the same namespace. Clearly this can only happen when the "clusterNamespace" is specified in the pattern definition and one of the services contains an embedded namespace definition.
Node:
The node resource is also updated to contain the cluster namespace so that it is displayed in the node related CLI commands.
Terminology:
Cluster scoped agent - An OH agent installed in an edge cluster node where the agent has permission to deploy services into any namespace.
Namespace scoped agent - An OH agent installed in an edge cluster where the agent has permission to deploy services into ONLY the namespace where it is installed.
DevOps user - a conflation of roles found in the practice of DevOps; e.g. service developer, or service deployer.
Usage scenarios:
As a DevOps user, I want to install the OH agent into one or more namespaces that I have permission to use for my project.
As a service deployer, I want to select the namespace into which a service is deployed, for both cluster scoped and namespace scoped agents.
As a service deployer, I want to use a node's namespace as (one of) the criteria for selecting deployment targets.
As a service deployer, I want to know if my deployment policy will deploy to a namespace or cluster scoped node.
There are no new commands or verbs introduced by this design, but the behavior of some existing CLI commands is changed to accommodate this feature.
hzn exchange deployment addPolicy -f <policy_definition>
If the policy specifies "clusterNamespace" and the property openhorizon.kubernetesNamespace is present in the constraint expression, a warning is provided telling the user that this deployment policy may result in no service deployment. The user will be directed to use the hzn deploycheck command to verify whether or not a deployment will result.
hzn deploycheck
There are several situations outlined in the design above where deployment may or may not occur as expected, depending on the depth of knowledge of the user. The hzn deploycheck command is enhanced to identify:
- a semantic conflict if the deployer specifies a
"clusterNamespace"and uses openhorizon.kubernetesNamespace in a constraint expression - deployment target selection (or not) in the absence of the
"clusterNamespace"field, and a namespace definition embedded in the operator definition - deployment target selection (or not) in the presence of the
"clusterNamespace"field, and a namespace definition embedded in the operator definition - correct deployment target selection as defined in the Deployment section above
hzn exchange service publish
If the service definition contains an embedded namespace definition, provide a warning to the user that due to the use of an embedded namespace definition they are not following best practices and therefore the service might not be deployable. Use the hzn deploycheck command before this service is deployed to ensure that the policy will select nodes as expected.
hzn exchange pattern...
...
When a namespace scoped agent is installed, it is given kubernetes privileges within its specific namespace but not beyond that scope.
...
The Exchange API is updated to support the new namespace clusterNamespace field in a deployment policy resourceresource.
The Exchange is unaware of node, service or deployment properties within its schema definitions.
...
- Authoring edge cluster services: Add a note that packaging a namespace definition inside an operator definition is not considered a best practice. Service developers should allow deployers to choose the target namespace in the deployment policy.
- Deploying edge cluster services: Document the new namespace clusterNamespace field in the deployment policy, node and pattern. Describe how it's used and how it plays into the algorithm used by the Agbot to determine where edge cluster services are placed.
- Policy: Document the new built-in property openhorizon.kubernetesNamespace for edge node policies.
- Installing edge cluster agent: Document the new --namespace flag in the agent-install script.
...