Skip to content

Latest commit

 

History

History
484 lines (385 loc) · 15.1 KB

README.org

File metadata and controls

484 lines (385 loc) · 15.1 KB

Kubernetes Operators

Topics

Custom Resource Definition

./pic/kubernetes_operators.jpg

  1. User creates CustomResourceDefinition named EtcdCluster - this adds new API Endpoint in k8s cluster.
  2. Create EtcdOperator which reacts to objects on new API Server endpoint.
  3. Create Custom Resource (CR) - unique instance of EtcdCluster.
  4. EtcdOperator reacts on new object awesomeETCD and spawns pods in k8s according to definition in CR.

Kubernetes Custom Resource Definition

  • A Resource - is an endpoint that stores a collection of API objects of a certain kind For example, the built-in pods resource contains a collection of Pod objects.
  • A Custom Resource - is an object that extends the Kubernetes API. It allows you to introduce your own API into a project or a cluster.
  • A Custom Resource Definitions (CRD) - is a file that describes your own object kinds and lets the Kubernetes API server handle the entire lifecycle. Deploying a CRD into the cluster causes the Kubernetes API server to begin serving the specified custom resource.

Example:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct

Deploy etcd cluster

We will now try to deploy etcd cluster into our Kubernetes. We will use etcd operator for it. As we want to do it properly we want our operator to be managed by a Lifecycle Operator. Lifecycle Operator enables management of operators inside Kuberentes clusters. It’s mostly responsible for:

  • Deploying Operators into namespaces
  • Updating Operators
  • Defines channels for operators delivery (Testing/Prod)
  • Package Operators
  • Operators inventory

You can read more about Operator Lifecycle Manager.

Deploying Operator Lifecycle Manager

git clone https://github.com/prgcont/operator-lifecycle-manager
kubectl create -f operator-lifecycle-manager/deploy/upstream/manifests/0.7.1

Check that Operator Lifecycle Manager is up and running:

kubectl -n olm get pods

You should see output similar to:

NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-6b999d5844-smtqg   1/1     Running   0          9m35s
olm-operator-b757b5f49-j57rs        1/1     Running   0          9m35s
package-server-88cdfb8f8-w5h6n      1/1     Running   0          9m35s

Deploying etcd operator

When Lifecycle Manager Operator is successfully deployed and is managing our cluster we can deploy etcd operator to a newly created etc namespace.

kubectl create namespace etc

Note: If you want to operate on etc namespace you need to pass namespace option to kubectl so it will look like kubectl -n etc get pods. Note: If you are running on shared cluster, change etc namespace to your <CUSTOM_NAMESPACE>.

Deploy operator here by applying InstallPlan, this will assure that etcd operator is deployed to our etc namespaces and is watch its crd object.

apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  namespace: etc
  name: etcd
spec:
  clusterServiceVersionNames:
  - etcdoperator.v0.9.2
  approval: Automatic

Tasks:

  • Check that etcd operator is deployed
  • List all available crd

Install etcd Cluster using etcd Operator

To install etcd cluster we need to apply following CRD object.

apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
  name: "example-etcd-cluster"
spec:
  size: 3
  version: "3.2.13"
  repository: "docker.io/prgcont/etcd"

Verify the state of deployed etcd cluster

kubectl -n etc  describe etcdcluster example-etcd-cluster

Tasks

  • Check that etcdclusters.etcd.database.coreos.com CRD is available

Check the health of etcd Cluster

Exec into one etcd pod

# Get arbitrary pod name using
kubectl -n etc get po -l etcd_cluster=example-etcd-cluster

# Exec into etcd pod
kubectl -n etc exec -it <POD_NAME> -- sh

# In container:
# Update env variable
export ETCDCTL_API=3

# List etcd members
etcdctl member list

# Write and read record
etcdctl put /here test
etcdctl get /here

Tasks:

  • Scale up Currently deployed etcd cluster and verify that record you made into the DB still exists
  • Deploy second etcd cluster in ‘etc2’ namespace
  • Check that both clusters are independent (contains different data)

Note on Cluster wide operators

Note: This can lead to security issues and render you cluster to be hard to maintain

The above example created etcd-operator and etcd Cluster in same namespace (etc). By default etcd Operator reacts only on etcdcluster objects that are in same namespace. This behavior can be changed by passing arg -cluster-wide to etcd-operator and creating etcdcluster object with annotation: etcd.database.coreos.com/scope: clusterwide.

From our example:

apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
  name: "example-etcd-cluster"
  annotations:
    etcd.database.coreos.com/scope: clusterwide
spec:
  size: 3
  version: "3.2.13"
  repository: "docker.io/prgcont/etcd"

Note: You need to update RBAC rules if you want etcd operator to manage resources across all kubernetes cluster.

Operator Framework

Operator Framework is set of tools that simplifies creation management of k8s operators.

The operators created by Operator Framework are using same primitives like k8s controller which can be found in this diagram:

./pic/operator_sdk_internals.jpeg

This framework is really good choice if you are golang developer or your applications stack is golang based, its benefit for other apps maybe is not good enough to learn is as operator can be created in almost any language and it can still be managed by Lifecycle Operator Manager.

Operator SDK helps you a lot with:

  • generating CRD for you
  • monitoring changes in CRD/Kubernetes cluster (you can register watchers and handlers easily)
  • package and deploy operator into cluster

Advance task: