In this tutorial you'll get a full tour through Keptn. Before we get started you'll get to know what you will learn while you walk yourself through this tutorial.

What you'll learn

You'll find a time estimate for completing this tutorial in the right top corner of your screen - this should give you guidance how much time is needed for each step.

In this tutorial, we are going to install Keptn on a Kubernetes cluster, along with Istio for traffic routing and ingress control.

The full setup that we are going to deploy is sketched in the following image.
demo setup

Keptn can be installed on a variety of Kubernetes distributions. Please find a full compatibility matrix for supported Kubernetes versions here.

Please find tutorials how to set up your cluster here. For the best tutorial experience, please follow the sizing recommendations given in the tutorials.

Please make sure your environment matches these prerequisites:

Download the Istio command line tool by following the official instructions or by executing the following steps.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.11.2 sh -

Check the version of Istio that has been downloaded and execute the installer from the corresponding folder, e.g.:

./istio-1.11.2/bin/istioctl install

The installation of Istio should be finished within a couple of minutes.

This will install the Istio default profile with ["Istio core" "Istiod" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete

Every release of Keptn provides binaries for the Keptn CLI. These binaries are available for Linux, macOS, and Windows.

There are multiple options how to get the Keptn CLI on your machine.

Now, you should be able to run the Keptn CLI:

To install the latest release of Keptn with full quality gate + continuous delivery capabilities in your Kubernetes cluster, execute the keptn install command.

keptn install --endpoint-service-type=ClusterIP --use-case=continuous-delivery

Installation details

By default Keptn installs into the keptn namespace. Once the installation is complete we can verify the deployments:

kubectl get deployments -n keptn

Here is the output of the command:

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
api-gateway-nginx             1/1     1            1           2m44s
api-service                   1/1     1            1           2m44s
approval-service              1/1     1            1           2m44s
bridge                        1/1     1            1           2m44s
configuration-service         1/1     1            1           2m44s
helm-service                  1/1     1            1           2m44s
jmeter-service                1/1     1            1           2m44s
lighthouse-service            1/1     1            1           2m44s
litmus-service                1/1     1            1           2m44s
mongodb                       1/1     1            1           2m44s
mongodb-datastore             1/1     1            1           2m44s
remediation-service           1/1     1            1           2m44s
shipyard-controller           1/1     1            1           2m44s
statistics-service            1/1     1            1           2m44s

We are using Istio for traffic routing and as an ingress to our cluster. To make the setup experience as smooth as possible we have provided some scripts for your convenience. If you want to run the Istio configuration yourself step by step, please take a look at the Keptn documentation.

The first step for our configuration automation for Istio is downloading the configuration bash script from Github:

curl -o configure-istio.sh https://raw.githubusercontent.com/keptn/examples/release-0.9.0/istio-configuration/configure-istio.sh

After that you need to make the file executable using the chmod command.

chmod +x configure-istio.sh

Finally, let's run the configuration script to automatically create your Ingress resources.

./configure-istio.sh

What is actually created

With this script, you have created an Ingress based on the following manifest.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
  name: api-keptn-ingress
  namespace: keptn
spec:
  rules:
  - host: <IP-ADDRESS>.nip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-gateway-nginx
            port:
              number: 80

Please be aware, when using OpenShift 3.11, instead of using the above manifest, use the following one, as it uses an already deprecated apiVersion.

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
  name: api-keptn-ingress
  namespace: keptn
spec:
  rules:
  - host: <IP-ADDRESS>.nip.io
    http:
      paths:
      - backend:
          serviceName: api-gateway-nginx
          servicePort: 80

In addition, the script has created a gateway resource for you so that the onboarded services are also available publicly.

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      name: http
      number: 80
      protocol: HTTP
    hosts:
    - '*'

Finally, the script restarts the helm-service pod of Keptn to fetch this new configuration.

In this section we are referring to the Linux/MacOS derivatives of the commands. If you are using a Windows host, please follow the official instructions.

First let's extract the information used to access the Keptn installation and store this for later use.

KEPTN_ENDPOINT=http://$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')/api
KEPTN_API_TOKEN=$(kubectl get secret keptn-api-token -n keptn -ojsonpath='{.data.keptn-api-token}' | base64 --decode)
KEPTN_BRIDGE_URL=http://$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')/bridge

Use this stored information and authenticate the CLI.

keptn auth --endpoint=$KEPTN_ENDPOINT --api-token=$KEPTN_API_TOKEN

That will give you:

Starting to authenticate
Successfully authenticated

If you want, you can go ahead and take a look at the Keptn API by navigating to the endpoint that is given via:

echo $KEPTN_ENDPOINT

api

For enabling the Keptn Quality Gates and for production monitoring, we are going to use Dynatrace as the data provider. Therefore, we are going to setup Dynatrace in our Kubernetes cluster to have our sample application monitored and we can use the monitoring data for both the basis for evaluating quality gates as well as a trigger to start self-healing.

If you don't have a Dynatrace tenant yet, sign up for a free trial or a developer account.

To monitor a Kubernetes environment using Dynatrace, please setup dynatrace-operator as described below, or visit the official Dynatrace documentation.

For setting up dynatrace-operator, perform the following steps:

  1. Log into your Dynatrace environment
  2. Open Dynatrace Hub (on the left hand side, scroll down to Manage and click on Hub)
  3. Within Dynatrace Hub, search for Kubernetes
    Dynatrace Hub
  4. Click on Kubernetes, and select Monitor Kubernetes at the bottom of the screen
  5. In the following screen, select the Platform and click on Create tokens to generate PaaS and API tokens.
  6. Select options appropriate for your cluster:

    • By default, most Kubernetes clusters will only offer a self-signed certificate. In such cases, please select Skip SSL Security Check when deploying the Dynatrace OneAgent Operator.
    • When deploying the Dynatrace OneAgent Operator to a cluster running on a Container-Optimized OS (cos), which includes GKE, Anthos, CaaS and PKS environments, please select the Enable volume storage option.
      Dynatrace Kubernetes Monitoring
  7. Copy the generated code and run it in a terminal/bash
  8. Optional: Verify if all pods in the dynatrace namespace are running. It might take up to 1-2 minutes for all pods to be up and running.
    kubectl get pods -n dynatrace

    NAME                                          READY   STATUS    RESTARTS   AGE
dynakube-kubemon-0                            1/1     Running   0          11h
dynatrace-oneagent-operator-cc9856cfd-hrv4x   1/1     Running   0          2d11h
dynatrace-oneagent-webhook-5d67c9bb76-pz2gh   2/2     Running   0          2d11h
dynatrace-operator-fb56f7f59-pf5sg            1/1     Running   0          2d11h
oneagent-gc2lc                                1/1     Running   0          35h
oneagent-w7msm                                1/1     Running   0          35h
    Note: If you are on newer versions of OneAgent / Dynatrace Operator, pods might look as follows:
    NAME                                          READY   STATUS    RESTARTS   AGE
dynakube-classic-d2ckw                        1/1     Running   0          1d13h
dynakube-kubemon-0                            1/1     Running   0          15h
dynakube-routing-0                            1/1     Running   0          23h
dynatrace-operator-fb56f7f59-pf5sg            1/1     Running   0          1d13h
    Note: In case any pods are crashing with CrashLoopBackOff or Error, please double check that you ticked Enable volume storage. Alternatively, please take a look at the official OneAgent troubleshooting guide.
  9. Optional: Verify in your Dynatrace Environment under the section Kubernetes that your cluster is monitored.
  1. Create a Dynatrace API TokenLog in to your Dynatrace tenant and go to Settings > Integration > Dynatrace API. Then, generate a new API token with the following permissions:
    • Access problem and event feed, metrics, and topology
    • Read log content
    • Read configuration
    • Write configuration
    • Capture request data
    • Read metrics
    • Ingest metrics
    • Read entities
    Take a look at this screenshot to double check the right token permissions for you.Dynatrace API V2 TokenDynatrace API V1 Token
  2. Store your credentials in a Keptn-managed secret by executing the following command. The DT_TENANT has to be set according to the appropriate pattern:
    • Dynatrace SaaS tenant (this format is most likely for you): {your-environment-id}.live.dynatrace.com
    • Dynatrace-managed tenant: {your-domain}/e/{your-environment-id}
    If running on a Unix/Linux based system, you can use variables for ease of use. Naturally, it is also fine to just replace the values in the keptn command itself.
    DT_TENANT=yourtenant.live.dynatrace.com
DT_API_TOKEN=yourAPItoken
    If you used the variables, the next command can be copied and pasted without modifications. If you have not set the variables, please make sure to set the right values in the next command.
    keptn create secret dynatrace --from-literal="DT_TENANT=$DT_TENANT" --from-literal="DT_API_TOKEN=$DT_API_TOKEN"
  1. The dynatrace-service integrates Dynatrace into Keptn. The latest version may be installed using the helm chart available in the Releases section of the GitHub project. Please use the same namespace for the dynatrace-service as you are using for Keptn, e.g. keptn:
    helm upgrade --install dynatrace-service -n keptn https://github.com/keptn-contrib/dynatrace-service/releases/download/0.17.1/dynatrace-service-0.17.1.tgz --set dynatraceService.config.keptnApiUrl=$KEPTN_ENDPOINT --set dynatraceService.config.keptnBridgeUrl=$KEPTN_BRIDGE_URL --set dynatraceService.config.generateTaggingRules=true --set dynatraceService.config.generateProblemNotifications=true --set dynatraceService.config.generateManagementZones=true --set dynatraceService.config.generateDashboards=true --set dynatraceService.config.generateMetricEvents=true
  2. Once installed, use the following command to configure Keptn to use Dynatrace for monitoring. This will also set up monitoring in your Dynatrace environment.
    keptn configure monitoring dynatrace
    The output of the command will tell you what has been set up in your Dynatrace environment:
    ID of Keptn context: 79f19c36-b718-4bb6-88d5-cb79f163289b
Dynatrace monitoring setup done.
The following entities have been configured:
    
...
---Problem Notification:--- 
  - Successfully set up Keptn Alerting Profile and Problem Notifications
...

Verify Dynatrace configuration

Since Keptn has configured your Dynatrace tenant, let us take a look what has be done for you:

A project in Keptn is the logical unit that can hold multiple (micro)services. Therefore, it is the starting point for each Keptn installation.

To get all files you need for this tutorial, please clone the example repo to your local machine.

git clone --branch release-0.9.1 https://github.com/keptn/examples.git --single-branch

cd examples/onboarding-carts

Create a new project for your services using the keptn create project command. In this example, the project is called sockshop. Before executing the following command, make sure you are in the examples/onboarding-carts folder.

Recommended: Create a new project with Git upstream:

To configure a Git upstream for this tutorial, the Git user (--git-user), an access token (--git-token), and the remote URL (--git-remote-url) are required. If a requirement is not met, go to the Keptn documentation where instructions for GitHub, GitLab, and Bitbucket are provided.

Let's define the variables before running the command:

GIT_USER=gitusername
GIT_TOKEN=gittoken
GIT_REMOTE_URL=remoteurl

Now let's create the project using the keptn create project command.

keptn create project sockshop --shipyard=./shipyard.yaml --git-user=$GIT_USER --git-token=$GIT_TOKEN --git-remote-url=$GIT_REMOTE_URL

Alternatively: If you don't want to use a Git upstream, you can create a new project without it but please note that this is not the recommended way:

keptn create project sockshop --shipyard=./shipyard.yaml

For creating the project, the tutorial relies on a shipyard.yaml file as shown below:

apiVersion: "spec.keptn.sh/0.2.0"
kind: "Shipyard"
metadata:
  name: "shipyard-sockshop"
spec:
  stages:
    - name: "dev"
      sequences:
        - name: "delivery"
          tasks:
            - name: "deployment"
              properties:
                deploymentstrategy: "direct"
            - name: "test"
              properties:
                teststrategy: "functional"
            - name: "evaluation"
            - name: "release"
        - name: "delivery-direct"
          tasks:
            - name: "deployment"
              properties:
                deploymentstrategy: "direct"
            - name: "release"

    - name: "staging"
      sequences:
        - name: "delivery"
          triggeredOn:
            - event: "dev.delivery.finished"
          tasks:
            - name: "deployment"
              properties:
                deploymentstrategy: "blue_green_service"
            - name: "test"
              properties:
                teststrategy: "performance"
            - name: "evaluation"
            - name: "release"
        - name: "rollback"
          triggeredOn:
            - event: "staging.delivery.finished"
              selector:
                match:
                  result: "fail"
          tasks:
            - name: "rollback"
        - name: "delivery-direct"
          triggeredOn:
            - event: "dev.delivery-direct.finished"
          tasks:
            - name: "deployment"
              properties:
                deploymentstrategy: "direct"
            - name: "release"

    - name: "production"
      sequences:
        - name: "delivery"
          triggeredOn:
            - event: "staging.delivery.finished"
          tasks:
            - name: "deployment"
              properties:
                deploymentstrategy: "blue_green_service"
            - name: "release"
        - name: "rollback"
          triggeredOn:
            - event: "production.delivery.finished"
              selector:
                match:
                  result: "fail"
          tasks:
            - name: "rollback"
        - name: "delivery-direct"
          triggeredOn:
            - event: "staging.delivery-direct.finished"
          tasks:
            - name: "deployment"
              properties:
                deploymentstrategy: "direct"
            - name: "release"

        - name: "remediation"
          triggeredOn:
            - event: "production.remediation.finished"
              selector:
                match:
                  evaluation.result: "fail"
          tasks:
            - name: "get-action"
            - name: "action"
            - name: "evaluation"
              triggeredAfter: "15m"
              properties:
                timeframe: "15m"

This shipyard contains three stages: dev, staging, and production. Later in the tutorial, deployments will be made in three corresponding Kubernetes namespaces: sockshop-dev, sockshop-staging, and sockshop-production.

Let's take a look at the project that we have just created in the Keptn's Bridge. To access it, visit the URL contained in $KEPTN_BRIDGE_URL using the command:

echo $KEPTN_BRIDGE_URL

You can view the Keptn Bridge credentials using the following commands:

echo Username: $(kubectl get secret -n keptn bridge-credentials -o jsonpath="{.data.BASIC_AUTH_USERNAME}" | base64 --decode)
echo Password: $(kubectl get secret -n keptn bridge-credentials -o jsonpath="{.data.BASIC_AUTH_PASSWORD}" | base64 --decode)

You will find the just created project in the bridge with all stages.
bridgebridge

After creating the project, services can be created for our project.

  1. Create the carts service using the keptn create service and keptn add-resourcecommands:
    keptn create service carts --project=sockshop
keptn add-resource --project=sockshop --service=carts --all-stages --resource=./carts.tgz --resourceUri=helm/carts.tgz
  2. After creating the service, tests (i.e., functional- and performance tests) need to be added as basis for quality gates in the different stages:
    • Functional tests for dev stage:
    keptn add-resource --project=sockshop --stage=dev --service=carts --resource=jmeter/basiccheck.jmx --resourceUri=jmeter/basiccheck.jmx
    • Performance tests for staging stage:
    keptn add-resource --project=sockshop --stage=staging --service=carts --resource=jmeter/load.jmx --resourceUri=jmeter/load.jmx
    Note: You can adapt the tests in basiccheck.jmx as well as load.jmx for your service. However, you must not rename the files because there is a hardcoded dependency on these file names in the current implementation of Keptn's jmeter-service.

Since the carts service requires a mongodb database, a second service needs to be created.

Take a look in your Keptn's Bridge and see the newly created services.
bridge services

After creating the services, a built artifact of each service can be deployed.

  1. Deploy the carts-db service by executing the keptn trigger delivery command:
    keptn trigger delivery --project=sockshop --service=carts-db --image=docker.io/mongo --tag=4.2.2 --sequence=delivery-direct
  2. Deploy the carts service by specifying the built artifact, which is stored on DockerHub and tagged with version 0.13.1:
    keptn trigger delivery --project=sockshop --service=carts --image=docker.io/keptnexamples/carts --tag=0.13.1
  3. Go to Keptn's Bridge and check which events have already been generated.
    bridge
  4. Optional: Verify the pods that should have been created for services carts and carts-db:
    kubectl get pods --all-namespaces | grep carts-

    sockshop-dev          carts-77dfdc664b-25b74                            1/1     Running     0          10m
sockshop-dev          carts-db-54d9b6775-lmhf6                          1/1     Running     0          13m
sockshop-production   carts-db-54d9b6775-4hlwn                          2/2     Running     0          12m
sockshop-production   carts-primary-79bcc7c99f-bwdhg                    2/2     Running     0          2m15s
sockshop-staging      carts-db-54d9b6775-rm8rw                          2/2     Running     0          12m
sockshop-staging      carts-primary-79bcc7c99f-mbbgq                    2/2     Running     0          7m24s
  1. Get the URL for your carts service with the following commands in the respective namespaces:
    echo http://carts.sockshop-dev.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')

    echo http://carts.sockshop-staging.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')

    echo http://carts.sockshop-production.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')
  2. Navigate to the URLs to inspect the carts service. In the production namespace, you should receive an output similar to this:

carts in production

Now that the service is running in all three stages, let us generate some traffic so we have some data we can base the evaluation on.

Change the directory to examples/load-generation/cartsloadgen. If you are still in the onboarding-carts directory, use the following command or change it accordingly:

cd ../load-generation/cartsloadgen

Now let us deploy a pod that will generate some traffic for all three stages of our demo environment.

kubectl apply -f deploy/cartsloadgen-base.yaml

The output will look similar to this.

namespace/loadgen created
deployment.extensions/cartsloadgen created

Optionally, you can verify that the load generator has been started.

kubectl get pods -n loadgen

NAME                            READY   STATUS    RESTARTS   AGE
cartsloadgen-5dc47c85cf-kqggb   1/1     Running   0          117s

During the evaluation of a quality gate, the dynatrace-service can be used to fetch the values for the SLIs that are referenced in an SLO configuration. In our example, we are going to customize the SLIs made available to Keptn by adding an SLI configuration file.

Prior to executing the next step, please make sure you are in the correct folder examples/onboarding-carts. If not, please change the directory accordingly, e.g., with cd ../../onboarding-carts/.

Next, add a global SLI configuration file to the project for all services and stages we create.

keptn add-resource --project=sockshop --resource=sli-config-dynatrace.yaml --resourceUri=dynatrace/sli.yaml

For your information, this is what the file looks like:

---
spec_version: '1.0'
indicators:
  throughput: "metricSelector=builtin:service.requestCount.total:merge(\"dt.entity.service\"):sum&entitySelector=type(SERVICE),tag(keptn_project:$PROJECT),tag(keptn_stage:$STAGE),tag(keptn_service:$SERVICE),tag(keptn_deployment:$DEPLOYMENT)"
  error_rate: "metricSelector=builtin:service.errors.total.count:merge(\"dt.entity.service\"):avg&entitySelector=type(SERVICE),tag(keptn_project:$PROJECT),tag(keptn_stage:$STAGE),tag(keptn_service:$SERVICE),tag(keptn_deployment:$DEPLOYMENT)"
  response_time_p50: "metricSelector=builtin:service.response.time:merge(\"dt.entity.service\"):percentile(50)&entitySelector=type(SERVICE),tag(keptn_project:$PROJECT),tag(keptn_stage:$STAGE),tag(keptn_service:$SERVICE),tag(keptn_deployment:$DEPLOYMENT)"
  response_time_p90: "metricSelector=builtin:service.response.time:merge(\"dt.entity.service\"):percentile(90)&entitySelector=type(SERVICE),tag(keptn_project:$PROJECT),tag(keptn_stage:$STAGE),tag(keptn_service:$SERVICE),tag(keptn_deployment:$DEPLOYMENT)"
  response_time_p95: "metricSelector=builtin:service.response.time:merge(\"dt.entity.service\"):percentile(95)&entitySelector=type(SERVICE),tag(keptn_project:$PROJECT),tag(keptn_stage:$STAGE),tag(keptn_service:$SERVICE),tag(keptn_deployment:$DEPLOYMENT)"

Configure the already onboarded project with the new SLI provider for Keptn to create some needed resources (e.g., a configmap):

keptn configure monitoring dynatrace --project=sockshop

Keptn requires a performance specification for the quality gate. This specification is described in a file called slo.yaml, which specifies a Service Level Objective (SLO) that should be met by a service. To learn more about the slo.yaml file, go to Specifications for Site Reliability Engineering with Keptn.

To activate the quality gates for the carts service, navigate to the examples/onboarding-carts folder and upload the slo-quality-gates.yaml file using the add-resource command:

keptn add-resource --project=sockshop --stage=staging --service=carts --resource=slo-quality-gates.yaml --resourceUri=slo.yaml

This will add the SLO.yaml file to your Keptn - which is the declarative definition of a quality gate. Let's take a look at the file contents:

---
spec_version: "1.0"
comparison:
  aggregate_function: "avg"
  compare_with: "single_result"
  include_result_with_score: "pass"
  number_of_comparison_results: 1
filter:
objectives:
  - sli: "response_time_p95"
    key_sli: false
    pass:             # pass if (relative change <= 10% AND absolute value is < 600ms)
      - criteria:
          - "<=+10%"  # relative values require a prefixed sign (plus or minus)
          - "<600"    # absolute values only require a logical operator
    warning:          # if the response time is below 800ms, the result should be a warning
      - criteria:
          - "<=800"
    weight: 1
total_score:
  pass: "90%"
  warning: "75%"

You can take a look at the currently deployed version of our "carts" microservice before we deploy the next build of our microservice.

  1. Get the URL for your carts service with the following commands in the respective stages:
    echo http://carts.sockshop-dev.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')

    echo http://carts.sockshop-staging.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')

    echo http://carts.sockshop-production.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')
  2. Navigate to http://carts.sockshop-production.YOUR.DOMAIN for viewing the carts service in your production environment and you should receive an output similar to the following:

carts service
  1. Use the Keptn CLI to deploy a version of the carts service, which contains an artificial slowdown of 1 second in each request.
    keptn trigger delivery --project=sockshop --service=carts --image=docker.io/keptnexamples/carts --tag=0.13.2
  2. Verify that the slow build has reached your dev and staging environments by opening a browser for both environments. This may take 5 to 10 minutes. Get the URLs with these commands:
    echo http://carts.sockshop-dev.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')

    echo http://carts.sockshop-staging.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')

carts service

carts service

After triggering the deployment of the carts service in version v0.13.2, the following status is expected:

Take a look in the Keptn's bridge and navigate to the last deployment. You will find a quality gate evaluation that got a fail result when evaluation the SLOs of our carts microservice. Thanks to this quality gate the slow build won't be promoted to production but instead automatically rolled back.

To verify, the Keptn's Bridge shows the deployment of v0.13.2 and then the failed test in staging including the roll-back.

Keptn's bridge
  1. Use the Keptn CLI to send a new version of the carts artifact, which does not contain any slowdown:
    keptn trigger delivery --project=sockshop --service=carts --image=docker.io/keptnexamples/carts --tag=0.13.3
  2. To verify the deployment in production (it may take a couple of minutes), open a browser and navigate to the carts service in your production environment. As a result, you see Version: v3.
  3. Besides, you can verify the deployments in your Kubernetes cluster using the following commands:
    kubectl get deployments -n sockshop-production

    NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
carts-db        1         1         1            1           63m
carts-primary   1         1         1            1           98m

    kubectl describe deployment carts-primary -n sockshop-production

    ...
Pod Template:
Labels:  app=carts-primary
Containers:
  carts:
    Image:      docker.io/keptnexamples/carts:0.13.3
  4. Take another look into the Keptn's Bridge and you will see this new version passed the quality gate and thus, is now running in production!

Next, you will learn how to use the capabilities of Keptn to provide self-healing for an application with feature flags based on the Unleash feature toggle framework.

To quickly get an Unleash server up and running with Keptn, follow these instructions:

  1. Make sure you are in the correct folder of your examples directory:
    cd examples/unleash-server
  2. Create a new project using the keptn create project command:
    keptn create project unleash --shipyard=./shipyard.yaml
  3. Create a unleash and unleash-db service using the keptn create service and keptn add-resource commands:
    keptn create service unleash-db --project=unleash
keptn add-resource --project=unleash --service=unleash-db --all-stages --resource=./unleash-db.tgz --resourceUri=helm/unleash-db.tgz

keptn create service unleash --project=unleash
keptn add-resource --project=unleash --service=unleash --all-stages --resource=./unleash.tgz --resourceUri=helm/unleash.tgz
  4. Send new artifacts for unleash and unleash-db using the keptn trigger delivery command:
    keptn trigger delivery --project=unleash --service=unleash-db --image=postgres:10.4 
keptn trigger delivery --project=unleash --service=unleash --image=docker.io/keptnexamples/unleash:1.0.0
  5. Get the URL (unleash.unleash-dev.KEPTN_DOMAIN):
    echo http://unleash.unleash-dev.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}')
  6. Open the URL in your browser and log in using the following credentials:
    • Username: keptn
    • Password: keptn

You should be able to login using the credentials keptn/keptn.

In this tutorial, we are going to introduce feature toggles for two scenarios:

  1. Feature flag for a very simple caching mechanism that can speed up the delivery of the website since it skips the calls to the database but instead replies with static content.
  2. Feature flag for a promotional campaign that can be enabled whenever you want to run a promotional campaign on top of your shopping cart.

To set up both feature flags, please use the following scripts to automatically generate the feature flags that we need in this tutorial.

UNLEASH_TOKEN=$(echo -n keptn:keptn | base64)
UNLEASH_BASE_URL=$(echo http://unleash.unleash-dev.$(kubectl -n keptn get ingress api-keptn-ingress -ojsonpath='{.spec.rules[0].host}'))

curl --request POST \
  --url ${UNLEASH_BASE_URL}/api/admin/features/ \
  --header "authorization: Basic ${UNLEASH_TOKEN}" \
  --header 'content-type: application/json' \
  --data '{
  "name": "EnableItemCache",
  "description": "carts",
  "enabled": false,
	"strategies": [
    {
      "name": "default",
      "parameters": {}
    }
  ]
}'

curl --request POST \
  --url ${UNLEASH_BASE_URL}/api/admin/features/ \
  --header "authorization: Basic ${UNLEASH_TOKEN}" \
  --header 'content-type: application/json' \
  --data '{
  "name": "EnablePromotion",
  "description": "carts",
  "enabled": false,
	"strategies": [
    {
      "name": "default",
      "parameters": {}
    }
  ]
}'

Optionally verify the generated feature flags

If you want to verify the feature flags that have been created, please login to your Unleash server - or if you are already logged in - refresh the browser.

unleash

Now, everything is set up in the Unleash server. For Keptn to be able to connect to the Unleash server, we have to add a secret with the Unleash API URL as well as the Unleash tokens.

  1. In order for Keptn to be able to use the Unleash API, we need to add the credentials as a secret to our Keptn namespace. In this tutorial, we do not have to change the values for UNLEASH_SERVER, UNLEASH_USER, and UNLEASH_TOKEN, but in your own custom scenario this might be needed to change it to your actual Unleash URL, user and token.
    As said, in this tutorial we can use the following command as it is:
    kubectl -n keptn create secret generic unleash --from-literal="UNLEASH_SERVER_URL=http://unleash.unleash-dev/api" --from-literal="UNLEASH_USER=keptn" --from-literal="UNLEASH_TOKEN=keptn"
  2. Install the Unleash action provider which is responsible for acting upon an alert, thus it is the part that will actually resolve issues by changing the stage of the feature flags.
    kubectl apply -f https://raw.githubusercontent.com/keptn-contrib/unleash-service/release-0.3.2/deploy/service.yaml -n keptn
  3. Switch to the carts example (cd examples/onboarding-carts) and add the following remediation instructions
    apiVersion: spec.keptn.sh/0.1.4
kind: Remediation
metadata:
  name: carts-remediation
spec:
  remediations:
    - problemType: Response time degradation
      actionsOnOpen:
        - action: toggle-feature
          name: Toogle feature flag
          description: Toogle feature flag EnableItemCache to ON
          value:
            EnableItemCache: "on"
    - problemType: Failure rate increase
      actionsOnOpen:
        - action: toggle-feature
          name: Toogle feature flag
          description: Toogle feature flag EnablePromotion to OFF
          value:
            EnablePromotion: "off"
    using the following command. Please make sure you are in the correct folder examples/onboarding-carts.
    keptn add-resource --project=sockshop --service=carts --stage=production --resource=remediation_feature_toggle.yaml --resourceUri=remediation.yaml
    Note: The file describes remediation actions (e.g., featuretoggle) in response to problems/alerts (e.g., Response time degradation) that are sent to Keptn.
  4. We are also going to add an SLO file so that Keptn can evaluate if the remediation action was successful.
    keptn add-resource --project=sockshop --stage=production --service=carts --resource=slo-self-healing-dynatrace.yaml --resourceUri=slo.yaml
  5. Start the load generation script for this use case:
    kubectl apply -f ../load-generation/cartsloadgen/deploy/cartsloadgen-prod.yaml

Now that everything is set up and we hit it with some load, next we are going to toggle the feature flags.
  1. In this tutorial, we are going to turn on the promotional campaign, which purpose is to add promotional gifts to about 30 % of the user interactions that put items in their shopping cart.
  2. Click on the toggle next to EnablePromotion to enable this feature flag.enable-toggle
  3. By enabling this feature flag, a not implemented function is called resulting in a NotImplementedFunction error in the source code and a failed response. After a couple of minutes, the monitoring tool will detect an increase in the failure rate and will send out a problem notification to Keptn.
  4. Keptn will receive the problem notification/alert and look for a remediation action that matches this problem. Since we have added the remediation.yaml before, Keptn will find a remediation action and will trigger the corresponding action by reaching out to the action provider that will disable the feature flag.
  5. Finally, take a look into the Keptn's Bridge to see that an open problem has been resolved. You might notice that also the other stages like dev, and staging received the error. The reason is that they all receive the same feature flag configuration and all receive traffic from the load generator. However, for dev and staging there is no remediation.yaml added and thus, no remediation will be performed if problems in this stages are detected. If you want to change this behaviour, go ahead and also add the remediation.yaml file to the other stages by executing another keptn add-resource command. For this tutorial, we are fine by only having self-healing for our production stage!bridge unleash
  6. 10 minutes after Keptn disables the feature flag, Keptn will also trigger another evaluation to make sure the trigger remediation action did actually resolve the problem. In case the problem is not resolved and the remediation file would hold more remediation actions, Keptn would go ahead and trigger them. For our tutorial Keptn has resolved the issue already, so no need for a second try!

Congratulations

Thanks for taking a full tour through Keptn!
Although Keptn has even more to offer that should have given you a good overview what you can do with Keptn.

What we've covered

Keptn can be easily extended with external tools such as notification tools, other SLI providers, bots to interact with Keptn, etc.
While we do not cover additional integrations in this tutorial, please feel fee to take a look at our integration repositories:

Please visit us in our Keptn Slack and tell us how you like Keptn and this tutorial! We are happy to hear your thoughts & suggestions!

Also, make sure to follow us on Twitter to get the latest news on Keptn, our tutorials and newest releases!