Deploy Pulse-IT to AWS EKS

1. Requirements:

1.1 EKS cluster:

• Version: 1.29
• Following add-ons installed and service account configured:
  • EBS (doc)
  • AWS LoadBalancer controller (doc)
  • Metrics server enabled (steps in README.md)
  • AWS Secret Store (doc)
  • AWS fluent-bit for cloudwatch installed (steps in README.md)
  • AWS EFS CSI (doc)
  • ExternalsDns (doc)

These Add-ons can be installed via terraform. Refer to provided terraform scripts

1.2 AWS Services:

• Postgres 12.x from RDS Database
• RabbitMQ 3.12.x from Amazon MQ
• Redis 6.2. from Amazon ElastiCache

1.3 Secrets stored on AWS Secrets manager:

Store the following credentials in AWS Secrets Manager:

• Postgres username and password
• RabbitMQ username and password
• Pulse-IT superuser username and password

1.4 Kubernetes tools

• kubectl configured to connect with the EKS Cluster
• Helm

1.5 Embrace credentials for DockerHub saved on cluster secret

The Embrace images are kept in a private repository. To retrieve the images, you'll need to store the Embrace repository credentials in a Kubernetes secret. For detailed instructions, please consult the official Kubernetes documentation

kubectl create secret docker-registry emb-registry-cred  --docker-server=https://index.docker.io/v1/  --docker-username=embracecustomer  --docker-password=<password> -n embrace

1.6 A registered domain name on AWS Route 53

See official AWS Route 53 documentation.

2. Deploy Pulse-IT with Helm

2.1 Edit values to match with your environment:

• Create a new file named values.local.yml next to the Pulse-IT Chart

touch values.local.yaml

• Copy these minimal configurations into values.local.yaml by Replacing <variable> to match with your environment values

# values.local.yaml
imagePullSecrets:
- name: emb-registry-cred

database:
  host: "<rds-name>.<rds-id>.<aws-region>.rds.amazonaws.com"
  port: "<rds-postgres-port>"
  name: "<rds-postgres-database-name>"

rabbitmq:
  host: "<amq-id>.mq.<aws-region>.amazonaws.com"
  port: "<amq-rabbitmq-port>"
  useSSL: true

redis:
  host: "<xxxx>.cache.amazonaws.com"

embApp:
  serviceAccount:
    create: true
    annotations: {
      eks.amazonaws.com/role-arn: arn:aws:iam::<awsAccountId>:role/<AWSEksSecretCSIRole>
    }

ingress:
  annotations: {
    alb.ingress.kubernetes.io/scheme: internet-facing,
    alb.ingress.kubernetes.io/target-type: ip,
    alb.ingress.kubernetes.io/load-balancer-name: <aws-load-balancer-name>,
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]',
    alb.ingress.kubernetes.io/ssl-redirect: "443",
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:<aws-region>:<aws-account>:certificate/<ssl-certificate-id>
  }

conf:
  protocol: "https"
  hostName: "<application-hostname.com>"
  kubernetesHost: "https://<EKS-CLUSTER-ID>.<xx>.<aws-region>.eks.amazonaws.com"

secrets:
  database:
    name: <RDS-secret-key>
  su:
    name: <application-super-user-secret-key>
  rabbitmq:
    name: <AMQ-rabbitmq-secret-key>

storage:
  shared:
    enabled: true
    volumeName: 'efs-emb-app'
    className: efs-sc
    claimName: emb-efs-pvc

    containerMountPath: /mnt/efs-store

    provisioner: efs.csi.aws.com
    fileSystemId: <efs-file-system-id>
    provisioningMode: efs-ap

    accessMode: ReadWriteMany
    size: <efs-storage-size> # ex: 1Gi

2.2 Deploy Pulse-IT using Helm

helm install <my-release> \
    oci://registry-1.docker.io/embracemedia/pulse_it_chart \
    --version 3.2.1906 \
    --values=values.local.yaml \
    -n embrace

< my-release > can be replaced by "pulse-it" as release name for example.

Replace 3.2.1906 by latest stable Pulse-IT release.

2.3 Verify installation:

After running helm install, Pulse-IT components should be created on the Kubernetes cluster. These components can be listed using this command:

kubectl get all -n embrace

Output:

NAME                                     READY   STATUS    RESTARTS      AGE
pod/emb-app-79666d6bdc-xjt9q             2/2     Running   0             25h
pod/emb-celery-beat-7776d9844-8jnd9      1/1     Running   0             25h
pod/emb-celery-worker-5c56574788-bzn6z   1/1     Running   0             25h
pod/emb-dbg-7b697c686f-v2s6g             1/1     Running   0             25h

NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)                      AGE
service/emb-app-svc        LoadBalancer   172.20.108.85   sandbox-embrace-server-lb-5495603babb2e259.elb.eu-west-3.amazonaws.com   80:31993/TCP,443:30370/TCP   34d
service/emb-debugger-svc   ClusterIP      172.20.11.99    <none>                                                                   6000/TCP                     34d

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emb-app             1/1     1            1           34d
deployment.apps/emb-celery-beat     1/1     1            1           34d
deployment.apps/emb-celery-worker   1/1     1            1           34d
deployment.apps/emb-dbg             1/1     1            1           34d

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/emb-app-5cd46c85c9             1         1         1       13d
...
replicaset.apps/emb-celery-beat-54498c9d84     1         1         1       12d
...
replicaset.apps/emb-celery-worker-54995dfb57   1         1         1       25h
...
replicaset.apps/emb-dbg-56bc5dd44              1         1         1       25h
...

2.4 Access Pulse-IT on your browser

Access Pulse-IT using hostname you've specified in the value.local.yaml file.

!! It may take a few minutes for Pulse-IT to become accessible due to DNS updates.

3. Worker Nodes

3.1 Deploy Worker Node's requirements Charts

3.1.1 Edit values to match with your environment:

• Create a new file named values.local.yml next to the Worker Node Chart

touch values.local.yaml

• Copy these minimal configurations to created file by replacing <variable> to match with your environment values

# values.local.yaml
conf:
  debug: False
  logLevel: Info

rabbitmq:
  host: "<amq-rabbitmq-host>"
  port: "<amq-rabbitmq-port>"
  useSSL: true

secrets:
  su:
    name: <application-super-user-secret-key>
  rabbitmq:
    name: <AMQ-rabbitmq-secret-key>

3.1.2 Deploy Worker Nodes requirements using Helm

helm install <my-release> \
    oci://registry-1.docker.io/embracemedia/pulse_it_worker_chart \
    --version 3.2.701 \
    --values=values.local.yaml \
    -n embrace

< my-release > can be replaced by "pulse-it-worker" as release name for example.

Replace 3.2.701 by latest stable Pulse-IT worker release.

3.2 Create Worker Node from Pulse-IT

• Access the Pulse-IT interface and navigate to the "Monitor" menu, then select "Nodes".
• Click on the "+" button
• Modify the value of the "Number of nodes" field to specify the desired quantity of nodes to be created.
• If desired, you can add a "Display name prefix".
• Initiate the creation of nodes by clicking on the validation button labeled "New nodes".

4.Upgrade Pulse-IT

4.1 Activate maintenance mode:

• Open the Pulse-IT interface and go to the "Monitor" menu, then choose "Nodes".
• Click on the Pulse-IT Node in the diagram.
• In the right panel, locate the "Maintenance" field and click on the "Deactivate" button.
• In the modal that appears, click on "Stop all workflows and activate".

4.2 Upgrade Pulse-IT

4.2.1. Retrieve last values.local.yaml

If you can't see the old file, create a new one by consulting section 2.2

4.2.2 Upgrade Pulse-IT using Helm upgrade

helm upgrade  <my-release> \
    oci://registry-1.docker.io/embracemedia/pulse_it_chart \
    --version 3.2.1906 \
    --values=values.local.yaml \
    -n embrace

< my-release > can be replaced by "pulse-it" as release name for example.

Replace 3.2.1906 by latest statble Pulse-IT release.

4.3 Upgrade Worker Nodes

4.3.1 Delete old Worker Nodes

• Open the Pulse-IT interface and go to the "Monitor" menu, then choose "Nodes".
• Select all nodes and click on the "trash" button

4.3.2 Create new nodes

Refer to section 3.2

4.4 Deactivate maintenance mode

• Open the Pulse-IT interface and go to the "Monitor" menu, then choose "Nodes".
• Click on the Pulse-IT Node in the diagram.
• In the right panel, locate the "Maintenance" field and click on the "Activated" button.

5. Troubleshooting

5.1 Pod's state is ImagePullBackOff:

1- Verify whether any secret containing Embrace credentials have been created within the embrace namespace.

Command:

kubectl get secret -n embrace

Output:

NAME                TYPE                             DATA   AGE
emb-registry-cred   kubernetes.io/dockerconfigjson   1      3s

2- Ensure that the provided repository credentials are correct.

• Delete existing secret

kubectl delete secret emb-registry-cred -n embrace

• Recreate secret: Refer to section 1.5 to recreate secret
• Deploy Pulse-IT using helm (see section 2.2)

5.2 Pulse-IT interface is not accessible

5.2.1 Review the applications logs.

Inspect the logs of every Pulse-IT component to identify errors.

- Review emb-app logs

kubectl logs -n embrace -l component=emb-app

- Review celery beat logs

kubectl logs -n embrace -l component=emb-celery-beat

- Review celery worker logs

kubectl logs -n embrace -l component=emb-celery-worker

5.3 Pulse-IT workers state icon is red

Inspect the logs of Pulse-IT worker container

kubectl logs -n embrace <pulse-it-worker-name> -c workflow-player

6. Utils commands:

Apply to LDAP/Active Directory configuration

kubectl rollout restart deployment emb-app emb-celery-worker -n embrace

Manually scale emb-app deployment

kubectl scale deployment emb-app --replicas=2 -n embrace

--replicas=2 can be replaced by desired number of replicas

Get logs for aws-load-balancer-controller pods

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

Get logs for karpenter pods

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter