Skip to main content
Version: 4.2

Deploy Pulse-IT to AWS EKS

1. Requirements:​

1.1 EKS cluster:​

  • Version: 1.31
  • Following add-ons installed and service account configured:
    • EBS (doc)
    • AWS LoadBalancer controller (doc)
    • Metrics server enabled (steps in README.md)
    • AWS Secret Store (doc)
    • AWS fluent-bit for cloudwatch installed (steps in README.md)
    • AWS EFS CSI (doc)
    • ExternalsDns (doc)
    • Karpenter

These Add-ons can be installed via terraform. Refer to provided terraform scripts

1.2 AWS Services:​

  • Postgres 16.x from RDS Database
  • RabbitMQ 3.13.x from Amazon MQ
  • Redis 6.2. from Amazon ElastiCache

1.3 Secrets stored on AWS Secrets manager:​

Store the following credentials in AWS Secrets Manager:

  • Postgres username and password
  • RabbitMQ username and password
  • Pulse-IT superuser username and password

1.4 Kubernetes tools​

  • kubectl configured to connect with the EKS Cluster
  • Helm

1.5 Embrace credentials for DockerHub saved on cluster secret​

The Embrace images are kept in a private repository. To retrieve the images, you'll need to store the Embrace repository credentials in a Kubernetes secret. For detailed instructions, please consult the official Kubernetes documentation

kubectl create secret docker-registry emb-registry-cred  --docker-server=https://index.docker.io/v1/  --docker-username=embracecustomer  --docker-password=<password> -n embrace

1.6 A registered domain name on AWS Route 53​

See official AWS Route 53 documentation.

2. Deploy Pulse-IT with Helm​

2.1 Edit values to match with your environment:​

  • Create a new file named values.local.yml next to the Pulse-IT Chart
touch values.<my_env>.yaml

Replace <my_env> with an environment name of your choice

  • Copy these minimal configurations into values.local.yaml by Replacing <...> to match with your environment values
# values.<my_env>.yaml
imagePullSecrets:
- name: emb-registry-cred

kubernetesHostType: "eks"

database:
  host: "<rds-instance-url>.amazonaws.com"
  port: "5432"
  name: "ait_factory"

rabbitmq:
  host: "<rabbitmq-endpoint>"
  port: "5671"
  useSSL: true

redis:
  host: "<elasticache-redis-master-endpoint>"
  port: "6379"

embApp:
  serviceAccount:
    create: true
    user: <AmazonEKSSecretCSIRoleIAM>
    annotations: {
      eks.amazonaws.com/role-arn: <AmazonEKSSecretCSIRoleIAM>
    }
  service:
    serviceAnnotations: {
      alb.ingress.kubernetes.io/healthcheck-path: /workflow_template/test
    }


embAppNginx:
  serviceNginx:
    serviceAnnotations: {
      alb.ingress.kubernetes.io/healthcheck-path: /static/img/pulseit/pulseit_logo.png
    }

ingress:
  enabled: true
  className: "alb"
  annotations: {
    alb.ingress.kubernetes.io/scheme: internet-facing,
    alb.ingress.kubernetes.io/target-type: ip,
    alb.ingress.kubernetes.io/load-balancer-name: <aws-alb-name>,
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]',
    alb.ingress.kubernetes.io/ssl-redirect: "443",
    alb.ingress.kubernetes.io/certificate-arn: <acm-certificate-arn>
  }

conf:
  debug: "False"
  logLevel: "INFO"
  protocol: "https"
  hostName: "<pulse-it-hostname>"
  kubernetesHost: "<eks-API-server-endpoint>"
  kubernetesCluster: "<EKS-Cluster-Name>"
  timeZone: "Europe/Paris"
  WebSocketTimeout: "60"
  NodeMaxDayLogs: "6"
  ChannelLayerCapacity: "10000"
  ChannelLayerHttpRequestCapacity: "5000"
  ChannelLayerWebSocketSendCapacity: "10000"
  PlannedWorkflowProcessorsCount: "4"
  WorkflowEventsProcessorsCount: "4"
  sslEnabled: "True"

secrets:
  provider: external
  providerType: aws

  external:
    database:
      name: <RDS-secret-key-name>
    su:
      name: <application-super-user-secret-key-name>
    rabbitmq:
      name: <AMQ-rabbitmq-secret-key>

storage:
  enabled: true

  shared_storages:
    - volumeName: efs-emb-app
      claimName: emb-efs-pvc
      containerMountPath: /mnt/efs-store

efs_storage:
  enabled: true
  className: efs-sc
  claimName:  emb-efs-pvc
  provisioner: efs.csi.aws.com
  fileSystemId: <efs-file-system-id>
  provisioningMode: efs-ap
  accessMode: ReadWriteMany
  size: <efs-storage-size>

2.2 Deploy Pulse-IT using Helm​

helm install <my-release> \
    oci://registry-1.docker.io/embracemedia/pulse_it_chart \
    --version 4.2.2306 \
    --values=values.local.yaml \
    -n embrace

<my-release> can be replaced by "pulse-it" as release name for example.

Replace 4.2.2306 by latest stable Pulse-IT release.

2.3 Verify installation:​

After running helm install, Pulse-IT components should be created on the Kubernetes cluster. These components can be listed using this command:

kubectl get all -n embrace

Output:

NAME                                     READY   STATUS    RESTARTS      AGE
pod/emb-app-79666d6bdc-xjt9q             2/2     Running   0             25h
pod/emb-celery-beat-7776d9844-8jnd9      1/1     Running   0             25h
pod/emb-celery-worker-5c56574788-bzn6z   1/1     Running   0             25h
pod/emb-dbg-7b697c686f-v2s6g             1/1     Running   0             25h

NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)                      AGE
service/emb-app-svc        LoadBalancer   172.20.108.85   sandbox-embrace-server-lb-5495603babb2e259.elb.eu-west-3.amazonaws.com   80:31993/TCP,443:30370/TCP   34d
service/emb-debugger-svc   ClusterIP      172.20.11.99    <none>                                                                   6000/TCP                     34d

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emb-app             1/1     1            1           34d
deployment.apps/emb-celery-beat     1/1     1            1           34d
deployment.apps/emb-celery-worker   1/1     1            1           34d
deployment.apps/emb-dbg             1/1     1            1           34d

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/emb-app-5cd46c85c9             1         1         1       13d
...
replicaset.apps/emb-celery-beat-54498c9d84     1         1         1       12d
...
replicaset.apps/emb-celery-worker-54995dfb57   1         1         1       25h
...
replicaset.apps/emb-dbg-56bc5dd44              1         1         1       25h
...

2.4 Access Pulse-IT on your browser​

Access Pulse-IT using hostname you've specified in the value.local.yaml file.

!! It may take a few minutes for Pulse-IT to become accessible due to DNS updates.

3. Worker Nodes​

3.1 Deploy Worker Node's requirements Charts​

3.1.1 Edit values to match with your environment:​

  • Create a new file named values.local.yml next to the Worker Node Chart
touch values.<my_env>.yaml

Replace <my_env> with an environment name of your choice

  • Copy these minimal configurations to created file by replacing <...> to match with your environment values
# values.<my_env>.yaml
conf:
  debug: False
  logLevel: INFO
  timeZone: Europe/Paris

rabbitmq:
  host: "<rabbitmq-endpoint>"
  port: "5671"
  useSSL: true

secrets:
  provider: external # or kubernetes

  external:
    su:
      name: <application-super-user-secret-key-name>
    rabbitmq:
      name: <AMQ-rabbitmq-secret-key>

3.1.2 Deploy Worker Nodes requirements using Helm​

helm install <my-release> \
    oci://registry-1.docker.io/embracemedia/pulse_it_worker_chart \
    --version 4.2.801 \
    --values=values.local.yaml \
    -n embrace

<my-release> can be replaced by "pulse-it-worker" as release name for example.

Replace 4.2.801 by latest stable Pulse-IT worker release.

3.2 Create Worker Node from Pulse-IT​

  • Access the Pulse-IT interface and navigate to the "Monitor" menu, then select "Nodes".
  • Click on the "+" button
  • Modify the value of the "Number of nodes" field to specify the desired quantity of nodes to be created.
  • If desired, you can add a "Display name prefix".
  • Initiate the creation of nodes by clicking on the validation button labeled "New nodes".

4.Upgrade Pulse-IT​

4.1 Activate maintenance mode:​

  • Open the Pulse-IT interface and go to the "Monitor" menu, then choose "Nodes".
  • Click on the Pulse-IT Node in the diagram.
  • In the right panel, locate the "Maintenance" field and click on the "Deactivate" button.
  • In the modal that appears, click on "Stop all workflows and activate".

4.2 Upgrade Pulse-IT​

4.2.1. Retrieve last values.local.yaml​

If you can't see the old file, create a new one by consulting section 2.2

4.2.2 Upgrade Pulse-IT using Helm upgrade​

helm upgrade  <my-release> \
    oci://registry-1.docker.io/embracemedia/pulse_it_chart \
    --version 4.2.2306 \
    --values=values.local.yaml \
    -n embrace

<my-release> can be replaced by "pulse-it" as release name for example.

Replace 4.2.2306 by latest stable Pulse-IT release.

4.3 Upgrade Worker Nodes​

4.3.1 Delete old Worker Nodes​

  • Open the Pulse-IT interface and go to the "Monitor" menu, then choose "Nodes".
  • Select all nodes and click on the "trash" button

4.3.2 Create new nodes​

Refer to section 3.2

4.4 Deactivate maintenance mode​

  • Open the Pulse-IT interface and go to the "Monitor" menu, then choose "Nodes".
  • Click on the Pulse-IT Node in the diagram.
  • In the right panel, locate the "Maintenance" field and click on the "Activated" button.

5. Troubleshooting​

5.1 Pod's state is ImagePullBackOff:​

1- Verify whether any secret containing Embrace credentials have been created within the embrace namespace.

Command:

kubectl get secret -n embrace

Output:

NAME                TYPE                             DATA   AGE
emb-registry-cred   kubernetes.io/dockerconfigjson   1      3s

2- Ensure that the provided repository credentials are correct.

  • Delete existing secret
kubectl delete secret emb-registry-cred -n embrace
  • Recreate secret: Refer to section 1.5 to recreate secret
  • Deploy Pulse-IT using helm (see section 2.2)

5.2 Pulse-IT interface is not accessible​

5.2.1 Review the applications logs.​

Inspect the logs of every Pulse-IT component to identify errors.

- Review emb-app logs​
kubectl logs -n embrace -l component=emb-app
- Review celery beat logs​
kubectl logs -n embrace -l component=emb-celery-beat
- Review celery worker logs​
kubectl logs -n embrace -l component=emb-celery-worker

5.3 Pulse-IT workers state icon is red​

Inspect the logs of Pulse-IT worker container

kubectl logs -n embrace <pulse-it-worker-name> -c workflow-player

6. Utils commands:​

Get logs for pulse-it app pods​

kubectl logs -n embrace -l component=emb-app

Get logs for celery worker pods​

kubectl logs -n embrace -l component=emb-celery-worker

Restart to apply LDAP/Active Directory configuration​

kubectl rollout restart deployment emb-app emb-celery-worker -n embrace

Manually scale emb-app deployment​

kubectl scale deployment emb-app --replicas=2 -n embrace

--replicas=2 can be replaced by desired number of replicas

Get logs for aws-load-balancer-controller pods​

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

Get logs for karpenter pods​

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter