Camunda External Task Workers and Quarkus

In process automation with Camunda Platform 7, the external task pattern is often used to decouple the orchestration of tasks from their execution. In this pattern, the task execution is carried out by a so-called external task worker, i.e. a smaller application that can be deployed and run separately from the process engine. Thus, it can be scaled independently and therefore more precisely based on the task's needs. To actually profit from this feature, the external task worker application needs to be lightweight enough to allow for fast up and down scaling without causing huge costs in the process. The Java framework Quarkus is an ideal fit to create small, lightweight Java applications, and our new Quarkus extension makes it much easier to implement external task workers within this framework.

 A German version of this blog post can be found here.

Camunda architecture: Decpoupling

In process automation, Camunda more and more favours the decoupling of task orchestration and execution. At the start of 2022, Camunda updated their architecture recommendations to reflect the fact that they now favour a separation of process engine and execution code as opposed to their earlier preferred architecture of process engines being integrated into process applications. Camunda Co-Funder Bernd Rücker explains their reasoning behind this change of mind in an insightful blog post. For Camunda Platform 7 this decoupling is achieved by using the external task worker pattern, where tasks are executed asynchronously by independent applications called external task workers. This pattern offers a number of advantages, such as the separation of concerns, better scaling or - to a certain degree- independence of programming languages.  
NB: Camunda's decision to favour decoupled orchestration and execution of processes is very much in line with the release of Camunda 8 in April 2022. In this cloud-based product, the separation of engine and execution is enforced by design. For the next couple of years, Camunda 8 and Camunda 7 will co-exist as two separate but related products. However, using external tasks in Camunda 7 (which our Quarkus extension is built for) will make a potential future transition to Camunda 8 much easier, as is explained in yet another blog post by Bernd Rücker. 

The basic principles of External Task Workers

We have already explained the concept of External Task Workers in quite some detail in one of our earlier blog posts Camunda External Task Worker – The Basics. In short: A BPMN process is being automated using Camunda's Process engine. The engine takes care of the correct order of steps within the process and makes sure that at any point the correct task is executed, after which the process instance continues on the correct path. In classic process applications, the execution code for these tasks would usually be part of the very same Java application that already contains the process engine (or at least be running on the same application server). This is not the case for the external task worker pattern. Instead, the execution code is part of a separate, independent application which uses the process engine's REST API to obtain open tasks for a specific topic, do the work specified within the task and then return the result again using the REST API. Therefore, the process engine is no longer concerned with how tasks are executed, but only that they are executed.

You might also be interested in our blogpost Camunda External Tasks - Error-Handling and Retry-Behavior.

External Task Worker and the Cloud: A match made in heaven

One of the key advantages of the External Task Pattern is that the worker applications are (in general) much smaller than the usually quite large full process applications. Additionally, they can be scaled up and down independently from the process engine, such that e.g. many instances are available in times of heavy workload and no instances are running if there is no work available in the foreseeable future. Both of these features imply that External Task Worker applications are ideal candidates for deployment in the cloud, e.g. in Kubernetes clusters, where the scaling of small(-ish) applications is absolutely essential. 

Implementation of External Task Workers

As the external task workers only need to be able to communicate with the process engine via REST, they can be implemented in just about every programming language available. However, developers who have worked with Camunda over the last few years are usually used to working with Java. Thus, many of them will choose to implement External Task Workers in that language, too. Being aware of that, Camunda offers the Camunda External Task Client, a Java library which takes care of most of the overhead caused by the use of the REST API as well as of the continuous polling of new tasks. With this library, the implementation of an External Task Worker application boils down to the following few steps:

1. Create and configure an instance of the class ExternalTaskClient.

2. Implement the interface ExternalTaskHandler and within its execute method carry out whatever necessary for the business logic of the task in question.

3. Register an instance of the class implemented in Step 2 with the instance of the ExternalTaskClient under the targeted task's topic.

Once started, the ExternalTaskClient will continuously poll the process engine for new tasks for that topic. Whenever new tasks are available, it will get them and use the registered TaskHandler to execute and complete said tasks. Additional useful features such as a backoff functionality for the task polling are also part of this library.

Step 1 and 3 are essentially boilerplate, as they need to be implemented  for every new External Task Worker application, despite never really changing. Therefore, Camunda also offers a Spring Boot Starter which takes care of steps 1 and 3 and allows to easily register ExternalTaskHandler implementations via an annotation. This way, implementing External Task Worker applications becomes quite comfortable. Unfortunately, Java applications and Spring Boot applications in particular are known to be rather large and slow, especially at startup. Therefore, this setup is not really well-suited to make use of the advantages of the external task worker pattern mentioned above, which arise from applications being lightweight and fast.

Quarkus to the rescue

The fact that Spring Boot applications are not ideal for deployment in Kubernetes clusters due to their sluggish behaviour, is not big news. To tackle this problem and to provide smaller and faster Java applications, several new frameworks have come out over the years, notably Micronaut, Spring Native and Quarkus . These frameworks all promise to provide extremely small and fast starting applications using a concept called native compilation. However, already with normal compilation, they lead to smaller applications with faster startup times in normal JVM deployments than Spring Boot can offer, cf. for example here.

Quarkus-Camunda-External-Task-Client Extension

To enable developers to more easily implement External Task Worker applications in Quarkus, we have created a small Quarkus extension. It is based on Camunda's ExternalTaskClient library and just like Camunda's own Spring Boot starter it essentially takes care of steps 1 and 3 mentioned above, i.e. the configuration and instanciation of an ExternalTaskClient as well as the automatic registration of ExternalTaskHandler beans for a topic using a suitable annotation. Additionally, the instanciated ExternalTaskClient is made available as a bean in Quarkus application context to e.g. register new handlers or start/stop the execution of tasks at runtime.

DOes It Work?

Setup

To test out our extension we automated the small BPMN process depicted below with Camunda.

We settled on the following scenario:

• The process definition above is deployed on a Camunda process engine which is starting a process instance every 20 seconds
• The first service task is handled by an external task worker built with Quarkus using our new extension
• The second service task is handled by an external task worker built with Spring Boot
• The process engine and the task handlers are running as pods on a K8s-Cluster
• We started the setup a total of 5 times and average over these runs

Results

The two key metrics we are comparing are the startup time of the worker applications and their RAM usage while continuously completing their service tasks.

The following table represents the worker applications' startup times over our 5 experiment runs in seconds as well as their average:

As we can see, the Quarkus applications started significantly faster than their Spring Boot counterparts.

Similarly, we tracked the applications' RAM usage throughout our experiments.

Again it is easy to see that Quarkus performs quite a bit better in this metric, too.

Using the extension

As usual, to use our new Quarkus extension, it has to provided as a dependency, e.g. in the pom.xml of a Maven project:

The application.properties (or application.yaml) can then be used to configure the extension and in particular to configure the connection to the process engine as well as a worker-id, e.g.

de.viadee.camunda.client.base-url=http://localhost:8080/engine-rest

de.viadee.camunda.client.worker-id=my-example-worker

There are quite a few more options to configure, but this should do for now. As always, we now have to implement an ExternalTaskHandler taking care of the actual business logic we want to achieve with this worker.

Only the first two lines are actually noteworthy here. Firstly, we use @ApplicationScoped to make sure that Quarkus creates a bean for this class (just as you might use @Component in Spring Boot). The annotation @ExternalTaskSubscription is part of our extension. On application startup, all beans whose class is annotated with it will be registered with the ExternalTaskClient under their topic specified by topicName. Then, the client will start polling tasks and using the handler to execute and complete them - that's it!

Open Source

All of the source code of our extension can be found on GitHub. The folder /example contains a small workable example on how to use it. The External Task Client Extension for Quarkus is an open source project published under the BDS3-Clause License. Of course, everyone is welcome to participate on GitHub either by making suggestions for improvement using issues or by directly working on the code via pull requests.

Next Stop: Native Compilation

As explained above, Quarkus can also be used for native compilation with the GraalVM to build extremely small and fast applications running without a JVM. However, some extra effort is required to actually get natively compiled Quarkus applications to work. In particular, all classes on which reflection is used at runtime need to be known at build time already. In particular, this affects the use of essentially all (de-)serialization libraries. 
Our External Task Worker extension is not yet fully compatible with native compilation and applications using it will face some difficulties. However, it is definitely a goal for one of the upcoming versions.