AWS StepFunction and the Importance of Idempotent Functions

AWS StepFunction allows you to build decoupled applications by letting you orchestrate microservices in an asynchronous and event-driven manner. It lets you focus on the application logic, while handling failures/retries/delays without you having to write any code. However, in order to truly take advantage of these features, you need to understand the concept of idempotence and apply it when building your microservices.

What is Idempotence?

In simple terms, idempotence refers to an operation that produces the same result (for a given input) irrespective of how many times it is applied. For example, in mathematics, the abs (absolute value) function is an idempotent function since

abs( abs( abs(x) ) ) = abs(x)

No matter how many times we perform the abs operation on x, the result is the same as doing it once.

At this point, I bet you are wondering, what does it have anything to do with StepFunctions. Let me explain by giving an example.

Let’s say we have a Step Function that allows a user to check out a book from a Library.

Step Function : Non Idempotent

• The Step Function is invoked when a user initiates the checkout.
• The ISBN of the book is provided as input to the function.
• The “Get Book Count” step invokes ${GetBookCountFunction}(Line 7) Lambda function that returns book count.
• If $.bookCount > 0 then we proceed to “Check out the book” step or else move on to “Book not available”.
• The “Checkout the book” step takes the book from the Library and assigns it to the user.
• If any step within the workflow fails, we retry until we are successful.

Now, in this setup, what happens when we run into an error executing “Get Book Count” step? We keep retrying, right? But maybe the error was transient and in our third or fourth attempt we are successful. No harm is done since each time it returns the book count at that instant of time. Hence, ${GetBookCountFunction} is idempotent.

Next, after a successful attempt, and assuming we have the book available, we proceed over to the next step: “Check out the book”.

Now comes the fun part. What happens when we run into an error executing “Check out the book” step? Can we retry and can we do that safely? Retrying in this case would cause issues. When we check out the book, we do not check if the book is available since we have already done that in the previous step. And that’s fine as long as we are successful in our attempt to check out the book. However, if we do run into an error and retry, it is possible that someone else might check out the book before our current user, essentially leaving no more book available for checkout. Thus, ${CheckoutBookFunction} (Line 39) is not idempotent.

With a simple change, here is how we can fix this problem. Let’s combine “Get Book Count” and “Check out the book” into “Check out Book if available” and run through the scenario again.

Step Function : Idempotent

Just like before, if an error occurs we keep retrying. But unlike before, we check for the availability and checkout the book at the same instant and if there is an error, we handle it gracefully.

The following code snippet illustrates the implementation of ${CheckoutBookIfAvailableFunction} (Line 5)

try:
   if get_book_count() > 0:
      checkout_book();
   else:
      raise BookNotAvailableError()
except:
   handle_error()

Hence, no matter how many times we retry we can expect a consistent result. If the book is available we attempt to checkout. If we succeed, we end the workflow. If there are no books left, we raise BookNotAvailableError and move on to “Book not available” step.

Aside from the obvious advantage described above, idempotence has led to a lot simpler and cleaner code as well.

Idempotence is a very powerful concept and there are many strategies you can use to make your functions idempotent. In my forthcoming blogs, I’ll demonstrate some of these strategies.

Hope you liked this article. Thanks for reading!

More content at plainenglish.io