Simulate failures

Recover from a server crash

Unlike many modern applications that require complex processes and external databases to handle failure, Temporal automatically preserves the state of your Workflow even if the server is down.

1. Make sure your Worker is stopped. Press CTRL+C in the Worker terminal.
2. Switch back to the terminal where your Workflow ran. Start the Workflow again with python run_workflow.py.
3. Verify the Workflow is running in the UI.
4. Shut down the Temporal Server by pressing CTRL+C in the server terminal.
5. After the Cluster has stopped, restart it and reload the UI.

Your Workflow is still listed:

If the Temporal Cluster goes offline, you can pick up where you left off when it comes back online.

Recover from an unknown error in an Activity

This demo makes a call to an external service in an Activity. If that call fails due to a bug, the Activity produces an error.

To test this out, simulate a bug in the deposit() Activity method. Let your Workflow continue to run but don't start the Worker yet.

1. Open the activities.py file and switch out the comments on the return statements so that the deposit() method calls self.bank.deposit_that_fails.
2. Save your changes and switch to the Worker terminal.
3. Start the Worker again:

python run_worker.py

You'll see the Worker complete withdraw() but error on deposit() - and keep retrying:

2024/02/12 10:59:09 INFO  Started Worker
2024/02/12 10:59:09 Withdrawing $250 from account 85-150.

2024/02/12 10:59:09 Depositing $250 into account 43-812.

2024/02/12 10:59:09 ERROR Activity error. ActivityType Deposit Attempt 1 Error This deposit has failed.
2024/02/12 10:59:10 Depositing $250 into account 43-812.

2024/02/12 10:59:10 ERROR Activity error. ActivityType Deposit Attempt 2 Error This deposit has failed.
2024/02/12 10:59:12 Depositing $250 into account 43-812.

2024/02/12 10:59:12 ERROR Activity error. ActivityType Deposit Attempt 3 Error This deposit has failed.

...

The Workflow keeps retrying using the RetryPolicy defined earlier.

View more in the Web UI:

Your Workflow is running, but only withdraw() has succeeded. In any other application, the whole process would be abandoned. With Temporal, you can debug and fix the issue while the Workflow is running.

Pretend that you found a fix. Switch the comments back on the return statements of the deposit() method and save your changes.

To restart the Worker, cancel it with CTRL+C, then restart:

python run_worker.py

On the next scheduled attempt, the Worker picks up right where the Workflow was failing and executes the fixed deposit() Activity:

Transfer complete.
Withdraw: {'amount': 250, 'receiver': '43-812', 'reference_id': '1f35f7c6-4376-4fb8-881a-569dfd64d472', 'sender': '85-150'}
Deposit: {'amount': 250, 'receiver': '43-812', 'reference_id': '1f35f7c6-4376-4fb8-881a-569dfd64d472', 'sender': '85-150'}

Visit the Web UI again and you'll see the Workflow has completed:

You have just fixed a bug in a running application without losing the state of the Workflow or restarting the transaction.

Conclusion

You now know how to run a Temporal Workflow and understand some of the value Temporal offers. You explored Workflows and Activities, you started a Workflow Execution, and you ran a Worker. You also saw how Temporal recovers from failures and retries Activities.

Key advantages Temporal offers:

1. Temporal gives you full visibility in the state of your Workflow and code execution.
2. Temporal maintains the state of your Workflow, even through server outages and errors.
3. Temporal lets you time out and retry Activity code using options that exist outside your business logic.
4. Temporal enables you to perform "live debugging" of your business logic while the Workflow is running.

Further exploration

Try the following before moving on:

• Change the Retry Policy in workflows.py so it only retries 1 time. Then change the deposit() Activity in activities.py so it uses the refund() method. Does the Workflow place the money back into the original account?