After profiling our system performance some months ago, we realized that a significant bottleneck of the workflow latency was a particular dependent service, which accounted for the vast majority of the overall latency. Workflow execution occurs on worker machines; each worker would pick up the state of the operation, attempt to run the relevant state-specific logic, and then handle a state transition to the next state based on the result of that logic. When we later went to profile our system latency, it didn’t occur to me that this was a parameter that could be changed – it just appeared as “black box” framework latency.