Replay Seattle 2024

This year we will introduce a range of new features to both the open source Temporal project as well as our commercial solution, Temporal Cloud. 

Everyone has seen Generative AI demonstrations and prototypes. But 2024 is the year of production, with real customers, real products, and real return. In this talk we’ll cover how AWS sees the Generative AI landscape, and how developers and customers are moving to product and business success. Going from prototype to product requires a crystalized understanding of product needs, a rock solid infrastructure, and some modern engineering practices to ensure great outcomes. We’ll cover how some of the surprising challenges of Generative AI are quite familiar to cloud scale engineers. Hear how AWS and partners like Temporal are charting the path to GenAI product success.

Netflix uses Temporal to orchestrate cloud operations for Spinnaker, a multi-cloud continuous delivery platform, which includes dozens of cloud operations written for various cloud providers. Netflix also uses Temporal to orchestrate myriad other infrastructure control planes across the business including real-time data infrastructure, online datastores, and CDNs.

Netflix aims to allow each team to own their own namespace and call in each other to build abstractions on top of each other’s workflows. When a durable execution spans teams, you often want to delegate a portion of the overall workflow to another team with workflows running in a different namespace, but want a clean service contract that doesn’t share too much.

Previously, this required building custom API proxies and webhooks to facilitate cross-namespace collaboration. Teams had to handle complex tasks like setting up authentication and authorization (AuthZ), ensuring security, and managing cross-namespace failover scenarios. This was a lot of undifferentiated heavy lifting that could be simplified.

In this session you will learn about the future of a fully integrated Temporal experience. Discover how new enhancements will simplify durable execution across team and namespace boundaries, making collaboration easier and more efficient.

How many workflow systems do you have at your company? If you are like most companies, the answer is too many. Twilio has various workflow engines driving everything from routine maintenance  jobs to critical workflows that drive the company. Learn Twilio’s journey from homegrown systems to various third party solutions to Temporal Cloud. Learn not only about Twilio’s journey to Temporal Cloud, but also the challenges we discovered and how we overcame them.

In this session, we will delve into the intricacies of operating self-hosted Temporal clusters, focusing on achieving high availability and cost-efficient scalability. Temporal, an open-source workflow orchestration engine, is renowned for its flexibility and robustness, but mastering its deployment and management in a self-hosted environment requires specialized knowledge and strategies. We would like to share Salesforce’s experience with managing and operating self-hosted Temporal clusters over the past 3 years and our learnings along the way.

We will explore key topics including:

• High Availability: Best practices for configuring Temporal clusters to ensure continuous operation and resilience against failures.
• Scalability: Techniques to efficiently scale your Temporal clusters to handle increasing workloads without compromising performance.
• Cost Efficiency: Strategies to optimize resource usage and minimize operational costs while maintaining high service levels.

Through real-world examples and our experience, attendees will gain a comprehensive understanding of how to effectively manage and optimize their self-hosted Temporal clusters. Whether you are new to Temporal or looking to enhance your current setup, this talk will equip you with the knowledge and tools to achieve a high-performing, secure, and cost-efficient orchestration environment.

When services need to communicate with other services, there are several options which might include APIs, queues such as Kafka / AWS SQS, etc. If latency must be low, communication via APIs probably makes sense. If the communication between services can be asynchronous and latency can be high, queues might make sense.

When communicating between services across different clouds, communication between services can be more difficult. Exposing APIs across the public internet is obviously more difficult than exposing APIs internally in a single cloud. The most common pattern for integrating services across clouds are public APIs but several issues can arise with APIs.

Cross cloud API communication requires: The API service must expose public (or somewhat public) api endpoints. These endpoints probably need to enable: Encryption Authentication / access rights Rate limiting (429s with too many calls) API clients must be able to access the public api (possible via IP allow-listing, security credentials etc) API clients must handle api failures by retrying.

In this session, we explore practical applications of Temporal newest features in handling payments. First, we showcase the transformation of our checkout process using Workflow Update, highlighting key improvements and practical applications. Then we discuss various patterns of using child workflows. Finally, you’ll learn how WF Replayer helps us ensure workflow version compatibility during CI/CD deployments.

Details:

1. Handling Payments During Checkout: Explore the business case of the checkout process and how Temporal streamlined its implementation.
   • Checkout Workflow: Understand the purpose, structure, and challenges of our checkout workflow.
   • Advanced Workflow Techniques: Learn about using signals, Redis, and internal state machines to advance the workflow.
   • Enhancing Workflows with Workflow Update: Discover how we removed Redis, replaced signals with updates, and addressed new challenges like concurrent workflow updates.
   • Command Queue and Sequential Execution: How we improved Workflow Update with a command queue to deal with concurrency issues.
2. Child Workflows: Delve into patterns and usages, including task partitioning, parallel execution, sync/async execution, and async compensation.
   • StartWithUpdate Feature: Learn how to replace async child workflows with the StartWithUpdate feature.
3. Workflow Compatibility Verification: Get a comprehensive solution for verifying workflow compatibility with Workflow Replayer.
   • CI/CD Integration: Understand how we ensure workflow version compatibility using a CI/CD pipeline.

Paradigm shifts are easy to recognize in hindsight.  The trick is recognizing them in their early stages.  Evolution, even in software, tends towards an equilibrium until something dramatically better comes along to disrupt that balance.  This talk compares past "software regime changes" (i.e. epochal design disruptions) to the new abstraction that's at the heart of Temporal, with analogies to the evolution of life. You will come away with knowledge about how this abstraction will reshape the ecosystem and change how you design and build software, and with a clear vision of the future of Durable Execution.

TBD

Yum! brands has been a presenter at every Replay and this year, we return to talk about our progress with Temporal and explicitly around some new use cases. This year, we will go into detail about our promotion authoring and data sync use cases and explicitly how we use parent workflows and schedules to throttle load on various internal and external systems. 

We will dive into a general overview of a few workflows and their activities and then into code examples where we present some best practices for this pattern. Along the way, we will speak to the challenges we encountered and hopefully help with some practical tips and tricks you can walk away with.

What happens when you mix the best open source GenAI stack with the best durable execution platform? You get a production ready application. Learn how the power of GenAI running on containers can be enhanced with Temporal's durable execution platform.

Having a production ready service is always important in modern services and becoming more and more important everyday, and it is always important to make sure every service in your organization meets the standard of production ready. Not all teams in the company have the same level of expertise on what the production ready means, and it is important to help each service owner teams to make their service more production ready. There are some interesting production readiness metrics, like unit tests coverage, CI/CD pipeline setup, any known vulnerabilities or if the service is resilient in multiple clusters in case of disaster.

That’s what we do in Cloudflare is to have a production readiness dashboard, that will display your service health and production readiness information and what users could do to improve it. We have tons of services and repositories in Cloudflare, and for one big instance to check all services would not be scalable and reliable, so that’s where Temporal comes into play to help us solve this problem at scale, and run production readiness checks in parallel to save time. 

We used Temporal to help coordinate a list of instances to distribute the workload of those production readiness checks, and also used Temporal schedule to help check the metrics on a defined interval, and update our production readiness dashboard in real time.


With the help of Temporal, we are able to coordinate and complete the production readiness checks in just a few hours for thousands of repositories and well before the next scheduled refresh, and are able to configure a much shorter refresh interval if needed. Not to mention the automatic retries provided by Temporal out of the box for the workflows that help us retry failures automatically when needed. This gives us the scalability and reliability we need for our dashboard.

“Workflows” in Temporal are much more than a sequence of steps. In addition to traditional, DAG-based workflows, they may represent a Finite State Machine, an Actor, or any other process.

At NVIDIA, we operate a fleet of GPUs deployed across public and private clouds. Many developers use Temporal to deal with unreliable services. At NVIDIA we use it to manage unreliable hardware, networks, and configuration, and our solution involves all of the above Workflow types.

Not only does Temporal provide the building blocks for these various types of Workflows, they are interoperable via Signals, Child Workflows, Queries, and Updates. This means that a traditional (DAG) Workflow may represent an action taken by an Agent; or that a Signal to an Actor may form one step of a traditional Workflow; or that a State Machine Workflow may guard resources modified by another Workflow. All of these interaction patterns are used in our design.

The scale of Temporal allows us to represent each resource in our system as a Workflow, all the way from racks down to individual GPUs. Finally, because these fine-grained Workflows store little data and are usually idle, it is economical to run them on Temporal Cloud.

The Control Plane is the ‘brain’ behind Temporal Cloud that ensures its smooth operation, manages resources and workloads, rolls out updates to the software stack, and performs many other critical operations. In this talk, Sergey will share more details about the behind-the-scenes Control Plane functionality and how it continues to evolve over time. 

1. Recap of high-level architecture: DP/CP/IP
2. Rings and rollouts
   1. Recap
   2. Ring100
   3. Different kinds of rollouts: normal, patch, config, alerts, infra, auth
   4. How this approach improves reliability
3. Release process
   1. How it used to be (OSS → CDS → Cloud, patches)
   2. Bi-weekly deployments (Cloud → OSS)
   3. How this approach improves reliability
4. Migration
   1. Across Cells with different capacity (shard count)
   2. For load balancing
   3. Daycare Cells
   4. How this approach improves reliability 
5. MRN
   1. Managing two-replica namespaces
   2. Leveraging migration milage
   3. Automatic failover? [when it’s ready]
   4. Three-way replication during migration
6. Multi-cloud
   1. We ran into some differences between GCP and AWS, but nothing major
   2. CP had a notion of cloud provider from the start but found a couple of misses
   3. We are not running a piece of CP in GCP yet, TBD

Following on from last year’s Replay talk Temporal @ HashiCorp, this talk dives into several use cases for which durable execution can give superpowers to your platform organization, enabling you to deal with the inevitable complexity when your architecture (and team) needs to scale.

We will share specific use cases where Temporal has empowered us to manage the complexities of growing architectures and teams. With detailed code examples, attendees will learn how Temporal workflows have enabled us to minimize operational risk and streamline operations. We will also explore the human aspect of durable execution, detailing its impact on our stakeholders, and our team’s productivity and morale.

Internal platforms built to enable developer self-service often mask messy ownership and security concerns in order to present an ergonomic user experience, resulting in limited extensibility, operational risk, and a productivity tax carried by platform documentation and support processes. Furthermore, many of these issues worsen exponentially as the size of your team and your architecture grows. Attendees will take away actionable insights into building scalable, secure platforms using durable execution, and a clear understanding of how Temporal can be leveraged to reduce operational risks and eliminate the productivity tax on developers.

At the heart of every platform are workflows exposed as services, and Temporal helps teams to focus on workflows rather than the accidental complexity introduced by technology. Since its inception, the HashiCorp Cloud Platform has been built on durable execution.

Now, with Temporal as the foundation of our internal platform we have been able to create a reliable, secure, self-documenting and team-agnostic approach to virtually any platform use case required by our development teams.