Replay Seattle 2024

Media processing workflows are inherently complex, often requiring extensive state management and continuous updates to ensure media encodes remain current. At Netflix, we have developed the Plato media workflow platform, a key component of our larger Cosmos media processing system. This talk will delve into how we process millions of media workflow events daily, ensuring durability and scalability while enhancing the developer experience.


Enhancing RPC Durability in Media Workflows

Cosmos is a microservices-based platform, where each processing component is implemented as a microservice or as a serverless function. This necessitates the workflows to make RPC calls to execute the tasks at scale asynchronously. Plato’s unique approach to handling remote procedure calls (RPCs) using message-passing techniques makes the flaky RPC calls more durable and reliable. This adaptation allows our users to build on a resilient RPC client foundation, mitigating the impacts of potential failures on workflow continuity.


Scaling to Millions of Workflow Events

The media processing domain is characterized by its bursty nature of work, where the demand for producing encodes often exceeds available compute resources. To address this, Plato incorporates features like priority-based task queues, execution avoidance, and a combination of dynamic and static graph execution models. Together, these features enable us to process millions of workflow events daily. We will present real-world scenarios that showcase how these technologies allow Plato to efficiently scale up and durably execute millions of workflow events.


Prioritizing Developer Experience
While ensuring durability is crucial for our users, it cannot come at the cost of developer experience. The Plato platform allows users to seamlessly bring their own strongly typed data models. This feature ensures that workflow execution state can be stored and retrieved reliably, testing workflows with strong contracts, and lowering the barrier to entry for our users by enhancing the platform’s usability. We will highlight case studies that demonstrate how Plato provides a good developer experience and discuss some of the open challenges we are working on.


Conclusion

This talk will provide an overview of how Netflix implements durable executions to process media encodes at scale. Attendees will gain insights into the challenges and techniques that Netflix uses in the media processing space, with practical examples from the Plato platform that highlights our approach to durability, scalability, and developer experience.

References

• For an overview of the underlying technologies and design principles of the Cosmos platform, please refer to our blog post here.

TBD

Yum! brands has been a presenter at every Replay and this year, we return to talk about our progress with Temporal and explicitly around some new use cases. This year, we will go into detail about our promotion authoring and data sync use cases and explicitly how we use parent workflows and schedules to throttle load on various internal and external systems. 

We will dive into a general overview of a few workflows and their activities and then into code examples where we present some best practices for this pattern. Along the way, we will speak to the challenges we encountered and hopefully help with some practical tips and tricks you can walk away with.

Is your company failing to get the expected ROI on its cloud investments? Are your software engineers spending far too much time thinking about containers, pub/sub topics, and cloud functions instead of focusing on the business problems their software is meant to solve? If so, you are not alone.

Many large enterprises face disappointment with their cloud investments, failing to achieve the expected return on investment. This issue often stems from focusing too much on the easier aspects of cloud adoption, such as setting up infrastructure, while neglecting the more challenging task of adapting software architecture for the cloud’s unique demands.

In traditional data centers, hardware is reliable and failure-resistant, which shapes the way software is designed. However, the cloud environment is inherently ephemeral, with transient virtual machines and distributed storage that requires applications to be resilient and fault-tolerant. This shift necessitates a new approach to software architecture, one that many enterprises overlook, leading to inefficiencies and increased development times.

The cloud often shifts complexity rather than reducing it, placing new burdens on software engineers. They must now handle transient failures, ensure data durability, and maintain distributed consistency. These challenges not only erode the anticipated benefits of cloud migration but also increase cognitive load, distracting engineers from writing good software and focusing on business logic. Furthermore, this complexity shift often benefits cloud providers from a billing and revenue perspective, as organizations may end up paying more for additional services and resources to manage these complexities.

Temporal offers a solution by providing an abstraction layer that simplifies the development of resilient cloud applications. It includes built-in primitives for managing state, retries, timeouts, and long-running workflows, allowing engineers to focus on business logic rather than infrastructure complexities. This results in applications that maintain high availability and fault tolerance, essential for cloud environments.

By using Temporal, software engineers are relieved from the burdens of cloud-specific challenges, improving productivity and accelerating the delivery of cloud-native applications. This alignment with modern software development practices, such as microservices and event-driven architectures, ensures that enterprises can modernize their application portfolios effectively.

The key to maximizing cloud investment ROI lies in rethinking software architecture to embrace the cloud’s unique characteristics. Temporal enables this transformation, ensuring cloud applications are as resilient and dependable as traditional on-premises systems, thereby realizing the full benefits of cloud migration. By addressing the true complexities of the cloud, enterprises can avoid unnecessary costs and achieve their strategic goals more effectively.

Today, iterating on workflow implementations requires careful, backwards-compatible patching that, if done incorrectly, can result in stuck workflows and challenging remediations. Together, Temporal and Datadog are working to improve this process.

Temporal is developing APIs that allow teams to manage their deploys of Temporal Workers. With these APIs, short-running workflows will be guaranteed to run to completion on a single version of code with no patching, and developers will have better and safer tools to migrate even long-running workflows to new versions.

As a design partner, Datadog has been building an open-source Kubernetes controller based on these APIs. This controller will manage the rollouts of worker deployments and automatically reconcile running workflows with existing worker versions.
In this session, you will learn about the future of safely managing worker rollouts, see a demo of the worker Kubernetes controller, and join a community of developers creating new best practices for deploying stateful applications.

What happens when you mix the best open source GenAI stack with the best durable execution platform? You get a production ready application. Learn how the power of GenAI running on containers can be enhanced with Temporal's durable execution platform.

The fundamental pricing model of Temporal Cloud is simple to understand, just executed actions and utilized storage, yet it’s easy to underestimate how many actions your workflow might run, or how large the event history may grow. We’ll help clarify what workflow design choices may be responsible for high action or storage costs.

This talk will compare the impacts of several design decisions made as part of a simple but realistic workflow. We’ll examine a month of simulated activity and use it to illustrate the effects of each seemingly minor change to the workflow.

Along with simple changes (e.g shorter lived workflows) we’ll also review some more dramatic design choices (e.g actor model execution) that aim to minimize costs in scenarios where Temporal Cloud might appear prohibitively expensive.

Whether you’re looking for a way to optimize your cloud costs, or increase the accuracy of your next cost estimation, this talk is sure to be of interest.

Topics Will Include: 

• Unexpected Sources Of Action Consumption: - Unnecessary parallelism / Child workflows - Low timeout heartbeats - High interactivity / incoming signals - Short timers
• Unexpected Sources Of Storage Utilization: - Returning large state from activities - Signal payloads - Overall history length - Unnecessarily Long Lived Workflows - Sleeping With A Long History
• Illustrated Design Impacts Will Include: - Skipped short timers - Encoded workflow payloads - Actor style workflows - “Deactivating” workflows - Optimistically continue-as-new for “history compaction”

Having a production ready service is always important in modern services and becoming more and more important everyday, and it is always important to make sure every service in your organization meets the standard of production ready. Not all teams in the company have the same level of expertise on what the production ready means, and it is important to help each service owner teams to make their service more production ready. There are some interesting production readiness metrics, like unit tests coverage, CI/CD pipeline setup, any known vulnerabilities or if the service is resilient in multiple clusters in case of disaster.

That’s what we do in Cloudflare is to have a production readiness dashboard, that will display your service health and production readiness information and what users could do to improve it. We have tons of services and repositories in Cloudflare, and for one big instance to check all services would not be scalable and reliable, so that’s where Temporal comes into play to help us solve this problem at scale, and run production readiness checks in parallel to save time. 

We used Temporal to help coordinate a list of instances to distribute the workload of those production readiness checks, and also used Temporal schedule to help check the metrics on a defined interval, and update our production readiness dashboard in real time.


With the help of Temporal, we are able to coordinate and complete the production readiness checks in just a few hours for thousands of repositories and well before the next scheduled refresh, and are able to configure a much shorter refresh interval if needed. Not to mention the automatic retries provided by Temporal out of the box for the workflows that help us retry failures automatically when needed. This gives us the scalability and reliability we need for our dashboard.

Temporal has six officially supported SDKs and a few more that are available in the community. They are a critical part of the project as they provide idiomatic language support so you can code for reliability in your apps and services.

Each SDK we strive to make feel as natural to the language as possible.  All of them have a common set of base functionality, and provide the ability to rebuild your workflow's state from history, enabling you to write durable, fault-tolerant code in your language of choice.

In this talk, I'll explain how the insides of the SDKs make the magic of Temporal state recreation work. This talk assumes familiarity with the basics of Temporal Workflows.

We will present two use cases where we have leveraged Temporal at Turo.

For car rentals, guests would like to defer payment until the start of their reservation. Ensuring that the payment is triggered at the right time is crucial. We’ll describe our specific use case and detail its implementation and workflow design decisions. For example, use of notifications, scheduling, and error handling.

In order to showcase our selection of cars accurately at Turo, we maintain a catalog of car data for each country we operate in. We build this dataset by integrating with multiple data providers. The datasets change frequently as new cars are released and the import process for each dataset runs on a fixed schedule. Temporal has helped us process data in a reliable and efficient way. We will show how workflows have enabled us to easily spread the work across our application instances, how we are notified and act on workflow errors, and how the Temporal UI has assisted in monitoring the status of our long running jobs.

Let's eat!

“Workflows” in Temporal are much more than a sequence of steps. In addition to traditional, DAG-based workflows, they may represent a Finite State Machine, an Actor, or any other process.

At NVIDIA, we operate a fleet of GPUs deployed across public and private clouds. Many developers use Temporal to deal with unreliable services. At NVIDIA we use it to manage unreliable hardware, networks, and configuration, and our solution involves all of the above Workflow types.

Not only does Temporal provide the building blocks for these various types of Workflows, they are interoperable via Signals, Child Workflows, Queries, and Updates. This means that a traditional (DAG) Workflow may represent an action taken by an Agent; or that a Signal to an Actor may form one step of a traditional Workflow; or that a State Machine Workflow may guard resources modified by another Workflow. All of these interaction patterns are used in our design.

The scale of Temporal allows us to represent each resource in our system as a Workflow, all the way from racks down to individual GPUs. Finally, because these fine-grained Workflows store little data and are usually idle, it is economical to run them on Temporal Cloud.

In the realm of software development, scheduling and executing tasks precisely when needed is a universal challenge—particularly as systems scale. This session will provide an in-depth exploration of how timed events are managed with traditional tools such as Redis, databases, and queues, which many developers are already familiar with.

However, despite their capabilities, these tools often require complex integrations and extensive manual maintenance to handle dynamic scheduling effectively. This talk will dissect these challenges and demonstrate how they can impede application scalability and reliability.

Contrasting this, we will introduce Temporal, a robust solution that simplifies the scheduling and management of timed events. Attendees will learn about Temporal’s innovative approach to workflows and timers, which provides out-of-the-box scalability and flexibility, allowing developers to adapt to changing business requirements with minimal effort.

The session aims to equip developers with the knowledge to:

• Understand the limitations of traditional scheduling tools.
  
• Leverage Temporal’s workflows for dynamic, reliable task execution.
  
• Seamlessly integrate scalable timed events into full stack applications.
  

By the end of this presentation, attendees will not only grasp how Temporal can streamline their development processes but will also gain practical insights into transforming their current event-handling implementations for better performance and ease of maintenance. Join us to elevate your application’s functionality and discover a simpler path to robust, timed task execution.

Exponential Growth and Team Expansion
Salesforce has experienced a remarkable 36% quarter-over-quarter growth since adopting Temporal, demonstrating the platform’s scalability and reliability. This growth is a direct result of over 20 teams within Salesforce utilizing Temporal to foster a culture of excellence and continuous improvement.

Operational Mastery: Executing at Scale
Our operational capabilities are showcased by the management of over 70+ clusters. This reflects our dedication to customer excellence and our ability to handle complex scale effectively.

Innovation and Customization: The Salesforce Edge
The synergy between Temporal’s robust features and our bespoke enhancements has significantly advanced our business process automations, giving us a distinct advantage in the marketplace.

Key Innovations

• Seamless Customer Onboarding: We’ve streamlined the onboarding process to under an hour, thanks to automated namespace management and the use of the Open Source Terraform Temporal provider. Our active contributions to the OSS community underscore our commitment to collective advancement.
• Collaboration at its Core: The integration of Temporal Clusters with Slack has improved collaboration and notifications, providing rich, timely updates and reducing the need for constant workflow monitoring.

At Salesforce, we’re not just keeping pace with technological advancements; we’re setting the pace, pushing the boundaries to deliver exceptional value to our customers.

Discern, Inc. presents Kibu: An open-source developer productivity toolkit developed in Go and powered by Temporal.

This presentation will focus on Kibu’s unique capabilities to simplify and expedite the development process by automating the generation of essential plumbing code. This automation allows developers to concentrate more on business logic rather than boilerplate, significantly reducing development time and effort.

A key feature of our session will be to demonstrate how quickly you can build service endpoints using plain old Go code (no proto files). We will showcase several examples, including starting workflows, performing queries, handling updates, and generating plumbing code that makes your endpoints durable by default.

Attendees will learn how Temporal can be the backbone of your entire system and how Kibu empowers developers to rapidly build and deploy robust applications.

So you want to try Temporal, or you want to use Temporal more in your code bases, but you’re working with legacy systems. Example repos and docs may have great ideas for greenfield applications or simple set up for applications that can rely on workflow-to-workflow communication, but your systems are...not as “straightforward”.

Stitch Fix has been integrating Temporal into its older distributed systems, getting creative with applications that are a mix of languages, architectures, and stimulus styles -- from async endpoint calls to message queues. This presentation will give you a brief overview of ways to carve out use cases for Temporal based on features and functionality that already exist in your ecosystem, and simple ideas of how to integrate Temporal tooling in tricky situations.

The Control Plane is the ‘brain’ behind Temporal Cloud that ensures its smooth operation, manages resources and workloads, rolls out updates to the software stack, and performs many other critical operations. In this talk, Sergey will share more details about the behind-the-scenes Control Plane functionality and how it continues to evolve over time. 

1. Recap of high-level architecture: DP/CP/IP
2. Rings and rollouts
   1. Recap
   2. Ring100
   3. Different kinds of rollouts: normal, patch, config, alerts, infra, auth
   4. How this approach improves reliability
3. Release process
   1. How it used to be (OSS → CDS → Cloud, patches)
   2. Bi-weekly deployments (Cloud → OSS)
   3. How this approach improves reliability
4. Migration
   1. Across Cells with different capacity (shard count)
   2. For load balancing
   3. Daycare Cells
   4. How this approach improves reliability 
5. MRN
   1. Managing two-replica namespaces
   2. Leveraging migration milage
   3. Automatic failover? [when it’s ready]
   4. Three-way replication during migration
6. Multi-cloud
   1. We ran into some differences between GCP and AWS, but nothing major
   2. CP had a notion of cloud provider from the start but found a couple of misses
   3. We are not running a piece of CP in GCP yet, TBD

This talk will cover:

• 3 pitfalls of DIY-ing your own durable execution platform,
• 3 benefits of leveraging a mature solution like Temporal, and
• 3 keys to success in getting started with your implementation project.

Redirecting The BuildersMark will also share his experience with redirecting the talent and energy of these brilliant builders toward enhancing and building on top of the mature and battle-hardened durable execution features that Temporal provides rather than getting mired in reinventing the wheel.

What Can Participants ExpectThis talk will help participants make strong business and technical arguments for adopting Temporal rather than doing DIY and also why leveraging Temporal Cloud could be a great fit rather than self-hosting.

In an increasingly connected world, handling traffic spikes effectively is paramount. This talk delves into several autoscaling approaches for Temporal worker clusters.

We’ll explore scaling methodologies, such as using Temporal Cloud metrics, PaaS metrics, or scheduled scaling. We’ll discuss how companies leverage these various options and the pros and cons associated with them. By dissecting these approaches, you’ll gain insights into scaling strategies for meeting demand in real time while optimizing resource allocation in Temporal environments.

We’ll also dive into suggested server metrics for autoscaling Temporal workers. Metrics are the bedrock for informed decision-making, ensuring that resources are allocated judiciously to meet fluctuating demands while maintaining cost efficiency.

Scaling a Temporal worker cluster can be daunting, depending on your requirements and experience level. I’ll walk through the different approaches and provide real-world solutions used to scale some of the production systems Bitovi has the privilege of contributing to.

Finally, I’ll walk you through a live demo showcasing the use of Temporal Cloud metrics to scale a worker cluster dynamically. Attendees will witness firsthand how this scaling strategy adapts to real-time metrics, ensuring minimal latency and optimal resource utilization.

Walk away with tangible strategies to help you scale with Temporal and make smart decisions about how to do it.

Following on from last year’s Replay talk Temporal @ HashiCorp, this talk dives into several use cases for which durable execution can give superpowers to your platform organization, enabling you to deal with the inevitable complexity when your architecture (and team) needs to scale.

We will share specific use cases where Temporal has empowered us to manage the complexities of growing architectures and teams. With detailed code examples, attendees will learn how Temporal workflows have enabled us to minimize operational risk and streamline operations. We will also explore the human aspect of durable execution, detailing its impact on our stakeholders, and our team’s productivity and morale.

Internal platforms built to enable developer self-service often mask messy ownership and security concerns in order to present an ergonomic user experience, resulting in limited extensibility, operational risk, and a productivity tax carried by platform documentation and support processes. Furthermore, many of these issues worsen exponentially as the size of your team and your architecture grows. Attendees will take away actionable insights into building scalable, secure platforms using durable execution, and a clear understanding of how Temporal can be leveraged to reduce operational risks and eliminate the productivity tax on developers.

At the heart of every platform are workflows exposed as services, and Temporal helps teams to focus on workflows rather than the accidental complexity introduced by technology. Since its inception, the HashiCorp Cloud Platform has been built on durable execution.

Now, with Temporal as the foundation of our internal platform we have been able to create a reliable, secure, self-documenting and team-agnostic approach to virtually any platform use case required by our development teams.