Flows
Introduction to Flows
A data warehouse typically has two kinds of business processes for data processing: Flows and temporary queries. Flows handle data updates for stable data models, while temporary queries handle flexible data querying and processing.
Compared with temporary queries, Flows typically run repeatedly on a schedule and process and update data incrementally whenever possible. The computation is designed in layers with dependencies between layers, and intermediate-layer results are persisted during computation so that downstream tasks can use them. This keeps metric definitions consistent and data from the same period consistent, and it saves computing costs.
The complexity of a Flow mainly shows in its multi-layer paths and multiple branches. Therefore, in data development, a business data model is split into multiple Flows. Some Flows clean data from a certain type of data source as it's loaded, while others merge data from multiple sources into multi-dimensional wide business tables. Splitting things up this way reduces maintenance complexity, makes it easier to divide work, and makes partial refactoring and debugging easier.
The AE DataOps Platform supports building batch-processing Flows with an advanced visual Workflow. This chapter describes in detail how to use Flows.
This chapter covers:
- Create a Flow: how to create a Flow, and the release model for the Flow's dual environments
- Flow nodes: the task node types available in a Flow and how to use them
- Flow scheduling settings: how to configure Flow scheduling, including cron and interval scheduling policies for Flows and custom exception handling policies
- Flow parameter settings: how to flexibly reference workspace parameters and Flow parameters in a Flow
- Flow release: how to release a Flow
- Flow management and operations: how to manage Flows and operate Flow instances