Flow nodes

Last updated 10/05/2026

Overview

In the AE DataOps Platform, Flow nodes are the basic units of a data processing workflow and can be understood as abstract representations of data processing tasks. Each node has clearly defined input, processing logic, and output.

Flow nodes connect the entire data processing workflow, and the dependencies between nodes keep data flowing and being processed in order. Nodes support visual orchestration, making it easy for developers to design and manage data processing workflows. Node statuses and execution results let you track data processing in real time and find and fix problems promptly.

In the Flow's Dev Mode, click the Task List button in the sidebar tool area to view all task nodes in the current Flow.

Create a Flow node

In the sidebar tool area of the Flow's Dev Mode, click + Add Node to see all task node types.

To add a task node, drag a node card onto the DAG.

After you drag a node onto the DAG, the Create node dialog appears, where you can set a name for the new node and select its node type.

Click Create and edit to go directly to the node's edit page and fill in its content.

Edit a Flow node

Double-click to select a node, then click Edit at the top of the page to enter edit mode.

In edit mode, you can change the current node's content. Click Debug to run a trial test.

You can also use AI (TIKI) to interpret logs and modify statements.

Flow node types

Trino SQL node

The Trino SQL node supports standard SQL syntax. With the Trino SQL node, you can quickly build efficient data query and processing logic in a Flow, which is especially useful when you need to integrate data across data sources.

You can run complex SQL transformations (such as window functions, JOINs, and aggregations) to generate intermediate results or write to target tables.

Offline sync plan node

In a Flow in the DataOps Platform, the Offline sync plan node transfers data in batches between different data sources and supports scheduled runs and full / incremental sync.

You can mount a created Offline Sync plan to a Flow for scheduled runs.

Its core usage and notes are as follows:

1. Cross-source data migration
   Supports syncing data from sources such as relational databases (for example, MySQL), file systems (for example, HDFS), and data warehouses (for example, Hive) to target storage (for example, ClickHouse and Doris).

2. Full / incremental sync

   • Full sync: overwrites the target table on each run (suitable for small tables or initialization).
   • Incremental sync: performs incremental updates based on conditions such as timestamps or auto-increment IDs (for example, WHERE update_time > '${bd}').

3. Data transformation
   Supports simple ETL operations such as field mapping, type conversion, and filter conditions (for example, converting int to bigint).

Task Instance Check node

A check node is a special type of task node. Its main function is to periodically check whether an object that meets certain conditions exists. If the check passes, the task node continues running; if the check fails, it doesn't continue (and the Flow may stop as a result). Unlike a SQL node, a check node is configured mainly through a form. The main configuration items of a check node are:

1. Check object: what is checked, and the most important part. Multiple check objects are supported: click Add object to add more check objects (up to 20) and check whether multiple objects exist at the same time. You can switch the condition between objects to and or or
2. Check object filter: some check nodes check more specific content. For example, when checking a partitioned table, besides checking whether the table exists, you may also need to check whether a specific partition exists. This example is a scenario that needs an additional condition: it checks whether an instance of a task/Flow exists for a specific base time
3. Pass condition: generally, the check passes when the check object exists or is in a certain state
4. Check frequency: how often to check, in minutes
5. Stop policy: when the check node stops checking. Set it in Check count under Check strategy. If the check still hasn't passed after the set number of checks, the check fails and stops (Check frequency under Check strategy is the previous item)

Flow Instance Check node

The Flow Instance Check node checks the running status of instances of the current Flow or other Flows to make sure Flow tasks run as expected.

By configuring check conditions, you can periodically check the Current task flow or other task flows.

Shell node

The Shell node runs custom Shell scripts, giving you flexibility for tasks such as data processing, file operations, and tool calls.

With Shell nodes, you can integrate custom operations into a Flow to cover what the platform's built-in nodes can't, and meet complex data processing and ops automation needs.

Task node statuses

Task status	Description	Can change to
Pending Releasing	Tasks that were saved but never successfully released Tasks that were never successfully released, after the Flow goes offline	Releasing Deleted
Releasing	Release in progress	Released Pending Releasing: rolled back after a failed release
Online	Changes to "Released" after a successful release Rolls back to "Pending Releasing" when the Flow goes offline	Pending Releasing Released (pending deletion)
Released (pending deletion)	A "Released" task moved to the recycle bin becomes "Released (pending deletion)" A "Released (pending deletion)" task restored from the recycle bin becomes "Released"	Deleted Released
Deleted	Deleting a "Pending Releasing" task changes it to "Deleted" Deletion can't be undone. This is a final state.	-

Flow node run instances

You can view all task run instances and their statuses in Ops → Task Node Instance.