Configure a Redis data source
Redis is an open-source in-memory data structure store that is widely used as a database, cache, and message broker.
This article describes the data synchronization capabilities supported for Redis.
Supported versions
redis 3.0+
Limitations
Currently, only the following configuration is supported:
- Writing from AE Built-in Warehouse workspace tables to Redis (offline write)
Note:
- Reading from Redis data sources (offline read) isn't supported yet
Supported field types
| Field type | Offline write (Writer) |
|---|---|
| tinyint | Supported |
| smallint | Supported |
| integer | Supported |
| bigint | Supported |
| varchar | Supported |
| boolean | Supported |
| double | Supported |
| date | Supported |
| timestamp | Supported |
array(tinyint) array(smallint) array(integer) array(bigint) array(varchar) array(boolean) array(double) | Supported |
| array(others) | Not supported |
| row | Supported |
| map | Supported |
| varchar | Supported |
Create a Redis data source
In the DataOps Platform - Integration module, you can choose to add a Redis data source.
Data source configuration parameters
Fill in the configuration required by the data source and pass the connectivity test to create the Redis data source.
Parameters whose names start with * are required; parameters without * are optional.
| Field name | Description |
|---|---|
| Basic Information | |
| *Datasource Name | Must be unique within the DataOps Platform space. Can contain only letters, digits, and underscores, and can't start with a digit or an underscore |
| Remarks | Optional |
| Data source configuration | |
| not distinguish environment / Independent setting | Choose one: not distinguish environment means the production and development environments share one configuration; Independent setting means the two environments are configured independently |
| *Mode |
|
| *Server Address/IP | Address of the server where the Redis database runs For cluster deployment, click Add to enter information for multiple nodes |
| *Port | Port used to access Redis |
| Username | Username with permission to access the database |
| *Password | Password of the username |
| Advanced | Other advanced parameters required to connect to the database; you can customize them |
Create an offline sync task
After you create the Redis data source and pass the connectivity test as described above, you can configure a Redis offline write task for your scenario.
Redis as the Data Target
Select Redis as the Data Target and configure the following parameters:
| Field name | Description |
|---|---|
| *Source Type | Select Redis as the target type of the Data Target |
| *Datasource Name | A Redis data source registered on the data source management page; select it from the drop-down list. If you haven't created the data source yet, click the Data sources management button to create a Redis data source. |
| *Database | Redis db slot index that identifies the database *This parameter is available only for sources in non-cluster mode |
*Written Data Type | Data type of the Redis Value Four types are supported: string, set, hash, and list. |
| *Write Mode |
|
| Batch size | Batch size: Amount of data submitted per batch; 1000 records by default |
| Expiration Time | When writing data to Redis, you can set an expiration time for keys, in days, hours, minutes, or seconds. The switch is off by default |
Parameters whose names start with * are required; parameters without * are optional.
Field mapping
After configuring the data source and the target, parse the fields and configure how Redis Keys are stored.
Index key concatenation strategy
Redis is a key-value store, and the design of keys directly affects query efficiency and maintainability.
For ease of use, key concatenation and storage support the following:
- Writing specified fields (that is, not the entire table)
- Key prefix and suffix configuration
- Inserting Parameter (space parameters)
- Inserting Special Characters
- The key as a whole must pass rule validation
Index key configuration
| Key settings | ||
|---|---|---|
| Field name | Description | Required |
| Key prefix/suffix | Supports custom prefixes and suffixes Only [0-9A-Za-z-_] are allowed, to prevent pollution of Redis keys. Example: student:tuition:${bd} | No |
| Insert field value | Only fields already checked in the source table on the left can be selected | No |
| Parameter | Supports preset space parameters and custom space parameters | No |
| Insert special characters | For example: \u0001 | No |
| Data preview | Shows 3 records by default | |
Basic information settings
Finally, set the basic information of the integration plan, including the plan name, owner, synchronization rate, and remarks.
When you're done, click Save to create the integration plan.
Note: The plan name can't be changed after it's saved
The details page looks like this:
Mount an offline sync plan on a Flow
Mount on Flow
1. Start mounting
- On the integration plan details page, click Mount on task flow in the upper-right corner
2. Select a Flow
-
Select the target Flow from the drop-down menu
-
To create a new Flow:
- Click the New Flow shortcut button below the drop-down menu
- Or go to the Dev module to create one
-
Tip: If the target Flow isn't shown, click the ↻ refresh button on the right
3. Create a sync node
- In the Flow, create a node of the Offline sync plan type
- The node is linked to the current integration plan. When the node runs, it triggers a run of that integration plan
4. Complete mounting
- Click Create node and mount on it to complete the configuration
- After mounting succeeds, click Go to flow page to view the result right away
Note: The task node created by mounting is in the unreleased state. We recommend going to the Flow and releasing the node.
Unmount from a Flow
To unmount an offline sync plan from a Flow:
- If the Flow hasn't been released yet, go to the Flow that the plan is mounted on and delete the Offline sync plan task node in Dev Mode.
- If the Flow has already been released, after deleting the Offline sync plan task node in Dev Mode, release the Flow again. This also unmounts the node from the Production environment.