Model Mapping Assets Configuration
This is a value-added feature. Make sure the corresponding value-added service is enabled for your system version.
1. Feature overview
Model Mapping Assets Configuration: Converts cross-source data sources into data assets
With a series of configuration settings, Model Mapping Assets Configuration reads "non-AE-native data" and converts it into "data assets in AE"
For example: tags, events and event properties, entity properties, and so on
"Non-AE-native data" includes: external tables connected through the Link View feature, tables generated by the Integration feature, and tables generated by the Flow feature
| Asset Type | Usage scenarios | Downstream asset management |
|---|---|---|
| Tag table | Converts external data structures of the wide table or tall table type into AE model mapping tags | View on the Tags page |
| Event Table | Converts a wide table that records event information into events and event properties in AE | View in Source asset management under Events and Event Properties |
| Entity table | Converts a wide table that records user information into entity properties (user properties) in AE | View in Source asset management under User Properties |
| Dimension table | Converts a wide table of dimension relationships into dimension properties of a property | View on the Event Properties or User Properties page |
2. List page
Management page description
The fields are described as follows
| Field name | Description |
|---|---|
| Configuration Name | The name of the model-mapping configuration. It must start with a lowercase letter and can contain only lowercase letters, digits, and underscores
|
| Display Name | Chinese and English supported, fewer than 80 characters |
| Description | Chinese and English supported, fewer than 200 characters |
| Validation Status |
|
| Asset Type | Supported: event table, tag table, entity table, dimension table
|
| Data Source | The name of the data source in the DataOps Platform. Enter the full name in the format catalog.schema.table |
| Data source description | The data source description read from the DataOps Platform |
| Table type | The logic used to parse the data source. The enum values are
|
| Configuration Details | Key configuration information for parsing assets |
| Created at | The time the model-mapping configuration was created |
| Action |
|
- With bulk upload, you can edit configuration information in Excel and generate multiple model-mapping configurations directly. For details, see Bulk upload of model-mapping configurations
3. Create a configuration on the page
Click [Create] in the upper-right corner of the list page to start creating a model-mapping configuration on the page. Select the type of model mapping asset to create
Select the model mapping asset type
Creating on the page takes three steps: Data Source, Model-Mapping Configuration, and Basic information
3.1 Configure the data source
Model-mapping configuration flow 1
3.1.1 Select a data source
Select a data source that the DataOps Platform has authorized for the current project
If the data source list is empty or you can't find the data source you need, contact your DataOps team to confirm whether the data source has been authorized for the current project
Initial state
Data source list
After you select a data source, click [Preview] to get an overview of the data source
Including: data preview, table structure preview, and SQL statement
3.1.2 Data preview
The system randomly queries 30 sample rows from the data source and displays them, so you can check the content format of the values
For example: the text format of name data and the precision of time data
Data preview
3.1.3 Table structure preview
Shows information about all columns in the data source, including name, data type, and description, so you can check whether values need type conversion
For example: "88" (string) can't be used in arithmetic operations and needs to be converted to 88.0 (number) with try_cast()
Table structure preview
3.1.4 SQL statement
Shows the SQL statement that queries the data source, so you can copy it directly into SQL IDE
For example: add a where condition to find abnormal data, or test SQL functions for value processing
SQL statement preview
3.2 Model-mapping configuration
Model-mapping configuration flow 2
This step configures two kinds of information:
- How key fields in the data source are parsed
- The list of model mapping assets to output
Preview of model-mapping fields and the asset list
3.2.1 Configure fields
Because the parsing logic differs, each asset type has its own configuration fields and logic, which are described in the following documents:
- Model-mapping tag table
- Model-mapping dimension table
- Model-mapping entity table
- Model-mapping event table
3.2.2 Asset list
The asset list generates the corresponding assets after the data source is read
For example: after reading a tag data source, multiple tags are generated, and each tag needs settings such as name, value acquisition method, output type, description, and enum values
⚠ Different asset types may have extra fields during configuration. Only the common features are described here. The extra parts are explained by scenario in the document for each type
Asset list preview
a. Add assets
The system provides two ways to add assets
🟦 Scan Assets
- After you fill in the configuration fields, the system scans the data source according to the corresponding asset logic and generates an initial asset list, which you can then edit or delete from
- After the scan, the "original asset type" follows the type in the data source and is filtered by the types AE allows. For unsupported types, the system shows the underlying type name in the data source, such as object or binary
🟦 Add New
- Click it to add a blank record to the first row of the asset list, where you can fill in the asset information manually
- When you add an asset manually, the "original asset type" is empty by default and is updated automatically after validation
b. Count overview and error display
- The upper-right corner of the feature area shows "total data" and "incorrect data", giving you an overview of the current asset list
- When you select Show only abnormal assets, the system filters and shows the entries in the list with input errors
⚠ Error validation covers only the "current configuration". If assets entered here conflict with other assets in the project, the error details are validated and shown when you submit
Abnormal asset display preview
c. List fields and features
Depending on the asset type, list fields may be named "tag", "property", and so on. For ease of description, they are collectively called "assets" below
Each configuration outputs one model mapping asset. The configuration information covers three categories: input, processing, and output
🔵 Original asset name
The original input asset that the model mapping asset corresponds to
For example, in a wide-table tag structure, each column corresponds to one tag, so enter the original column name here
🔵 Original asset type
After an asset scan or validation, the system automatically reads and shows the original column data type of the data source, such as string, number, or date
⚠ For unsupported types, the system shows the underlying type name in the data source, such as object or binary. In this case, handle them with a value acquisition method
🔵 Output asset name & output asset display name
The name and display name of the model mapping asset generated by the system. They must meet the naming requirements of the corresponding asset type
For example: tag names can be up to 80 characters, and event names up to 60 characters
🔵 Value Acquisition Method
If left blank, the system reads values directly by the "original asset name"
To process values, enter an SQL function together with column names in the data source
For example: use the try_cast( pay_amount as double ) function to read the pay_amount column in the data source and convert it to a number type
⚠ After you enter a value acquisition method, the system ignores the "original asset name", which enables logic such as concatenating multiple columns or splitting a single column
⚠ Child properties of an Array Row don't support value acquisition methods. Special handling must be set at the parent property level
🔵 Output type
By default, keep the output asset type the same as the original asset type
⚠ After you enter a Value Acquisition Method, select the processed data type here. For example, if the original type of col is string and it's processed with try_cast( col as double), select Number here
🔵 Expansion configuration
Asset information other than the above is collapsed into Expansion configuration. Click it to edit in a dialog. The common expansion information is Description
⚠ Different types of model mapping assets add extra expansion fields in specific scenarios. These are described in the document for the corresponding asset type
Expansion configuration dialog preview
🔵 Enum Value Logic
When you build reports and dashboards with analysis models and filter by an asset, the system shows a drop-down list of the asset's enum values
Property enum value drop-down preview
When you configure model mapping assets, you can set their enum values with the enum value logic to make filtering easier for business members
When editing, you can download the CSV reference template first, modify it, and upload it directly
⚠ If you export CSV files with other tools, make sure to set them to UTF-8 encoding
⚠ Only string, number, and list types support enum values. For lists, the enum values apply to the elements in the list
Enum value logic preview
🔵 Status
The validation status of the current model mapping asset. The default status is Not Validated, and the result is shown after validation
Possible errors include: a column doesn't exist in the data source, an error in the value acquisition method or output type, and so on
⚠ Errors in the value acquisition method or output type need to be adjusted based on the actual data and SQL statement. Preview the data source before filling in, and copy the SQL statement into <SQL IDE> to verify it
🔵 Action
Each asset configuration has two actions on the right: Copy and Delete
-
Copy
- Copies the current asset configuration and adds the "_copy" suffix to the name of the new asset configuration
-
Delete
-
Deletes the current asset configuration. The result appears as a floating message on the page with an undo button
Delete tip preview
-
3.3 Basic information of the model-mapping configuration
Model-mapping configuration flow 3
Identifying information that every model-mapping configuration must have
Basic information fields
-
Model-Mapping Configuration Name
- The name of the current Model Mapping Assets Configuration
- Must start with a lowercase letter and can contain only lowercase letters, digits, and underscores. Fewer than 80 characters
-
Model-mapping configuration display name
- The display name of the current Model Mapping Assets Configuration
- Fewer than 80 characters
-
Model-mapping configuration description
- The description of the current Model Mapping Assets Configuration
- Fewer than 200 characters
4. Data types
- Names such as Number, String, and Boolean on the event and property pages are the "select_type", used by members to make choices in business scenarios and in analysis models
- Names such as bigint and double in the underlying data warehouse tables are the "raw_type", the data types of the underlying tables
- The mapping is as follows:
| UI type (select_type) | Underlying type (raw_type) |
|---|---|
| Numeric | tinyint,smallint,int,bigint,double,decimal |
| Text | varchar |
| Boolean | boolean |
| Time | datetime |
| List | array(varchar/timestamp/boolean/bigint/double... ) |
| Object | row( col_str varchar, col_num double, col_array array<varchar> ) |
| Object group | array( row( col_str varchar, col_num double, col_array array<varchar>) ) |
Data types supported inside List: Number, String, Boolean, and Date. Data types supported inside Row and Array Row: Number, String, Boolean, Date, and List (only String inside).
5. Limits
-
A single project can have up to 200,000 model mapping assets (regardless of asset type)
Q: How do I see which model mapping assets currently exist?
A: Go to the Assets usage governance page and filter by Model Mapping Assets to view the detailed list
6. FAQ
How do I check the status of a model mapping asset?
- After creating the asset, click Validate in the list. The system reads the configuration, checks availability, and returns the validation result
Why can't I use a model mapping asset after creating it?
- Assets in Not Validated or Validation Failed status can be used to create reports and dashboards, but no valid results are returned during calculation
- Run validation, modify the configuration as prompted, and only after the status becomes Validated can the related assets be calculated and return valid results
Validation failed with the message "xxxx". What's the cause, and how do I fix it?
-
Field validation
- Fine-grained validation based on the configuration fields, which returns relatively clear messages
- For example:
Data source does not exist or no permission,事件名称列 event_name 不存在, and so on - In this case, modify as prompted
-
SQL validation
- For a Value Acquisition Method that involves SQL execution, validation takes into account factors such as SQL syntax, data types, and calculation logic, so the returned results are more complex
- For example:
Cannot apply operator: varchar(4) + integer, and so on - In this case, check the asset configurations that use SQL statements, and troubleshoot with information such as the error message, asset type, and SQL function