Create and manage workspace tables
Overview
Workspace tables are the core assets of the DataOps Platform and the scope of data that can be used directly in Flows of the current workspace, that is, the tables (Table) or views (View) created directly in Dev and Catalog of the DataOps Platform. They exist in both the development and production environments, support dual-environment release, lifecycle management, hot/cold data separation, and more, and are suitable for primary data storage.
In version 5.1, the former workspace tables, former system tables, and link views are displayed together.
This section describes in detail how to create, delete, edit, and query workspace tables, helping you quickly get started with creating workspace tables and Table Management.
Get to know Catalog
The new Catalog page has five functional areas: the feature switching area, dictionary area, tab area, page content area, and drawer area.
For an introduction to each area, see Catalog
Create a workspace table
First, go to the Catalog page. You can create a workspace table in three ways on this page:
- Click the + button at the top of the dictionary area
- Click the + button at the end of the tab area
- Click + Table in the upper-right corner of the page content area
The Create New Table or View page opens
The management mode defaults to Workspace-Task table. You can choose a table type:
- Table: lets you define the table structure, storage format, storage method, and more for a physical table.
- View: lets you create a view based on base tables such as workspace task tables, system tables, and linked sources.
Note: In the content area of the Catalog page, view icons have a "👁️" so you can quickly tell them apart.
In Catalog, a newly created table goes into the development environment by default, and goes into the production environment after it's released. After a table or view is released, you can click Edit to switch to the development environment.
Create a table (Table)
On the new workspace table page, select Table as the table type, then click Next to go to the new table page (development environment)
Mode switching
Tables support two creation methods: Form Mode and DDL Mode. The table schema is parsed in both directions: you can preview the DDL in Form Mode, or parse it in DDL Mode and go back to Form Mode to edit it again.
- Form Mode and DDL Mode are two views of the same configuration, and they share one set of settings
- You can create an Internal Table / External Table as needed (note: once a table is created, its table type can't be changed)
- Once an external table path is saved, it can't be changed
- In DDL Mode, you must Parse before you can switch to Form Mode
Table configuration information
| Basic table information | |
|---|---|
| Field name | Description |
| Remarks | Remarks of the table |
| Lifecycle | The manageable lifecycle of the current workspace table |
| Owner | Owner of the workspace table |
| Basic Information | |
| Field name | Description |
| Table Name | Unique within the space. Must start with a lowercase letter, can contain lowercase letters, numbers, and underscores, and can't exceed 80 characters |
| Type | Table, View (View isn't selectable yet) |
| Comment | Display name (comment) of the table |
| Table Configuration | |
| Partition |
|
| Storage Type |
Once released, the table type can't be changed |
| Storage Method |
|
| Storage Format | Supports multiple storage formats, such as orc, parquet, text, csv, and json |
| Storage Path |
|
Note: Once a table is saved, its storage type, storage path, and so on are fixed and can't be changed.
Custom table properties
You can customize some table properties. The allowlist is as follows:
"format"
"partitioned_by"
"external_location"
"location"
"bucket_count"
"bucketed_by"
"sorted_by"
"bucketing_version"
Of these, format, partitioned_by, external_location, and location are parsed by the system.
Save options
- If you use Form Mode: after you finish configuring the fields in Form Mode, click Preview DDL in the upper-right corner to view the table's DDL statement and confirm the table structure.
- If you use DDL Mode: after you finish the DDL statement, click Parse in the upper-right corner and confirm the table structure in the field area at the lower right of the page. You can also go back to Form Mode to edit it manually.
Finally, after you finish configuring the workspace table, click Save only or Release in the upper-right corner of the page to create the workspace table. The difference between them is:
- Save only: creates the workspace table only in the development environment. You can release the production environment workspace table later once the configuration is finalized
- Release: creates the workspace table in both the development and production environments at the same time. It's equivalent to saving and releasing, and is suitable when all settings are complete.
Create a view (View)
You can use tables in custom Catalogs mounted to AE Trino as base tables to create views that can be used in the workspace. Views can be used in all modules of the DataOps Platform, for example, queried in Flows or authorized to AE analytics modules.
Views can be created one at a time or in batches.
Create a single view
On the new workspace table page, select View as the table type, then click Next
The new view page (development environment) opens:
You can create a view in two ways:
-
Enter the DDL statement directly, then click Debug
-
Click Dictionary in the upper-right corner to help you write the statement
- In the dictionary area, select the catalog, source database, and source table (or enter the table's fully qualified name to parse it)
Note: The catalog drop-down menu shows the display names of the Catalogs available for linking (the display name can be defined during configuration), the Catalog types, and their logos. If the Catalog you want to link doesn't appear in the drop-down menu, contact ThinkingAI staff to configure it.
- Then select the fields you need and click Generate Query in the upper-right corner of Table Field. The statement is automatically copied to your clipboard. Or click Generate DDL to copy it to the clipboard, then paste it into the DDL statement IDE to debug
After debugging succeeds, the field preview, data preview, and source base tables appear for you to check.
You can implement more complex view structures by writing SQL statements.
Finally, you need to rename the view. In the generated statement, the name defaults to VIEW + source table name. Change it in the statement as needed
Note: A new view also becomes a workspace table in the workspace, so it can't have the same name as an existing workspace table.
After you complete all the steps above, click Save only or Release in the upper-right corner of the page to create the view. The difference between them is:
- Save only: creates the workspace table only in the development environment. You can release the production environment workspace table later once the configuration is finalized
- Release: creates the workspace table in both the development and production environments at the same time. It's equivalent to saving and releasing, and is suitable when all settings are complete.
Create in batches
Batch release, batch delete, and other operations on views are currently supported.
In this version, go to Link Management for further operations. See Batch create link views
Table Management
The page content area shows all workspace tables (tables and views) in the current workspace, and provides a rich set of operations to help you find and use tables quickly and efficiently.
- Click the funnel button at the top to filter. The following filter conditions are currently supported:
- Changed
- I owned
- Expiring Soon
- Unpublished
- Management Mode: Workspace-Task table, Workspace-IDE table, System table, External authorized table
- Table Type: Physical table, View
- Owning entity
- Owner
- You can perform batch operations:
Hover over Batch Operation in the upper-right corner of the list to expand the batch operation menu. Batch renewal, batch release, and batch recycling are currently supported.
After you select a batch operation, the list shows the workspace tables on which that operation can be performed.
Note: Views don't need to be renewed
- You can also use Search to search table names and table comments to find the workspace table you need
Workspace table details page
The page content area of Table Management offers two ways to view the details of a workspace table:
- Click a blank area of the workspace table to open the drawer, where you can preview, renew, recycle, and quickly edit some basic information (table comment, owner, remarks). Click Edit to go to the details page.
- Click the table name to go to the details page. Different table types and statuses show different pages. Taking a standard workspace table as an example, the details page contains the following core information:
- Basic information: table name, environment, warehouse, catalog, database, and so on
- Field structure: column name, type, comment
- DDL statement: the table creation statement
- Data preview: the first N rows of data
- Partition details (for partitioned tables): partition data volume and storage size
Other table types may show different fields.
For Hive external partitioned tables, you can also configure automatic or manual partition sync. For details, see Partition table sync.
- The table management display items are listed below:
| Column name | Description |
|---|---|
| Table Name | Name of the workspace table. Click it to open the workspace table details page |
| Comment | User-defined description of the table |
| Warehouse name | Name of the warehouse that the workspace table belongs to |
| Catalog name | Name of the catalog that the workspace table belongs to |
| Database name | Name of the database that the workspace table belongs to |
| Status | Release status of the workspace table: Online or Not online |
| Management Mode | Defaults to Workspace-Task table |
| Related task name | Tasks related to the workspace table |
| Owning entity name | The space entity that the workspace table belongs to |
| Storage Type | Internal Table or External Table |
| Storage Format | Storage format of the workspace table, such as ORC |
| Storage Method | Storage method of the workspace table, such as HDFS |
| Storage Path | Storage path of the workspace table |
| Lifecycle Status | Lifecycle status of the workspace table: Normal or Expiring Soon. Filterable (a table enters the Expiring Soon status when its remaining lifecycle is less than 7 days; if the table's maximum lifecycle is less than 7 days, it always stays in the Expiring Soon status) |
| Lifecycle Days | Can be renewed |
| Owner | Owner of the workspace table |
| Remarks | User-defined remarks |
| Creator | Creator of the workspace table |
| Created at | Creation time of the workspace table |
| Last Releaser | Last releaser of the workspace table |
| Last Releasing time | Time of the workspace table's last release |
| Field preview | Field list of the workspace table |
| DDL | DDL statement of the workspace table |
| Data preview | Shows the data in the workspace table. Supports refresh |
| Usage scenarios (hidden by default) | Which scenarios in the DataApp module the workspace table can be used in |
| Create Time (hidden by default) | Creation time of the workspace table |
| Action - Edit | Click to open the workspace table edit page |
| Action - Renew | Click to manually renew the workspace table and refresh its lifecycle days |
| Action - Release | Click to release a workspace table that has changes or hasn't been released |
| Action - Recycle | Click to recycle the workspace table. Recycled workspace tables can be viewed in the workspace table recycle bin, where you can restore or permanently delete them |
- The view management display items are listed below:
| Column Name | Description |
|---|---|
| View Name | Name of the created view. Click the jump button to open the view details page |
| Comment | User-defined description of the view |
| Warehouse name | Name of the warehouse that the workspace table belongs to |
| Catalog name | Name of the catalog that the workspace table belongs to |
| Database name | Name of the database that the workspace table belongs to |
| Status | Release status of the view: Online or Not online |
| Management Mode | Defaults to Workspace-Task table |
| Owner | Owner of the view |
| Remarks | User-defined remarks |
| Lifecycle Status | Lifecycle status of the workspace table. Defaults to Normal for views |
| Creator | Creator of the workspace table |
| Created at | Creation time of the workspace table |
| Last Modifier | Last modifier of the workspace table |
| Last Modified Time | Time of the workspace table's last modification |
| Field preview | Field list of the workspace table |
| DDL | DDL statement of the workspace table |
| Data preview | Shows the data in the workspace table. Supports refresh |
| Source Tables | Fully qualified name of the source base table. Click it to open the table details page |
| Action - Edit | Click to open the link view edit page |
| Action - Release | Click to release an unreleased link view |
| Action - Recycle | Click to recycle the link view. Recycled views can be viewed in the workspace table recycle bin of the Dev module, where you can restore or permanently delete them |

