LogBus Windows version guide
This section describes how to use the Windows version of the data transfer tool LogBus:
Before you start the integration, read the data rules. After you are familiar with the TE data format and data rules, follow this guide to complete the integration.
Data uploaded by LogBus must follow the TE data format
Download LogBus for Windows
Latest version: 1.3.0
Update time: 2021-10-19
1. About LogBus
LogBus imports backend log data into the TE backend in real time. Its core working principle is similar to Flume: it monitors the file streams in the log directories on the server. When new data is written to any log file in a directory, LogBus validates the new data and sends it to the TE backend in real time.
We recommend LogBus for the following types of users:
- Users of server-side SDKs, who upload data through LogBus
- Users with high requirements for data accuracy and dimensions, whose data needs cannot be met by client SDKs alone, or for whom integrating client SDKs is inconvenient
- Users who do not want to develop their own backend data push process
- Users who need to transfer large volumes of historical data
2. Prepare data before use
1. First, use ETL to convert the data to be transferred into the TE data format, and write it to local files or send it to a Kafka cluster. If you use the consumer of a server-side SDK that writes to local files or Kafka, the data is already in the correct format and does not need to be converted.
2. Determine the directory where the data files to upload are stored, or the Kafka address and topic, and configure LogBus accordingly. LogBus monitors file changes in the directory (new files are created or existing files are tailed), or subscribes to data in Kafka.
3. Do not directly rename data logs that are stored in the monitored directory and have already been uploaded. Renaming a log is equivalent to creating a new file, and LogBus may upload these files again, which causes duplicate data.
4. Because the LogBus data transfer component includes a data buffer, the LogBus directory may take up a fair amount of disk space. Make sure that the node where LogBus is installed has enough disk space: reserve at least 10G of storage for each project you transfer data to (that is, for each APP_ID you add).
3. Install and upgrade LogBus
3.1 Install LogBus
1. Download the LogBus package and unzip it.
2. Directory structure after unzipping:
- bin: Startup program folder
- conf: Configuration file folder
- lib: Function library folder
4. Configure LogBus parameters
1. Go to the unzipped conf directory, which contains the configuration file logBus.conf.Template. This file contains all LogBus configuration parameters. Before you use LogBus for the first time, rename it to logBus.conf.
2. Open the logBus.conf file and configure the parameters
4.1 Project and data source configuration (required)
- Project APP_ID
Do not configure the same APP_ID more than once
##APPID is the token from the tga official website. Get the APPID of the project from the project configuration page in the TE backend and enter it here. Separate multiple APPIDs with ","
APPID=APPID_1,APPID_2
- Monitored file configuration (choose one; required)
4.1.1. When the data source is local files
##Path and file name of the data files that LogBus reads (fuzzy matching is supported for file names). Read permission is required
##Separate different APPIDs with commas, and different directories of the same APPID with spaces
##File names in TAIL_FILE support wildcard matching
TAIL_FILE=C:/path1/dir*/log.*,C:/path3/txt.*
TAIL_FILE supports monitoring multiple files in multiple subdirectories under multiple paths
The corresponding parameter configuration is:
APPID=APPID1,APPID2
TAIL_FILE=C:/root/log_dir1/dir_*/log.* C:/root/log_dir*/log*/log.*,C:/test_log/*
The rules are as follows:
- Multiple monitoring paths for the same APP_ID are separated by spaces
- Monitoring paths for different APP_IDs are separated by commas "
,", and the comma-separated paths correspond to the APP_IDs in order - Directories in monitoring paths can be matched with wildcards
- File names can be matched with wildcards
- You can use "
/" or "\\\\" as the path separator. Do not use "\". For example:C:/root/_.logorC:\\\\root\\\\_.log
Do not store the log files to be monitored in the root directory of the server.
4.1.2. When the data source is Kafka
To monitor multiple topics with the KAFKA_TOPICS parameter, separate the topics with spaces. If there are multiple APP_IDs, separate the topics monitored for each APP_ID with half-width commas. The KAFKA_GROUPID parameter must be unique. The KAFKA_OFFSET_RESET parameter sets the Kafka kafka.consumer.auto.offset.reset parameter. Valid values are earliest and latest. The default is earliest.
Note: The Kafka version of the data source must be 0.10.1.0 or later
Single APP_ID example:
APPID=appid1
######kafka configuration
#KAFKA_GROUPID=tga.group
#KAFKA_SERVERS=localhost:9092
#KAFKA_TOPICS=topic1 topic2
#KAFKA_OFFSET_RESET=earliest
Multiple APP_IDs example:
APPID=appid1,appid2
######kafka configuration
#KAFKA_GROUPID=tga.group
#KAFKA_SERVERS=localhost:9092
#KAFKA_TOPICS=topic1 topic2,topic3 topic4
#KAFKA_OFFSET_RESET=earliest
4.2 Transfer parameter configuration (required)
##Transfer settings
##Destination URL
##For HTTP transfer, use
PUSH_URL=https://global-receiver-ta.thinkingdata.cn/logbus
##If you use an on-premises deployment, change the transfer URL to: http://YOUR_RECEIVER_URL/logbus
##Maximum number of records per transfer
#BATCH=10000
##Transfer at least once within this interval (unit: seconds)
#INTERVAL_SECONDS=600
##Number of transfer threads. Single-threaded by default. Recommended when network conditions are poor; multiple threads consume more memory and CPU
#NUMTHREAD=1
##Compression format for file transfer: gzip,snappy,none
#COMPRESS_FORMAT=none
4.3 Converter configuration (optional)
##Currently supported converter types: json csv regex splitter
#PARSE_TYPE=json
##Additional fixed properties, in the format: name value,name1 value1
#LABELS=
##Property names and types, applicable when PARSE_TYPE is csv, regex, or splitter, in the format: name type,name1 type1
##Supported types: float int string date list bool
#SCHEMA=
##Separator. Cannot be empty when PARSE_TYPE is csv or splitter
#SPLITTER=
##Separator for the list type, applicable when a list type exists. Default: ,
#LIST_SPLITTER=,
##Regular expression. Cannot be empty when PARSE_TYPE is regex
#FORMAT_REGEX=
4.4 Monitored file deletion configuration (optional)
# Deletion of files in monitored directories. Uncomment to enable file deletion
# Files can only be deleted by day (day) or by hour (hour)
# UNIT_REMOVE=hour
# How old files must be before they are deleted
# OFFSET_REMOVE=20
# Interval in minutes for deleting monitored files that have been uploaded
# FREQUENCY_REMOVE=60
4.5 Configuration file example
##################################################################################
## logBus configuration file, the transfer tool of the thinkingdata data analytics platform
##Uncommented parameters are required, and commented parameters are optional. Configure them
##as appropriate for your situation
##Requirements: java8+. For details, see the tga official website
##https://docs.thinkingai.cn/zh/integration/logbus_legacy_installation
##################################################################################
##APPID is the token from the tga official website
##Separate different APPIDs with commas. Do not configure the same APPID more than once
APPID=from_tga1,from_tga2
#-----------------------------------source----------------------------------------
######file-source
##Path and file name of the data files that LogBus reads (fuzzy matching is supported for file names). Read permission is required
##Separate different APPIDs with commas, and different directories of the same APPID with spaces
##File names in TAIL_FILE support wildcard matching
TAIL_FILE=C:/path1/log.* C:/path2/txt.*,C:/path3/log.* C:/path4/log.* C:/path5/txt.*
######kafka-source
#KAFKA_GROUPID=tga.flume
#KAFKA_SERVERS=
#KAFKA_TOPICS=
#KAFKA_OFFSET_RESET=earliest
#------------------------------------sink-----------------------------------------
##Transfer settings
##Destination URL
##If you use an on-premises deployment, change the transfer URL to: http://${RECEIVER_URL}/logbus (RECEIVER_URL is the data collection URL)
##PUSH_URL=https://global-receiver-ta.thinkingdata.cn/logbus
PUSH_URL=http://${RECEIVER_URL}/logbus
##Maximum number of records per transfer
#BATCH=10000
##Transfer at least once within this interval (unit: seconds)
#INTERVAL_SECONDS=60
##### HTTP transfer
##Compression format for file transfer: gzip,snappy,none
#COMPRESS_FORMAT=none
##Whether to add a uuid property to each record
#IS_ADD_UUID=true
#------------------------------------parse----------------------------------------
##Currently supported converter types: json csv regex splitter
#PARSE_TYPE=json
##Additional fixed properties, in the format: name value,name1 value1
#LABELS=
##Property names and types, applicable when PARSE_TYPE is csv, regex, or splitter, in the format: name type,name1 type1
##Supported types: float int string date list bool
#SCHEMA=
##Separator. Cannot be empty when PARSE_TYPE is csv or splitter
#SPLITTER=
##Separator for the list type, applicable when a list type exists. Default: ,
#LIST_SPLITTER=,
##Regular expression. Cannot be empty when PARSE_TYPE is regex
#FORMAT_REGEX=
#------------------------------------other----------------------------------------
##Deletion of files in monitored directories. Uncomment the fields (both fields below must be uncommented) to enable file deletion. The deletion program runs once an hour
##Delete files older than offset, in units of unit
##How old files must be before they are deleted
#OFFSET_REMOVE=
##Only deletion by day (day) or hour (hour) is accepted
#UNIT_REMOVE=
5. Start LogBus
Before you start LogBus for the first time, perform the following checks:
1. Check the Java version
Go to the bin directory, which contains two scripts: check_java.bat and logbus.bat
check_java checks whether the Java version meets the requirements. Run the script. If the Java version does not meet the requirements, a message such as Java version is less than 1.8 or Can't find java, please install jre first. appears
You can update the JDK version, or see the next section to install a separate JDK for LogBus
2. Install a separate JDK for LogBus
Use this feature if the JDK version on the LogBus deployment node does not meet LogBus requirements because of the environment, and cannot be replaced with a JDK version that does.
Go to the bin directory, which contains install_logbus_jdk.bat.
Running this script adds a java directory to the LogBus working directory. LogBus uses the JDK environment in this directory by default.
3. Complete the logBus.conf configuration and run the parameter and environment check command
For how to configure logBus.conf, see the Configure LogBus parameters section
After the configuration is complete, run the env command to check whether the configuration parameters are correct
logbus.bat env
If red error messages are output, the configuration has problems. Modify it until no errors are reported for the configuration file.
After you modify the logBus.conf configuration, restart LogBus for the new configuration to take effect
4. Start LogBus
logbus.bat start
After startup, a logkit.exe process starts. Do not close it; otherwise, data may be uploaded repeatedly
6. LogBus commands
6.1 Help information
If you run the command without arguments or with --help or -h, the help information is displayed
The LogBus commands are as follows:
usage: logbus <command|auxiliary command> [options]
Commands:
start Start logBus.
restart Restart logBus.
stop Safely quit logBus.
reset Reset logBus read records.
stop_atOnce Force quit logBus.
Auxiliary commands:
env Check the runtime environment.
server [-url <url>|-url <url> -appid <appid>] Test the network connection to the receiver
show_conf Show the current logBus configuration.
version Show the version number.
update Update logbus to the latest version.
Options:
-appid <appid> Project appid
-h,--help Show the help and exit.
-path <path> Absolute path of the test file
-url <url> URL to test
Examples:
logbus.bat start Start logBus.
logbus.bat stop Safely quit logBus.
logbus.bat restart Restart logBus.
logbus.bat server -url http://${YOUR_RECEIVER_URL}/logbus -appid ***** Test the network connection to the receiver
6.2 Check the transfer channel server -url
After you complete the format check, you also need to check whether the data channel is connected. You can use the server -url command to check it, and you can enter the APP_ID you received from the TE platform at the same time. Note that the APP_ID is bound to your project, so make sure that the APP_ID you enter corresponds to your project
logbus.bat server -url http://${YOUR_RECEIVER_URL}/logbus -appid ${appid}
6.3 Show the configuration show_conf
You can use the show_conf command to view the LogBus configuration. The output is shown in the following figure:
logbus.bat show_conf
6.4 Check the startup environment env
You can use env to check the startup environment. If any output line ends with an asterisk, the configuration has a problem. Modify it until no asterisks appear.
logbus.bat env
6.5 Upgrade LogBus update
You can use update to upgrade online. This command updates LogBus to the latest version
logbus.bat update
6.6 Start start
After you complete the format validation, the data channel check, and the environment check, you can start LogBus to upload data. LogBus automatically detects whether new data has been written to your files and, if so, uploads it.
logbus.bat start
6.7 Stop stop
To stop LogBus, use the stop command. This command takes some time, but no data is lost.
logbus.bat stop
6.8 Stop immediately stop_atOnce
To stop LogBus immediately, use the stop_atOnce command. This command may cause data loss.
logbus.bat stop_atOnce
6.9 Restart restart
You can use the restart command to restart LogBus, for example, to apply new configuration after you modify configuration parameters.
logbus.bat restart
6.10 Reset reset
Running reset resets LogBus. Use this command with great caution: it clears the file transfer records, and LogBus uploads all data again. If you use this command when you are not sure of the situation, your data may be duplicated. We recommend that you consult ThinkingAI staff before you use it.
logbus.bat reset
After you run the reset command, run start to start transferring data again
6.11 View the version number version
To find out the version number of the LogBus you are using, run the version command. If your LogBus does not have this command, you are using an early version
logbus.bat version
7. ChangeLog
Version 1.3.0 --- 2021/10/19
- Supported uploading data that is not in TE format
Version 1.2.0 --- 2021/05/26
- Supported cygWin mode
Version 1.1.0 --- 2020/08/28
- Supported adding #UUID
- Supported #event_id and #first_check_id
- Supported multi-threaded sending
- Supported separator parsing and regex parsing
Version 1.0.0 --- 2020/06/25
- Released LogBus-Windows