LogBus guide

Last updated 10/03/2026

This section describes how to use the data transfer tool LogBus:

For the Windows version of LogBus, see LogBus Windows version guide

Before you start the integration, read the data rules. After you are familiar with the TE data format and data rules, follow this guide to complete the integration.

Data uploaded by LogBus must follow the TE data format

Download LogBus

Latest version: 1.5.17

Update time: 2022-11-08

Download link

Download link for the Linux ARM version

Upgrade notes:

• Version 1.5.0 and later:

First run the ./logbus stop command to stop LogBus. After it has stopped, run the ./logbus update command to upgrade to the latest version

• If you use a version earlier than 1.5.0 and need to upgrade to the new version, contact ThinkingAI staff

1. About LogBus

LogBus imports backend log data into the TE backend in real time. Its core working principle is similar to Flume: it monitors the file streams in the log directories on the server. When new data is written to any log file in a directory, LogBus validates the new data and sends it to the TE backend in real time.

We recommend LogBus for the following types of users:

1. Users of server-side SDKs, who upload data through LogBus
2. Users with high requirements for data accuracy and dimensions, whose data needs cannot be met by client SDKs alone, or for whom integrating client SDKs is inconvenient
3. Users who do not want to develop their own backend data push process
4. Users who need to transfer large volumes of historical data

2. Prepare data before use

1. First, use ETL to convert the data to be transferred into the TE data format, and write it to local files or send it to a Kafka cluster. If you use the consumer of a server-side SDK that writes to local files or Kafka, the data is already in the correct format and does not need to be converted.

2. Determine the directory where the data files to upload are stored, or the Kafka address and topic, and configure LogBus accordingly. LogBus monitors file changes in the directory (new files are created or existing files are tailed), or subscribes to data in Kafka.

3. Do not directly rename data logs that are stored in the monitored directory and have already been uploaded. Renaming a log is equivalent to creating a new file, and LogBus may upload these files again, which causes duplicate data.

4. Because the LogBus data transfer component includes a data buffer, the LogBus directory may take up a fair amount of disk space. Make sure that the node where LogBus is installed has enough disk space: reserve at least 10G of storage for each project you transfer data to (that is, for each APP_ID you add).

3. Install and upgrade LogBus

3.1 Install LogBus

1. Download the LogBus package and unzip it.

2. Directory structure after unzipping:

1. bin: Startup program folder
2. conf: Configuration file folder
3. lib: Function library folder

3.2 Upgrade LogBus

If you use version 1.5.0 or later, first run the ./logbus stop command to stop LogBus, then run the ./logbus update command to upgrade LogBus to the latest version, and then restart LogBus

3.3 Docker version

To use LogBus in a Docker container, see the LogBus Docker guide

4. Configure LogBus parameters

1. Go to the unzipped conf directory, which contains the configuration file logBus.conf.Template. This file contains all LogBus configuration parameters. Before you use LogBus for the first time, rename it to logBus.conf.

2. Open the logBus.conf file and configure the parameters

4.1 Project and data source configuration (required)

• Project APP_ID

##APPID is the token from the tga official website. Get the APPID of the project from the project configuration page in the TE backend and enter it here. Separate multiple APPIDs with ","

APPID=APPID_1,APPID_2

• Monitored file configuration (choose one; required)

4.1.1. When the data source is local files

##Path and file name of the data files that LogBus reads (fuzzy matching is supported for file names). Read permission is required
##Separate different APPIDs with commas, and different directories of the same APPID with spaces
##File names in TAIL_FILE support two modes: standard Java regular expressions and wildcards
TAIL_FILE=/path1/dir*/log.* /path2/DATE{YYYYMMDD}/txt.*,/path3/txt.*
##TAIL_MATCHER specifies the fuzzy matching mode for TAIL_FILE paths: regex (regular expression) or glob (wildcard).
##regex is the regular expression mode. It supports standard Java regular expressions, but fuzzy matching only works for one directory level and the file name
##glob is the wildcard mode. It supports fuzzy matching across multiple directory levels, but does not support DATE{} matching
##regex matching is used by default
#TAIL_MATCHER=regex

TAIL_FILE supports monitoring multiple files in multiple subdirectories under multiple paths. The following figure shows an example:

APPID	Monitored directory path	Subdirectories of the monitored directory	Monitored files	TAIL_FILE
APPID1	/root/log_dir1	dir_a、dir_b、dir_c	log.1 in each subdirectory	/root/log_dir1/dir_*/log.*
APPID1	/root/log_dir2/log	20190101、20190102	log.1 in 20190101; log.1, log.2, and log.3 in 20190102	/root/log_dir2/log/DATE{YYYYMMDD}/log.*
APPID2	/test_log	—	a.log	/test_log/*

View original image

The corresponding parameter configuration is:

APPID=APPID1,APPID2

TAIL_FILE=/root/log_dir1/dir_*/log.* /root/log_dir2/log/DATE{YYYYMMDD}/log.*,/test_log/*

The rules are as follows:

• Multiple monitoring paths for the same APP_ID are separated by spaces
• Monitoring paths for different APP_IDs are separated by half-width commas, and the comma-separated paths correspond to the APP_IDs in order
• Subdirectories in monitoring paths (that is, the directories where the files are located) can be matched by date format, regular expression, or wildcard
• File names can be matched by regular expression or wildcard

Do not store the log files to be monitored in the root directory of the server.

Rules for date-format subdirectories (regex mode only):

A date-format subdirectory must wrap the date template in DATE{}, and DATE must be uppercase. The following examples show several recognizable date templates and the files they monitor. Other templates also work, as long as the date template is a standard date format.

1. /root/logbus_data/DATE{YYYY-MM-DD}/log.* ---> /root/logbus_data/2019-01-01/log.1
2. /root/logbus_data/DATE{YYMMDD}/log.*---> /root/logbus_data/190101/log.1
3. /root/logbus_data/DATE{MM_DD_YYYY}/log.*---> /root/logbus_data/01_01_2019/log.1
4. /root/logbus_data/DATE{MM*DD}/log.*---> /root/logbus_data/01*01/log.1

---

4.1.2. When the data source is Kafka

Since version 1.5.2, the KAFKA_TOPICS parameter no longer supports regular expressions. To monitor multiple topics, separate the topics with spaces. If there are multiple APP_IDs, separate the topics monitored for each APP_ID with half-width commas. The KAFKA_GROUPID parameter must be unique. Version 1.5.3 added the KAFKA_OFFSET_RESET parameter, which sets the Kafka kafka.consumer.auto.offset.reset parameter. Valid values are earliest and latest. The default is earliest.

Note: The Kafka version of the data source must be 0.10.1.0 or later

Single APP_ID example:

APPID=appid1

######kafka configuration
#KAFKA_GROUPID=tga.group
#KAFKA_SERVERS=localhost:9092
#KAFKA_TOPICS=topic1 topic2
#KAFKA_OFFSET_RESET=earliest

#Optional cloud providers: tencent/ali/huawei
#KAFKA_CLOUD_PROVIDER=
#KAFKA_INSTANCE=
#KAFKA_USERNAME=
#KAFKA_PASSWORD=

#SASL authentication
#KAFKA_JAAS_PATH=
#KAFKA_SECURITY_PROTOCOL=
#KAFKA_SASL_MECHANISM=

Multiple APP_IDs example:

APPID=appid1,appid2

######kafka configuration
#KAFKA_GROUPID=tga.group
#KAFKA_SERVERS=localhost:9092
#KAFKA_TOPICS=topic1 topic2,topic3 topic4
#KAFKA_OFFSET_RESET=earliest

#Optional cloud providers: tencent/ali/huawei
#KAFKA_CLOUD_PROVIDER=
#KAFKA_INSTANCE=
#KAFKA_USERNAME=
#KAFKA_PASSWORD=

#SASL authentication
#KAFKA_JAAS_PATH=
#KAFKA_SECURITY_PROTOCOL=
#KAFKA_SASL_MECHANISM=

4.2 Transfer parameter configuration (required)

##Transfer settings
##Destination URL

##For HTTP transfer, use
PUSH_URL=https://global-receiver-ta.thinkingdata.cn/logbus
##If you use an on-premises deployment, change the transfer URL to: http://YOUR_RECEIVER_URL/logbus

##Whether to enable APPID checking. Disabled by default
#IS_CHECK_APPID=false

##Maximum number of records per transfer
#BATCH=10000
##Transfer at least once within this interval (unit: seconds)
#INTERVAL_SECONDS=60
##Number of transfer threads. The actual number of threads is the configured number + 1. Two threads by default
#NUMTHREAD=1

##Whether to add a uuid property to each record (enabling it reduces transfer efficiency)
#IS_ADD_UUID=true

##Compression format for file transfer: gzip,snappy,none
#COMPRESS_FORMAT=none

4.3 Flume memory parameter configuration (optional)

# Flume channel capacity settings
# Channel capacity. Set it based on the configuration of the deployment machine.
CAPACITY=1000000

# Transfer volume from the channel to the sink. Must be greater than the BATCH parameter
TRANSACTION_CAPACITY=10000

##Maximum memory for starting Flume, in MB.
#MAX_MEMORY=2048

# Flume channel type: file or memory (optional; file is used by default)
# CHANNEL_TYPE=file

4.4 Monitored file deletion configuration (optional)

# Deletion of files in monitored directories. Uncomment to enable file deletion
# Files can only be deleted by day (day) or by hour (hour)
# UNIT_REMOVE=hour
# How old files must be before they are deleted
# OFFSET_REMOVE=20
# Interval in minutes for deleting monitored files that have been uploaded
# FREQUENCY_REMOVE=60

4.5 Custom parser (optional)

Starting from version 1.5.9, you can use a custom data parser to convert the format when the raw data format differs from the TE data format.

Details:

1. Add the following dependency

Maven:

<dependency>
	<groupId>cn.thinkingdata.ta</groupId>
	<artifactId>integration-common-api</artifactId>
	<version>1.0.9</version>
</dependency>

Gradle:

// https://mvnrepository.com/artifact/cn.thinkingdata.ta/logbus-custom-interceptor
compile group: 'cn.thinkingdata.ta', name: 'integration-common-api', version: '1.0.9'

2. Implement the transFrom method of the CustomInterceptor interface. The first parameter of the method is the raw data content, and the second parameter is sourceName (used when you need to distinguish APP_IDs. It is assigned by APP_ID: for example, if you configure only one APP_ID, sourceName is r1; if you configure multiple APP_IDs, sourceName is r1, r2, and so on, in order). The transferToList method converts one record into multiple records. To use it, add INTERCEPTOR_ONE_TO_MORE=true to the configuration. You can use only one of the two methods.

public interface CustomInterceptor {

    TaDataDo transFrom(String var1, String var2) throws Exception;

    List<TaDataDo> transferToList(String var1, String var2) throws Exception;
}

Example implementation of the interface method:

public TaDataDo transFrom(String s, String s1) {
    return JSONObject.parseObject(s, TaDataDo.class);
}

3. Configure the following two fields

##The following two fields enable the custom parser (both fields must be set)
##Fully qualified name of the custom parser (package name + class name). If it is not set, the default parser is used
#CUSTOM_INTERCEPTOR=cn.thinkingdata.demo.DemoCustomInterceptor
##Absolute path of the custom parser jar (including the jar file name)
#INTERCEPTOR_PATH=/var/interceptor/custom-interceptor-1.0-SNAPSHOT.jar
##To convert one record into multiple records, enable the following field
#INTERCEPTOR_ONE_TO_MORE=true

4.6 Assign projects by #app_id in the data (optional)

Note: This feature requires TE 3.1 or later

##Version 1.5.14 added the APPID_IN_DATA setting. To distribute data to projects by #app_id in the data, set APPID_IN_DATA=true.
##In this case, you do not need to configure APPID, and TAIL_FILE can only be configured with one level.
##Note: TE 3.1 or later is required
#APPID_IN_DATA=false

4.7 Automatically distribute data to projects by configuration (optional)

##Version 1.5.14 changed this feature. It must be used with TE 3.1 or later.
##Name of the property to map, for example, name. Use it together with the APPID_MAP field
##APPID_MAP_ATTRIBUTE_NAME=name
##Mapping between APPIDs and property values. For example, the following configuration means that when the value of the name field in the properties field of the data is a or b, the data is distributed to appid_1, and when it is c or d, the data is distributed to appid_2
##APPID_MAP={"appid_1":["a","b"],"appid_2":["c","d"]}
##DEFAULT_APPID is the project that data is distributed to when the data has no name field or the value of the name field is not in the configuration above
##DEFAULT_APPID=appid_3

4.8 Configuration file example

##################################################################################
##             logBus configuration file, the transfer tool of the thinkingdata data analytics platform
##Uncommented parameters are required, and commented parameters are optional. Configure them
##as appropriate for your situation
##Requirements: java8+. For details, see the tga official website
##https://docs.thinkingai.cn/zh/integration/logbus_legacy_installation
##################################################################################

##APPID is the token from the tga official website
##Separate different APPIDs with commas
APPID=from_tga1,from_tga2

#-----------------------------------source----------------------------------------
######file-source
##Path and file name of the data files that LogBus reads (fuzzy matching is supported for file names). Read permission is required
##Separate different APPIDs with commas, and different directories of the same APPID with spaces
##File names in TAIL_FILE support two modes: standard Java regular expressions and wildcards
#TAIL_FILE=/path1/log.* /path2/txt.*,/path3/log.* /path4/log.* /path5/txt.*
##TAIL_MATCHER specifies the fuzzy matching mode for TAIL_FILE paths: regex (regular expression) or glob (wildcard).
##regex is the regular expression mode. It supports standard Java regular expressions, but fuzzy matching only works for one directory level and the file name
##glob is the wildcard mode. It supports fuzzy matching across multiple directory levels, but does not support DATE{} matching
##regex matching is used by default
#TAIL_MATCHER=regex

######kafka-source
##kafka: topics use regular expressions
#KAFKA_GROUPID=tga.flume
#KAFKA_SERVERS=ip:port
#KAFKA_TOPICS=topicName
#KAFKA_OFFSET_RESET=earliest/latest


#------------------------------------sink-----------------------------------------
##Transfer settings
##Destination URL
##PUSH_URL=http://${RECEIVER_URL}/logbus
##Maximum number of records per transfer (a transfer request is sent when this number is reached)
#BATCH=1000
##Transfer at least once within this interval (unit: seconds) (when the interval is reached before the batch is full, the current records are sent)
#INTERVAL_SECONDS=60

##### HTTP transfer
##Compression format for file transfer: gzip,snappy,none
#COMPRESS_FORMAT=none

#------------------------------------other-----------------------------------------
##Deletion of files in monitored directories. Uncomment the fields (both fields below must be uncommented) to enable file deletion. The deletion program runs once an hour
##Delete files older than offset, in units of unit
##How old files must be before they are deleted
#OFFSET_REMOVE=
##Only deletion by day (day) or hour (hour) is accepted
#UNIT_REMOVE=

#------------------------------------interceptor-----------------------------------
##The following two fields enable the custom parser (both fields must be set)
##Fully qualified name of the custom parser. If it is not set, the default parser is used
#CUSTOM_INTERCEPTOR=
##Location of the custom parser jar
#INTERCEPTOR_PATH=

5. Start LogBus

Before you start LogBus for the first time, perform the following checks:

1. Check the Java version

Go to the bin directory, which contains two scripts: check_java and logbus

check_java checks whether the Java version meets the requirements. Run the script. If the Java version does not meet the requirements, a message such as Java version is less than 1.8 or Can't find java, please install jre first. appears

You can update the JDK version, or see the next section to install a separate JDK for LogBus

2. Install a separate JDK for LogBus

Use this feature if the JDK version on the LogBus deployment node does not meet LogBus requirements because of the environment, and cannot be replaced with a JDK version that does.

Go to the bin directory, which contains install_logbus_jdk.sh.

Running this script adds a java directory to the LogBus working directory. LogBus uses the JDK environment in this directory by default.

3. Complete the logBus.conf configuration and run the parameter and environment check command

For how to configure logBus.conf, see the Configure LogBus parameters section

After the configuration is complete, run the env command to check whether the configuration parameters are correct

./logbus env

If red error messages are output, the configuration has problems. Modify it until no errors are reported for the configuration file, as shown in the figure above.

After you modify the logBus.conf configuration, restart LogBus for the new configuration to take effect

4. Start LogBus

./logbus start

If the startup succeeds, the message in the figure above appears. If it fails, an error message appears, as shown in the following figure

6. LogBus commands

6.1 Help information

If you run the command without arguments or with --help or -h, the following help information is displayed

The LogBus commands are as follows:

usage: logbus <command|auxiliary command> [options]
Commands:
        start                                                        Start logBus.
        restart                                                      Restart logBus.
        stop                                                         Safely quit logBus.
        reset                                                        Reset logBus read records.
        stop_force                                                   Force quit logBus.
Auxiliary commands:
        env                                                          Check the runtime environment.
        data_debug                                                   Validate the data format of files in the configured directories in detail.
        show_conf                                                    Show the current logBus configuration.
        version                                                      Show the version number.
        update                                                       Update logbus to the latest version.
        progress                                                     Show the transfer progress of the current files
        status                                                       Show the current transfer speed, memory usage, CPU consumption, and more

Options:
 -appid <appid>   Project appid
 -h,--help        Show the help and exit.
 -path <path>     Absolute path of the test file
 -url <url>       URL to test
Examples:
   ./logbus start                                                    Start logBus.
   ./logbus stop                                                     Safely quit logBus.
   ./logbus restart                                                  Restart logBus.
   ./logbus data_debug                                               Validate the data format of files in the configured directories in detail.

6.2 Check the file data format data_debug

When you use LogBus for the first time, we recommend that you validate the format of your data before you formally upload it. The data must comply with the data format specification. You can use the data_debug command to validate the data format, as follows:

This feature consumes cluster resources. Each run is limited to 10000 records, split evenly among the files, and validation starts from the beginning of each file.

./logbus data_debug

When the data format is correct, a message indicates that the data is correct, as shown in the following figure:

If the data format has problems, a format error warning appears with a brief description of the errors:

6.3 Show the configuration show_conf

You can use the show_conf command to view the LogBus configuration. The output is shown in the following figure:

./logbus show_conf

6.4 Check the startup environment env

You can use env to check the startup environment. If any output line ends with an asterisk, the configuration has a problem. Modify it until no asterisks appear.

./logbus env

6.5 Start start

After you complete the format validation, the data channel check, and the environment check, you can start LogBus to upload data. LogBus automatically detects whether new data has been written to your files and, if so, uploads it.

./logbus start

6.6 Stop stop

To stop LogBus, use the stop command. This command takes some time, but no data is lost.

./logbus stop

6.7 Stop immediately stop_force

To stop LogBus immediately, use the stop_force command. This command may cause data loss.

./logbus stop_force

6.8 Restart restart

You can use the restart command to restart LogBus, for example, to apply new configuration after you modify configuration parameters.

./logbus restart

6.9 Reset reset

Running reset resets LogBus. Use this command with great caution: it clears the file transfer records, and LogBus uploads all data again. If you use this command when you are not sure of the situation, your data may be duplicated. We recommend that you consult ThinkingAI staff before you use it.

./logbus reset

After you run the reset command, run start to start transferring data again

Since LogBus 1.5.0, the following confirmation message appears, and LogBus is reset only after you confirm

6.10 View the version number version

To find out the version number of the LogBus you are using, run the version command. If your LogBus does not have this command, you are using an early version

./logbus version

6.11 Upgrade logBus update

LogBus 1.5.0 added online upgrade. Run this command to upgrade LogBus to the latest version

./logbus update

6.12 View the current upload progress progress

LogBus 1.5.9 added a feature for viewing the current upload progress. Run this command to query the current transfer progress. You can use -appid to specify a project, or append the full path of a file to query that file.

./logbus progress /data/logbus-1.log /data/logbus-2.log -appid {APPID}

6.13 Check for common logBus problems doctor

LogBus 1.5.12 added a command that checks for common logBus problems. Run this command to check whether the current logBus has problems

./logbus doctor

6.14 View the current logBus upload speed and status status

LogBus 1.5.13 added a command for viewing the current logBus upload speed and status. Run this command to view the current upload speed and status of logBus

./logbus status

7. ChangeLog

Version 1.5.17 --- 2022/11/08

New:

• Supported one-to-many conversion in custom parsers

Version 1.5.16.2 --- 2022/07/05

Fixed:

• Upgraded fastjson to 1.2.83 because of a vulnerability in the old version

Version 1.5.16.1 --- 2022/05/24

Fixed:

• Upgraded fastjson to 1.2.80 because of a vulnerability in the old version

Version 1.5.16 --- 2022/04/01

Optimized:

• Optimized support for heterogeneous data

Version 1.5.15.7 --- 2021/12/10

Fixed:

• Upgraded log4j to 2.15.0 because of a vulnerability in the old version

Version 1.5.15.6 --- 2021/12/06

New:

• Added Kafka configuration items to the instance configuration

Version 1.5.15.5 --- 2021/12/02

New:

• Added SASL authentication for Kafka

Version 1.5.15.4 --- 2021/11/23

Fixed:

• Fixed upgrade errors in some cases

Version 1.5.15.3 --- 2021/11/19

Fixed:

• Fixed an issue where appid was not checked in some cases

Version 1.5.15.2 --- 2021/10/15

Fixed:

• Fixed several issues

Optimized:

• Optimized the LogBus shutdown process

Version 1.5.15.1 --- 2021/08/17

Fixed:

• Fixed some issues with reading data from Kafka sources

Version 1.5.15 --- 2021/06/02

Fixed:

• Fixed an error when reading empty lines

New:

• Added support for SASL authentication with Huawei Cloud and Alibaba Cloud Kafka, and changed the previous authentication scheme for Tencent Cloud CKafka

Version 1.5.14.3 --- 2021/04/29

Fixed:

• Fixed an issue where the install_logbus_jdk script could not download the JRE

Optimized:

• Attempts to repair itself in abnormal states

Version 1.5.14.2 --- 2021/03/16

Fixed:

• Fixed upgrade failures from some old versions

Version 1.5.14.1 --- 2021/03/03

Fixed:

• Fixed an issue where modifying NUMTHREAD when upgrading some old versions prevented normal transfer
• Fixed duplicate file uploads caused by unreleased file handles when a data file was deleted and then regenerated

Version 1.5.14 --- 2021/02/02

Optimized:

• Optimized the memory scheme, which greatly increases transfer speed when network conditions are good

New:

• Added the APPID_IN_DATA setting to match projects by the #app_id field in the data

Version 1.5.13.1 --- 2020/12/23

Fixed:

• Updated jackson to 2.11.2 because of a vulnerability in the old version

Version 1.5.13 --- 2020/11/27

Optimized:

• Optimized the progress command: you can query a specified file, and the display text is improved
• Optimized the Docker container
• Optimized the concurrency mechanism so that the concurrency is easier to change

New:

• Added the status command to view the current upload speed and status
• Added the repair_channel command and optimized how channels are repaired

Version 1.5.12 --- 2020/08/28

Optimized:

• Ensured data reliability when MemoryChannel is used and the data source is Kafka
• Fixed an issue where appid could not be removed
• Optimized the env command to fully check configuration properties, and improved the display text

New:

• Added sending in a specified property order
• Added the doctor command to check for some common logbus problems
• Added #event_id and #first_check_id

Version 1.5.11 --- 2020/06/01

Optimized:

• Ensured data reliability when MemoryChannel is used and the data source is local files

New:

• Added mapping of APP_IDs based on a specified property in the data.
• Added health checks for the LogBus daemon.
• Added status monitoring for the Flume process.
• A uuid property can be added to data.
• LogBus now deletes logs older than 30 days.
• Added APP_ID checking.

Deprecated:

• Deprecated the ftp transfer method of old versions.

Version 1.5.10 --- 2020/03/31

Fixed:

• Fixed a bug where exact path matching failed.

Optimized:

• Upgraded fastjson to 1.2.67 to fix deserialization and SSRF vulnerabilities.

Version 1.5.9 --- 2020/03/25

New:

• Added custom parsers.
• Added the progress command to view the current transfer progress.

Optimized:

• In taildir mode, fuzzy matching at any single directory level is supported.
• Optimized support for macOS.

Version 1.5.8 --- 2020/02/20

New:

• Added a retry strategy for packet loss caused by network fluctuations.

Version 1.5.7 --- 2020/02/13

Optimized:

• Optimized the distribution strategy of the USER data channel.

Version 1.5.6 --- 2020/01/03

Optimized:

• Optimized the storage location of the pid file and the status lock file.
• Optimized concurrency so that it can also be increased when multiple projects are configured.
• Optimized JVM parameters.
• Optimized reading of local files to skip hidden files.

New:

• Supported a separate JDK.
• Added disk usage scanning to the daemon, which stops LogBus when disk space is insufficient.
• Added the data_debug feature to check for detailed errors in the content of files in the configured directories.
• Added offset position recording for Kafka data sources.

Deprecated:

• Deprecated the format_check feature of old versions.

Version 1.5.5 --- 2019/09/23

Optimized:

• Optimized the JDK check script to support validation of JDK 10 and later.
• Optimized the internal startup order.
• Optimized the Flume runtime environment to avoid environment conflicts.
• Added a download progress bar.
• Optimized the server IP allowlist message.

Version 1.5.4 --- 2019/06/25

Optimized:

• Optimized configuration file parameter validation logic and instruction text.
• Added logic to automatically stop LogBus when the number of files read exceeds the maximum limit.

Version 1.5.3.1 --- 2019/05/22

Fixed:

• Fixed the data transfer strategy under network exceptions: instead of interrupting the transfer, LogBus now keeps retrying.

Version 1.5.3 --- 2019/04/25

Optimized:

• Optimized data transfer logic when a large number of files are transferred at the same time
• Optimized LogBus data transfer logs, which are split into info and error logs to make it easier to monitor how LogBus is running
• Upgraded the underlying Flume component to the latest version 1.9.0

Changes:

• Added offset configuration for the Kafka connector: sets the Kafka kafka.consumer.auto.offset.reset parameter. Valid values are earliest and latest. The default is earliest

Version 1.5.2.2 --- 2019/04/10

Fixed:

• Fixed some system compatibility issues
• Fixed the issue with the maximum number of open files
• Fixed position file errors in some extreme cases

Version 1.5.2.1 --- 2019/03/29

Fixed:

• Fixed LogBus runtime errors in some extreme cases

Version 1.5.2 --- 2019/03/14

New features:

• Kafka topics support multiple APP_IDs: With multiple APP_IDs, you can monitor multiple Kafka topics (for details, see Kafka parameter configuration)

Version 1.5.1 --- 2019/03/02

New features:

• HTTPS support: The transfer URL parameter PUSH_URL supports HTTPS
• Subdirectory monitoring: Monitors files in multiple subdirectories under one or more directories (for details, see TAIL_FILE parameter configuration). You can configure it with date templates or regular expressions

Version 1.5.0 --- 2018/12/26

New features:

• Multiple APP_IDs: A single LogBus can transfer data to multiple projects (multiple APP_IDs), and with multiple APP_IDs you can monitor multiple log file directories at the same time
• Online update command: Added the update command. Run it to upgrade LogBus to the latest version

Optimized:

• Added a prompt when you run the reset command

Version 1.4.3 --- 2018/11/19

New features:

• Multiple file directory monitoring: Supports monitoring multiple log file directories (for details, see the TAIL_FILE parameter configuration). The FILE_DIR and FILE_PATTERN parameters are deprecated. When you upgrade from an old version, you must configure TAIL_FILE

Changes:

• Flume monitoring was changed to the custom CustomMonitor, so you no longer need to configure the FM_PORT parameter (this parameter is deprecated)

Optimized:

• Fixed an error when checking Java version 10 and later

Version 1.4.2 --- 2018/09/03

New features:

• New data transfer method: Added ftp transfer

Version 1.4.0 --- 2018/07/30

New features:

• Multiple instances: You can deploy multiple LogBus instances on the same server.

Install multiple LogBus tools in different directories and configure a separate configuration file for each LogBus.

• Multi-threaded transfer: Thread safety is implemented.

Set the number of threads with the NUMTHREAD parameter in the configuration file

• The sink supports multiple compression formats: gzip, snappy, and no compression (none)

From left to right, the compression ratio decreases. Choose one based on your network environment and server performance.

Changes:

• When you start LogBus for the first time or after you modify the configuration file, run the env command to apply the configuration file so that LogBus works properly.

Optimized:

• Optimized the check messages before startup.

Version 1.3.5 --- 2018/07/18

Optimized:

• Optimized the output messages of the file format check command.
• Optimized the file transfer output messages.
• Added a backup to the checkpoint to prevent errors caused by frequent checkpoint reads and writes.
• Added channel watermark control so that the channel full warning no longer occurs.
• Added a network socktimeout for the sink, set to 60s.
• Added LogBus monitoring that automatically restarts LogBus when the sink hangs.

Version 1.3.4 --- 2018/06/08

Changes:

• Data filtering: Only empty lines and non-JSON data are filtered out.
• File deletion changed from deleting at a fixed time every day to deleting at regular intervals.

Configuration file:

• Optimized the configuration file format, which is divided into four parts: source, channel, sink, and others.
• Added the FREQUENCY_REMOVE parameter to delete uploaded directory files at regular intervals. Unit: minutes.
• Removed the TIME_REMOVE parameter.

New features:

• Added optimized startup scripts for automation tools (mainly Ansible) in the bin/automation directory, with three commands: start start, stop stop, and stop immediately stop_atOnce.

Performance optimization:

• Optimized the memory required for startup to reduce memory requirements.

Version 1.3 --- 2018/04/21

• Added support for Kafka data sources
• Fixed known bugs

Version 1.0 --- 2018/03/29

• Released LogBus