Skip to content

Commit

Permalink
Editorial updates
Browse files Browse the repository at this point in the history
  • Loading branch information
Meggielqk committed Dec 25, 2024
1 parent b32abe9 commit 8bc4d51
Show file tree
Hide file tree
Showing 6 changed files with 215 additions and 205 deletions.
193 changes: 100 additions & 93 deletions en_US/data-integration/data-bridge-sqlserver.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,75 +42,13 @@ The data integration with Microsoft SQL Server offers a range of features and be

## Before You Start

This section describes the preparations you need to complete before you start to create the Microsoft SQL Server data integration, including how to install and connect to the Microsoft SQL Server, create database and data tables, and install and configure the ODBC driver.
This section describes the preparations you need to complete before you start creating the Microsoft SQL Server data integration, including how to install and configure the ODBC driver, install and connect to the Microsoft SQL Server, ands create database and data tables.

### Prerequisites

- Knowledge about EMQX data integration [rules](./rules.md)
- Knowledge about [data integration](./data-bridges.md)

### Install and Connect to Microsoft SQL Server

This section describes how to start Microsoft SQL Server 2019 on Linux/MacOS using Docker images and use `sqlcmd` to connect to Microsoft SQL Server. For other installation methods of Microsoft SQL Server, please refer to [Microsoft SQL Server Installation Guide](https://learn.microsoft.com/en-us/sql/database-engine/install-windows/install-sql-server?view=sql-server-ver16).

1. Install Microsoft SQL Server via Docker, and then start the docker image with the command below. Use `mqtt_public1` as the password. For the password policy of Microsoft SQL Server, see [Password Complexity](https://learn.microsoft.com/en-us/sql/relational-databases/security/password-policy?view=sql-server-ver16#password-complexity).

Note: By starting a Docker container with the environment variable `ACCEPT_EULA=Y` you agree to the terms of Microsoft EULA, see also [End-User Licensing Agreement](https://go.microsoft.com/fwlink/?linkid=857698).

```bash
# To start the Microsoft SQL Server docker image and set the password as `mqtt_public1`
$ docker run --name sqlserver -p 1433:1433 -e ACCEPT_EULA=Y -e MSSQL_SA_PASSWORD=mqtt_public1 -d mcr.microsoft.com/mssql/server:2022-CU15-ubuntu-22.04
```

2. Access the container.

```bash
docker exec -it sqlserver bash
```

3. Enter the preset password to connect to the server in the container. The characters are not echoed when entering the password. Click `Enter` directly after entering the password.

```bash
$ /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P mqtt_public1 -N -C
1>
```

::: tip

The `mssql-tools18` package have been installed in the Microsoft SQL Server container provided by Microsoft, but the executable file is not in `$PATH`. Therefore, you need to specify the executable file path for `sqlcmd` before proceeding. As for the Docker deployment in this example, the file path should be `/opt`.

For more information on how to use `mssql-tools18`, see [sqlcmd-utility](https://learn.microsoft.com/en-us/sql/tools/sqlcmd/sqlcmd-utility?view=sql-server-ver16).

:::

So far, the Microsoft SQL Server 2022 instance has been deployed and can be connected.

### Create Database and Data Tables

Use the connection created from the previous section and the following SQL statements to create data tables.

- Create the following data table for storing the MQTT message, including the message ID, topic, QoS, payload, and publish time of each message.

```sql
CREATE TABLE dbo.t_mqtt_msg (id int PRIMARY KEY IDENTITY(1000000001,1) NOT NULL,
msgid VARCHAR(64) NULL,
topic VARCHAR(100) NULL,
qos tinyint NOT NULL DEFAULT 0,
payload VARCHAR(100) NULL,
arrived DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP);
GO
```

- Create the following data table for recording the online/offline status of clients.

```sql
CREATE TABLE dbo.t_mqtt_events (id int PRIMARY KEY IDENTITY(1000000001,1) NOT NULL,
clientid VARCHAR(255) NULL,
event_type VARCHAR(255) NULL,
event_time DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP);
GO
```

### Install and Configure ODBC Driver

You need to configure the ODBC driver to be able to access the Microsoft SQL Server database. You can use either FreeTDS or the msodbcsql18 driver provided by Microsoft as the ODBC driver.
Expand Down Expand Up @@ -173,6 +111,7 @@ Check that the DSN Name in `odbcinst.ini` should be `ms-sql` if you install the
This section introduces how to install and configure FreeTDS as an ODBC driver on some of the mainstream distributions.

Install and configure FreeTDS ODBC driver on MacOS:

```bash
$ brew install unixodbc freetds
$ vim /usr/local/etc/odbcinst.ini
Expand All @@ -185,6 +124,7 @@ FileUsage = 1
```

Install and configure FreeTDS ODBC driver on Centos:

```bash
$ yum install unixODBC unixODBC-devel freetds freetds-devel perl-DBD-ODBC perl-local-lib
$ vim /etc/odbcinst.ini
Expand All @@ -199,6 +139,7 @@ FileUsage = 1
```

Install and configure FreeTDS ODBC driver on Ubuntu (Take Ubuntu20.04 as an example, for other versions, please refer to the official ODBC documentation):

```bash
$ apt-get install unixodbc unixodbc-dev tdsodbc freetds-bin freetds-common freetds-dev libdbd-odbc-perl liblocal-lib-perl
$ vim /etc/odbcinst.ini
Expand All @@ -210,6 +151,74 @@ Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
FileUsage = 1
```

### Install and Connect to Microsoft SQL Server

This section describes how to start Microsoft SQL Server 2019 on Linux/MacOS using Docker images and use `sqlcmd` to connect to Microsoft SQL Server. For other installation methods of Microsoft SQL Server, please refer to [Microsoft SQL Server Installation Guide](https://learn.microsoft.com/en-us/sql/database-engine/install-windows/install-sql-server?view=sql-server-ver16).

1. Install Microsoft SQL Server via Docker, and then start the docker image with the command below. Use `mqtt_public1` as the password. For the password policy of Microsoft SQL Server, see [Password Complexity](https://learn.microsoft.com/en-us/sql/relational-databases/security/password-policy?view=sql-server-ver16#password-complexity).

Note: By starting a Docker container with the environment variable `ACCEPT_EULA=Y` you agree to the terms of Microsoft EULA, see also [End-User Licensing Agreement](https://go.microsoft.com/fwlink/?linkid=857698).

```bash
# To start the Microsoft SQL Server docker image and set the password as `mqtt_public1`
$ docker run --name sqlserver -p 1433:1433 -e ACCEPT_EULA=Y -e MSSQL_SA_PASSWORD=mqtt_public1 -d mcr.microsoft.com/mssql/server:2022-CU15-ubuntu-22.04
```

2. Access the container.

```bash
docker exec -it sqlserver bash
```

3. Enter the preset password to connect to the server in the container. The characters are not echoed when entering the password. Click `Enter` directly after entering the password.

```bash
$ /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P mqtt_public1 -N -C
1>
```

::: tip

The `mssql-tools18` package have been installed in the Microsoft SQL Server container provided by Microsoft, but the executable file is not in `$PATH`. Therefore, you need to specify the executable file path for `sqlcmd` before proceeding. As for the Docker deployment in this example, the file path should be `/opt`.

For more information on how to use `mssql-tools18`, see [sqlcmd-utility](https://learn.microsoft.com/en-us/sql/tools/sqlcmd/sqlcmd-utility?view=sql-server-ver16).

:::

So far, the Microsoft SQL Server 2022 instance has been deployed and can be connected.

### Create Database and Data Tables

Use the connection created from the previous section and the following SQL statements to create data tables.

::: tip

Due to ODBC interface limitations, if you need to write Unicode characters, such as CJK characters or Emoji, you need to use a function to convert them into binary format before inserting them. When creating a table, set the column type that stores Unicode characters to `NVARCHAR`.

:::

- Create the following data table for storing the MQTT message, including the message ID, topic, QoS, payload, and publish time of each message.

```sql
CREATE TABLE dbo.t_mqtt_msg (id int PRIMARY KEY IDENTITY(1000000001,1) NOT NULL,
msgid VARCHAR(64) NULL,
topic VARCHAR(100) NULL,
qos tinyint NOT NULL DEFAULT 0,
payload VARCHAR(100) NULL,
arrived DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP);
GO
```

- Create the following data table for recording the online/offline status of clients.

```sql
CREATE TABLE dbo.t_mqtt_events (id int PRIMARY KEY IDENTITY(1000000001,1) NOT NULL,
clientid VARCHAR(255) NULL,
event_type VARCHAR(255) NULL,
event_time DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP);
GO
```

## Create a Connector

This section demonstrates how to create a Connector to connect the Sink to the Microsoft SQL server.
Expand Down Expand Up @@ -251,6 +260,22 @@ This section demonstrates how to create a rule in the Dashboard for processing m

::: tip

Due to ODBC interface limitations, if you need to write Unicode characters, such as CJK characters or Emoji, you need to use a function to convert them into binary format before inserting them.

You can use built-in functions to convert strings to UTF-16-little-endian encoded binary strings when creating rules. For example:

```sql
SELECT
sqlserver_bin2hexstr(str_utf16_le(payload)) as payload
*
FROM
"t/#"
```

:::

::: tip

If you are a beginner user, click **SQL Examples** and **Enable Test** to learn and test the SQL rule.

:::
Expand All @@ -271,6 +296,18 @@ This section demonstrates how to create a rule in the Dashboard for processing m
insert into dbo.t_mqtt_msg(msgid, topic, qos, payload) values ( ${id}, ${topic}, ${qos}, ${payload} )
```

::: tip

Due to ODBC interface limitations, if you need to write Unicode characters, such as CJK characters or Emoji, you need to use a function to convert them into binary format before inserting them.

You can use the `CONVERT` function in the SQL template to convert the corresponding binary data into a string by Microsoft SQL Server.

```sql
insert into dbo.t_mqtt_msg(msgid, topic, qos, payload) values ( ${id}, ${topic}, ${qos}, CONVERT(NVARCHAR(100), ${payload}) )
```

:::

If a placeholder variable is undefined in the SQL template, you can toggle the **Undefined Vars as Null** switch above the **SQL template** to define the rule engine behavior:

- **Disabled** (default): The rule engine can insert the string `undefined` into the database.
Expand All @@ -295,36 +332,6 @@ You have now successfully created the rule for the Microsoft SQL Server Sink. Yo

You can also click **Integration** -> **Flow Designer** to view the topology and you can see that the messages under topic `t/#` are sent and saved to Microsoft SQL Server after parsing by rule `my_rule`.

Due to ODBC interface limitations, if you need to write Unicode characters, such as CJK characters or Emoji, you need to use a function to convert them into binary format before inserting them.

- When creating a table, set the column type that needs to store Unicode characters to `NVARCHAR`.

```sql{5}
CREATE TABLE dbo.t_mqtt_msg (id int PRIMARY KEY IDENTITY(1000000001,1) NOT NULL,
msgid VARCHAR(64) NULL,
topic VARCHAR(100) NULL,
qos tinyint NOT NULL DEFAULT 0,
payload NVARCHAR(100) NULL,
arrived DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP);
GO
```

- Use built-in functions to convert strings to UTF-16-little-endian encoded binary strings when creating rules.

```sql{2}
SELECT
sqlserver_bin2hexstr(str_utf16_le(payload)) as payload
*
FROM
"t/#"
```

- Use the `CONVERT` function in the SQL template to convert the corresponding binary data into a string by Microsoft SQL Server.

```sql
insert into dbo.t_mqtt_msg(msgid, topic, qos, payload) values ( ${id}, ${topic}, ${qos}, CONVERT(NVARCHAR(100), ${payload}) )
```

## Create a Rule with Microsoft SQL Server for Events Recording

This section demonstrates how to create a rule for recording the clients' online/offline status and storing the events data to the Microsoft SQL Server table `dbo.t_mqtt_events` via a configured Sink.
Expand Down
2 changes: 1 addition & 1 deletion zh_CN/data-integration/data-bridge-clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ ClickHouse 数据集成是 EMQX 中的开箱即用功能,旨在结合 MQTT 的
::: tip
如果您初次使用 SQL,可以点击 **SQL 示例****启用调试**来学习和测试规则 SQL 的结果
如果可能,应该始终启用此选项;禁用该选项仅用于确保向后兼容性
:::
Expand Down
2 changes: 1 addition & 1 deletion zh_CN/data-integration/data-bridge-dynamo.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ DynamoDB 数据集成是 EMQX 中的开箱即用功能,它结合了 EMQX 的

::: tip

如果您初次使用 SQL,可以点击 **SQL 示例****启用调试**来学习和测试规则 SQL 的结果
如果可能,应该始终启用此选项;禁用该选项仅用于确保向后兼容性

:::

Expand Down
2 changes: 1 addition & 1 deletion zh_CN/data-integration/data-bridge-mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ CREATE TABLE emqx_client_events (

::: tip

如果您初次使用 SQL,可以点击 **SQL 示例****启用调试**来学习和测试规则 SQL 的结果
如果可能,应该始终启用此选项;禁用该选项仅用于确保向后兼容性

:::

Expand Down
Loading

0 comments on commit 8bc4d51

Please sign in to comment.