Stream IoT Data into the Record Evolution Cloud Data Platform

Image by Hike Shaw

This article provides a quick introduction on how to securely stream IoT data into the Record Evolution cloud data platform. To stream IoT data, you first need to have an account and a Data Pod in the platform. Sign up for a free Data Pod here.

1. The IoT Messaging Setup
2. Publish Some Data to Test
3. Connect and Listen with Record Evolution

1. The IoT Messaging Setup

In a first step, we are going to set up a basic IoT Messaging environment from scratch. Typically, you should have some other setup running productively in the field. You can simply skip the steps you do not need. In our example, we are using the WAMP messaging protocol.

A basic IoT messaging architecture consists of a message router node and a messaging client.

The Router

We are going to use crossbar for the WAMP router. To install the router on your local computer, you have two options. If you have a docker installed, you can open a terminal and simply do the following to download and run crossbar in a container:

docker run -p 8080:8080 crossbario/crossbar

If you have a Python3 installation on your computer, you can also do the following to get the router running:

pip install crossbar
crossbar init
crossbar start

Now the router is running. However, the router is running on your local machine only and is not accessible from the internet. To temporarily give it a publicly accessible IP address, you can use tunneling with ngrok. Once you have signed up for this service, you can use ngrok for free. Download the tool, unzip it in a folder, and go to that folder with your terminal to do the following:

./ngrok http 8080

Your output will look like this:

Make note of the temporary address you have been assigned. In the upper example, this would be the following:

f79e3bc8.ngrok.io

However, your temporary address will be different.

2. Publish Some Data to Test

To receive data into the Record Evolution environment, we first need physical data that is being streamed. For instance, you can have sensor temperatures that are being streamed in real-time.

This is how you publish an incremental counter streaming data on a second-by-second basis:

re.example.counter

You need to have node installed on your computer. Now you can start a terminal and install the WAMP client “autobahn” as follows:

npm install autobahn

Now you can create a file publish_counter.js with the following content:

Some data publisher

Now you can start the publisher with:

node publish_counter.js

The following output should be displayed now:

Connected to Router
published on 're.example.counter' : 1
published on 're.example.counter' : 2
published on 're.example.counter' : 3
...

3. Connect and Listen with Record Evolution

Now our demo IoT sensor is streaming numbers to the router. We only need to connect to the router and start listening. If you have not signed up for Record Evolution yet, you can do this for free here and obtain a free Data Pod to practice.

Figure 1. Select a Data Pod

Once you have signed up, wait until your free demo Data Pod has been created. You can activate the demo Pod or any other Pod you have created for this demo by pressing the Start button. Once your Pod has been activated, you can enter the Pod by clicking on the Enter button to set up your IoT Import.

Now go to the Import Panel and click the IoT button:

Figure 2. Open the IoT Import Setup

Click the small orange plus sign in the Connections list to create a new connection to an IoT router.

Figure 3. Create a New Connection

In the next step, you need to fill out the form as shown below:

Figure 4. Import from IoT Routers

Enter the temporary URL that was provided by ngrok (see above) in the URL input cell. For this demonstration, we will use the following:

f79e3bc8.ngrok.io

Create a full URL by prefixing ws://and suffixing/ws as follows:

ws://f79e3bc8.ngrok.io/ws

Note: The prefix refers to the WebSocket protocol whereas the suffix refers to the standard route to which the router makes WebSocket connections.

Once you have hit Save, you should be able to see your new connection MyLocal in the connections list:

Figure 5. Activate the Connection to Your Router

Now you can click the green play button displayed next to your connection:

After some processing time, the button will transform into a stop button (red square) indicating that the connection has been successfully established.

Since one router can handle thousands of different topics, we need to set up a listener for our specific topic re.example.counter to receive relevant data. This can be done in the Subscriptions list which is located below the Connections list.

To create a new subscription item in the Subscriptions list, you can click on the orange plus sign:

Figure 6. Create a New Subscription

Now you can fill in the form as shown in the screenshot below:

Figure 7. Fill Out Subscription Form

A new Raw Table S_MY_COUNTER will be created to store the messages received with this subscription:

Figure 8. Activate the Subscription

When you click the checkbox next to your new subscription MyCounter, the Data Pod will subscribe to the topic re.example.counter related to the connection MyLocal. Every ten seconds, or every 100 received messages, a new data package will be inserted into the Raw Table S_MY_COUNTER.

You can follow this process in real-time by zooming in the graphical representation of the data packages within the Import Panel:

Figure 9. Watch the Packages Arriving Every 10 Seconds

Click on any of the grey stripes to inspect the content of the package in the pop-up grid as displayed below:

Figure 10. Inspect One Received Data Package

The column payload_args displays the received message content. The purple color indicates that the data type of this column is in a JSON format. This is how, along with structured data, you can also store unstructured data. You can now also create a Workbook in the Results panel and perform your data query with native PostgreSQL:

Figure 11. Query Your IoT Stream Data

Outlook

Now you are continuously receiving data into the Raw Table S_MY_COUNTER. Below are a few suggestions on where to go from here:

1. Set some reasonable package retention in the raw table settings of S_MY_COUNTER by clicking on the table name in the Import Panel. This helps you avoid unwanted data accumulation and improves overall system performance.
2. Add derived columns in the Raw Table settings to extract some info from the payload_args JSON column. In our case, a good transform would be payload_args[1]::text::numeric; check it out here).
3. Create a Data Pipe to process the data into a long-term Core Table. Click here for a quick tutorial on how to do this.