This article provides a quick introduction on how to securely stream IoT data into the Record Evolution cloud data platform. To stream IoT data, you first need to have an account and a Data Pod in the platform. Sign up for a free Data Pod here.
- The IoT Messaging Setup
- Publish Some Data to Test
- Connect and Listen with Record Evolution
1. The IoT Messaging Setup
In a first step, we are going to set up a basic IoT Messaging environment from scratch. Typically, you should have some other setup running productively in the field. You can simply skip the steps you do not need. In our example, we are using the WAMP messaging protocol.
A basic IoT messaging architecture consists of a message router node and a messaging client.
We are going to use crossbar for the WAMP router. To install the router on your local computer, you have two options. If you have a docker installed, you can open a terminal and simply do the following to download and run crossbar in a container:
docker run -p 8080:8080 crossbario/crossbar
If you have a Python3 installation on your computer, you can also do the following to get the router running:
pip install crossbar
Now the router is running. However, the router is running on your local machine only and is not accessible from the internet. To temporarily give it a publicly accessible IP address, you can use tunneling with ngrok. Once you have signed up for this service, you can use ngrok for free. Download the tool, unzip it in a folder, and go to that folder with your terminal to do the following:
./ngrok http 8080
Your output will look like this:
Make note of the temporary address you have been assigned. In the upper example, this would be the following:
However, your temporary address will be different.
2. Publish Some Data to Test
To receive data into the Record Evolution environment, we first need physical data that is being streamed. For instance, you can have sensor temperatures that are being streamed in real-time.
This is how you publish an incremental counter streaming data on a second-by-second basis:
You need to have node installed on your computer. Now you can start a terminal and install the WAMP client “autobahn” as follows:
npm install autobahn
Now you can create a file publish_counter.js with the following content:
Now you can start the publisher with:
The following output should be displayed now:
Connected to Router
published on 're.example.counter' : 1
published on 're.example.counter' : 2
published on 're.example.counter' : 3
3. Connect and Listen with Record Evolution
Now our demo IoT sensor is streaming numbers to the router. We only need to connect to the router and start listening. If you have not signed up for Record Evolution yet, you can do this for free here and obtain a free Data Pod to practice.
Once you have signed up, wait until your free demo Data Pod has been created. You can activate the demo Pod or any other Pod you have created for this demo by pressing the Start button. Once your Pod has been activated, you can enter the Pod by clicking on the Enter button to set up your IoT Import.
Now go to the Import Panel and click the IoT button:
Click the small orange plus sign in the Connections list to create a new connection to an IoT router.
In the next step, you need to fill out the form as shown below:
Enter the temporary URL that was provided by ngrok (see above) in the URL input cell. For this demonstration, we will use the following:
Create a full URL by prefixing
/ws as follows:
Note: The prefix refers to the WebSocket protocol whereas the suffix refers to the standard route to which the router makes WebSocket connections.
Once you have hit Save, you should be able to see your new connection MyLocal in the connections list:
Now you can click the green play button displayed next to your connection:
After some processing time, the button will transform into a stop button (red square) indicating that the connection has been successfully established.
Since one router can handle thousands of different topics, we need to set up a listener for our specific topic
re.example.counter to receive relevant data. This can be done in the Subscriptions list which is located below the Connections list.
To create a new subscription item in the Subscriptions list, you can click on the orange plus sign:
Now you can fill in the form as shown in the screenshot below:
A new Raw Table
S_MY_COUNTER will be created to store the messages received with this subscription:
When you click the checkbox next to your new subscription MyCounter, the Data Pod will subscribe to the topic
re.example.counter related to the connection MyLocal. Every ten seconds, or every 100 received messages, a new data package will be inserted into the Raw Table
You can follow this process in real-time by zooming in the graphical representation of the data packages within the Import Panel:
Click on any of the grey stripes to inspect the content of the package in the pop-up grid as displayed below:
payload_args displays the received message content. The purple color indicates that the data type of this column is in a JSON format. This is how, along with structured data, you can also store unstructured data. You can now also create a Workbook in the Results panel and perform your data query with native PostgreSQL:
Now you are continuously receiving data into the Raw Table
S_MY_COUNTER. Below are a few suggestions on where to go from here:
- Set some reasonable package retention in the raw table settings of
S_MY_COUNTERby clicking on the table name in the Import Panel. This helps you avoid unwanted data accumulation and improves overall system performance.
- Add derived columns in the Raw Table settings to extract some info from the
payload_argsJSON column. In our case, a good transform would be
payload_args::text::numeric; check it out here).
- Create a Data Pipe to process the data into a long-term Core Table. Click here for a quick tutorial on how to do this.