Using NodeJS as MQTT client for Toon
Moderators: marcelr, TheHogNL, Toonz
Re: Using NodeJS as MQTT client for Toon
You don't need mosquitto stuff. But you indeed do need to install mqtt via npm.
Are there different versions of the device? I have installed NodeJS packages a couple of times. I haven't observed the behaviour during the installs though, but the installations succeeded without any intervention. Could also be that I don't have any other stuff running on to maybe?
Are there different versions of the device? I have installed NodeJS packages a couple of times. I haven't observed the behaviour during the installs though, but the installations succeeded without any intervention. Could also be that I don't have any other stuff running on to maybe?
Re: Using NodeJS as MQTT client for Toon
No hardware specs differences as what I am aware of.
The NPM install took about 5 minutes and at least a disabled qt-gui to have enough memory. I do have some extra tiles/apps on the qt-gui running so yeah maybe I was just having a bit too much.
But also running the test script with node took at least 10 seconds to run and took also a big chunk of memory. Keep sending any interesting data towards any mqtt broker will then probably just take too much memory/cpu.
Isn't there a more lightweight mqtt client which can be installed/compiled? Even a small esp8266 can run a mqtt client so there should be something less memory eating than using node.js
The NPM install took about 5 minutes and at least a disabled qt-gui to have enough memory. I do have some extra tiles/apps on the qt-gui running so yeah maybe I was just having a bit too much.
But also running the test script with node took at least 10 seconds to run and took also a big chunk of memory. Keep sending any interesting data towards any mqtt broker will then probably just take too much memory/cpu.
Isn't there a more lightweight mqtt client which can be installed/compiled? Even a small esp8266 can run a mqtt client so there should be something less memory eating than using node.js
Member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
Never mind. I'm running an install of mqtt now, and indeed it eats up all the memory. I also got disconnected from ssh. Also, it seems that mqtt was only partially installed. Just running npm --help eats up all the memory.
I just searched a bit for a lightweight version of npm, but did not find any. Thing is, for a specific project, we could probably do without npm. It's likely that once mqtt is installed, it could be distributed with the package, so you don't have to install it yourself. I've activated npm/nodejs packages before simply by copying the folder from node_modules. In this case, it might need some additional work, like creating symlinks. I re-ran the install of mqtt, and now it finished. So it will install eventually, but it's not very clean, and it is probably disruptive. Is it maybe possible to install an npm package using the oe-toolchain on another host, and create an mqtt add-on to nodejs that way?
I have sort of a proof of concept nodejs server running at the moment. It does connect to mqtt, but doesn't send anything yet. It uses 30% of memory according to top, net I'd say about 10MB. A couple of those MB's is probably mqtt. Without it was more like 3MB I think. So it's not very efficient as far as resources go. Since it doesn't do anything for most of the time, cpu utilization seems to not be a problem at the moment. For now, it reads the latest value from the database, with an interval of one minute. I haven't implemented the integer part yet. I do not use the xml config, because that seems to require another module. So it's not very clean, but it seems to do the trick.
Anyway, I'm not giving up just yet
I just searched a bit for a lightweight version of npm, but did not find any. Thing is, for a specific project, we could probably do without npm. It's likely that once mqtt is installed, it could be distributed with the package, so you don't have to install it yourself. I've activated npm/nodejs packages before simply by copying the folder from node_modules. In this case, it might need some additional work, like creating symlinks. I re-ran the install of mqtt, and now it finished. So it will install eventually, but it's not very clean, and it is probably disruptive. Is it maybe possible to install an npm package using the oe-toolchain on another host, and create an mqtt add-on to nodejs that way?
I have sort of a proof of concept nodejs server running at the moment. It does connect to mqtt, but doesn't send anything yet. It uses 30% of memory according to top, net I'd say about 10MB. A couple of those MB's is probably mqtt. Without it was more like 3MB I think. So it's not very efficient as far as resources go. Since it doesn't do anything for most of the time, cpu utilization seems to not be a problem at the moment. For now, it reads the latest value from the database, with an interval of one minute. I haven't implemented the integer part yet. I do not use the xml config, because that seems to require another module. So it's not very clean, but it seems to do the trick.
Anyway, I'm not giving up just yet
Re: Using NodeJS as MQTT client for Toon
As far as proof of concept goes, see attachment
- Attachments
-
- GrafanaPoC.png (55.54 KiB) Viewed 11997 times
Re: Using NodeJS as MQTT client for Toon
Can you share your proof of concept code?
Member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
Sure, see attachment.
It makes a list of all the dat files in the rra directory, puts the 30days ones in an array, and iterates over the array every minute, getting the latest value, and pushing it out over mqtt.
As people have stated earlier on in this post, the NodeJS/NPM/MQTT combination does seem quite 'heavy'. Also, not all data is written to the database, so the boiler modulation for example will still have to be retrieved in a different manner. Since the database data is not real time, I'm starting to wonder what the added value of mqtt is here, especially because of the effort it takes to get it on the device, and the resources it needs.
So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?
It makes a list of all the dat files in the rra directory, puts the 30days ones in an array, and iterates over the array every minute, getting the latest value, and pushing it out over mqtt.
As people have stated earlier on in this post, the NodeJS/NPM/MQTT combination does seem quite 'heavy'. Also, not all data is written to the database, so the boiler modulation for example will still have to be retrieved in a different manner. Since the database data is not real time, I'm starting to wonder what the added value of mqtt is here, especially because of the effort it takes to get it on the device, and the resources it needs.
So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?
- Attachments
-
- follow.js.zip
- (2.85 KiB) Downloaded 590 times
Re: Using NodeJS as MQTT client for Toon
Just noticed that Toon already has mosquitto-clients (mosquitto-pub and -sub) available for download via opkg (when connected to Eneco).
Very lightweight but ofcourse only commandline so only usable in a bash script or something like that.
But running a bash script from cron, which pulls some data from the rrd (using a curl http to localhost) and then using the mosquitto-pub to publish that (as mqtt messages) to a broker is probably consuming a less memory then the node-js solution.
Very lightweight but ofcourse only commandline so only usable in a bash script or something like that.
But running a bash script from cron, which pulls some data from the rrd (using a curl http to localhost) and then using the mosquitto-pub to publish that (as mqtt messages) to a broker is probably consuming a less memory then the node-js solution.
Member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
Re: Using NodeJS as MQTT client for Toon
Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
Just read the follow.js.zip and saw that HansS is reading the data directly from the RRD (so not using a http call to localhost) which ofcourse is a very nice job!Toonz wrote:Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
You can install the original Toon mosquitto from their feeds servers (you must have the openvpn running):
Code: Select all
opkg install http://feed.hae.int/feeds/qb2/oe/2.6.36-base-r10-qmf0/libmosquitto_1.4.9-r0_qb2.ipk
opkg install http://feed.hae.int/feeds/qb2/oe/2.6.36-base-r10-qmf0/mosquitto-clients_1.4.9-r0_qb2.ipk
Member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
One url is providing a lot of information already (except for example water temperature and boiler pressure): http://toon-ip/happ_thermstat?action=getThermostatInfoHansS wrote: So I expanded my iotserver to also get the data via http. The only drawback is that you cannot get all the metrics in one http call. So to get them all (assuming they're all interesting), it takes over 30 http calls every minute, which seems inefficient also. How are you guys doing this?
That is one call. I'm polling that using a python script currently, which then feeds a domoticz server.
This information is probably also available using the /qmf/bin/bxt command. So doing this in a bash script and then using the mosquitto_pub for the mqtt output could be a lightweight solution.
Member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
Adding to my own post,.. just figured out the correct bxt command to get the current thermostat info:
Need some more bash parsing.
Code: Select all
/qmf/bin/bxt -d :happ_thermstat -s specific1 -n webRequest -a action -v getThermostatInfo -c 2>&1 | grep u:webRequestResponse
Member of the Toon Software Collective
Re: Using NodeJS as MQTT client for Toon
Yeah, that's right. And as TheHogNL stated, I read the rrd files directly. But I use NodeJS for that, so that can be considered overhead as well. MQTT is more valuable if you can push out data the moment it changes. Which might still be possible. Not sure how it works yet, but I saw that the builtin thermostat app reacts to boiler info changes somehow. Don't know if that's also bound to minute interval polls. But maybe it's not. I believe that the thermostat displays the flame icon this way, and it doesn't take a minute to determine that it needs to be on.Toonz wrote:Doesn't really matter because you still need the http requests to get the internal data. Isn't mqtt purely overhead in that case?HansS wrote:Sounds interesting. Any way you can get your hands on those? Still seems to me that pushing values out via mqtt is better than doing a lot of http requests.
But, http for now. I do use the url that you mention, but indeed the boiler temperature values are not in there. Most of the values are not that interesting to me though. So for now I just get the boiler temperatures, so three http calls in total.
I tried installing the mosquito stuff, but I seem to be missing some libraries:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for libmosquitto:
* libssl1.0.0 (>= 1.0.2h) * libcrypto1.0.0 (>= 1.0.2h) *
Re: Using NodeJS as MQTT client for Toon
What's your FW version again?
Re: Using NodeJS as MQTT client for Toon
Avaiable from the same source, filenames are:HansS wrote: I tried installing the mosquito stuff, but I seem to be missing some libraries:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for libmosquitto:
* libssl1.0.0 (>= 1.0.2h) * libcrypto1.0.0 (>= 1.0.2h) *
Code: Select all
libssl1.0.0_1.0.2h-r2_qb2.ipk
libcrypto1.0.0_1.0.2h-r2_qb2.ipk
Member of the Toon Software Collective