After reading through the good, it appears that DomotiGa uses a single loop that runs most of modules, etc. What does a person need to do to prevent long running jobs/modules from causing the main loop to pause? Is there some sort of plan on the roadmap for that?
If not, here are some of the ideas I have thought of.
1) Use non blocking I/O and then be able to handle partial reads, etc.
2) Use threads... This appears to be a no go since gambas does not support threads.
3) Use multiple processes for device(s)..
I am not sure on when to split off the device to its own process -- Maybe it needs to be a checkbox on the configuration of the device or when the device needs to use polling. One of the questions that I have is how to handle the IPC of these processes. The database might work along with sockets, but I don't know how well that would scale.
Handling long running items without pausing the main loop
Moderator: RDNZL
-
RDNZL
- Forum Moderator
- Posts: 1008
- Joined: Sun Sep 24, 2006 1:45 pm
- Location: Dordrecht, The Netherlands
- Contact:
Re: Handling long running items without pausing the main loop
There is a program loop, it's derived from MisterHouse, that's how I started when I didn't had any Gambas experience.
It's only used to set time based global variables like New_Minute, New_Second Season etc. not for processing data from interfaces etc. those are based on events or timers like *_Read() *_Timer etc.
I would like the get rid of the loop altogether, or we could run it once every second instead of every 250mS like it does now.
Indeed Gambas doesn't support threads, that's a pity but is the choice of the developer, the reason is that GUI programs don't need to be capable of doing a lot of things at the same time (DomotiGa is not one of them, so I disagree
)
The split up of DomotiGa into DomotigaServer has done a lot of good things to stability and cpu usage.
It's runs days in a row without any problems (until I restart the process because of code changes).
Most if the i/o is "non blocking", and reads the data that's available at the time.
We could save a few cpu cycles to change some code which read/writes byte for byte into code that's reads all available data and then parse it, I already changes some routines to do that.
If you can be more specific about spots in DomotiGa which give the most delay I take a look at them.
I will install DomotiGaServer on a less powerfull board soon, so I can detect which parts are cpu hogs..
It's only used to set time based global variables like New_Minute, New_Second Season etc. not for processing data from interfaces etc. those are based on events or timers like *_Read() *_Timer etc.
I would like the get rid of the loop altogether, or we could run it once every second instead of every 250mS like it does now.
Indeed Gambas doesn't support threads, that's a pity but is the choice of the developer, the reason is that GUI programs don't need to be capable of doing a lot of things at the same time (DomotiGa is not one of them, so I disagree
)The split up of DomotiGa into DomotigaServer has done a lot of good things to stability and cpu usage.
It's runs days in a row without any problems (until I restart the process because of code changes).
Most if the i/o is "non blocking", and reads the data that's available at the time.
We could save a few cpu cycles to change some code which read/writes byte for byte into code that's reads all available data and then parse it, I already changes some routines to do that.
If you can be more specific about spots in DomotiGa which give the most delay I take a look at them.
I will install DomotiGaServer on a less powerfull board soon, so I can detect which parts are cpu hogs..
Regards, Ron.
Re: Handling long running items without pausing the main loop
I haven't seen a lot of issues yet, but I am looking into writing a driver for some modbus products from temco controls products (http://www.temcocontrols.com/).. I too have come from using misterhouse and am most likely carrying around
The things were I see issues is that modbus is a polling based network. Most of the time it is quick, but not always.. The normal timeout for modbus is about 1 second. The normal processing time that I have seen on test that I have run is about 250ms... Not big, but if a person adds several devices it adds up..
Using the non blocking I/O is not a hard deal with provided the all of the code being used is written in gambas... Some of the things that I was looking to exploit is using libraries that have already been test/developed... For example: https://code.launchpad.net/libmodbus for modbus communications. By doing this one doesn't have to reinvent everything for the given protocol..
So, if a person wants to use an external program to do most of the work talking to polled devices, there has not been any type of protocol setup/used to get the information back into the Domotiga? Is there any recommended method of getting the information back into Domotiga?
Steve
The things were I see issues is that modbus is a polling based network. Most of the time it is quick, but not always.. The normal timeout for modbus is about 1 second. The normal processing time that I have seen on test that I have run is about 250ms... Not big, but if a person adds several devices it adds up..
Using the non blocking I/O is not a hard deal with provided the all of the code being used is written in gambas... Some of the things that I was looking to exploit is using libraries that have already been test/developed... For example: https://code.launchpad.net/libmodbus for modbus communications. By doing this one doesn't have to reinvent everything for the given protocol..
So, if a person wants to use an external program to do most of the work talking to polled devices, there has not been any type of protocol setup/used to get the information back into the Domotiga? Is there any recommended method of getting the information back into Domotiga?
Steve
-
RDNZL
- Forum Moderator
- Posts: 1008
- Joined: Sun Sep 24, 2006 1:45 pm
- Location: Dordrecht, The Netherlands
- Contact:
Re: Handling long running items without pausing the main loop
As soon as I have my RMAed Micro-FieldCommander uFC9051 back from CER I want to add support for modbus to DomotiGa too.
Devices you have to poll kinda suck... sry... I have to poll the Xanura CTX35 device too once or twice a second otherwise it's very small buffer gets full
Some ideas:
Gambas has the ability to call external functions in system shared libraries.
Look here for some explanation http://gambasdoc.org/help/lang/extdecl
I haven't used it yet.
Use a async network socket call and use it's events to see if it has finished and what the answer was.
Example
Use external shell/exec commands and use it's _Read and _Kill events like here.
Example
Devices you have to poll kinda suck... sry... I have to poll the Xanura CTX35 device too once or twice a second otherwise it's very small buffer gets full
Some ideas:
Gambas has the ability to call external functions in system shared libraries.
Look here for some explanation http://gambasdoc.org/help/lang/extdecl
I haven't used it yet.
Use a async network socket call and use it's events to see if it has finished and what the answer was.
Example
Use external shell/exec commands and use it's _Read and _Kill events like here.
Example
Regards, Ron.
Re: Handling long running items without pausing the main loop
I saw the fact that gambas can call functions in th external shared libraries hence why I was thinking about using libmodbus.. The issue I see is that libmodbus appears to use blocking io for talking to the devices which messes with the loop.
Using an external program and network sockets was one of the things that I though of.. The question I had is if a person wants to go done the path where the external communications is some type of already defined standard like xAP or xPL or if a would rather make one that is defined for protocol and Domotiga itself?
Running an external program is doable, but just doesn't scale very well..
Using an external program and network sockets was one of the things that I though of.. The question I had is if a person wants to go done the path where the external communications is some type of already defined standard like xAP or xPL or if a would rather make one that is defined for protocol and Domotiga itself?
Running an external program is doable, but just doesn't scale very well..
-
RDNZL
- Forum Moderator
- Posts: 1008
- Joined: Sun Sep 24, 2006 1:45 pm
- Location: Dordrecht, The Netherlands
- Contact:
Re: Handling long running items without pausing the main loop
There is no global way of getting data into DomotiGa. (yet)
Since building an interface for specific hardware is only 20-50 lines of code (except the RFXCom part, but that one covers a lot of device types) I had no need to build something universal, someone offered to build Xpl support for DomotiGa a long time ago but I lost contact, send him a pm last week but didn't hear anything back. Maybe someone else needs something to do for the holidays and code me some xPL support?
Apart from that I think the code for supporting xPLl exceeds the number of lines of code we have now by a factor of 3 easily. Sure then we have something universal, but the whole xPL/xAP thing isn't that big as it was a year back, or I'm I wrong?
There is an outline on the website on how to integrate a new type of interface, and you can pick one of the current ones as an example.
And you are right, having to call an external command to poll every second isn't a wise thing to do.
Best is to find one of the many vb code snippets to do modbus and port that.
Which type of modbus are we talking about btw? The ASCII one?
The uFC interface I have came with a cd with al kind of modbus related code examples, will look it up for you if you want.
Since building an interface for specific hardware is only 20-50 lines of code (except the RFXCom part, but that one covers a lot of device types) I had no need to build something universal, someone offered to build Xpl support for DomotiGa a long time ago but I lost contact, send him a pm last week but didn't hear anything back. Maybe someone else needs something to do for the holidays and code me some xPL support?
Apart from that I think the code for supporting xPLl exceeds the number of lines of code we have now by a factor of 3 easily. Sure then we have something universal, but the whole xPL/xAP thing isn't that big as it was a year back, or I'm I wrong?
There is an outline on the website on how to integrate a new type of interface, and you can pick one of the current ones as an example.
And you are right, having to call an external command to poll every second isn't a wise thing to do.
Best is to find one of the many vb code snippets to do modbus and port that.
Which type of modbus are we talking about btw? The ASCII one?
The uFC interface I have came with a cd with al kind of modbus related code examples, will look it up for you if you want.
Regards, Ron.
Re: Handling long running items without pausing the main loop
I have some working perl code that I have used to talk to modbus devices already as well as the different sources that one can find on the net... That part isn't the problem..
The biggest thing that I wanted to do is to make sure that I handle it correctly so that later on it can extended to support more devices as things are being used.
The modbus stuff that I am looking at is RTU and not ASCII by default.. Modbus products are to required to support RTU while the ascii mode is optional.. Also, if I do this correctly, support Modbus over TCP should also work as well.
Steve
The biggest thing that I wanted to do is to make sure that I handle it correctly so that later on it can extended to support more devices as things are being used.
The modbus stuff that I am looking at is RTU and not ASCII by default.. Modbus products are to required to support RTU while the ascii mode is optional.. Also, if I do this correctly, support Modbus over TCP should also work as well.
Steve