A new event architecture

[hplus]

Happy new year everybody!

I wanted to talk about one of my main problem regarding event management in domotiga:
Here is an example to explain the situation:
Event_1: Trigger_A, Condition_B => Action_1
this event
case 1: will fire when Trigger_A BECOMES true and Condition_B IS true
case 2: will not fire if Trigger_A IS true and Condition_B BECOMES true
This is because of the definition of triggers and conditions. But most of the time I want the action to be performed also in the case 2. To do this i add :
Event_2: Trigger_B, Condition_A => Action_1
and this solves the problem.

Now imagine you have two conditions, this time you need:
Event_1: Trigger_A, Condition_B and Condition_C => Action_1
Event_1: Trigger_B, Condition_A and Condition_C => Action_1
Event_1: Trigger_C, Condition_A and Condition_B => Action_1

This makes a lot of events for the same action.

This complexity could be overcome if there was a possibility for multiple triggers and an advanced trigger editor (same as the condition editor). I think this was planned....

A deeper evolution of the event management could also be a really powerful solution to this issue and it solves also other problem. This is how i see things:

A trigger is a list of variables (globalvars, time, device value, ...), it triggers when one of the variable is updated
And conditions are integrated with actions.

Just an example to illustrate this concept:
Event :
Trigger(Device_45.Value1, TimeNow(0 * * * *), GlobalVar_HouseMode, GlobalVar_SetPoint)
Actionlist :

• Action1 : If( GlobalVar_HouseMode=="normal" And Device_45.Value1 < GlobalVar_SetPoint , setDevice(Device_33.Value1,"On"), setDevice(Device_33.Value1,"Off"))
• Action2 : SendEmail( "foo@domotiga.nl", "subject", "the temperature is %Device_45.Value1%")

This is one event. To perform the same actions with today's architecture you need 8 events !

Please tell me what do you think of this solution and if you see a limitation.

Hplus

[RDNZL]

Although it looks quite the same, it's totally different from the current mechanism.

In your picture, if one of the trigger list items changes, it has to 1) search all events and actions for that one trigger value, and 2) Eval them each and everyone.
I cannot tell now if this is feasible.

Can you describe some more events with you like to have more than one trigger?
Isn't the lack of more than two conditions the basic problem?

I will see if I can test somethings quickly.

[hplus]

Here is a real life example, it deals with temperature control.

I have a digimax thermostat which sends 3 values:

• value1: current temperature
• value2: temperature setpoint
• value3: demand for heat or no (this basically says if value1<value2)

I also have 2 global vars which can be modified through the main gui:

• Control_Mode: can have 3 values Local, Override, AlwaysOff
• Setpoint

The globalvar Control_Mode is used to say how we control the temperature, locally by changing the setpoint on the digimax (you have to be in your house) or by overriding this value and setting a new setpoint or just leaving the heater off.
The globalvar Setpoint is used when you override the local setpoint.

I need 13 events to control one heating point:

• Trigger(Control_Mode=AlwaysOff)=>Off
• Trigger(Control_Mode=Local),Condition(Digimax.Value3=NoHeat)=>Off
• Trigger(Control_Mode=Local),Condition(Digimax.Value3=Heat)=>On
• Trigger(Digimax.Value3=NoHeat),Condition(Control_Mode=Local)=>Off
• Trigger(Digimax.Value3=Heat),Condition(Control_Mode=Local)=>On
• Trigger(Control_Mode=Override),Condition(Digimax.Value1>Setpoint)=>Off
• Trigger(Control_Mode=Override),Condition(Digimax.Value1<Setpoint)=>On
• Trigger(Digimax.Value1<100),Condition(Digimax.Value1>Setpoint)=>Off //this trigger fires whenever the temp is updated
• Trigger(Digimax.Value1<100),Condition(Digimax.Value1<Setpoint)=>On
• Trigger(Setpoint<100),Condition(Digimax.Value1>Setpoint)=>Off //this trigger fires whenever the setpoint is updated
• Trigger(Setpoint<100),Condition(Digimax.Value1<Setpoint)=>On
• Trigger(TimeNow(0,10,20,30,40,50 * * * *)),Condition(Heater.Value1=Off)=>Off
• Trigger(TimeNow(0,10,20,30,40,50 * * * *)),Condition(Heater.Value1=On)=>On

There is event when you change the Control_Mode, event when the temperature change, event when the overriding setpoint change, event for watchdog resending each 10 minutes the device state.
I don't know if i am crazy, but i would like to create a lot more complex scenarios... but the huge amount of event needed each time i add something make this impossible to maintain...

With the new architecture you only need 1 event!!!!!:

Trigger(Control_Mode, Setpoint, Digimax.Value1, Digimax.Value3, TimeNow(0,10,20,30,40,50 * * * *))
Action:
setDevice(Heater.Value1,
IF((Control_Mode = "Local" AND Digimax.Value3=Heat) OR (Control_Mode="Override" AND Digimax.Value1 < Setpoint), "On", "Off"))

Notice the difference compared to my previous post, the action begins with the action type (setDevice) this is closer to today's architecture. What we need to add is an expression editor/evaluator (same as in the condition editor) in each tab of the action editor (setdevice, setglobal, sendemail .....)


I don't see any limitation... I only see that it would be a lot more powerful and will prevent the combinatorial explosion of events with todays paradigm.

[RDNZL]

Great ideas...
What happened to conditions?

I think about it, but i'm more into the following scenario:

1) Replace triggers with list of trigger types.
so "Device, Globalvar, etc... Time as you describe it is not so easy since it requires parsing/eval of this triggers to.

You need to add the current trigger conditon, to conditions.

2) Each trigger type checks if his trigger type is in list of triggers of each event, if it is, it will parse those events, and if conditions are met, it runs the actions for that event.

It's not so bad to keep the current conditions and actions as they are, is it?
So you can reuse those for other events...

Some caveats:
Currently you cannot run TimeCron() in a condition Eval, have to check this.
And you didn't taken into account the other trigger types like Received IR, IViewer etc.
This functionality is mandatory for your idea, so there are limitations for now for sure.

Or i have to parse them in the trigger, but this makes it not uniform working for all trigger types.

Like I said, I will try things, but the event code is kinda hard to change/create... you run soon into all kinds of pitfalls...

[RDNZL]

Screenshot-Edit Trigger.png (22.36 KiB) Viewed 13043 times

@hplus

I have build some test code for Dev_ and Var_ multi triggers.
Just to see the impact on sql load, and to see if this is correct way to do it.

Copy attached files over yours (make a backup first), then create triggers of type Multi, you can add one or more triggers.
And it will trigger on them.

Of course this trigger type 6 will be removed so it will be default (the only one) if all works well.
And all trigger conditions have to be moved to the conditions itself.

Just for this test Globalvar and DeviceChange events will be tested for normal AND multi triggers... so you have double event queries.
This will change of course.

Now i'm trying to find a way to add TimeCron, IR and IViewer to them.
So the attached code is not 100% functional for your scenario's I think.

Can you try this, and tell me if this is the direction you want to go?
Try it on dev/backup db, or be prepared to remove triggers with type 6 later.

[hplus]

I'll try this.

In my idea condition could be only inside action, but keeping conditions the way they are is ok and can be helpful for some type of events.
But being able to have a logical expression (same as in the condition) inside the action would open a new world!!

I am sure it will be possible to have a uniform trigger even for other type of interface. Sometimes we should rethink what a trigger means and what a condition means. And we should have a clear separation between both.

Thanks

[hplus]

By the way i think i noticed a bug:
I commonly use a trick to have a trigger fire each time a device value change by setting a trigger with lets say a < 100 criteria. This criteria will be always met and the trigger will fire each time this device value changes.
Then I put my real condition in the condition field. I do this because my condition involves a comparison with a globalvar and this kind of test can only be performed in the condition field.
The problem is that i think the internal state of the condition evaluator is not yet aware of the new device value, thus the condition is evaluated with to old device value.

This bug will be blocking in the new event architecture.

[hplus]

Ok this time i tried to solve the bug by myself not just reporting it. We should replace this (and for each value as well):
in the file Devices.module in function PUBLIC SUB ValueUpdate(....)

Code: Select all

   IF Comp(vValue, sValue, gb.Text) <> 0 THEN
      IF Main.bDevicesDebug THEN Main.WriteDebugLog(("[Devices] Device change triggered for device with id '") & iId & ("' and value1 '") & vValue & "'")
      EventLoop.DeviceChanged(iId, "Value", vValue, sLabel)
      bUpdated = TRUE
      IF rDevice!log THEN vValUpdated["value"] = vValue
==> rResult = Main.hDB.Exec("UPDATE devices SET lastchanged = &1, lastseen = &2, value = &3 WHERE id = &4", Format(Now(), "yyyy-mm-dd hh:nn:ss"), Format(Now(), "yyyy-mm-dd hh:nn:ss"), vValue, iId)
      IF rDevice!dimable OR rDevice!switchable THEN Main.RefreshControlPage()
    ENDIF

by this

Code: Select all

   IF Comp(vValue, sValue, gb.Text) <> 0 THEN
 ==>rResult = Main.hDB.Exec("UPDATE devices SET lastchanged = &1, lastseen = &2, value = &3 WHERE id = &4", Format(Now(), "yyyy-mm-dd hh:nn:ss"), Format(Now(), "yyyy-mm-dd hh:nn:ss"), vValue, iId)
      IF Main.bDevicesDebug THEN Main.WriteDebugLog(("[Devices] Device change triggered for device with id '") & iId & ("' and value1 '") & vValue & "'")
      EventLoop.DeviceChanged(iId, "Value", vValue, sLabel)
      bUpdated = TRUE
      IF rDevice!log THEN vValUpdated["value"] = vValue
      IF rDevice!dimable OR rDevice!switchable THEN Main.RefreshControlPage()
    ENDIF

[RDNZL]

By the way i think i noticed a bug:
I commonly use a trick to have a trigger fire each time a device value change by setting a trigger with lets say a < 100 criteria. This criteria will be always met and the trigger will fire each time this device value changes.
Then I put my real condition in the condition field. I do this because my condition involves a comparison with a globalvar and this kind of test can only be performed in the condition field.
The problem is that i think the internal state of the condition evaluator is not yet aware of the new device value, thus the condition is evaluated with to old device value.

This bug will be blocking in the new event architecture.

You shouldn't have to use that "< 100 " trick, because EventLoop.DeviceChange is only called when device value has changed to begin with:

Code: Select all

IF Comp(vValue, sValue, gb.Text) <> 0 THEN

Thats why the routine is named "DeviceChange"

I trigger on all device value changes all the time without problems...
If you mean you want to trigger all the time, even when value hasn't changed, you need to pick LastChanged as trigger field instead.

Also it's updated after event code on purpose after Eventloop.DeviceChange calls, because the new value is passed as vValue to this routine directly, and the event code has access to the old value via sql call.

If this was a bug basic events like when value changes from off->on send e-mail won't work, or it keeps sending e-mails.
And those work ok since I use a lot of those.

But like I wrote before, events are difficult, I'm happy they work quite nice now, and I'm open for your changes as long as we don't throw the baby out with the bathwater..

I don't if you are having this bug with old event code or new, in new you cannot specify a trigger value anymore, just a device field, so it's kinda confusing.
Please try to post (or e-mail directly) some more info/ debug logs output if I understood your problem wrongly.
Or open a ticket with that info on domotiga.nl and I will investigate.

[hplus]

I was talking about the 0.1.193 official version.
Let me explain once more the problem:
I want a trigger that compares a device.value against a globalvar. This is not possible in the trigger.
So I create a trigger = device.value < 100 => this will always be true because the value will never reach 100 ( i understand you first check if the new value is different from previous value)
And in the condition i do the real test which is device.value<globalvar.

The problem:
Here is the function call stack
EventLoop.DeviceChanged
=>Events.CheckDeviceChangeEvents
==>Events.DoEvent
===> Events.CheckConditions <= this function uses the DB values (old)

that why you should update the database at least before calling Events.CheckConditions. You could do it before EventLoop.DeviceChanged because old value is not used anymore after that point (IF Comp(vValue, sValue, gb.Text) <> 0 THEN)

[airox]

This is a situation which I also have experienced during building my software. I decided to allow actions to be done without a trigger and only be based on conditions. Downside is that you need to loop over two sets to evaluate to decide if there are some actions to be done for them.

Remember that using only conditions gives you a whole lot of flexibility. Consider the following sample from my software:

Code: Select all

<conditionBasedEvent id="OVERLOOP_LIGHTS_WHEN_NECESSARY" name="Zet lichten aan/uit op de overloop">
	<conditions logical="AND">
		<equal>
			<motion id="BEWEGING_OVERLOOP">1</motion>
		</equal>
		<equal>
			<flagValue id="PRESENCE">ja</flagValue>
		</equal>
		<equal>
			<flagValue id="LIGHTS_NEEDED">ja</flagValue>
		</equal>
	</conditions>
	<actionsWhenTrue>
		<switchOn device="OVERLOOPSCHAKELAAR" />
	</actionsWhenTrue>
	<actionsWhenFalse>
		<switchOff device="OVERLOOPSCHAKELAAR" />
	</actionsWhenFalse>
</conditionBasedEvent>

You can attach actions on the true side and on the false side. This is really what you want because often things need to be handled when the condition becomes false or true.

I thought about a few optimizations you can do when you need to fetch a lot of event data. This is something I never implemented because it's still fast enough for me (after adding indexes on all tables):
- Cache all condition and triggers in memory. When something needs to be evaluated you fetch a list of dates on which the conditions and triggers have changed. If something is changed (in my experience this almost never happens, HA is configured once and only altered on new devices or insights) you will update your cache. This limits the number of queries.
- Cache all condition and triggers and only refresh this every minute. This would allow for everything to be done in memory and without database communication (consider doing the logging stuff async).

I'm really interested in the experiences you guys also have in other event architectures.

The following are now supported by my software:
- Trigger (and conditions) based
- Conditions based
- Recurring based (every X minutes, hours, days)
- Sunset and sunrise based
- Incoming email based
- Date, time, day of week based

The following stuff I want to support, but haven't got the time for it:
- Incoming twitter based
- Incoming IRC message based
- Voice activation based
- Conditions over time (have actions connected to the following condition: "if within two minutes the car arrived home, there was movement outside and the frontdoor has opened then switch off the alarm")

[RDNZL]

===> Events.CheckConditions <= this function uses the DB values (old)

Where do you get this from?

It's not in here:
domotiga.nl/browser/trunk/DomotiGa/Even ... odule#L295
CheckConditions work with conditions and the AND/OR in between them, not device values.

Nor here:
domotiga.nl/browser/trunk/DomotiGa/Even ... odule#L194
sValue in there comes from subroutine argument, (all the way back from Devices.ValueUpdate()) not a db load as far as i can see.

I'm sure your running/have tested this bug-fix already, but it want to understand it because it could lead to other related problems later.
Almost all lines of code in events are there (and I mean the exact spot/line number) for a reason

[RDNZL]

Ok got it, had a lightbulb moment in my car on the way home.
Will investigate.

[hplus]

it's here
PRIVATE SUB AddDevicesToContext(cContext AS Collection)
this query the devices states from the database and replace them inside the condition formula....
the fix i proposed worked for me, and i didn't find any potential problem. But of course i don't know the whole baby enough...

by the way, i've tested the multi trigger, it seems to work!

[RDNZL]

You can attach actions on the true side and on the false side. This is really what you want because often things need to be handled when the condition becomes false or true.

I thought about a few optimizations you can do when you need to fetch a lot of event data. This is something I never implemented because it's still fast enough for me (after adding indexes on all tables):

I'm really interested in the experiences you guys also have in other event architectures.

Having actions for true and false conditions is a very good idea!
I now have 2 events for a lot of things, "if device value = on then" and seperate "if device value is off then"...
I'm going to try to implement that.

A step further like hplus suggested earlier -having conditions for each action- is almost like having scripting capabilities, but I have not succeeded to implement that (i can only eval() basic gambas code, which returns true/false)

About your event architecture:
If I understand correctly you parse all your event conditions whenever there is something going on, either a device value changes, a lastchanged/seen time changes, time changes... you don't have trigger types on which you filter as first step, right?

I did that so triggertype 1 = cron/time, 2 = device value.. x = globalvar changes (for e-mail counters, callerid etc) and then I only checked if device and field matched, before i continued the parse...
Now (thanks to suggestions of hplus) I introduced multitrigger (as a test) so events can be triggered on various types.

We must gtalk some time , but we had that idea earlier if I remember correctly, a bit lack of time here..