Copy Historical data (rra databases) between Toons

Everything about software tools, new software development and toolchains. For developers, mostly.

Moderators: marcelr, TheHogNL, Toonz

Toonz
Forum Moderator
Forum Moderator
Posts: 1873
Joined: Mon Dec 19, 2016 1:58 pm

Copy Historical data (rra databases) between Toons

Post by Toonz »

Hi all,

EDIT 29/3/2021 This method has been replaced by a much better solution further down on this thread :
viewtopic.php?f=103&t=12168&start=15#p96410

I would like to share my latest (small) activity: copy historical data from one Toon to another one.
Source and target Toons can either be a Toon 1 or a Toon 2.
It was my first Python script ever, so things can probably be done smarter, but hey, it works....

This script looks for matching topics in the rra database config files from both Toons and copy matching items from source to target with the target uuid's in the filename.
Items copied from one Toon to another: electricity and gas usage, electricity produced, OpenTherm boiler data and thermostat data (actual temperature, target temperature, program state)

Short 'manual':

- on your pc create three folders rraSource, rraTarget and rraOutput
- copy the contents of folder /qmf/var/hcb_rrd from the source Toon into the folder rraSource (typically about 80 files or more if you have smart switches)
- copy the contents of folder /qmf/var/hcb_rrd from the target Toon into the folder rraTarget (typically about 80 files or more if you have smart switches)
- copy the contents of folder /qmf/var/hcb_rrd from the target Toon into the folder rraOutput as well
- copy the contents of file /qmf/config/config_hcb_rrd.xml from the source Toon into the folder rraSource (contains the description of the files in rraSource)
- copy the contents of file /qmf/config/config_hcb_rrd.xml from the target Toon into the folder rraTarget (contains the description of the files in rraTarget)

- download the python script below and edit the first lines with actual folder names
- execute the Python script on your pc (e.g.: py CopyDataToon.py)

once executed, all matching data has been copied from the rraSource folder into the rraOutput folder (but renamed to the new uuid's of the target Toon)

- copy all files from rraOutput back to the folder /qmf/var/hcb_rrd on the target Toon
- sync and reboot Toon. That's all folks.

Have fun, comments are welcome of course.

Kind regardz,

Toonz
Attachments
CopyDataToon.zip
(817 Bytes) Downloaded 692 times
member of the Toon Software Collective
spattex
Starting Member
Starting Member
Posts: 20
Joined: Fri Jul 24, 2009 4:57 pm
Location: Netherlands

Re: Copy Historical data (rra databases) between Toons

Post by spattex »

Top!
TheHogNL
Forum Moderator
Forum Moderator
Posts: 2125
Joined: Sun Aug 20, 2017 8:53 pm

Re: Copy Historical data (rra databases) between Toons

Post by TheHogNL »

Great work. Does not contain the code to move the solar generated electricity yet.

Didn't knew it was so easy. Could be converted to a bash script which you run on one of both toons and which collect the necessary data itself.
Member of the Toon Software Collective
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Post by marcelr »

Probably it can be done in qml/javascript as well. Then it should be possible to transform it into a toonstore app.
Toonz
Forum Moderator
Forum Moderator
Posts: 1873
Joined: Mon Dec 19, 2016 1:58 pm

Re: Copy Historical data (rra databases) between Toons

Post by Toonz »

By all means, be my quest :)
I used Python to learn it a bit, this was more of a 'demo' project for me
member of the Toon Software Collective
Toonz
Forum Moderator
Forum Moderator
Posts: 1873
Joined: Mon Dec 19, 2016 1:58 pm

Re: Copy Historical data (rra databases) between Toons

Post by Toonz »

Solar data is copied as well but only if the target Toon also has a solar rra database.
If we want to copy 'new' rra databases to a Toon you also need to add that section in the config xml file. Haven't played with that option. Might work, haven't tried....
member of the Toon Software Collective
Rudolf
Member
Member
Posts: 136
Joined: Mon Dec 04, 2017 8:50 pm

Re: Copy Historical data (rra databases) between Toons

Post by Rudolf »

Do I have to pair Toon2 / rraTarget with a meteradapter first, or is that totally unrelated to the uuid's in the files rewritten in this script?
Toonz
Forum Moderator
Forum Moderator
Posts: 1873
Joined: Mon Dec 19, 2016 1:58 pm

Re: Copy Historical data (rra databases) between Toons

Post by Toonz »

Rudolf wrote:Do I have to pair Toon2 / rraTarget with a meteradapter first, or is that totally unrelated to the uuid's in the files rewritten in this script?
good question, depends when the consumption rra databases are being created. Before or after the pairing with the meteradapter.
I guess the latter but am not 100% sure.
member of the Toon Software Collective
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Post by marcelr »

I'm working on a piece of code that transfers data from an old toon into a new one, typically through the export function, but also through raw data transfer from an old toon. Everything seems to work, but I bump into issues when the old toon has single-tariff electricity and the new one has dual tariff (low/normal). This code eventually should find its way into the TSC menu.

So, I would like to get a dataset from someone who has had a toon for quite some time, and during that period changed his contract from single to dual tariff, or the other way around.
Specifically, I will need /HCBv2/config/config_hcb_rrd.xml and the entire contents of /qml/var/hcb_rrd/ (or /HCBv2/data/hcb_rrd/ in really old firmwares).

If you are ready to hand me those data, please contact me through PM or mail.
glsf91
Member
Member
Posts: 184
Joined: Fri Sep 15, 2017 9:25 pm

Re: Copy Historical data (rra databases) between Toons

Post by glsf91 »

marcelr wrote:I'm working on a piece of code that transfers data from an old toon into a new one, typically through the export function, but also through raw data transfer from an old toon. Everything seems to work, but I bump into issues when the old toon has single-tariff electricity and the new one has dual tariff (low/normal). This code eventually should find its way into the TSC menu.

So, I would like to get a dataset from someone who has had a toon for quite some time, and during that period changed his contract from single to dual tariff, or the other way around.
Specifically, I will need /HCBv2/config/config_hcb_rrd.xml and the entire contents of /qml/var/hcb_rrd/ (or /HCBv2/data/hcb_rrd/ in really old firmwares).

If you are ready to hand me those data, please contact me through PM or mail.
Are you sure that with a single tariff not elec_quantity_lt and elec_quantity_nt are both filled at the right time?
I had single tariff (but not from Eneco) and analog sensor last year and both are filled.
Where can I check where the tariff is defined?
(maybe that was defined wrong)
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Post by marcelr »

Are you sure that with a single tariff not elec_quantity_lt and elec_quantity_nt are both filled at the right time?
Probably.

My production toon has single tariff, and my test specimen has dual tariff. No idea where that choice is set. It has elec_quantity_nt completely filled, but that's only shown on the test specimen in the times where normal tariff applies.
Maybe both files are filled normally , but for some reason my production toon does not have that info.
I had single tariff (but not from Eneco) and analog sensor last year and both are filled.
Where can I check where the tariff is defined?
(maybe that was defined wrong)
See previous. No idea.
hafeez
Starting Member
Starting Member
Posts: 10
Joined: Thu Dec 21, 2017 10:26 am

Re: Copy Historical data (rra databases) between Toons

Post by hafeez »

I also want to copy my historical electricity and gas usage, solar electricity produced data from my old Toon to my new Toon2. Both are rooted.
Any idea when the app to accomplish this will be released?
Else I might start to copy it via the instructions in this topic but would like to take easy step if available already.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Post by marcelr »

It's in the testing phase right now. Give it another week and there should be a more mature version. If you can't wait, there's an alpha version available from the ftp server, for toon type 2 only. It's called transfer-logs-<long version-number>.ipk. Be sure to use the -b option (creating back-ups), before you begin. xml output is functional, but not very good-looking, will be fixed in later versions. The rra databases should be fine.
To use it, just open a shell and type transfer-logs. This will give you some instructions on how to use the code.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Post by marcelr »

Version 0.0.3 is now available for toon2. Prettyprint function added to get a proper xml layout in the output file for monthly usage.

And remember:

THOU SHALT MAKE BACK-UPS!

You will also need the libcurl5 package. Available from the same directory.
marcelr
Global Moderator
Global Moderator
Posts: 1153
Joined: Thu May 10, 2012 10:58 pm
Location: Ehv

Re: Copy Historical data (rra databases) between Toons

Post by marcelr »

Just removed the transfer-logs package. some things need fixing first.
Post Reply

Return to “Toon software development”