Toon app: Afvalwijzer

Forum about forum-provided applications on Toon

Moderators: marcelr, TerrorSource, Toonz, TheHogNL

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Mon May 29, 2017 8:45 pm

QtWebsockets isn't really for this purpose, but it allows tools like domoticz to interact directly with specific QML apps like sending realtime updated device info to Toon.
Let's start with wget :-)
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby marcelr » Mon May 29, 2017 9:19 pm

OK, done, see Downloads thread.
marcelr
Global Moderator
Global Moderator
 
Posts: 1145
Joined: May 2012
Location: Ehv

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Mon May 29, 2017 10:06 pm

That's quick. Working great, just need to use `--no-check-certificate' to skip validating the webserver certificate issuer authority.
Now need to start reading the cron wiki (never used it :oops: ) and need to find out the exact url for a specific waste zipcode in Eindhoven.
The parsing bit is easy.....
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby marcelr » Mon May 29, 2017 10:14 pm

cron is easy to use :-).
I think we just need to make a small shell script that takes care of the downloads (through wget) and then repeat that download every week or so, through a crontab entry.
marcelr
Global Moderator
Global Moderator
 
Posts: 1145
Joined: May 2012
Location: Ehv

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Mon May 29, 2017 10:34 pm

@marcelr: I'm struggling to get to your waste data via wget. First page asks you to enter postcode / housenr and then 'POST's the form data. As far as I know you cannot submit such form data via an URL (this works for forms with 'GET' method but not for 'POST'). Any tips? Could this be scripted otherwise?
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Tue May 30, 2017 10:32 pm

Hi all,

I enhanced the app with the waste provider 'cure-afvalbeheer.nl'.
As this provider is using 'https', a separate method of retrieving the data needs to be installed (will be posted separately by Marcelr).
This version is reading the file '/root/waste/waste_calendar.ics' to get the waste collection dates for Cure Afvalbeheer.
You therefore do not need to enter the postcode/housenr in the WastecollectionApp.qml app for this provider.

Configuration:
You need to edit WastecollectionApp.qml at the top to select one of the three supported waste providers
And edit your zipcode/house nr (not needed for Cure Avalbeheer).
This version is fully compatible with the thermostat_mod V1.2 posted earlier.

Currently supported:
"mijnafvalwijzer.nl"
"deafvalapp.nl"
"cure-afvalbeheer.nl"

Let me know if you want support for additional providers.

Regardsz,

Toonz
Attachments
wastecollection_V1.3.zip
(13.42 KiB) Downloaded 197 times
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby QuasaR » Wed May 31, 2017 1:41 pm

Can you implemente support for Cyclus ?

https://afvalkalender.cyclusnv.nl/form (2841 SP / 44)
QuasaR
Starting Member
Starting Member
 
Posts: 23
Joined: February 2017

Re: Toon app: waste collection (Afvalwijzer)

Postby marcelr » Wed May 31, 2017 4:04 pm

QuasaR wrote:Can you implemente support for Cyclus ?

https://afvalkalender.cyclusnv.nl/form (2841 SP / 44)

Should be easy; the site is an almost exact copy of the one of cure afvalbeheer; will post the download scripts and cron install manual later today.
marcelr
Global Moderator
Global Moderator
 
Posts: 1145
Joined: May 2012
Location: Ehv

Re: Toon app: waste collection (Afvalwijzer)

Postby marcelr » Wed May 31, 2017 5:56 pm

Installation of wget and cron, for waste data download.

This is the manual:

Code: Select all
###############################################################################

How to download "afvalwijzer" data from a https website, using wget and cron

20170513, marcelr, first draft

###############################################################################

Recently, Toonz created an app which displays the waste collection agenda for
a specific address on toon.

From version 1.3 onwards, it also supports cure-afvalbeheer (active in the
Eindhoven area).

The waste collection agenda provided by this company is available as an .ics
file, from their website. The website is served as a https service, offered
through service port 443 (secure http or https).

## getting acccess to files served through https (port 443) ##

Toon has no support for this protocol, by default. So, to gain access to this
site, the following is required:

1: (Access to) a rooted toon, with working ssh and (preferably) sftp servers.

2: Install package for wget, with ssl support. Available from the Downloads
   thread on this forum.

3: Some proficiency in using vi.

What to do:

1: Upload wget with ssl support (wget_1.12-r8.2_qb2.ipk) to toon and install:
   opkg install wget_1.12-r8.2_qb2.ipk

2: Open port 443 in the firewall:
   Locate the file  /etc/default/iptables.conf and add the following line:
   
   -A HCB-INPUT -p tcp -m tcp --dport 443 --tcp-flags SYN,RST,ACK SYN -j ACCEPT

3: Restart iptables by issueing the command:

   /etc/init.d/iptables restart

## Find the right URL for your waste agenda and adapt the download script ##

1: The agenda for the waste collection is given in the cure website:
   
   Fill in your address details (post code, house number) and click
   NAAR MAAND- EN JAARKALENDER > -> ZET INZAMELDATA IN UW AGENDA

   Locate the link to your agenda and copy it into the "get_data_from_cure.sh"
   script (it has the following shape):

   https://afvalkalender.cure-afvalbeheer.nl/ical/0yyyy000000xxxxx


#! /bin/sh
#
# get_data_from_cure.sh
#
# script to download calendar data from waste management company Cure
# (Eindhoven area, NL)
#
# call: get_data_from_cure.sh <address code>
#
# The address code is found at the cure website.
# fill in your address details (post code, house number) and click
# NAAR MAAND- EN JAARKALENDER > -> ZET INZAMELDATA IN UW AGENDA
#
# At this page, copy the link
# https://afvalkalender.cure-afvalbeheer.nl/ical/<whatever-your-number-is
# to the command line after the call to this script:
# get_data_from_cure.sh <address code>
#
# Hit <enter> and your personal calender is downloaded to the directory
# defined in $DIR_OUT.
# The address code is linked to your address, not to a time, and thus remains
# the same, so you can repeat downloads from the same URL, and get updates for
# your calendar.
#
# another option is to paste the URL into this script, replacing the first
# argument ($1) of this script, in the definition of $URL below.

WGET=/usr/bin/wget
FILE_OUT=waste_calendar.ics
DIR_OUT=~/waste/
#URL=$1
URL=<paste URL here>

mkdir -p $DIR_OUT

$WGET --no-check-certificate -O /$DIR_OUT/$FILE_OUT $URL &> /dev/null

# end of script


2: On toon, create the directory /root/bin:

   mkdir -p /root/bin

3: Upload the "get_data_from_cure.sh" script to toon, put it in /root/bin, and
   make sure it is executable:

   chmod 755 /root/bin/get_data_from_cure.sh

4: Test by issueing the command /root/bin/get_data_from_cure.sh
   This should give you a file called waste_calendar.ics, in the directory
   /root/waste

## Automation of the calendar download ##

To automate regular execution of programs, unix systems (like toon) use a
utility called cron. Cron is a daemon process which runs in the background,
and checks every minute whether something from the crontab file needs to be
done. The crontab file is a file which sets the rules for cron, what to
execute, and when. It is edited with the program crontab.

To automate waste agenda download, do the following:

1: Download and install cron_3.0pl1-r8_qb2.ipk

2: Edit the crontab for user root, by issueing the following command:

   crontab -e

This opens an editor (typically vi), paste the following into the file:

# crontab for root
#      The time and date fields are:
#
#             field          allowed values
#             -----          --------------
#             minute         0-59
#             hour           0-23
#             day of month   1-31
#             month          1-12 (or names, see below)
#             day of week    0-7 (0 or 7 is Sunday, or use names)
#
SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:                           
15 4 3 * * /root/bin/get_data_from_cure.sh

and save. You may omit the lines starting with a hash mark (#).

3: Start the cron daemon, by issueing:

   /etc/init.d/cron start

## Getting the information on toon ##

1: Install the app wastecollection v1.3 or later.

2: Edit the file WastecollectionApp.qml, to have the property wasteCollector
   reflect your waste collector:

   property string wasteCollector : "cure-afvalbeheer.nl"
   
   and save.

3: Add the wastecollection app to the file /HCBv2/qml/qb/base/Globals.qml.

4: Restart qt with killall qt-gui

5: All done, you should now be able to install a tile with your waste
   collection data from toon's tile selection menu.


and these are the scripts you need:

get_data_from_cure.sh:

Code: Select all
#! /bin/sh
#
# get_data_from_cure.sh
#
# script to download calendar data from waste management company Cure
# (Eindhoven area, NL)
#
# call: get_data_from_cure.sh <address code>
#
# The address code is found at the cure website.
# fill in your address details (post code, house number) and click
# NAAR MAAND- EN JAARKALENDER > -> ZET INZAMELDATA IN UW AGENDA
#
# At this page, copy the link
# https://afvalkalender.cure-afvalbeheer.nl/ical/<whatever-your-number-is
# to the command line after the call to this script:
# get_data_from_cure.sh <address code>
#
# Hit <enter> and your personal calender is downloaded to the directory
# defined in $DIR_OUT.
# The address code is linked to your address, not to a time, and thus remains
# the same, so you can repeat downloads from the same URL, and get updates for
# your calendar.
#
# another option is to paste the URL into this script, replacing the first
# argument ($1) of this script, in the definition of $URL below.

WGET=/usr/bin/wget
FILE_OUT=waste_calendar.ics
DIR_OUT=~/waste/
#URL=$1
URL=<paste URL here>

mkdir -p $DIR_OUT

$WGET --no-check-certificate -O /$DIR_OUT/$FILE_OUT $URL &> /dev/null

# end of script


crontab for root:

Code: Select all
# crontab for root
#      The time and date fields are:
#
#             field          allowed values
#             -----          --------------
#             minute         0-59
#             hour           0-23
#             day of month   1-31
#             month          1-12 (or names, see below)
#             day of week    0-7 (0 or 7 is Sunday, or use names)
#
SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:                           
15 4 3 * * /root/bin/get_data_from_cure.sh


Will post everything in Downloads as well, but that's for later, gotta run ...
marcelr
Global Moderator
Global Moderator
 
Posts: 1145
Joined: May 2012
Location: Ehv

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Wed May 31, 2017 5:58 pm

almost the same, collection of paper is called differently ('Oud papier' instead of 'Papier') and plastic is collected separately. Implementation will be easy. will do later today.
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Wed May 31, 2017 6:31 pm

Hi all,

I enhanced the app with the waste provider 'cyclusnv.nl'.
This site is almost a copy of cureafvalbeheer.nl.
As this provider is using 'https', a separate method of retrieving the data needs to be installed as posted by Marcelr earlier.
This version is reading the file '/root/waste/waste_calendar.ics' to get the waste collection dates for Cyclus NV.

Configuration:
You need to edit WastecollectionApp.qml at the top to select one of the four supported waste providers
And edit your zipcode/house nr (not needed for Cure Avalbeheer and Cyclus NV).
This version is fully compatible with the thermostat_mod V1.2 posted earlier.

Currently supported:
"mijnafvalwijzer.nl"
"deafvalapp.nl"
"cure-afvalbeheer.nl"
"cyclusnv.nl"

Let me know if you want support for additional providers.

Regardsz,

Toonz
Attachments
wastecollection_V1.4.zip
(13.47 KiB) Downloaded 172 times
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby Toonz » Wed May 31, 2017 6:55 pm

marcelr wrote:SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:
15 4 3 * * /root/bin/get_data_from_cure.sh


Hi QuasaR,

I noticed your waste calender contains only data for the next 4 weeks (CureAfvalbeheer gives more entries in the future).
You might consider retrieving the file every 2 weeks instead of once a month.

Regardz,

Toonz
member of the Toon Software Collective
User avatar
Toonz
Forum Moderator
Forum Moderator
 
Posts: 1481
Joined: December 2016

Re: Toon app: waste collection (Afvalwijzer)

Postby QuasaR » Tue Jun 06, 2017 4:55 pm

Toonz wrote:
marcelr wrote:SHELL=/bin/sh
# download .ics file for "afvalwijzer" app, every third day of the month,
# at 4:15 in the morning:
15 4 3 * * /root/bin/get_data_from_cure.sh


Hi QuasaR,

I noticed your waste calender contains only data for the next 4 weeks (CureAfvalbeheer gives more entries in the future).
You might consider retrieving the file every 2 weeks instead of once a month.

Regardz,

Toonz


Tnx Toonz.

So my entry for the crontab would be like this ?

15 4 3,17 * * /root/bin/get_data_from_cure.sh
QuasaR
Starting Member
Starting Member
 
Posts: 23
Joined: February 2017

Re: Toon app: waste collection (Afvalwijzer)

Postby marcelr » Tue Jun 06, 2017 6:12 pm

Yes, only the name of the script in the zip-file has changed (in the manuals/tutorials thread).
(get_waste_calendar_data.sh)
marcelr
Global Moderator
Global Moderator
 
Posts: 1145
Joined: May 2012
Location: Ehv

Re: Toon app: waste collection (Afvalwijzer)

Postby dennis071nl » Tue Jun 06, 2017 7:47 pm

Hi guys,

Where can i find a list of waste collection providers?? I live in Katwijk, actually Valkenburg gemeente Katwijk, but i can't find my area on the given sites...

Can anyone confirm this??

Thanks, Dennis
dennis071nl
Starting Member
Starting Member
 
Posts: 35
Joined: June 2017

PreviousNext

Return to Toon Apps

Who is online

Users browsing this forum: No registered users and 1 guest

cron