Last
updated 7/21/22
(This page is just
kind of a starter. Lots more details need
to get filled in. A better description is
being started on my Project Notes pages here.)
The system that
collects the data displayed on the front page of this web site has
been under development since late 2008 (with no end in sight :-) Its main purpose is to let me see how the
house is doing when I'm not there – especially when we're on vacation.
Collection/control
network
There are currently
about 14 data collection/control nodes on an RS-485 network running
all around the house (and out to the garage) at 57Kb over Cat5 cable. Three pairs carry unregulated ~9VDC to power
the nodes; the other pair is the (half-duplex) data.
Everything is just wired in parallel.
The old nodes run PIC
16F628s; the majority are now Arduino based. Some
talk I2C to LM75 temperature sensors, some just sense data bits, as
with the rain sensor and sump cycle sensor. Some
have output bits that can control stuff. Ones
that have wires that could get ground referenced (like water sensors
on the basement floor) have opto-isolators
between the external sensor and the local processor.
The one that reads the water meter has a long and
"interesting" history. Some notes here.
The master/slave
protocol on the wire is locally designed since it is not expected to
talk to anything else. (Sounds like
famous last words.) It's basically a sync
byte, from addr, to addr,
opcode, length, data, checksum. The master polls each of the nodes
(currently once/min) and the addressed node replies with its latest
info. Data – both length and meaning - is
device dependent, so the master program has a decode callback for each
device – identified uniquely by its address. The
protocol allows the master to send data to the slave nodes so it can
use them to control external devices.
Currently four nodes
use that 'control' capability: the
router/modem power cycler, the sprinkler controller, the house water
shutoff valve, and an inside/outside temperature/dew point display in
the bedroom. The first gives the computer
a chance to try to fix loss of connection to the internet by power
cycling the modem and main router, which occasionally gets confused.
Some details here. The sprinkler controller lets the main
computer control 8 channels of 24V valves that control watering of
various parts of the landscaping. The house water shutoff,
controlling a hacked Water Cop is obvious. The temp/dewpoint
display in the bedroom doesn't control anything - just displays the
data sent to it.
Main
collection program
Finally moved the main control perl program to
a 5 watt pink Pogoplug from its earlier
home on the main PC. (Trying to host a mission
critical application on a Windows machine is completely unworkable.
Don't get me started.) It's responsible for
polling the slaves, deciding when to give up on them, collects info
for 5 minutes, then tries to ftp the latest to the web site.
It can send email, or text messages to my phone for serious
problems like sensor failures, extended power outages, water where it
shouldn't be, etc. Some details of the
migration to the Pogo are here.
Control of the
sprinkler system is driven by a perl
module that reads a crontab-style control
file. On the list for years has been
using the system's knowledge of rain and temperature and a new soil
moisture sensor and maybe insolation to let it modulate the watering
intelligently. Someday. That same cron-type system
now also extracts special data from the stdout logs,
writes timestamps to make logs easier to read, etc. Of course
the real system cron currently controls 9 other tasks, with lots
of commented-out leftovers.
There is a little web
server on the Pogo, and it would be nice if I could serve the main
stats page from there as a backup to the web host.
No progress on that yet. It does host a day's
worth of security cam pictures, tho. The web
server (and sshd!)
are visible on non-standard ports from the Internet.
I'm using a dynamic DNS mechanism for that.
UPS
The whole system – and
especially its ability to sense and report power failures obviously
doesn't work well if the computer or network/internet connection go
down if power goes out. The big, old,
ugly sine wave UPS/power conditioner retired from work finally died
and was replaced with a current 1KW unit (11/2020).
The two 12V gel cells that used to live inside it have been
replaced with two large marine deep-cycle batteries sitting next to
the UPS. It's ugly, but it keeps the
computer and router and modem up for maybe 3 hours of power outage.
The new UPS has a
little more intelligence than the old one's contact closure on shut
down, but it isn't connected yet. On the
list is a battery voltage monitor that should let me shut the main PC
(the biggest power hog) down gracefully while there's
still lots of battery left to keep the Pogo, modem, and
router up. That monitor – which needs to
be opto isolated from the network, since
there's voltage relation between the battery terminals and the output
AC – will talk to the PC over the 485 network.
Interestingly, the batteries are protected from lack of maintenance by – my backup sump pump! That has a sensor and an alarm for "needs water" which beeps every couple of months. As long as I have water and tools out to fill that one, I always fill the UPS batteries as well. (That statement is a little out of date.)
That sump pump backup is based on a Basement Watchdog 'Big Dog' system with which I have a love/hate relationship. Its badly flawed charging system (which is a critical part of its job!) has caused me to buy unnecessary replacement deep cycle batteries as well as investing a huge amount of time and energy in monitoring the battery to protect myself. It even has its own graph on the home page now! I've come to believe my unit is just faulty and needs to be repaired (by driving 45 minutes to their shop). A semipermanent improvement is a HF float charger that does a WAY better job of maintaining the terrible Basement Watchdog unit. The whole saga will be written up some day.
File
transfer
The update data sent
to the web site consist of one ascii
line per sensor per sample. The main perl program attempts an ftp of the latest
updates every 5 minutes. If it fails, the
updates are remembered (in a file) and all will be sent when it next
succeeds in connecting.
With both my previous
and new hosting provider, there were occasional ftp failures which
truncated the ever-growing file on the web host, losing all previous
info. That rarely happens any more, but
the append-only file does grow to an unmanageable size, requiring
manual intervention. Over the years, I developed tools to warn
me about that and make intelligently trimming the file fairly
painless.
Alerts
The main perl
program currently can send email, and an SMS message by sending an
email to my phone text address. It alerts
me if a sensor fails to respond 5 times in a row or if there's a power
outage. When the
water sensors are in place, if they find undesired water that will
certainly send an alert, too. Note
that this is dependent on my internet link being up!
I've vaguely considered a dial-out backup, but it isn't
likely.
Ping stats
to ISP
While it's not part of
the 485 network, a separate perl program
does a ping every
5 seconds to the router at my ISP that my modem connects to
and keeps round trip time stats. This is
mostly to detect whether the internet connection is OK.
It could go to a few other stable sites outside my ISP to get
a better view. The stats from that bit of
perl are also collected by the main
program and sent to the web site so I can see how connectivity has
been. The collection program notices when
it thinks connectivity is bad – currently 20 consecutive missed pings
– and sets a flag that tells the main perl
program to tell one of the 485 nodes to power cycle the modem and
router. There's some detail here.
Web site
software
I found some php software that can produce graphs with auto scaling, etc. Some more php reads the datafile and gets the data to feed the graphs. The datafile analyzing software bucketizes the one-minute data into something that makes for smoother graphs. It's a 'work in progress'.
It used to have some config options on the front page, but that let some damn Pakistani hackers in to trash the site. After cleaning up that mess, I locked it down a lot more. If I want the display different, I can ssh in and change it that way.
There's a watchdog that alerts me if things go wrong: the web host doesn't get an update for a while, file size is too big, graphs are stale, security cam pics are stale, etc. After a few years' battle to keep my watchdog process running in the background (hosting providers don't like that) I finally found I could make it cron-based, and it's been stable for years now. That watchdog can send email or - via email - text messages.
I succeeded (6/2019)
in using wget on the web host to get a
nice ascii
table of rainfall as recorded by the USGS at Salt Creek here in
Elmhurst. That data is taken every 5
minutes – just like mine. It's integrated
with the current rainfall info from my hardware as another line on the
rainfall graph. It languished on the list for many
years, but it's finally in place.
I would really like to change the back end to maybe a real database and do periodic data compression/purging of old data. (I'll keep my one minute data for a while, but maybe reprocess to hourly (average + peak?) after 2 weeks or a month, and maybe daily after 6 months. I still have the original one minute data on the PC at home.) My new hosting provider (GoDaddy) supports mysql, so that makes it a little easier. And maybe I can even get around the ftp failures/file truncations with transactions to the db! The Round Robin Database tool looks pretty interesting, but I don't know if it will run on either the web host or the Pogo.
New
commercial home automation
I installed some Z-wave (Plus) switches and outlets and a Hubitat C7 to control them (scrapping all the old X-10/Insteon stuff) at the end of 2021. As always, there were wrinkles, but it's pretty stable, and both Alexa and Google Home learned to control the lights with little fuss. A remotely controllable thermostat and maybe front door lock are on the list. The interesting part is that (a year and a half later) there's still no connection between the two systems. There's no driving need, but it seems like two systems in my house should be able to talk to each other. And Google and Alexa as well.
…end…