Swimming, 22nd December 2021

It’s getting very quiet at the pool as Christmas approaches. I think there were at most seven swimmers in the session today. I wasn’t feeling too great but my 50m set actually went quite well in the end. I had a single failure on rep 14 and skipped the next, but I also had several reps below 42s which I’m quite happy with.

Perhaps as a result my 25m reps weren’t the most stunning, though I only had one failure, on rep 10. I skipped 11 and then ran out of time just as I was about to set off on the next. I suspect I got less swimming time than I was expecting because of having to faff about paying for my son at reception rather than online as I was able to earlier this week.

Posted in Swimming | Tagged , | Leave a comment

Second take on emoncms device template for RFXmeter

I felt I needed to revisit the template for my RFXmeter devices because what’s really required is kWh consumption figures between readings rather than cumulative values. Also there needs to be a way to handle the count values rolling over. Having read through some more of the emoncms forum, I believe the following is now a better version of the template, though I’m prepared to have to change it again.

{
  "name": "single-input RFXmeter",
  "category": "RFXcom",
  "group": "Power",
  "description": "1-input RFXmeter",
  "inputs": [
    {
      "name": "count",
      "description": "CT Power 1",
      "processList": [
        {
          "process": "scale",
          "arguments": {
            "type": "ProcessArg::VALUE",
            "value": 0.001
          },
          "process": "kwh_accumulator",
          "arguments": {
            "type": "ProcessArg::FEEDID",
            "value": "power"
          }
        }
      ]
    }
  ],
  "feeds": [
    {
      "name": "power",
      "engine": "Engine::PHPTIMESERIES",
      "unit": "kWh"
    }
  ]
}

Posted in Computing | Tagged , , | Leave a comment

systemd configuration file for RFXcom to MQTT Perl script

First job of the evening (at least, first indoor job of the evening) was to come up with a unit file to start the Perl script that takes power consumption data from my RFXcom unit and publishes it to MQTT. Fortunately that wasn’t too taxing and the following looks to work first time.

[Unit]
Description=rfxcom-to-mqtt

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/rfxcom-to-mqtt.pl

[Install]
WantedBy=multi-user.target

I copied that to /etc/systemd/system/rfxcom.service at which point

sudo systemctl daemon-reload
sudo systemctl start rfxcom.service

was sufficient to get it running. Then

sudo systemctl enable rfxcom.service

should make sure it starts on a reboot.

There are a few things that would be neat to sort out at some point, such as the direct reference in the code to /dev/ttyUSB0. I guess the correct thing to do would probably be to arrange for a symlink to something like /dev/rfxcom to be created in the udev rules when the relevant device is plugged in, and also add

TAG+="systemd"

to the same rule. Then adding

BindsTo=dev-rfxcom.device
After=dev-rfxcom.device

to the unit file in the Unit section might do the trick. Need to research that last bit a little more though, as I’m not certain that’s correct.

Posted in Computing | Tagged , | Leave a comment

Two steps forward, one step back

This evening I made rather better progress with emoncms than I did a couple of days ago. I decided to go all-in and set up supervisord to control the processes running in the Docker container and that seemed to work fine up to the point where the process importing from MQTT died 🙁

Further investigation revealed a bug or two in the import script which I bodged up to the point where the script ran happily and imported no data whatsoever. In fact as far as I could see, it wasn’t even being sent any data despite the fact I could see it with a standalone client.

It wasn’t for some time that the possibility occurred to me that perhaps an authenticated user might not be allowed to subscribe to topics that are available to unauthenticated users in Mosquitto. I added a specific entry in the ACL file for my emoncms user and data started arriving straight away!

There’s still a bit of work to do now however, hence the step back. First I need to find out which of the two near-identical scripts for importing data from MQTT I should actually be using and test/fix it appropriately if it doesn’t work.

I also need to understand how to import my Watt-hour figures properly. I suspect that to be able to get daily usage graphs then I need to push the data through some more feed processing, which means modifying the device templates that I already have.

Finally I need to create systemd configurations to start the Docker image and the process that reads the data off the RFXcom unit. At that point I should hopefully be able to document it all.

Posted in Computing | Tagged , , , | Leave a comment

Swimming, 20th December 2021

I didn’t get to swim on Sunday which was a disappointment. Apparently Everyone Active’s booking system “lost” the session details so they couldn’t be booked. They’d have let me in to swim if I’d turned up though. Shame the receptionist didn’t tell me that when I phoned up to check what was going on 🙁

Anyhow, today I got to swim with my son now he’s back from university for the holidays and for most of the session we had the fast lane to ourselves. I was really struggling for pace today though and didn’t really get on it until halfway through my 50m set. I have absolutely no idea why 🙁 At least it wasn’t too bad once I did get sorted, but I shall hope for better on Wednesday.

Posted in Swimming | Tagged , | Leave a comment

Why do Hermes still exist?

The world being the way it is, we’ve probably had more deliveries than usual this winter. Quite a number of them have, sadly, been misdelivered by Hermes. Some have merely been several days late on a “next day” or “48 hour” delivery. Others have been weeks late, or failed to arrive at all. And this evening I received an email saying that a parcel I was expecting had been delivered, but none of the six people in the house at the time had actually seen it. Obviously I checked the tracking information and found the proof of delivery photo. Here it is:

Now I realise that some delivery drivers may be a bit weak when it comes to speaking the English language and don’t really understand what the word means, but to my mind that doesn’t constitute proof of delivery. More proof of being dumped in a hedge, really. Moreover, it’s no part of my property that I recognise and I do know it fairly well having lived here the better part of eighteen years now.

I assume that many organisations use Hermes because they’re cheap, but I do wonder if their up-front cheapness isn’t offset by having to deal with the persistent fuck-ups.

It really must take some special kind of effort to manage to be reliably this shit.

Posted in Random | Tagged | Leave a comment

One step forward, two steps back

I’ve spent a happy (hah!) evening trying to work out why emoncms wasn’t recording any data from MQTT. The documentation seems to claim it “just works”, but I just couldn’t see anything happening, even when I’d checked that the code is all starting with the correct configuration.

It’s taken quite a few hours to realise that the reason it doesn’t work in the Docker image is that there’s a second process to manage the incoming data feed that isn’t enabled. What’s more, Docker is really only set up with the intention of running one main process per container.

There are a couple of potential ways around this. The first is to run a script that starts both processes as the main command in the image. That might be ok, but it’s probably not good if one of them dies. In fairness, Apache itself seems pretty unlikely to die, but the php code for handling the MQTT data might. So whilst I might do this just to get a bit further with testing, it doesn’t seem like the right way to go long-term.

The other possibility is to use supervisord to start everything off. That means adding it to the image, but I think in the long run it’s the better bet.

It’s late now though, so this is something to look into tomorrow.

Posted in Computing | Tagged , | Leave a comment

Publishing RFXmeter power sensor data to MQTT

More hunting around for helpful modules led me to believe that the easiest way to get data from my RFXmeter sensors into MQTT/mosquitto so it can be read by emoncms was to stick with Perl and use the Device::RFXCOM and Net::MQTT::Simple modules from CPAN. Had there been suitable Python modules available for handling RFXmeter devices I might have gone that way, but at the moment I’m not really of a mind to fix the broken one I found.

It didn’t take much coding effort to come up with this:

!/usr/bin/perl

use warnings;
use strict;
use Device::RFXCOM::RX;
use Net::MQTT::Simple;

my $DEVICE             = '/dev/ttyUSB0';
my $MQTT_HOST          = '<ip-address>';
my $MQTT_PORT          = 1883;
my $MQTT_USER          = 'power';
my $MQTT_PASS          = '<password>';
my $SENSORS_PER_METER  = 3;

# Allow unencrypted connection with credentials
$ENV{MQTT_SIMPLE_ALLOW_INSECURE_LOGIN} = 1;

my $mqtt;

sub
cleanUp
{
  $mqtt->disconnect();
  die "Quitting... $!";
}

my $rx = Device::RFXCOM::RX->new ( device => $DEVICE );

$mqtt = Net::MQTT::Simple->new ( "$MQTT_HOST:$MQTT_PORT" );
$SIG{INT} = \&cleanUp;
$SIG{TERM} = \&cleanUp;

$mqtt->login ( $MQTT_USER, $MQTT_PASS );

while (1) {
  my $data = $rx->read() or next;
  if ( !$data->duplicate ) {
    if ( $data->type eq 'rfxmeter' ) {

      # I'm expecting something like:
      #
      # 'messages' => [
      #   bless( {
      #     'measurement' => 'count',
      #     'device' => 'rfxmeter.02f2',
      #     'value' => 448770
      #   }, 'Device::RFXCOM::Response::Sensor' )
      # ],

      foreach my $msg ( @{$data->messages} ) {
        if ( $msg->{measurement} eq 'count' ) {
          if ( $msg->{device} =~ /rfxmeter.([0-9a-f]{2})[0-9a-f]{2}/ ) {
            # this works for me because my sensors appear to be numbered
            # 00f0, 01f1, 02f2, 03f3 and 04f4, the first three being on the
            # first RFXmeter and the last two being on the second.
            my $sensorNum = hex ( $1 );
            my $meterNum = int ( $sensorNum / $SENSORS_PER_METER ) + 1;
            $sensorNum %= $SENSORS_PER_METER;
            my $sensorName = "rfxmeter$meterNum-$sensorNum";
            $mqtt->publish ( "sensors/$sensorName/count", $msg->{value} );
          } else {
            print "failed match\n";
          }
        }
      }
    }
  }
}

If I connect to mosquitto in another window, using:

mosquitto_sub -h <ip-address> -v -t sensors/#

and run the above script, I get output of the form:

sensors/rfxmeter1-0/count 784628
sensors/rfxmeter1-1/count 373203
sensors/rfxmeter1-2/count 451528
sensors/rfxmeter1-0/count 784648
sensors/rfxmeter1-1/count 373213
sensors/rfxmeter1-2/count 451536
sensors/rfxmeter2-0/count 0
sensors/rfxmeter2-1/count 43623

which looks good to me.

Now I need to go back to emoncms and see if I can confirm that it is picking up the same data, but first I shall rename the sensors as I suggested towards the end of my previous post on this topic.

Posted in Computing | Tagged , , , | Leave a comment

Swimming, 16th December 2021

My shoulders felt very tired whilst I was warming up today so I wasn’t expecting to achieve much, but actually it didn’t turn out too badly in the end. I did fail two of my 50m reps, but the first failure was on rep 12, so one better than yesterday.

The 25m reps were quite a struggle afterwards though. By that time things were catching up with me a bit I think and I had two fails in this set as well. I can be happy with this performance given the situation though. It’s always a little tough coming back after being ill, even if it’s only mildly ill.

Posted in Swimming | Tagged , | Leave a comment

Adding support for new devices to emoncms

To be clear: I’m not certain this is the right way to go about this. It’s mostly based on guesswork and assumption from what I’ve scraped together reading around the discussions about emoncms. For the moment it appears to be working for me and that’s the best I can say. If you try the same and it blows up in your face, sets your hair on fire, eats your firstborn and sells your soul to Santa, I’m afraid there are no refunds. So, here we go…

The way to start once logged into the application is to create devices having selected that section from the menu on the left-hand side of the main page, but that appears to be much more straightforward if there are device templates to work with. After checking over a few of the existing templates, I realised that perhaps the system isn’t going to work as well as I’d like for the RFXcom units that may receive messages from multiple RFXmeters at once, so I decided to treat each sensor connected to an RFXmeter (which may have one, two or three connected) as a separate device. In the end, this is what I came up with:

{
  "name": "single-input RFXmeter",
  "category": "RFXcom",
  "group": "Power",
  "description": "1-input RFXmeter",
  "inputs": [
    {
      "name": "count",
      "description": "CT Power 1",
      "processList": [
        {
          "process": "log_to_feed",
          "arguments": {
            "type": "ProcessArg::FEEDID",
            "value": "power"
          }
        }
      ]
    }
  ],
  "feeds": [
    {
      "name": "power",
      "engine": "Engine::PHPTIMESERIES",
      "unit": "Wh"
    }
  ]
}

“name” (on the first line) is the entry that will appear in the devices menu for this specific template. “category” is the top-level menu item that it will appear under. In this case I’ve used the name of the company producing the device. “group” appears as a sub-menu of the category, so there might be RFXcom units for measuring gas and water use, say, and they could appear in the same category with different group names. Or a different RFXmeter configuration might appear under the same group as my new device.

As I’m treating each meter input as a separate device there’s only one entry in the “inputs” section. There, I believe “name” is the third element of the topic name to use with MQTT to retrieve data for this sensor, so the full value would be sensors/rfxmeter1/count for example. I’ve used the name “count” because that’s what the RFXcom units call the values they send.

The rest of the entry for this input details how to deal with the data:

      "processList": [
        {
          "process": "log_to_feed",
          "arguments": {
            "type": "ProcessArg::FEEDID",
            "value": "power"
          }
        }
      ]

In this case it’s going to be delivered to a “feed” so it can be turned into graphs and suchlike, and the arguments are just indicating that the “log_to_feed” process should be given a feed ID of “power”. That ID is the one that’s used as a name in the “feeds” section of the configuration:

  "feeds": [
    {
      "name": "power",
      "engine": "Engine::PHPTIMESERIES",
      "unit": "Wh"
    }
  ]

The options for “engine” appear to be Engine::PHPFINA for data that is delivered at regular intervals, or Engine::PHPTIMESERIES where it may be delivered randomly. As the data I’m getting can be a bit variable if the radio signals from the RFXmeter unit aren’t clear, I chose the latter. And in the case of the RFXmeter values, they’re watt-hours rather than kilowatt-hours, so I’ve made that clear in the “unit” field.

This file now needs adding into the Docker image. The existing templates appear to be stored in /var/www/emoncms/Modules/device/data. Looking at the existing image I suspect the directory names may be important, too, so at the top level of my copy of the emoncms Docker project I created a new directory devices/RFXcom. I also suspect the template filenames are important, so in that directory I put the above template file, named rfxcom-rfxmeter-1.json. I believe the first component of the filename (before the “-“) must match the directory name, but be in lower case. It’s possible the end directory name must also match the category name in the template file. I don’t know that for certain, but it looks like that might be the case with the other templates, so I kept to the same form.

Then it’s just a case of copying these files into the Docker image when it is created, which I did by adding the second COPY line in this section of the Dockerfile:

COPY docker.settings.ini /var/www/emoncms/settings.ini

COPY devices/* /var/www/emoncms/Modules/device/data

# Create folders & set permissions for feed-engine data folders (mounted as docker volumes in docker-compose)

After recreating the image and logging in again I was then able to go through the usual device creation process using my own template. I actually went through the process of adding my five power sensors as rfxmeter1, rfxmeter2, rfxmeter3, etc., but in retrospect I think I’ll rename them along the lines of “rfxmeter<meter-number><sensor-number>” so it’s clear which sensor input is associated with each RFXmeter unit.

The next stage would appear to be to start pushing data from the sensors into MQTT and see what happens. Life is never simple though, is it?

Presently I am reading data from the RFXcom unit using Perl and a module that can decode the data sent. Unfortunately there doesn’t seem to be a confidence-inspiring MQTT module for Perl. Python might be an alternative as it does have good MQTT integration, but the only RFXcom module I can find doesn’t work with versions of python after 3.6. I have discovered that there’s a set of Node-RED nodes that might possibly do everything for me, but that seems like a massively over-engineered way to read some data from a serial port, decode it and post it to an MQTT topic and I don’t know it would work for certain anyhow. More investigation is required here…

Posted in Computing | Tagged , | Leave a comment