WIP - issue/34 - use pymodbus decoding

.github/workflows/ci.yml

@@ -48,7 +48,7 @@ jobs:
         poetry run flake8 mate3/sunspec/models.py --ignore=E501 --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Check black has been run (excluding auto-generated code)
       run: |
-        poetry run black . --check --exclude='setup.py|models.py'
+        poetry run black . --check --exclude='setup.py|fields.py|models.py'
     - name: Test with pytest
       run: |
         poetry run pytest

CONTRIBUTING.md

@@ -1,21 +1,24 @@
 # Contributing
 
+## General design decisions
+
+- Keep it simple for now - both for developers and for users. Don't worry too much about weird edge cases that could arise, unless they're likely to occur (i.e. someone reports it).
+- General Intellisense behaviour is important since we've got so many fields. And having a type annotation on them is even better for auto-completion etc. Since we're using dynamically generated models, it'd be tempting (and probably cleaner) to make liberal use of `__getattr__`, but we don't, 'cos that'd mess with Intellisense.
+- Documentation and tests come when there's time/users. Tests are best for now, as they serve as documentation too.
+
 ## Brief overview of the code ...
 
 The key things to note are:
 
-- All the modbus information about devices/fields/etc. is auto-generated from `./sunspec/doc/OutBack.Power.SunSpec.Map.xlsx`. 
+- Stuff related to SunSpec should go in `./sunspec`.
+- `./sunspec/models.py` includes the field definitions, and is auto-generated from `./sunspec/scripts/code_generator.py`.
 - All the other code for reading/interacting then utilises these definitions.
-- If it seems weird that e.g. we have `Field`s and `FieldValue`s, this is why. The `Field` is the pure things parsed from the specification provided by Outback, which we want to keep clean and separate. The `FieldValue` is then the thing you actually interact with (and references an underlying `Field`). Maybe there's a nicer way of doing it, but for now this works.
-- Typing is handy.
-- General focus is on simple usage for the majority of use cases.
 - There are some basic tests - it'd be nice to have more!
 - Using the caching options in `Mate3Client` or the CLI are best for developing, as it avoids bricking your device.
 
 ## Code contributions
 
-If you wish to edit the mate3 source (contributions are gladly received!), 
-then you can get the project directly from GitHub:
+If you wish to edit the mate3 source (contributions are gladly received!), then you can get the project directly from GitHub:
 
 ```sh
 # Install poetry if you don't have it already (if you're unsure, you don't have it)

README.md

@@ -22,96 +22,24 @@ pip install mate3
 
 After this you should be able to run the `mate3` command. To access your Mate it must be connected to your local network using its ethernet port.
 
-## Background info you probably should know ...
+## Quickstart
 
-Reading this will help you understand this libary and how to interact with your Mate.
-
-### Modbus
-
-Hopefully, you don't need to worry about Modbus at all - this library should abstract that away for you. The key thing to note is that Modbus is a communication protocol, and this library works by interacting with the Mate3 physical devices using synchronous messages. So:
-
-- The information isn't 'live' - it's only the latest state since we last read the values. Generally, you should be calling `read` or `write` before/after any operation you make.
-- Don't over-communicate! If you start doing too many `read`s or `write`s you might brick the Modbus interface of your Mate (requiring a reboot to fix). As a rule of thumb, you probably don't want to be reading more frequently than once per second (and even then, preferably only specific fields, and not the whole lot). Since it's a communication protocol (and it's not actually clear what the latency is inherent in the Mate), there's not much point reading faster that this anyway.
-- Given the above, you might want to use the caching options in the `Mate3Client`, which can allow you to completely avoid interacting with/bricking your Mate while you're developing code etc. It's really tedious having to restart it every time your have a bug in your code.
-- Weird things happen when encoding stuff into Modbus. Hopefully you'll never notice this, but if you see things where your `-1` is appearing as `65535` then yeh, that may be it.
-
-### SunSpec & Outback & Modbus
-
-You can check out the details of how Outback implements Modbus in [./mate3/sunspec/doc](./mate3/sunspec/doc), but the key things to note are:
-
-- SunSpec is a generic Modbus implementation for distributed energy systems include e.g. solar. There's a bunch of existing definitions for what e.g. charge controllers, inverters, etc. should be.
-- Outback use these, but they have their own additional information they include - which they refer to as 'configuration' definitions (generally as that's where the writeable fields live i.e. things you can change). Generally, when you're using this library you might see e.g. `charge_controller.config.absorb_volts`. Here the `charge_controller` is the SunSpec block, and we add on a special `config` field which is actually a pointer to the Outback configuration block. This is to try to abstract away the implementation details so you don't have to worry about their being multiple charge controller things, etc.
-
-### Pseudo-glossary
-
-Words are confusing. For now, take the below as a rough guide:
-  - `Field` - this is a definition of a field e.g. `absorb_volts` is `Uint16` with units of `"Volts"` etc.
-  - `Model` - This is generally referring to a specific Modbus 'block' - which is really just a collection of fields that are generally aligned to a specific device e.g. an inverter model will have an output KWH field, which a charge controller model won't. (Again, it's confusing here as Outback generally have two models per device.) In the case above `charge_controller` represents one (SunSpec) model, and `charge_controller.config` another (Outback) model. 
-  - `Device` - this is meant to represent a physical device and is basically our way of wrapping the Outback config model with the SunSpec one.
-  - `FieldValue` - this is kind of like a `Field` but with data (read from Modbus) included i.e. "the value of the field". It includes some nice things too like auto-scaling variables ('cos floats aren't a thing) and simple `read` or `write` APIs.
-
-## More documentation?
-
-At this stage, it doesn't exist - the best documentation is the code and [the examples](./examples), though this only works well for those who know Python. A few other quick tips:
-
-- Turn intellisense on! There's a bunch of typing in this library, so it'll make your life much easier e.g. for finding all the fields accessible from your charge controller, etc.
-- [./mate3/sunspec/models.py](./mate3/sunspec/models.py) has all of the key definitions for every model, including all the fields (each of which has name/units/description/etc.). Error flags and enums are properly defined there too.
-
-## Using the library
-
-More documentation is needed (see above), but you can get a pretty code idea from [./examples/getting_started.py](./examples/getting_started.py), copied (somewhat) below.
+Here's how you'd update a value and then read the battery voltage every second in Python:
 
 ```python
-# Creating a client allows you to interface with the Mate. It also does a read of all devices connected to it (via the
-# hub) on initialisation:
-with Mate3Client("...") as client:
-        # What's the system name?
-        mate = client.devices.mate3
-        print(mate.system_name)
-        # >>> FieldValue[system_name] | Mode.RW | Implemented | Value: OutBack Power Technologies | Read @ 2021-01-01 17:50:54.373077
-
-        # Get the battery voltage. Note that it's auto-scaled appropriately.
-        fndc = client.devices.fndc
-        print(fndc.battery_voltage)
-        # >>> FieldValue[battery_voltage] | Mode.R | Implemented | Scale factor: -1 | Unscaled value: 506 | Value: 50.6 | ...
-         Read @ 2021-01-01 17:50:54.378941
-
-        # Get the (raw) values for the same device type on different ports.
-        inverters = client.devices.single_phase_radian_inverters
-        for port, inverter in inverters.items():
-            print(f"Output KW for inverter on port {port} is {inverter.output_kw.value}")
-        # >>> Output KW for inverter on port 1 is 0.7
-        # >>> Output KW for inverter on port 2 is 0.0
-
-        # Values aren't 'live' - they're only updated whenever you initialise the client, call client.update_all() or
-        # re-read a particular value. Here's how we re-read the battery voltage. Note the change in the last_read field
-        time.sleep(0.1)
-        fndc.battery_voltage.read()
-        print(fndc.battery_voltage)
-        # >>> FieldValue[battery_voltage] | Mode.R | Implemented | Scale factor: -1 | Unscaled value: 506 | Value: 50.6 | Read @ 2021-01-01 17:50:54.483401
-
-        # Nice. Modbus fields that aren't implemented are easy to identify:
-        print(mate.alarm_email_enable.implemented)
-        # >>> False
-
-        # We can write new values to the device too. Note that we don't need to worry about scaling etc.
-        # WARNING: this will actually write stuff to your mate - see the warning below!
-        mate.system_name.write("New system name")
-        print(mate.system_name)
-        # >>>  FieldValue[system_name] | Mode.RW | Implemented | Value: New system name | Read @ 2021-01-01 17:50:54.483986
-
-        # All the fields and options are well defined so e.g. for enums you can see valid options e.g:
-        print(list(mate.ags_generator_type.field.options))
-        # >>> [<ags_generator_type.AC Gen: 0>, <ags_generator_type.DC Gen: 1>, <ags_generator_type.No Gen: 2>]
-
-        # In this case these are normal python Enums, so you can access them as expected, and assign them:
-        mate.ags_generator_type.write(mate.ags_generator_type.field.options["DC Gen"])
-        # >>> ags_generator_type.DC Gen
+with Mate3Client(host="<your mate3 IP address>") as client:
+    # Rename the system 'cos why not?
+    mate.system_name.write(b"New system name")
+    # Now monitor the battery voltage:
+    voltage = client.fndc.battery_voltage
+    while True:
+        print(f"Battery voltage is {voltage.value} {voltage.units}")
+        if voltage < 48:
+            # Panic stations!
+            # ...
 ```
 
-## Using the command line interface (CLI)
-
-A simple CLI is available, with four main sub-commands:
+You can also use the CLI, which has four main sub-commands:
 
 - `read` - reads all of the values from the Mate3 and prints to stdout in a variety of formats.
 - `write` - writes values to the Mate3. (If you're doing anything serious you should use the python API.)
@@ -120,6 +48,16 @@ A simple CLI is available, with four main sub-commands:
 
 For each you can access the help (i.e. `mate3 <cmd> -h`) for more information.
 
+## More documentation?
+
+At this stage, it's not really complete, but there are a few avenues:
+
+- [./doc/general.md](./doc/general.md) has a dicussion about modbus/SunSpec/etc. and some other potentially useful things.
+- The best documentation is the code and [the examples](./examples), especially [./examples/getting_started.py](./examples/getting_started.py). However this only works well for those who know Python.
+- Turn intellisense on! There's a bunch of typing in this library, so it'll make your life much easier e.g. for finding all the fields accessible from your charge controller, etc.
+- [./mate3/sunspec/models.py](./mate3/sunspec/models.py) has all of the key definitions for every model, including all the fields (each of which has name/units/description/etc.). Error flags and enums are properly defined there too.
+- [./mate3/sunspec/doc](./mate3/sunspec/doc) has more detailed vendor information about modbus/SunSpec/OutBack details, though this is unlikely to be useful to you unless you're developing the library.
+
 ## Warnings
 
 First, the big one:
@@ -130,34 +68,7 @@ In addition, there are other edges cases that may cause problems, mostly related
 
 ## Troubleshooting
 
-Some ideas (which can be helpful for issues)
-
-### Set log-level to DEBUG
-
-See `mate3 -h` for the CLI, otherwise the following (or similar) for python code:
-
-```python
-from loguru import logger
-logger.remove()
-logger.add(sys.stderr, level="DEBUG")
-```
-
-### List the devices
-
-```sh
-$ mate3 devices --host ...
-name                                               address    port
-----                                               -------    ----
-Mate3                                              40069      None
-ChargeController                                   40986      4
-ChargeControllerConfiguration                      41014      4
-...
-```
-Are they all there?
-
-### Create a dump of the raw modbus values
-
-See `mate3 dump -h`. You can send the resulting JSON file to someone to help debug. (Just note that it includes all the data about the Mate, e.g. any passwords etc.)
+See [./doc/troubleshooting.md](./doc/troubleshooting.md).
 
 ## Writing data to Postgres
 

doc/general.md

@@ -0,0 +1,27 @@
+# Generally useful information
+
+Reading this will help you understand this libary and how to interact with your Mate.
+
+### Modbus
+
+Hopefully, you don't need to worry about Modbus at all - this library should abstract that away for you. The key thing to note is that Modbus is a communication protocol, and this library works by interacting with the Mate3 physical devices using synchronous messages. So:
+
+- The information isn't 'live' - it's only the latest state since we last read the values. When you create a client, it reads everything from all of your devices once, but then you should be calling `read` or `write` on a field as required.
+- Don't over-communicate! If you start doing too many `read`s or `write`s you might brick the Modbus interface of your Mate (requiring a reboot to fix). As a rule of thumb, you probably don't want to be doing a full read more frequently than once per second (and even then, preferably only specific fields, and not the whole lot). Since it's a communication protocol (and it's not actually clear what the latency is inherent in the Mate), there's not much point reading faster that this anyway.
+- Given the above, you might want to use the caching options in the `Mate3Client`, which can allow you to completely avoid interacting with/bricking your Mate while you're developing code etc. It's really tedious having to restart it every time your have a bug in your code.
+- Weird things happen when encoding stuff into Modbus. Hopefully you'll never notice this, but if you see things where your `-1` is appearing as `65535` then yeh, that may be it.
+
+### SunSpec & Outback & Modbus
+
+You can check out the details of how Outback implements Modbus in [./mate3/sunspec/doc](./mate3/sunspec/doc), but the key things to note are:
+
+- SunSpec is a generic Modbus implementation for distributed energy systems include e.g. solar. There's a bunch of existing definitions for what e.g. charge controllers, inverters, etc. should be.
+- Outback use these, but they also have their own additional information they include - which they refer to as 'configuration' definitions (generally as that's where the writeable fields live i.e. things you can change). Generally, when you're using this library you might see e.g. `charge_controller.config.absorb_volts`. Here the `charge_controller` is the SunSpec ChargeController block, and we add on a special `config` field which is actually a pointer to the Outback configuration block. This is to try to abstract away the implementation details so you don't have to worry about their being multiple charge controller things, etc.
+
+### Pseudo-glossary
+
+For now, take the below as a rough glossary:
+
+  - `Field` - this is a definition of a field e.g. `absorb_volts` is `Uint16` with units of `"Volts"` etc. It includes nice APIs for doing things like `read` and `write` and getting additional info about a field's value.
+  - `Model` - This is generally referring to a specific Modbus 'block' - which is really just a collection of fields that are generally aligned to a specific device e.g. an inverter model will have an output KWH field, which a charge controller model won't.
+  - `Device` - this is meant to represent a physical device and is basically our way of adding the 'config' model (e.g. `ChargeControllerConfigurationModel`) with the 'main' model (e.g. `ChargeControllerModel`) via a `.config` attribute.

doc/troubleshooting.md

@@ -0,0 +1,30 @@
+# Troubleshooting
+
+Some ideas (which can be helpful for issues)
+
+## Set log-level to DEBUG
+
+See `mate3 -h` for the CLI, otherwise the following (or similar) for python code:
+
+```python
+from loguru import logger
+logger.remove()
+logger.add(sys.stderr, level="DEBUG")
+```
+
+## List the devices
+
+```sh
+$ mate3 devices --host ...
+name                                               address    port
+----                                               -------    ----
+Mate3                                              40069      None
+ChargeController                                   40986      4
+ChargeControllerConfiguration                      41014      4
+...
+```
+Are they all there?
+
+## Create a dump of the raw modbus values
+
+See `mate3 dump -h`. You can send the resulting JSON file to someone to help debug. (Just note that it includes all the data about the Mate, e.g. any passwords etc.)