OWON OW18E on Python

[ullix]

I am using owon_multi_cli via subprocess in Python. It works, thx for the code! But some things create headaches.

I currently use this without sudo and have no problems. Can I rely on not needing sudo in the future, or does it depend on other settings? If I needed sudo, the use of owon_multi_cli would become impossible for me due to the need for inclusion in bigger project. Could you explain when and why it could be required?

As I need it in Python, I am wondering whether it may now or in the future become available as a "Pure-Python" program?

I am using subprocess in this way:

scmd = ["OWON/owon_multi_cli", '-a', 'A6:C0:80:E3:85:9D', '-t', 'ow18e', "-1"]
suboutput = subprocess.run(scmd, capture_output=True)
value = float((suboutput.stdout.decode('UTF-8').strip().split(" "))[1])

Code is ok, but it runs for 1 ... 1.5 sec! This is long. Any option for making it shorter?

[sercona]

hi,

I did not write that part of the code; it was already working even though its a bit indirect, the way its designed. pros: it lets an already-stable command do all the heavy lifting. with bluetooth, that's not at all trivial. I agree for purity it would be better to have a pure python solution, and if one can be found that is close to what this app needs, I'm all for trying to all that option. but currently there are no plans to convert to pure python. I'm not against it, I just dont have the time for that one right now. sudo is needed because that cli command that the app forks needs it. it should be possible to install 'setuid' so that any user can invoke that (and then) this app.

you are running this 'connectionless' style, so to speak, in that you are running this whole app just to get one reading, correct? and you want less 'startup tax' to get that single value? if that's the case, I'd suggest running this in the background and having the output go to a file. your other app (that does your real work) would essentially follow or 'tail -F' that log file and be able to get the values immediately. in fact, you could even just do a 'tail' from shell on that live, running log file and get the last line or last few lines.

alternately, to be more elegant, the app could multicast to a bus, like MQTT. let it blast it's heart out to mqtt and you can connect as a client and get values connectionless style, if you want, or connection-oriented if you want a bit less latency. there's no support for mqtt currently, but that's something that could be done without too much effort. if that would get you by, let me know.

[ullix]

Yes, you understood what my needs are.

The tail thing is an idea. I tested, but it does not work. To get the intended output like 1715244520.35 2.152 DC V I start with:
/owon_multi_cli -a A6:C0:80:E3:85:9D -t ow18e >owon.txt

Then I read with: tail -n 1 -F owon.txt
resulting in: 1715244452.58 2.1

which is an incomplete output. After a few moments this partial line is completed, and several hundred lines are also being put out, again ending in an incomplete line:

1715244452.58 2.165 DC V
#hundreds of lines ...
1715244681.33 2.165 DC V
1715244681.24 2.164 DC V
1715244681.42 2.162 DC V
1715244681.63 2.159 DC V
1715244681.84 2.156 DC V
1715244682.23 2.155 DC V
1715244682.44 2.15

Looks like OS caused text buffering before saving to file. Not an option, I'd say.

MQTT could work. My code is already set up to read MQTT. Or your script could blare to WiFi as a WiFi client; my code is also ready for it.

[ullix]

Here is the plot of duration of calls from Python with /owon_multi_cli -a A6:C0:80:E3:85:9D -t ow18e :

"Ambient" is the duration in millisec. The range is 200 ... 1200 ms in a clearly non-random pattern. Any idea why such a strange plot results?

[ullix]

One more question: I noticed that in the response to a subprocess.run(...) the returncode is always 1, while I'd have expected it being 0 (zero).

from the Python Doc: returncode:
Exit status of the child process. Typically, an exit status of 0 indicates that it ran successfully.

response:
args=['gtools/owon_multi_cli', '-a', 'A6:C0:80:E3:85:9D', '-t', 'ow18e', '-1'], returncode=1, stdout=b'1715254147.60 2.154 DC V\n', stderr=b''

Should it not be zero? Is it an indication of something being wrong though all seems fine?

[sercona]

you are probably running into python line buffering. I wrote about it in the README. python aggressively buffers. you can disable it by using -u on the command line (not in the shebang, it does not work there).

but mqtt is really the better way since you wont have any disk buffering to worry about and the response time should be acceptable.

[sercona]

One more question: I noticed that in the response to a subprocess.run(...) the returncode is always 1, while I'd have expected it being 0 (zero).

from the Python Doc: returncode:
Exit status of the child process. Typically, an exit status of 0 indicates that it ran successfully.

response: args=['gtools/owon_multi_cli', '-a', 'A6:C0:80:E3:85:9D', '-t', 'ow18e', '-1'], returncode=1, stdout=b'1715254147.60 2.154 DC V\n', stderr=b''

Should it not be zero? Is it an indication of something being wrong though all seems fine?

I found where exit(1) was called. it should be fixed in the main branch now (thanks).

[sercona]

Here is the plot of duration of calls from Python with /owon_multi_cli -a A6:C0:80:E3:85:9D -t ow18e :

"Ambient" is the duration in millisec. The range is 200 ... 1200 ms in a clearly non-random pattern. Any idea why such a strange plot results?

so that I understand this: you are timing how long it takes for the subprocess call to complete? that graph is about how long each api call takes, is that correct?

again, disk buffering could easily explain latency in syncing to disk.

see if the mqtt idea works for you. I might even look into adding it to the mainline since it would be useful to have as a feature.

understand that the meter is not a fast reading meter and this really would not be ideal as a DAQ for something that needed to ensure fast data and drop-free data. there are better solutions for data logging. (another project I'm working on uses the ut61e from uni-t brand and that has an optical link like many dmm's do and I have some code and 3d printed dongle for the meter that will read its values and not go thru bluetooth. one advantage of that older uni-t meter is that it never turns off on its own, so you can run it for as long as you give the dmm battery power. most meters turn off after a while so its not great for longer term data logging)

[ullix]

so that I understand this: you are timing how long it takes for the subprocess call to complete? that graph is about how long each api call takes, is that correct?

Actually, it is the complete call including the return with a value, like "3.123" (Volt), but the subprocess call itself takes 99% of that time.

My program GeigerLog (https://sourceforge.net/projects/geigerlog/ ) typically uses a cycle time of 1 sec. Ideally, anything called should be well under 1 sec duration. But while the owon_multi_cli manages a call in as fast as 0.2 sec, the high end can take 1.5 sec! This spread is more of a nuisance than the absolute time :-(

again, disk buffering could easily explain latency in syncing to disk.

What syncing? I am not redirecting stdout to anything. Is there some disk writing inside of owon_multi_cli? If so, can this be eliminated?

As said, GeigerLog can already handle MQTT. But if I am not mistaken, this is not (yet) present in your code?

[ullix]

you are probably running into python line buffering. I wrote about it in the README. python aggressively buffers. you can disable it by using -u on the command line (not in the shebang, it does not work there).

Just to note: this buffering issue did arise on the shell, NOT under Python!

And I saw you have a '-l' logging command. Tried this also, but same result with tail as with redirection.

[ullix]

I found some (simple) Python code to read Bluetooth. After enriching it with your C code, I get at least the voltage reading right. Here is the Python code:

#! /usr/bin/env python3
# -*- coding: UTF-8 -*-

"""
test Bluetooth @ OW18E
"""

import time
import asyncio
from bleak import BleakClient

ADDRESS             = "A6:C0:80:E3:85:9D"
CHARACTERISTIC_UUID = "0000fff4-0000-1000-8000-00805f9b34fb"        # Output like: 23 f0 04 00 5b 0f

def notification_handler(sender, data):

    global lasttime

    nowtime = time.time()
    print(f"{nowtime - lasttime:0.3f} sec   Data OW18E: {data.hex(' ')}", end = "  ")
    print("value: ", data[5] <<8 |data[4], end="   ")
    lasttime = nowtime

    # Owon meter logic (from: https://github.com/sercona/Owon-Multimeters/blob/main/owon_multi_cli.c)
    reading =  [0, 0, 0]

    # convert bytes to 'reading' array
    reading[0] = data[1] << 8 | data[0]
    reading[1] = data[3] << 8 | data[2]
    reading[2] = data[5] << 8 | data[4]

    # Extract data items from first field
    function = (reading[0] >> 6) & 0x0f
    scale    = (reading[0] >> 3) & 0x07
    decimal  = reading[0] & 0x07

    # Extract and convert measurement value (sign)
    if (reading[2] < 0x7fff): measurement = reading[2]
    else:                     measurement = -1 * (reading[2] & 0x7fff)
    print(f"measurement: {measurement / 10**decimal:0.3f}")


async def run(address):

    global lasttime

    client = BleakClient(address)
    async with BleakClient(address) as client:
        lasttime = time.time()
        await client.start_notify(CHARACTERISTIC_UUID, notification_handler)
        await asyncio.sleep(60.0)  #show data for a minute
        await client.stop_notify(CHARACTERISTIC_UUID)


if __name__ == "__main__":
    try:
        loop = asyncio.get_event_loop()
        loop.set_debug(False)
        loop.run_until_complete(run(ADDRESS))
    except KeyboardInterrupt:
        loop.stop()

and here output from this code:

0.060 sec   Data OW18E: 23 f0 04 00 c0 08  value:  2240   measurement: 2.240
0.392 sec   Data OW18E: 23 f0 04 00 bf 08  value:  2239   measurement: 2.239
0.508 sec   Data OW18E: 23 f0 04 00 bf 08  value:  2239   measurement: 2.239
0.512 sec   Data OW18E: 23 f0 04 00 c0 08  value:  2240   measurement: 2.240
0.180 sec   Data OW18E: 23 f0 04 00 c1 08  value:  2241   measurement: 2.241
0.208 sec   Data OW18E: 23 f0 04 00 c2 08  value:  2242   measurement: 2.242
0.420 sec   Data OW18E: 23 f0 04 00 c3 08  value:  2243   measurement: 2.243
0.512 sec   Data OW18E: 23 f0 04 00 c3 08  value:  2243   measurement: 2.243
0.300 sec   Data OW18E: 23 f0 04 00 c2 08  value:  2242   measurement: 2.242
0.388 sec   Data OW18E: 23 f0 04 00 c1 08  value:  2241   measurement: 2.241
0.212 sec   Data OW18E: 23 f0 04 00 c0 08  value:  2240   measurement: 2.240
0.508 sec   Data OW18E: 23 f0 04 00 c0 08  value:  2240   measurement: 2.240

Faster, but also not a speed champion, and with quite a variation in the speed from 60 ... 512 ms. Well, ok.

But now I am facing a different problem: my program GeigerLog is using PyQt5, which does not cooperate with asyncio :-((.

[sercona]

I wonder how much the meter's update rate (and bluetooth overhead) is slowing things down.

I think the sample rate of the meter is twice a second, so that would come close to your 512ms value.

remember, this is a low end hobby grade meter. if you need better data aquisition, I do suggest avoiding bluetooth and going direct serial.

a full native python app that does not have to keep forking another command will surely be faster, but it also adds a lot more complexity and failure modes. I have to admit I'm not a huge fan of BLE and I'd never use it for mission critical work.

if you can get your native python app to just keep sending data over mqtt, that level of client/server abstraction will probably work well for you. pend on a socket (regular tcp sockets) or modern websockets. something network based. that would be my choice if I had to use this meter and had to make it as fast as possible.

[sercona]

how about this as an intermediate step? you can get the BLE data via your method and then call the C routine to parse it and expand to human printable format.

it would mean converting this existing c code to be more like an api call or a standalone app that JUST converts the bytes you'd get from BLE into human printable output.

then you can use the existing logic in this program but skip its method of doing BLE.

another idea: have YOUR app send data to mqtt and this app, as a C program, could just recieve the mqtt BLE bytes (from that characteristic) and convert to printable format. a network service that converts the ble bytes to printable.

lots of ways to make this work.

[ullix]

The OW18E specs say it does 3 samples per sec. In the Bluetooth logs I find quite often 5 measurements per seconds, and each one having a different Voltage, i.e. these are new values. Over several thousand measurements I found a range of 19 ... 608 ms between log values. Quite a spread.

I do have an old DVM sending data via RS232, which is a lot more consistent with respect to sampling duration. Though, this is clumsy with its cables and USB-converters. But Bluetooth really is for masochists ;-).

I prefer Python for anything on the desktop, because the same code works on multiple systems, including Linux, Windows, or Mac. On microchips, of course I use C/C++.

You may have noticed that I have already taken from your code to implement the data conversion.

In my search I also found a Python offering for the GATT mechanism: https://pypi.org/project/pygatt/. I tried it out, and it works. However, it also needs sudo, which makes it impossible to implement in my main code.

Finally I settled for the bleak and asyncio code I posted above and have it run as a separate program which forwards the OW18E data to my main program GeigerLog as a WiFiClient (transfer the data in a GET call). For anyone interested it will be in the next (pre-)release here https://sourceforge.net/p/geigerlog/discussion/devel/

lots of ways to make this work.

Indeed! And thanks again for your code and comments, which finally made this work for me!

[ullix]

Yet another Python solution: SimplePyBLE https://pypi.org/project/simplepyble/
And neither a need for async nor for sudo. Here is code, running fast and well, and so far no failure observed:

import simplepyble
import time
import datetime         as dt

peripheral = None
lasttime   = None

def longstime():
    """Return current time as YYYY-MM-DD HH:MM:SS.mmm, (mmm = millisec)"""
    return dt.datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3] # ms resolution


def main():
    global peripheral, lasttime

    adapters = simplepyble.Adapter.get_adapters()
    adapter  = adapters[0] # I have only 1
    print(f"Selected adapter: {adapter.identifier()} [{adapter.address()}]")

    adapter.set_callback_on_scan_start(lambda: print("Scan started."))
    adapter.set_callback_on_scan_stop (lambda: print("Scan complete."))
    adapter.set_callback_on_scan_found(lambda peripheral: print(f"Found {peripheral.identifier()} [{peripheral.address()}]"))

    adapter.scan_for(1000)                      # Scan for N milliseconds
    peripherals = adapter.scan_get_results()

    BDMfound = False
    for peripheral in peripherals:
        if peripheral.identifier() == "BDM":
            BDMfound = True
            break

    if not BDMfound:
        print( "Peripheral BDM not found - exiting")
        return

    print(f"Connecting to: {peripheral.identifier()} [{peripheral.address()}]")
    peripheral.connect()

    services = peripheral.services()
    service_characteristic_pair = []
    for service in services:
        for characteristic in service.characteristics():
            service_characteristic_pair.append((service.uuid(), characteristic.uuid()))

    lasttime = time.time()
    try:
        choice = 7
        service_uuid, characteristic_uuid = service_characteristic_pair[choice]
        # contents = peripheral.notify(service_uuid, characteristic_uuid, lambda data: print(f"{time.time()} Notification: {data}"))
        contents = peripheral.notify(service_uuid, characteristic_uuid, printStuff)
        print("contents: ", contents)
    except Exception as e:
        print(e)

    while True: time.sleep(1)     # stop with CTRL C


def printStuff(data):
    # print(f"{time.time():0.8f} Notification: {data}", end="   ")

    global lasttime

    defname = "printStuff: "

    nowtime   = time.time()
    durtime = 1000 * (nowtime - lasttime)
    print(f"{longstime()} {defname}Dur:{durtime:3.0f} ms  OW18E:'{data.hex(' ')}'", end = "  ")
    lasttime = nowtime

    # Owon meter logic (from: https://github.com/sercona/Owon-Multimeters/blob/main/owon_multi_cli.c)
    # convert bytes to 'reading' array
    reading =  [0, 0, 0]
    reading[0] = data[1] << 8 | data[0]
    reading[1] = data[3] << 8 | data[2]
    reading[2] = data[5] << 8 | data[4]

    # Extract data items from first field
    function = (reading[0] >> 6) & 0x0f
    scale    = (reading[0] >> 3) & 0x07
    decimal  = reading[0]        & 0x07

    # Extract and convert measurement value (sign)
    if (reading[2] < 0x7fff):     measurement = reading[2]
    else:                         measurement = -1 * (reading[2] & 0x7fff)

    print(f"meas: {measurement}  fon: {function}  scl: {scale}  dec: {decimal}", end= "  ")
    voltage = measurement / 10**decimal
    print(f"Volt: {voltage:0.3f}")


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt as e:
        print()

    peripheral.disconnect()
    print("Disconnected")
    print()

giving such output:

2024-05-12 09:52:00.909 printStuff: Dur:388 ms  OW18E:'23 f0 0c 00 97 08'  meas: 2199  fon: 0  scl: 4  dec: 3  Volt: 2.199
2024-05-12 09:52:01.089 printStuff: Dur:180 ms  OW18E:'23 f0 0c 00 95 08'  meas: 2197  fon: 0  scl: 4  dec: 3  Volt: 2.197
2024-05-12 09:52:01.305 printStuff: Dur:216 ms  OW18E:'23 f0 0c 00 94 08'  meas: 2196  fon: 0  scl: 4  dec: 3  Volt: 2.196
2024-05-12 09:52:01.509 printStuff: Dur:204 ms  OW18E:'23 f0 0c 00 92 08'  meas: 2194  fon: 0  scl: 4  dec: 3  Volt: 2.194
2024-05-12 09:52:01.689 printStuff: Dur:180 ms  OW18E:'23 f0 0c 00 90 08'  meas: 2192  fon: 0  scl: 4  dec: 3  Volt: 2.192
2024-05-12 09:52:01.869 printStuff: Dur:180 ms  OW18E:'23 f0 0c 00 8c 08'  meas: 2188  fon: 0  scl: 4  dec: 3  Volt: 2.188
2024-05-12 09:52:02.109 printStuff: Dur:240 ms  OW18E:'23 f0 0c 00 89 08'  meas: 2185  fon: 0  scl: 4  dec: 3  Volt: 2.185
2024-05-12 09:52:02.321 printStuff: Dur:212 ms  OW18E:'23 f0 0c 00 88 08'  meas: 2184  fon: 0  scl: 4  dec: 3  Volt: 2.184
2024-05-12 09:52:02.741 printStuff: Dur:420 ms  OW18E:'23 f0 0c 00 89 08'  meas: 2185  fon: 0  scl: 4  dec: 3  Volt: 2.185
2024-05-12 09:52:02.921 printStuff: Dur:180 ms  OW18E:'23 f0 0c 00 8b 08'  meas: 2187  fon: 0  scl: 4  dec: 3  Volt: 2.187
2024-05-12 09:52:03.101 printStuff: Dur:180 ms  OW18E:'23 f0 0c 00 8a 08'  meas: 2186  fon: 0  scl: 4  dec: 3  Volt: 2.186
2024-05-12 09:52:03.309 printStuff: Dur:208 ms  OW18E:'23 f0 0c 00 89 08'  meas: 2185  fon: 0  scl: 4  dec: 3  Volt: 2.185

Just one problem: once it is run integrated into my big code GeigerLog it lasts for only 1 ... 100 sec before it crashes with a Segmentation fault! No idea what is going on :-((