Python tips – Simple sensor data handling

Say you have a system which collects data from a group of sensors. Let’s have three sensors giving integer values for the sake of argument, but this doesn’t affect what I’m about to show you. The easiest way to store this data (assuming you want some history) is as a list of tuples, so if the data you get from the sensors is (1, 5, 3) at t=0, (2, 5, 3) at t=1, (2, 4, 4) at t=2, and (2, 5, 2) at t=3 then you would be storing

[(1, 5, 3), (2, 5, 3), (2, 4, 4), (2, 5, 2)]

As a side-note, this lets you do a window fairly easily:

values = get_sample()
data.append(values)
if len(data) > WINDOW_SIZE:
    data.pop(0)

Now, say you want to process the data from each sensor separately, that is to end up with

[(1, 2, 2, 2), (5, 5, 4, 5), (3, 3, 4, 2)]

One way to do this would be to use:

result = []
for sensor in data[0]:
    result.append([])
    for sample in data:
        for sensor_id in range(len(sample)):
            value = sample[sensor_id]
            result[sensor_id].append(value)

That, however, is not very neat and nor does it tell you at a glance what it’s aiming to achieve. There is, however, another way to reach the same goal, and it uses only a single function call:

result = zip(*data)

Simple, no? Possibly some explanation is required here though. The idea behind zip is that it provides a way to iterate over two (or more) sequences, obtaining the first item from each, the second item from each, and so on. This means that

a = [1, 2, 3, 4]
b = [6, 7, 8, 9]
for x in zip(a, b):
    print x

should print out

(1, 6)
(2, 7)
(3, 8)
(4, 9)

So how does my sensor data magic work? Prepending an asterisk (*) to a list used as a function argument has the effect of passing each item in the list as a separate argument, so

def x(v1, v2, v3):
    return v1 + v2 + v3
a = [1, 2, 3]
print x(*a)

will print out 6 as the result. Each element of a was passed as a separate argument to x. I’m sure you can see where this is going now. zip will accept any number of sequences as arguments, so zip(*data) will pass each sample to zip separately, and then zip will happily do its job giving you the first item from each, the second item from each, and so on. In this case that gives you all the data from sensor 1, followed by all the data from sensor 2, and then all the data from sensor 3.

In part 2, I cover easily reading logged data in from a file.