Raspberry Pi Home Datalogger

[Static]

I'm working on a Raspberry Pi Datalogging System (Shameless plug here: http://medicforlife.blogspot.com/).
The goal is to create a very flexible device that is capable of logging and easily reporting data. I'm starting out with a simple BMP085 sensor to record temperature and pressure. Eventually I would like to be able to include additional temperature and pressure sensors, motion sensors, smoke/CO/gas sensors, even light and sound sensors. The goal isn't really to create the sensor device. Instead, I want to create a device that detect local events, log them, report them via a webpage, and possibly react to them.

I know would like to incorporate analog and digital sensors. I've already incorporated the I2C portion (LadyAda really did most of that work), and I'm looking for more sensors to include using I2C, as I think that has tremendous potential.

I'm using Python. I fell in love with it when I was in college (Dan Fleck, you were a great teacher). Back then, I was learning 2.7. Now I'm trying to update my skills by using 3.2. I may be having some trouble with that (more on that later).

First, I've written a couple of code snippets to take advantage of LadyAda's Adafruit_I2C and Adafruit_I2c code. I wrote some code that polls the sensor after a preset period (I'm using once per minute), and logs the result (temperature and pressure. I'm not interested in the altitude) using the Python Logging module. This code also updates a primitive webpage that is hosted on the Pi, showing the temperature and pressure (I'm hosting the page using Apache). Today a wrote some code that jumps through those log files and pulls the temperature data. It takes the temperature data and generates a list. My goal with this is to allow the user to access the webpage, click on the current data conditions and be presented with a graph of the last hour. Clicking on that graph would generate a graph of the last 6 hours. Clicking on that graph would generate a graph of the last 12 hours. Then 24 hours, then a week, a month and out to a year.

I was reading a book, Beginning Python Visualization (Shai Vaingast, Apress Publishing), that made it look simple to generate chart files to view the data. It really looked like I would need less than half a dozen lines of code to generate the chart. I actually laughed out loud when I read the chapter because it looked so easy.

I should have known better.

I'm trying to get matplotlib to work with my Python3 installation, and I'm not having any luck. I can import the module using Python 2.7, but Python 3 refuses to see the module. I really am an amateur with these things (This is the most I've used Linux in my 30 years with computers), and I know I'm missing something simple. If anyone has any ideas, I'm more than happy to listen. Even more, I'm more than happy to try out ideas.

[Static]

OK, so I got this one solved.

When you are trying to install modules for Python3, make sure you use the Python3 installer:

python3 setup.py install

I saw tons of tutorials about using matplotlib. I saw a fair number of people saying that they could get matplotlib working with Python3. I even saw a couple talking about using it on the Raspberry Pi. I never saw anyone put everything together. This is going to be important, as the Raspberry Pi lowers the bar for people to get involved with microprocessors, programming, hacking.

I woke up this morning after letting Python build matplotlib (it takes a REALLY long time), did some code tweaks, and generated my first chart.

Now, it's time to go to work.

[Static]

Smoked through a whole bunch of issues yesterday.
I had a basic math/logic issue with one of my formulas to determine how much of the datafile to parse into a graph. It was really frustrating because the program worked for my default values, but wouldn't accept anything else. Essentially I could plot 24 hours of data. If I tried to plot 18 or 36, I either got a string index error or an empty graph.

My next issue was trying to get all of the software running through SSH. I kept getting a Hostname and Display error. This was confusing because I had gone through the program and disabled all of the flow print points (code that printed something to the display to let me know the progress of the program). I had also set the program up to save the generated graph in the Apache data folder, so that I could pull the graph up from a browser.
It looks like Matplotlib works by invisibly generating a graph and then saving the graph. The system needs to display something, somewhere. By adding a single line of code at the start of the program, BEFORE importing matplotlib, this issue is worked around.
The system works. I'm running into recursion lag by repeatedly re-iterating over the datalog as it gets longer. I need to figure out a way to dynamically generate the graphs on demand so that the processor is not being taxed every minute just to generate a simple graph. I think I need to grab an HTML tutorial or something. It's weird saying that. I built my first webpage using a text editor and HTML1(?) many, many years ago. The last couple of webpages that I made used Dreamweaver, which I think may be overkill for this project.

A couple of lessons learned or reinforced:
Google is SO your friend. You get an error code, Google it and the product you are using. Google the error code and the circumstances. Google it! My tech support skills are very limited when I don't have access to Google. Learn to use Google with everything that you do. It's a great tool when you get frustrated or reach a dead-end.

Python includes comments from the code line at issue. No one ever told me this. I'm using a series of nested loops to iterate over the text file, so I have at least 3 or 4 lines that are identical. Commenting the code let me figure out where the problem was. Simple solution, but one I hadn't heard of before.

Oh yeah, document your code (Yeah, I need to do this more).

More things to do:
I learned a couple of "leaner" methods for parsing text documents. I'm going to try to incorporate them into generating the Pressure Charts.
I also looked at the CSV reader/writers. I'm going to look at them and see how much of a memory/performance hit I take switching to them.
Google has announced that you can host websites using Google Drive. I may look at that, and see if I can get an external web presence.
I need to re-write the timed execution system. Right now it simply runs a "For Loop" after sleeping every couple of seconds. I would rather have it check the system time and see if the predetermined time has elapsed. This will allow me to regularly schedule certain aspects of the program (like moving data from the "short term log" to the "long term log", and checking available memory).

I need to mess around with the other I2C components. The board that I'm using now has a 9 or 10 degree of freedom (I don't know how it got 10) sensor on it. I'd like to see if it is sensitive enough to detect seismic events. If it is, I need to generate another polling scheme. I don't care that there is a constant 9.8m/s^2 pull in one direction. I am VERY interested if that value changes. This is really interesting because it is similar to how different body senses work (phasic versus tonic sensors). It would be interesting to make two basic modules, phasic and tonic. Each one takes the input from a sensor, but how it handles the input is different. To put it in perspective: Right now I have a thermal sensor reporting the temperature every minute. I could take that same sensor and plug it into a different software module and have it start screaming if the temperature ever exceeds 50 degrees Celsius. This takes the sensor outside of the realm of human comfort, but before a fire has started burning the circuit board (or have it respond to a combination of measured temperature and rate of change, etc).

Also, I'm waiting on my new order from Adafruit. I grabbed one of the microSD card adapters, an I2C ADC, and the Pi Cobbler T plate. I'll let you know how they look.

[Static]

Got the I2C ADC, Pi T Cobbler Board, and mini-SD adapter.

I haven't played with the I2C ADC yet.
The mini-SD adapter looks pretty good. I need to figure out how to transfer all of my files from the 8GB card to the new 16GB card. I'm wondering if I can use my Nikon as a drive for the purposes of cloning the SD card. I'll try that later.

The T Cobbler board really makes the whole operation easier. It is very easy to see which pins you are using.
Pay VERY special attention to orientation of the ribbon connectors. It's not a big deal if you screw it up, but if you put it together using the default method, you're going to have the ribbon cable crossing one of the boards (either the Pi or the Cobbler). I need to post up on the product page.

I did some re-writes yesterday. The timed execution system now runs according to the system time, not according to arbitrary sleeps. At the top of every minute, the program polls the sensor, logs it, and updates the chart.
I also rewrote the chart generator. It now goes backwards through the logfile to generate a chart instead of iterating through the chart to get to the end. I also set the chart generator up to create a chart in a one hour section, by default. However, the user can specify a longer or shorter period if it is desired.

Found out I've got pneumonia yesterday. That'll give me some downtime to work on this, but I keep falling asleep while reading datasheets and tutorials. Grrrr.

I'm going to also look at getting the pressure charts done. That should be almost the same text file slicing. With the revamped chart generator that should be a snap.

[Static]

I think my I2C ADC board may be a bad one. I need to check my solder joints today, but I can't detect the board with the Raspberry Pi I2C detection code.
An interesting issue came up as I was trying to troubleshoot the I2C bus.
First, I'm a real beginner to all of this. I've been playing around for awhile, but I avoided I2C because I really didn't want to get too in depth with the datasheets.
I've never hooked up a real I2C databus. It's always been one master, one slave. I needed to add pull-up resisters from the voltage line to the SDA and SCL lines. It's simple, when you get down to it, but it seemed to simple. Sure enough, when I hooked up the two I2C boards, I couldn't see the new board. I checked the connections. Powered down and powered up, the whole nine yards. Nothing.
I tried the I2C ADC I'd gotten from Adafruit. Nothing. I could still see that weird Chinese "locator board" that included the BMP085 (It shows up with numerous I2C addresses). I had the Adafruit BMP085 board, but that would show up on address 77 as well. That's no good. I cast about my pile of parts and found a Sparkfun TMP102 thermal sensor. On a whim, I tried it out. BAM, it popped right up, along with the other addresses from the "locator board".

So now I really think I'm out to lunch. My TMP102 is an abused looking board. It's one of the first breakout boards that I soldered. I've recovered it from all sorts of places (I found out that one of the cats likes to grab it. Don't ask me.... The cat is the devil). I honestly wasn't expecting it to work.

So I tore the Raspberry Pi down. Rebuilt everything from start. I then added I2C devices one at a time. It turns out that one of the other devices that I was trying to see has the same I2C address as one of the devices on the "locator board". The ADC board just isn't coming up on the I2C bus (I'm going to troubleshoot that later. I need to pull some tools out to examine it).

I'll post up some more on that when I find something out.

[Static]

On a different note, I played around with the RPIO Python module this morning. I wanted to experiment a little, and see how the software reported the states of GPIO pins relative to what the Pi Cobbler labeled them as

Code: Select all

#!/usr/repos/pilog/
import RPi.GPIO as io
io.setmode(io.BOARD)

'''
Cobbler Designation          RPIO Pin
#22                          15
#27 V2  #21 V1               13
#17                          11
#4                           7
#25                          22
#24                          18
#23                          16
#18                          12



'''


for i in range (26):

    try:
        io.setup(i,io.IN,pull_up_down=io.PUD_UP)

    except:
        continue

    try:
        print(io.input(i),i)

    except:
        print('invlaid pin')
        print(i)
print('Complete')

~


I only tested the pins marked as general use on the Cobbler. I can't afford to blow out my board yet (As soon as the holidays are done, I'm putting in for two more Raspberry Pi's).
I commented my findings in the software, but here's a quick list as well:
Cobbler RPIO
#4 7
#17 11
#18 12
#21 V1 13
#22 15
#23 16
#24 18
#25 22
#27 V2 13

I think my next task is going to be an IR motion detector. I'll be able to use one of the GPIO pins for that, and poll it every couple of seconds. I need to figure out how to keep track of, and log, when the motion is detected.

I was thinking about driving an LED with the GPIO pin, but then I remembered that I have a BlinkM LED module that runs over I2C. I think it will be more fun (and productive) to learn how to drive that over the I2C bus.

[Static]

I cloned my 8GB drive over to a microSD 16GB drive. The model I'm using is the PNY Class 10 16GB. So far, no problems at all.
I used EaseUS Partition Master Home Edition after checking out a couple of reviews. The free home user edition worked for me, but I had some hurdles getting the drive to clone over. If anyone reads this and has issues with it, let me know. I can probably walk you through it. The process was slightly counter-intuitive, but that could also be user error.

I posted about some strange start-up behavior. I'm currently running the Raspberry Pi on my office TV. It sports both an old style video input (the yellow plug) and HDMI ports. I've got the Pi hooked up to both.
When I power down the Raspberry Pi board and re-power it, the old style AV video port is the one that is active, only. The HDMI port continually gives me the display "No Signal".
However, when I command line reboot the system, the HDMI port comes online. No data appears to be sent over the AV port (again, "No Signal").

I have no idea how to address this. I have almost no hangs with the Raspberry Pi board. Well, no permanent hangs. I've slowed the board down considerably with some of my antics, but it's just a matter of killing the offending process (I'm usually logged in to a couple of PuTTY windows from my desktop).

[Static]

I got a couple of issues resolved.

First, I set the system up so that I can see the hosted webpage from outside of my network. Next, I need to have one of my real IT-guru friends attack it and see if he can break in. I'm....not looking forward to that.

Second, I fixed a charting issue. Since I set up the charting script, it's performed bizarrely. I created a function that would build a chart from the data (default one day deep), or take optional arguments to build a chart to a custom size. I then set the logging program up to automatically update a webpage with the charts.
The issue was that the chart function would chart all of the data, no matter what parameters I passed to the function. I was pretty sure that this was a function of readline/readlines and the index pointers associated with that. I was REALLY stumped. I'm talking hours across several days I looked at this, and tried attacking it from a new direction with different readline/readlines functions. I was trying to write loops that would iterate over the sections of the log file that I didn't care about. I just couldn't get it to work correctly.
Finally, I just looked at it and realized that I could structure my for...loop to slice the data file in exactly the right spots. It was an answer that was overly-simplistic, and I glossed right over it. I put the new solution in, commented out four lines of code, and bam, done.

Finally, tonight I wrote a function that examines the GPIO state, and returns the state as a dictionary (RPIO pin number and the state of the pin). I played with the logging functions a bit so that the system automatically logs just one pin (I set up a Parallax IR motion sensor on the pin). I got the programs recording the state of the pin to the log file. My next step will be to graph this piece of data from the log file to a chart and the website.

[Static]

Sparkfun order came in (Sorry Lady Ada. I needed some other odds and ends).

I grabbed a couple of RJ-45 jacks and breakout boards. I was wondering if I could use CAT-5E cable to connect sensors to my Raspberry Pi. I picked CAT5 because I have tons of it, the jacks and breakout boards are cheap, and the form factor is incredibly easy to work with. One jack gets me 8 lines to play with. I'm going to give four over to the I2C bus, and I'll figure out what to do with the other four later.

How far can you stretch an I2C bus? I'll let you know in a bit. Time to cut some cables.

[Static]

OK, so the answer to that is "More than two meters"*

Right now I have two devices on the I2C bus. The BMP085, and a TMP102. I'm trying to build a Python module to use the TMP102. Again, I'm not a computer or electronics engineer, so this is going to take some doing. I'm trying to deconstruct Lady Ada's Python module for the BMP085. I'm getting data from the TMP102, but it has nothing to do with a real temperature (I'm getting -45.3 when the sensor is sitting right next to me. I'm wearing shorts).

I'm comparing the datasheets, here's the product page for the Sparkfun TMP102 breakout board:
https://www.sparkfun.com/products/9418
There's a link off of that to the datasheet and a bildr blog. It's Arduino code (which I like, but I feel I can do more with a Raspberry Pi and Python), and dirt simple at that. I'm just having some trouble working through it today (I'm being distracted by coronavirus... long story).

Damn... I have enough trouble getting my English and Spanish straight and I use that everyday. When I get something working I'll post up. It strikes me that it would be pretty useful to read one temp, but even more useful to read two (or more).

[Static]

This project thread is very much a parallel of the actual project.
It goes something like this:

I build and tinker.
I encounter a problem.
I mumble to myself in a semi-coherent fashion.
I find a solution
Wash, Rinse, Repeat as Necessary

I tried Googling solutions to the TMP102 problem last night. Boy did that muddy the waters.
This method works:
http://donalmorrissey.blogspot.co.uk/2012/09/raspberry-pi-i2c-tutorial.html
but I can't figure out how to get that little snippet of code to work in a Python program. I think I understand the LSB and MSB portion of the library. This will help me write my own Python function.
This method doesn't work:
http://www.raspberrypi.org/phpBB3/viewt ... 44&t=11319
or this one:
http://www.bootc.net/archives/2012/05/1 ... pberry-pi/
This last one gives me an invalid argument error when I try using the "echo" based command.

[Static]

OK, I'm going with this code, based off of Adafruit's I2C code. It feels like something isn't right, but I'm honestly guessing here.

Code: Select all

#!/usr/bin/python

import time
from Adafruit_I2C import Adafruit_I2C

class TMP102 :
  i2c = None


  # Constructor
  def __init__(self, address=0x48, mode=1, debug=False):
    self.i2c = Adafruit_I2C(address)

    self.address = address
    self.debug = debug
    # Make sure the specified mode is in the appropriate range
    if ((mode < 0) | (mode > 3)):
      if (self.debug):
        print "Invalid Mode: Using STANDARD by default"
      self.mode = self.__BMP085_STANDARD
    else:
      self.mode = mode



  def readRawTemp(self):
    "Reads the raw (uncompensated) temperature from the sensor"
    raw = self.i2c.readU16(0x48)     #The TMP102 returns 12-bits, I think
    #raw = self.i2c.reverseByteOrder(raw)  #I need to get the bit order reveresed
    if (self.debug):
      print "DBG: Raw Temp: 0x%04X (%d)" % (raw & 0xFFFF, raw)
    return raw


  def readTemperature(self):
    "Gets the compensated temperature in degrees celcius"

    RawBytes = self.readRawTemp()  #get the temp from readRawTemp (above)
    RawBytes = (RawBytes >> 4)


    temp = float(RawBytes) *0.0625  #this is the conversion value from the data sheet.
    if (self.debug):
      print "DBG: Calibrated temperature = %f C" % temp
   
   
    return temp


I left the debug code in there and tried to comment in what I was thinking. I really was trying to think my way through this.
Right now the sensor is working in tandem with the BMP085. The BMP085 is sitting on my desk. The TMP102 is connected through a ~3meter CAT-5 cable out the window. It's an eastern exposure, and it's sitting in a plastic tupperware container. I got a huge spike this afternoon (around 40c while the ambient was probably around 15c), but that's probably a greenhouse effect.

I'm testing some new code across the board right now. I'm trying to get the motion sensor to log when there is activity. This will really be interesting because I'll be accessing the GPIO pins and the I2C bus at the same time. It looks like it is working, but I'm going to let it run for the next couple of days before I really call it.

I'm working on commenting my code. Then I'll work on getting it posted up here. It's broken down into functions, so it will take a couple of posts to do it.

[adafruit_support_rick]

You can save yourself a step like this. I don't know what python does with right shifts on signed values (i.e., the raw reading), but this ought to avoid any problems. You might also try a double instead of a float…

Code: Select all

    RawBytes = self.readRawTemp()  #get the temp from readRawTemp (above)
    temp = double(RawBytes) *(double(0.0625) / 16.0)  #adjust the conversion value from the data sheet to account for the left-shifted raw data


[tldr]

I don't know what python does with right shifts on signed values (i.e., the raw reading),

apparently it's an arithmetic shift...

Code: Select all

Python 2.7.2 (default, Aug 19 2011, 20:41:43) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> -4>>2
-1
>>> exit()


... but it was an unsigned 12 bit quantity to start with.

that said, there's no reson to fear integer multiply and divide like on an arduino, since they'll happen in hardware on the pi.

but, of course, you're right. might as well do it all at once, anyway. and the interpreter should take care of 0.0625 / 16.0 on the first pass through the code and it will never have to happen, again.

[Static]

I think that's it!

I'm going to do a test run for a couple of hours while I get some sleep between shifts.
I'll post up as soon as I have some time off.
Thank you folks, very much.