Object detection on Jetson Nano

I’ve been learning about AI and computer vision with my Jetson Nano. I’m hoping to have it use my cameras to improve my home automation. Ultimately, I want to install external security cameras which will detect and scare off the deer when they approach my fruit trees. However, to start with I decided I would automate a ‘very simple’ problem.

Take out the garbage reminder

I have for some time had a reminder to bring out the garbage, to bring it in, and a thank you message once someone brings it in. This is done with a few WebCore pistons:

In order to decide if the garbage is in the garage or not I’ve attached a trackr tile which is detected by my Raspberry Pi 3. Unfortunately, if the battery dies or gets too cold it’s stops working. I could attach a larger battery to the tile, but it needs to be attached to my bin, so I don’t want something too big. So decided it should be trivial to have a camera learn if the garbage bin is present and then update the presence in SmartThings. It took me but a few minutes to train an object classification on https://teachablemachine.withgoogle.com/, so I thought this was doable.

First I mounted a USB camera to the ceiling in the garage and attached to the Raspberry Pi. I then spent a few days learning how to access the camera, and my options to stream from it, etc.. ultimately, I decided to use fswebcam to grab the images.

fswebcam --quiet --resolution 1920 --no-banner --no-timestamp --skip 20 $image

Once I had a collection of images, I installed labelImg on my nano. This is because for this project I didn’t just want to do image classification but object detection. In hindsight, it would have been much simpler to crop my image to the general area where the bins reside and then train an object detector.

After assembling about 20 images I then copied around scripts to create all the supporting files for TensorFlow. I went from text to csv to xml to protocol buffers. In the end, I had something ready to train. I attempted to train on the Nano, but soon came to the realization it was never going to work. My other PCs don’t have a modern GPU for running AI tasks, so my hope was to get it to work the with Nano. I learned about renting servers but that was going to add costs and complications. I then learned about Google Colab, which (for now), gives you free runtimes with a good GPU or TPU. Once running you’ll find out what kit your runtime has. I’ve gotten different hardware on different runs. My last run used the Tesla P100-PCIE-16GB. That’s a $5,000 card which not even NVidia is going to let me try out for free.

It look me a long time to get the pieces together in one notebook to be able to train my model. Certainly not the drag and drop of the Teachable Machine.

One thing which helped a lot was tuning the augmentation items. I know the camera is fixed so I don’t need to have it flip or crop the image. Since the garage has windows the lighting can change a lot depending on the time of day. I didn’t setup TensorBoard, but it quickly goes from 0.5% loss after a few steps. I have a small sample and a fixed camera, which helps.

  data_augmentation_options {
    random_adjust_brightness {
    }
  }
  data_augmentation_options {
    random_adjust_saturation {
    }
  }

Once running in the notebook I then spend another few days getting the model to run on my Jetson Nano. NVidia did not make this easy. Ultimately, I downgraded to TensorFlow 1.14.0 and patched one of the model files. Eventually I got it running, then I just needed to get it to work with SmartThings. Since the bins are really only going to move when the garage doors open, I don’t need to do this detection in real time. I want WebCore to query the garage when it detects the doors open or close. I have it do this by querying a web service on my Raspberry Pi:

On the Raspberry Pi, I want it to snap an image, and send it to the Jetson for analysis. I wrote the world’s dumbest web service, installing it with inetd:

#!/bin/sh

0<&-
image=$(mktemp /var/images/garage.XXXXXXX.jpg)

/bin/echo -en "HTTP/1.0 200 OK\r\n"
fswebcam --quiet --resolution 1920 --no-banner --no-timestamp --skip 20 $image
/bin/echo -en "Content-Type: application/json\r\n"

curl --silent -H "Tranfer-Encoding: chunked" -F "file=@$image" http://egge-nano.local:5000/detect > $image.txt
/bin/echo -en "Content-Length: $(wc -c < ${image}.txt)\r\n"
/bin/echo -en "Server: $(hostname) $0\r\n"
/bin/echo -en "Date: $(TZ=GMT date '+%a, %d %b %Y %T %Z')\r\n"
/bin/echo -en "\r\n"
cat $image.txt
chmod a+r $image

I keep a copy of the image and the response in case I need to retain the model. The image is sent over the jetson, where I have a Flask app running. I wasted a ton of time trying to get Flask to work, basically, if you use debug mode, then OpenCV doesn’t work because of different context loading. I could not seems to get Flask to keep the GPU opened for the life of the request, so on each request I open the GPU and load the model. This is quite inefficient as you may imagine. I also experimented with having the Raspberry Pi stream the video all the time over rtsp and then having ffmpeg save an image when it needs it. The problem seemed to be ffmpeg wasn’t always reliable. If I ran it for a single snapshot, it would not always capture an image. If I ran it continually, after some time it would exit. I have it trained to recognize four objects. If use my tool bucket as a source of truth. If it sees that, then I can assume it’s working, otherwise, I don’t have reliable enough information.

The scripts which I adapted are here: https://github.com/brianegge/garbage_bin

I’d like to use a ESP Cam to detect if a I have a package on my front steps. Maybe this will be my next project before I work on detecting deer.

Boiler Room Pipe Temperatures

I run SmartThings and Konneced for my home automation. I decided I could get some data on my boiler and hot water usage by monitoring the pipe temperatures with some cheap DS18B20 probes off Amazon.

Parts:
DS18B20 Five for $11.99 on Amazon
20′ of Shielded Low Voltage Security Alarm Wire
6′ of Aluminum tape
1 Mini PCB Prototype Board
1 4K7 resistor
A few shrink tubings

I used a Konnected add on board, put and connected my security wire to it. I tied the yellow wire to Pin 6, the black to the adjacent ground and the red to the +5v via a dupont wire. Next I ran the security wire over to my indirect hot water heater, where I connected two DS18B20’s and another cable over to my boiler. I used a prototype board because it was not an easy place to solder and though, I guess I could have done the soldering on the bench and then run the wire, as I did with my second run. I added the 4K7 pull up resistor here. I couldn’t get on of the yellow wires to insert into the prototype board, so I pushed in a header.

On my workbench I soldered three DS18B20 to one security wire and shrink tubed each wire plus a shink tube over all three. Effectively I have a star design.

I placed the probes on the pipe an attached with aluminum tape. I then wrapped some insulation over the taped section.

I configured Konnected to poll every minute instead of every three. The devices appeared SmartThings shortly after I configured pin 6 to be a temperature probe.

My next task was to get the data recorded in my Raspberry Pi. For that I’m using InfluxDB and Grafana, following this guide: http://codersaur.com/2016/04/smartthings-data-visualisation-using-influxdb-and-grafana/

Smart Air Freshener

My wife asked for us to have an air freshener installed in the bathroom. I don’t like the plug in types, even if they don’t burn your house down. At my office we have air fresheners which run on a schedule, or maybe run 24×7, but seem to spray every fifteen minutes. I found a model on Amazon which was similar:

SVAVO Automatic LCD Fragrance Dispenser

This would probably work OK an in office, where you program it 9-5 M-F, but at home the schedule is not so easy. For one, we don’t want it going off when we’re asleep or not home. That’s trivial to set up a home automation to do that, but I could find no air fresheners which would connect to SmartThings.

I decided to order the device and hack the motor to be controlled via SmartThings. Opening the device up, I found it ran on 3.2v via 2 AA batteries and had a simple PCB with two wires for the battery and two for the PCB. The PCB even had pads which I assume one could reprogram the controller. If the controller had a radio, my approach my have been to try to hack it. However, I assumed it didn’t, so I unsoldered the green(-) and yellow(+) wires from the motor.

It’s difficult to have a wifi device connected via batteries, so I decided I’d convert the device to run off of 5V micro-usb. This was easily powered via an ethernet cable and POE adaptor dropped down from my attic.

Wemos D1 Mini inside battery cabinet

Fortunately, the battery compartment had a generous amount of space. I decided to use the Wemos D1 Mini because of its small size and I flashed the Konnected firmware on. Using Konnected allowed for quick integration into SmartThings.

Once I had the software / hardware working, I mounted it on the wall. Because SmartThings has connections to Alexa and Google home, it was easy to get the voice assistants to activate the air freshener as well.

I created a basic piston to run it once an hour when my wife is home and not asleep. I also setup a routing to run it once when she first arrives home.

The Final Product!

Parts List:

I spent $35.97 on the air freshener and sprays, $21.64 on the parts for a total of $57.61. Most of the cost was my POE power supply and adaptor.

Connecting Novostella 20W Smart LED Flood Lights to SmartThings

I purchased of pair of LED flood lights for my home from Amazon. I’ve looked at the Philips Hue lights which look nice but are very expensive ($330). The Novostella were $35 each when I purchased them. The main problem with lights like this is they come with an app, and they can only be controlled from that app or applications which work with it’s cloud account. Changing the firmware should be easy and would allow it to work with any app or home automation system.

20W is very bright!

They appear to be ESP8266 based, so I should be able to flash them OTA using Tuya OTA. I used my Raspberry Pi 3 for the OTA flashing following this guide. The only issue I ran into is I plugged my lamp in too soon as it went out of the flashing light mode. There are no switches on the lamp, so the procedure is to plug in, unplug, plug in, unplug, plug in. Then it will resume blinking and the OTA software will work.

I found it’s quite important to attach the antennas before starting, otherwise, it may work but will be quite slow.

I checked my router for the device in the DHCP and connected to the web server. I setup the template as follows:

{"NAME":"Generic","GPIO":[0,0,0,0,37,41,0,0,38,40,39,0,0],"FLAG":0,"BASE":18}

The web UI lets you adjust the brightness and the white balance, but not the color. I tested the color command and got a nice blue:

Color 1845FF0000

Next, I wanted to connect to SmartThings. I installed this DHT https://github.com/GaryMilne/Tasmota-RGBCCT-DH-for-SmartThings-Classic-with-MQTT

I forked and installed the “Holiday Color Lights” SmartApp to automate changing the color of the lights with the season. It needs some work to be able to handle relative dates, like Fourth Thursday of the month. I modified it to use “white” for default when there isn’t a holiday.

I think the end result looks pretty good. I’ll be ordering two more of these.