RTL_POWER data in Hadoop/HDFS using Impala and GNUPLOT

I am running rtl_power from 12 RTL dongles now 100Mhz per device with a resolution of 1.3Khz over 1 minute.

Today I found something interesting (finally, not just the boring stuff).  See post here .

Anyway, after finding that, I decided to put "Impala" to use for the first time for real since I setup the Hadoop/HDFS/Hive/Impala Cluster.

Every hour I import the rtl_power data that's been post processed and placed in LZO compressed files into Hadoop's HDFS using a Hive Loader. (more on that some other time) That puts the data into a Table within Hive.  Impala can search these LZO compressed logs VERY quickly!  So I decided to put this process to the test today (finally).

Using the following commands within about 6 seconds I created the below graph using GNUPLOT which shows what I think are NOAA weather Satellite Transmissions.

• impala-shell -d prod -r -i node2 -f wx_sat.sql -o /tmp/wx_sat.dat --output_delimiter=, -B
• gnuplot wx_sat
• gimp /var/www/mufmon/wx_sat.2014-02-22.20.png

wx_sat.sql looks like this:

SELECT ldate,freq,dbm

FROM powerlog

WHERE logdate='2014-02-22'

AND loghour >= '17'

AND hz_start = 100000000

and freq between 137100000 and 137800000;

The gnuplot for this looks like this:

The white line within the graph was when I was testing and calibrating the RTL so it was not generating any rtl_power data during that time.

The two Dark spaces near the bottom are when the LNA was turned off for mainenance.

The 'doppler-like' lines I believe are NOAA and/or MET weather sat's.