Today I found something interesting (finally, not just the boring stuff). See post here .
Anyway, after finding that, I decided to put "Impala" to use for the first time for real since I setup the Hadoop/HDFS/Hive/Impala Cluster.
Every hour I import the rtl_power data that's been post processed and placed in LZO compressed files into Hadoop's HDFS using a Hive Loader. (more on that some other time) That puts the data into a Table within Hive. Impala can search these LZO compressed logs VERY quickly! So I decided to put this process to the test today (finally).
Using the following commands within about 6 seconds I created the below graph using GNUPLOT which shows what I think are NOAA weather Satellite Transmissions.
- impala-shell -d prod -r -i node2 -f wx_sat.sql -o /tmp/wx_sat.dat --output_delimiter=, -B
- gnuplot wx_sat
- gimp /var/www/mufmon/wx_sat.2014-02-22.20.png
wx_sat.sql looks like this:
SELECT ldate,freq,dbm
FROM powerlog
WHERE logdate='2014-02-22'
AND loghour >= '17'
AND hz_start = 100000000
and freq between 137100000 and 137800000;
The gnuplot for this looks like this:
The white line within the graph was when I was testing and calibrating the RTL so it was not generating any rtl_power data during that time.
The two Dark spaces near the bottom are when the LNA was turned off for mainenance.
The 'doppler-like' lines I believe are NOAA and/or MET weather sat's.