BigData Analytics: Color Vision in a Cat Whisker World

Seeing with New Digital Eyes

"He who fights with monsters should look [into] ... himself ... if you gaze long into [the] abyss, the abyss also gazes into you." -- Friedrich Nietzsche (1844-1900), from Beyond Good and Evil -- Aphorism 146

Natural Analog Question: For jungle predators, what is the advantage of seeing in color -- when all your prey only see in black & white -- or maybe only have cat whiskers and no eyes?  

Now, thinking in metaphorical terms: What are the computational insight advantages of BigData Analytics -- and social media data curation & Data Mining -- over everyone else? Are you and your electronic Doppelganger food for BigData predators?  How can you detect stalkers?  Are there taste testing teeth marks on your electronic doppleganger?

Ross Anderson of Cambridge University offered up a form of these questions in his 2017 YouTube interview -- entitled "The Threat." Mr Anderson illustrated 30+ years of trends for potentially predatory BigData aggregation and exploitation -- and Mr Anderson's odd journey thru "technical gettos" -- political gaming of technology -- and vetting the social/ psychological/ economic landscape of "digital life". 

One glaring exploitable aspect in a world gone Gaga with "free" social media? Economic & Political manipulation of "digitally curated" social media relationships.  Too scary to believe? Hear Ross Anderson's perspectives:

2017: "The Treat" -- 30+ year trends in

in computing, privacy, social adaptions (or

not) to BigData Surveillance Capitalism

Deep Nerd Hints: You can download audio only of this interview -- and the audio contains 98% of the key info -- via the command line tool YouTube-DL.  See Nerd Bonus addition at end of this posting.

In 2013 -- long before the Facebook and Cambridge Analytica controversy arose, Maxine Waters hinted at the ice berg under the digital waters -- for naive sailors of the digital seas:

2013 Maxine Waters perspectives

on Database Nation & politics.

Uncharted Ocean? No. Surprise History and 17th Century BigData: Just before the 1789 French revolution, the King's secret police were "monitoring" the "news" of Paris.  How?  In a time when most folks were illiterate?  Street vendors would "sing the news" -- to customers and each other.  News traveled by song -- and by word-of-mouth -- and was a mixture of true events and "fake news" (salacious gossip).

The King's secret police would write up a Zeitgeist summary of this street "news" -- apparently for "actionable" intelligence.  Effective?  17th century BigData did not save the French King from the revolution. And illustrates how BigData machinations of economics & politics is not new. One amazing character and secret police monitor was Jean-Charles-Pierre Lenoir -- escaped the worst of the French revolution -- and left valuable historical insights into the approach and unfolding of the French Revolution -- from "loser side".

Not paranoid? Yet?  Check for teeth marks on your electronic doppleganger.  Some are as subtle as search engine stovepiping of your query results. Many are changes in online Ads boiling up in web page visits.  Tone and flavor of bulk mail is another indicator. And more regular & routine ID theft challenges. And more ID / authentication friction at post office, banks, credit-debit card use, good & bad police interaction.  Surveillance capitalism data trafficking -- coupled w/ rouge BigGov actors -- begets more ID theft, which begets more surveillance friction, which beget more ID theft, etc etc.

Upshot?  Action?  Lacking deep resources to thwart BigData with your own BigOther magic?  If you must be on the internet, then first know this: We cannot stop the data gathering -- only pollute it.  Or at least blow a smoke screen.  

Other Options?

(1) Grow some color vision -- learn to detect teeth marks on your electronic Doppleganger.  (2) Learn how to use some digital camouflage -- and -- metaphorically speaking -- learn to use the old Star Trek technique of "rotating your shield frequency" -- to throw off BigData alien tractor beams molesting your electronic doppleganger: Create "spare" online profiles.  (3) Use app browsers that thwart BigData tracking -- like Firefox Klar -- or on WinXX machines routinely flush Google Chrome browser cache (amazing how your search results change!) -- avoid "logging in" -- and (4) Boot and cruise internet with "stock" live systems.  

Welcome to a Brave New World, Mr Savage (;))!

Nerd Bonus & Hints: How to fetch audio only track from YouTube videos:

From your *NIX command line: 

# =============================================
# --- Fetch audio track from YouTube URL
# --- install YouTube-DL if not already onboard
sudo curl -L \
  https://yt-dl.org/downloads/latest/youtube-dl \
  -o /usr/local/bin/youtube-dl 
sudo chmod a+rx /usr/local/bin/youtube-dl 

#
# --- Query YT for stored stream containers,
# --- "shopping list" output results below.
# --- Note selection of container 171 for audio
/usr/local/bin/youtube-dl -F rSw0fDCSfX8

#
# --- Fetch audio only track:
/usr/local/bin/youtube-dl -f 171 rSw0fDCSfX8

[youtube] rSw0fDCSfX8: Downloading webpage
[youtube] rSw0fDCSfX8: Downloading video info webpage
[youtube] rSw0fDCSfX8: Extracting video information
[download] Destination: The Threat by Ross Anderson-rSw0fDCSfX8.webm
[download] 100% of 26.35MiB in 00:36

# ===============================================
# --- Typical output: Query YouTube for shopping
# --- list of video container IDs and digital 
# --- formats -- for help, use "-h" with youtube-dl
/usr/local/bin/youtube-dl -F rSw0fDCSfX8

[youtube] rSw0fDCSfX8: Downloading webpage
[youtube] rSw0fDCSfX8: Downloading video info webpage
[youtube] rSw0fDCSfX8: Extracting video information
[info] Available formats for rSw0fDCSfX8:
format code  extension  resolution note
249          webm       audio only DASH audio   52k , opus @ 50k, 15.80MiB
250          webm       audio only DASH audio   72k , opus @ 70k, 19.07MiB
171          webm       audio only DASH audio   89k , vorbis@128k, 26.35MiB
251          webm       audio only DASH audio  129k , opus @160k, 34.69MiB
140          m4a        audio only DASH audio  130k , m4a_dash container, mp4a.40.2@128k, 39.80MiB
278          webm       248x144    144p  105k , webm container, vp9, 30fps, video only, 29.10MiB
160          mp4        248x144    144p  107k , avc1.4d400c, 30fps, video only, 15.71MiB
242          webm       414x240    240p  242k , vp9, 30fps, video only, 49.02MiB
133          mp4        414x240    240p  258k , avc1.4d400d, 30fps, video only, 45.79MiB
243          webm       622x360    360p  459k , vp9, 30fps, video only, 96.26MiB
134          mp4        622x360    360p  616k , avc1.4d401e, 30fps, video only, 70.79MiB
17           3gp        176x144    small , mp4v.20.3, mp4a.40.2@ 24k
36           3gp        320x186    small , mp4v.20.3, mp4a.40.2
18           mp4        622x360    medium , avc1.42001E, mp4a.40.2@ 96k

43           webm       640x360    medium , vp8.0, vorbis@128k (best)

Unlimited Soothing Surf & Stimulating Thinking w/ Brownian Noise

Challenge: Fighting noise with noise?  How to source unlimited soothing sounds to aid heavy thinking -- and block out highway noise?  And avoid use of privacy invasive apps / websites -- and run on "old" hardware?  

First a quick solution example -- and then second -- detailed background discussion.  

On almost any *NIX system -- using SoX -- the following commands will generate "Surf Noise" -- sound that mimics ocean waves breaking & retreating on a beach -- much more pleasing than just un-modulated stream of noise:

# =======================================================
# -- Option #1: Using Sound Exchange (SoX) tools 
# -- Unlimited "real time" play with surf waves (brown noise)  
# -- with period 8 secs (0.125 Hz) and 10% zero offset
/usr/bin/play -n synth brownnoise mix synth sine amod 0.125 10 

# 
# --- Or write 40 secs audio to a WAV file at CD quality of 44100: 
/usr/bin/sox -r 44100 \
             -n surf_out.wav synth 40 brownnoise  \
             mix synth sine amod 0.125 10 

Hear & see "surf_out.wav" -- via YouTube -- video rendering generated via FFMPEG over piano keyboard scale (showcqt filter):  

40 secs of brown noise modulated by 

"ocean waves" with 8 sec period

Alternate tool? A fun visual frequency spectrum -- generated using FFMPEG / FFPLAY -- and synthetic input "device" anoisesr  -- and Tremolo modulated -- to simulate beach waves:

# =====================================================
# -- Generate Ocean Surf wave noise & spectrum display 
# -- Hints from obscure FFMPEG development notes:
#  http://k.ylo.ph/2015/11/30/endless-seashore.html 
#  
# -- for audio only, set "-showmode 0"
# 

/usr/bin/ffplay -f lavfi -showmode 2 \
  -i 'anoisesrc=d=40:color=brown:r=44100:a=0.5' \
  -af tremolo=f=0.125:d=0.9 
# 
# --- save to file with ffmpeg 
# 
/usr/bin/ffmpeg -y -f lavfi \
       -i 'anoisesrc=d=40:color=brown:r=44100:a=0.5' \
       -af tremolo=f=0.2:d=0.9  \
        surf_out.wav

BACKGROUND:  Why these examples? Why unlimited ocean wave noise?  Sourced for work & study? 

During undergraduate school, I discovered that background music was soothing help -- while doing complex math & physics homework.  And subjectively seemed to aid problem solving. There is evidence appropriate noise levels help ADHD children -- and according the Wall St Journal, a little noise help normal folks.  See also end of script notes.

Beyond math, for this author, when writing was required -- different brain circuits were involved -- and music -- especially if there were lyrics.  The lyrics mangled the word smithing process.  Use of noisy fans -- or better -- noise from rain / thunderstorms -- provided brain stimulation -- minus the cross wiring of thoughts from lyrics. 

Since childhood days, this author has often used small electric fans or noisy space heaters -- to generate a background of "pink noise" -- and help drown out the spurious urban & suburban sounds.  My spouse regularly uses fans to aid sleep. Using noise sources to aid sleep is so common many commercial solutions exist.

One possible commercial noise

machine (from wikipedia pixs)

Natural Sources: This author enjoys ambient audio recording of passing thunderstorms -- for later soothing playback during long dry spells -- and to provide low level brain stimulation for mentally concentrated work -- or sleep assistance.  One of the down sides of recording in suburban environments is persistent low frequency traffic rumble -- mostly wheel intermod rumble under 30 Hz.  Depending upon atmospheric conditions, this low freq rumble can persist for many miles from traffic sources -- and can be difficult to filter out -- and retain the pleasing deep thunder boom harmonics.

Synthetic Solution?  There are herds smartphone apps --  interesting websites -- and many YouTube videos -- that provide soothing "white/pink" background noise -- and help drown out traffic / urban noise pollution.  Each of these have their up & down sides.  Enter a low cost option: Most laptops -- even 6-8 year old laptops -- have excess CPU capacity to generate soothing ocean wave noise.  Why not give new life to an old laptop -- and avoid buying a white noise machine?

UPSHOT? FOSS tools like SoX and FFMPEG / FFPLAY -- and from experiments illustrated here -- great for generating soothing background "surf" or ocean wave noise.  To paraphrase an old 1960s TV show: Your mission, Jim -- should you choose to accept it -- is to give these tools a shot -- you will be surprised at their power and flexibility -- beyond "trivial" synthetic surf sounds.  As always, should your computer be caught or killed, my secretary will dis-avow any knowledge of your actions.  Good luck Jim!