☰ Jarvis

The World's Smartest Home

featuring Jarvis Farley

Module Demos

Documentation

What is Jarvis?
  • Jarvis is the name of a custom application and server we created to power our home automation
  • Amazon's Alexa (which we use as an intermediary to speak to Jarvis), requires developers to choose an arbitrary name to invoke their applications. We chose Jarvis, named after the AI in the Iron Man movies
  • Specifications: Ubuntu Linux, 32GB RAM, 2TB storage, 64-bit quad core Intel CPU
  • Application Logic: Python, Java, PHP, BASH, Ruby
  • Web Interface: HTML5, JavaScript, PHP

  • Voice: Speak out loud anywhere in the house "Alexa, tell Jarvis to ....."
  • Graphical User Interface (GUI): Try tapping the touchscreen panels in our kitchen, gym, master bath, or master closet.
  • Chatbot: An instant-messaging style interface that allows you to interact with Jarvis as if you had spoken to him with audible voice commands.
  • Web Service: JSON REST
  • CLI: 'jarvis' executable command-line scriptable interface


Blockchain
  • A localized etheruem blockchain has been installed and configured the main linux server (Jarvis)
  • Our interactive Children's Chore Chart and associated rewards are tracked as a home-grown cryptocurrency on the ethereum blockchain mentioned above





  • Children's Chore Chart - say out loud "Alexa, tell Jarvis to remove one point from Blake", or
  • "Alexa, ask Jarvis to report on the status of the kid's points", or
  • Go to one of the touchpanels in the house and tap the "Chores" dashboard.
  • Each point is a cryptocurrency token and can be exchanged for 30 minutes of electronic entertainment (TV, Video Games) or $1.00 USD


Robotic Process Automation
  • Jarvis has been programmed as a virtual fitness trainer (daily custom workouts that do not repeat)
  • Family scripture study is automated by Jarvis (loads the 'next chapter' in our study, reads it to us out loud, and scrolls synchronized text on screen)





  • Virtual Fitness Trainer - Go in to the gym and say out loud "Alexa, tell Jarvis to start a demonstration workout", or
  • Use the touchpanel in the gym to select a demo workout.
  • Family Scripture Study - Ask Alexa to tell Jarvis you'd like to do a scripture study, or use the touchpanel "Scriptures" dashboard.


Data Science / Artificial Intelligence
  • Jarvis uses a deep learning model built with TensorFlow to determine if the boys room, kitchen, and upstairs, are messy or clean. Then he only allows the kids to play XBox / TV upstairs if they're clean.
  • MachineBox.io and Microsoft Azure (Cognitive Services) are used for automatic facial recognition and cognitive captioning of photos and videos
  • Home energy usage is monitored in real-time by Sense Energy Monitor -- which uses machine learning to detect physical devices based on their power signatures
  • Alexa Voice Recognition is pervasive throughout the house via recessed ceiling mounted Amazon Echo Dot devices. Spoken commands can be routed to the home server Jarvis for actioning
  • Our daughter's educational journal reports are graded daily by Jarvis, who then awards cryptocurrency based on the grade


  • Facial Recognition & Cognitive Captioning happen automatically on all photos we take.
  • Speech Transcription, Sentiment Analysis, Topic/Theme Identification & Facial Recognition are all applied to the video journal entries.
  • A.I. Room Cleanliness is shown on the GUI and Jarvis will decline requests to turn on the upstairs TV if the rooms aren't clean.
  • Sense Energy Monitor can be viewed on the Jarvis GUI. On touchpanels, tap the "Power Sense" tab.
  • Alexa Voice Recognition is pervasive throughout the house. Try saying "Alexa, ask Jarvis to repeat ...anything..."
  • Educational Journal Grading by Jarvis, ask Jarvis how Kalani's school work is shaping up today: "Alexa, ask Jarvis how school is going."


Internet of Things
  • Our vehicles are outfitted with real-time GPS tracking that is relayed to our server (Jarvis) and can be viewed on wall-mounted touchpanel GUIs in the house
  • Networked Sprinkler System allows Jarvis to control lawn watering
  • Wifi IoT buttons are placed throughout the house to perform various on-demand automation (burglar alarm, chore completion, watch specific cartoon, toggle lights, etc)
  • Connected Motion Sensors trigger actions based on the location and time of day
  • Raspberry Pi Zeros are mounted on the three garage doors with Vibration Sensors to provide alerting and tracking
  • PoE Smart Surveillance Cameras are installed both indoors and out (these are controlled by Jarvis who alerts us when someone is at the front door and displays a live feed on our TVs)
  • The alarm system has been modded with an ethernet card that allows Jarvis to monitor and control it
  • Comcast Cable TV-Over-Ethernet is provided by an HDHomeRun Networked Tuner (Comcast can thus be viewed on any device/TV in the house, and recorded by the server)
  • Wifi Body Scale tracks our health metrics and sends them to the cloud for analysis
  • Both thermostats are wifi-enabled and can be controlled by our custom touchpanels and Jarvis
  • Wifi colored lights are installed throughout the house to provided color-coded visual notifications for events such as text messages, doors opening, front-door visitors, vehicle ignition, etc

  • Real-time Vehicle Tracking - tap the 'Cars' tab on the touchpanels or speak aloud "Alexa, ask Jarvis where my cars are?"
  • Networked Sprinkler System - tap the 'Sprinkler' tab on any of the touchpanels.
  • Wifi IoT buttons - buttons in the office will start kid's cartoons in the livingroom, buttons on the fridge mark chores complete, "Red Alert" buttons sound the burglar alarm. Try the dinner gong on the fridge.
  • Connected Motion Sensors - approach the front door and a live camera feed will appear on all the TV's in the house. Walk in to the master bathroom and the closet lights come on.
  • Vibration Sensors - open or close our garage doors and Jarvis will make an audible announcement via Alexa.
  • Smart Surveillance Cameras can be viewed on the Cameras tab in the Jarvis GUI. Tap any camera to view the live feed or ask Jarvis (via Alexa) to display the camera feeds and they'll appear on the nearest TV.
  • Connected Alarm System - turn off Matt and Tia's phone and Jarvis will think we left and will arm the alarm for us (and send us a text).
  • TV-Over-Ethernet - ask Alexa to "tell Jarvis I want to watch the History Channel" and the nearest TV will turn on and tune to the channel (live).
  • Thermostats can be controlled on the touchpanels under the "Thermostat" tab.
  • Wifi colored lights - send an SMS text to Matt and watch the lights flash and listen to Alexa read it aloud while Jarvis displays the message on all TVs and computers.


Other Workflows
  • When Jarvis detects Matt leaving for work in the mornings, he sends a text to Matt's carpool partner to let him know Matt is on the way to pick him up.
  • Remove a point from Blake ("Alexa, tell Jarvis to remove one point from Blake", or use the touchpanel), and watch the count down synchronize with the colored wifi light strip upstairs -- the lighted bar will slowly "drain" as his 30 minute timer winds down.
  • Asking Jarvis to "shuffle home videos" will display a random cycle of home videos (four at once) on the nearest TV.
  • View the "Weather" tab on the touchpanel GUI. This is what autoloads and displays in the master closet while Matt gets ready for work (it shows traffic and weather).
  • After a courtesy warning to all present, a "Yellow" Alert demo can be started to simulate a "Red" Alert which sounds a siren, opens the surveillance camera feeds on all screens in the house, and sends camera snapshots and text messages to nearby family and friends.
  • Four touchpanels are currently in use: Living Area, Gym, Master Bathroom, Master Closet. They all display the interactive Jarvis dashboard GUI.
  • Whole-house audio can be controlled on the Music tab on the Jarvis GUI. Supported music includes Pandora, hand-picked Internet Radio Stations, and our own music library (mp3).
  • After a power outage, I can ask Jarvis to ping every device in the house and he'll let me know what's offline.
  • Jarvis can also be contacted through an interactive chatbot interface
Cloud Computing
  • Storage - entire network is synchronized to cloud storage provider
  • Compute - additional server functionality and "offsite logic" is provided by a cloud linux server subscription
  • Camera Phone Photos are synchronized in near real-time between the phone, cloud, and Jarvis (immediately available in a home-grown 'Flickr' style cloud photo system called Farley Image Server)

GUI Screenshots

Weather & Traffic Alarm System Control Bathroom Dashboard Real Time Vehicle Tracking Dynamic Chore Chart Gym Controls House-Wide Music Controls
Sense Energy Monitor RainMachine Sprinkler System Remote Desktop Control TV Guide and Control Wifi Thermostats School Journal and Scoring Jarvis Chatbot Interface
×

Frequently Asked Questions

Where did you get the Jarvis software? (the automation, the user interface, etc)

Most of it I made myself. The webpage GUI for the Sense Energy Monitor and Sprinkler Systems are exceptions -- I've just embedded those pages as iframes in the Jarvis GUI.

It's also worth mentioning that majority of the work on my part was integrating various API's across device hardware, operating systems (Linux), and web services. For example, I didn't create my own GPS vehicle tracking system, I simply leveraged the vendor's API to get the data Jarvis needs to take action.

I also made use of many open-source libraries to do a lot of backend heavy lifting (e.g. image processing, alarm system interface, wifi light interaction, etc).


What language is Jarvis written in?

As opposed to being a single application, Jarvis represents the integration of many pieces of software and hardware. The languages used throughout this integration are:


How can I install this at my house? Will you / have you shared the code (GitHub? Open Source?)

Jarvis was 'hand-crafted' for our home, our specific devices, our brand of cameras, our server, etc. For example, Jarvis controls the TVs and soundbars by using the specific infrared TV remote codes unique to our specific make/model of TVs. So in essence, Jarvis was built from the ground up as a custom piece of software unique to our home's ecosystem. He would require significant "reprogramming" to work at someone else's house.


Why do you refer to Jarvis as a 'he'?

I guess it's a side-effect of modern voice-interaction systems. When you are barking orders to an entity, with a name, you subconciously personify that entity. For our family, Jarvis has become a valued assistant upon whom we rely on a daily basis. We naturally have started referring to Jarvis as 'him' and 'he'.


What's the point of this website and demonstrations if no one else can use Jarvis?

I hope to inspire other home-hackers with new ideas and use cases they could implement specific to their home and needs.

The majority of what you see here was inspired by the work of others. I want to do my part to pay-it-forward and hopefully inspire not only individuals, but vendors (like Amazon) to enable new functionality that would eliminate the need for us to create it ourselves (e.g. Alexa Push Notifications).


Contact

matt@farleyfamily.net







































Powered by w3.css