Facebook CEO Zuckerberg AI experiment yields good news and bad news.
Facebook CEO Mark Zuckerberg built “Jarvis,” an artificial intelligence system, called Jarvis, to run his own home.
In a post on Facebook, Zuckerberg explained that he spent about 100 hours of his spare time over the last year building Jarvis. The simple AI system, customized to his own home, is now capable of choosing and playing music, turning lights on and off and recognizing visitors at the door.
As Zuckerberg noted, this was his personal challenge. He designed it so that he can learn more about the state of AI, he explained.
He said the system uses several AI techniques, “including natural language processing, speech recognition, face recognition, and reinforcement learning, written in Python, PHP and Objective C.”
While reading about how he built Jarvis, five things struck me.
Tinkerer in chief
First, his blog reaffirms that the Facebook CEO is a deft, capable software engineer.
Given their engineering backgrounds, many CEOs in the tech industry, I’m sure, can probably be found in the basement tinkering on weekends.
I wrote almost two years ago about Sehat Sutardja, then Marvell Technology’s CEO, who spent his spare time developing a new interconnect technology called MoChi.
The Jarvis project places Zuckerberg in the same class as Sutardja as a master tinker.
Not so connected home
Second— the most interesting part of his blog— Zuckerberg’s AI project exposes fundamental problems, not with AI, but with the “connected home.”
To Zuckerberg, AI was the easy part.
Much more complicated than he expected was “simply connecting and communicating with all of the different systems in my home.”
So, even before embarking on the development of an AI system, Zuckerberg had to spend hours and hours writing code to connect home appliances that don’t speak the same languages and don’t use the same protocols.
At Zuckerberg’s home, for example, he said, “We use a Crestron system with our lights, thermostat and doors, a Sonos system with Spotify for music, a Samsung TV, a Nest cam for Max, and of course my work is connected to Facebook’s systems.”
Zuckerberg noted, “I had to reverse engineer APIs for some of these to even get to the point where I could issue a command from my computer to turn the lights on or get a song to play.”
For us non-coders, there’s no route to the “connected home” other than, say, buying every appliance based on Samsung’s SmartThings, or Apple’s HomeKit. Of course, that’s not going happen!
Further, there’s an issue even bigger than the non-interoperability among “smart home appliances.” As Zuckerberg discovered, “most appliances aren’t even connected to the internet yet.”
I was amused by Zuckerberg’s discussion on toasters. He wrote: “It’s possible to control some of these using internet-connected power switches that let you turn the power on and off remotely. But often that isn’t enough.” He discovered, “It’s hard to find a toaster that will let you push the bread down while it’s powered off so you can automatically start toasting when the power goes on.”
Solution? He ended up finding a 60-year-old (analog) toaster and rigging it up with a connected switch.
AI discoveries
Third, once he was able to get going with the AI system development, Zuckerberg actually offers a number of useful AI-related discoveries and nuggets in his blog post.
Next page: Zuckerbutt
Facebook CEO Zuckerberg AI experiment yields good news and bad news.
They include:
- Understanding context is important for any AI application
- Using Facebook Messenger to build a Messenger bot to communicate with Jarvis. It allowed Zuckerberg to text anything to his Jarvis bot from his mobile phone, and it will instantly be replayed to his Jarvis server and processed.
- Text vs. voice. Zuckerberg found himself communicating with Jarvis more via text than voice
- A two-step process he installed to recognize a person at the door (first, it runs face detection to see if anyone has come into view, and then, if it finds a face, runs face recognition to identify who it is)
- Text is more important in communicating with AI than most people think, but voice is useful, because it’s “very fast.” He wrote, “You don’t need to take out your phone, open an app, and start typing — you just speak.” But to enable voice for Jarvis, he said, “I needed to build a dedicated Jarvis app that could listen continuously to what I say. The Messenger bot is great for many things, but the friction for using speech is way too much.
State of AI
Fourth, I found his conclusion priceless because he is pretty straightforward about what he discovered about the state of AI today.
He concluded:
I spent about 100 hours building Jarvis this year, and now I have a pretty good system that understands me and can do lots of things. But even if I spent 1,000 more hours, I probably wouldn’t be able to build a system that could learn completely new skills on its own — unless I made some fundamental breakthrough in the state of AI along the way.
Zuckerbutt
Fifth, reading through what Jarvis can now do at Zuckerberg’s home, I couldn’t help but think “Is that it?”
In short, any one of the things he accomplished by connecting devices and teaching Jarvis seems like putting a lot of labor into saving labor that ain’t that hard without Jarvis. I mean, really, what’s faster and more efficient — spending hundreds of hours writing custom code that turns on the audio system, answers the door, starts the toaster and peeks into the kid’s room? Or just taking a few seconds to get off your unconnected Zuckerbutt and doing that same stuff “manually?”
— Junko Yoshida, Chief International Correspondent, EE Times 
