When Mark Zuckerberg built the first version of Facebook in his college dorm room at Harvard, he imagined it as a window that would allow people to look in on the lives of other users. If Google was a search engine for information then Facebook, by contrast, was a search engine for people. Fifteen years later, Facebook has taken this ambition to the next level. By creating Portal and Portal+, its line of screen-enhanced smart speakers, launched in November 2018, the social media giant has established a far more literal window, letting Facebook users to make video calls to one another.
The Portal smart speakers literalize another Facebook dream, too. Where Facebook was, in essence, a search engine for people, Portal actually does search them out: with a roving 12-megapixel camera, boasting a 140-degree field of view, which follows you around the room to see what you’re doing. As Digital Trends put it in our review, “if you’re busy moving about the kitchen while asking Grandma how to make her famous meatballs, you can keep busy while listening to her talk.”
What exactly is the smart technology that drives Portal? And how does Facebook think it’s cracked the challenge of making regular video chat feel as personal as sitting down for a real conversation? The answer involves some impressive artificial intelligence — and an added human touch.
Making cameras smarter
Right from the start, Facebook knew that the core to its Portal experience would be the so-called “Smart Camera” system. The idea of the Smart Camera was to move beyond the kind of static shot that services like Skype have been offering us for years, and to play a more creative role in the process. Just as a movie director or cinematographer knows when to employ a wide shot or when to zoom in for an intimate close-up, so Facebook challenged its engineers to imitate this same ability with Portal.
To give this camera the necessary human touch, Facebook worked with filmmakers to figure out the best way of distilling their wisdom into machine learnable insights. In one case, it asked them to demonstrate how they might shoot a scene in which it was impossible to capture all the relevant information from one fixed angle.
Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.
In another, Facebook engineers looked at the different photographic elements that camera operators prioritize in portrait and landscape shots. These observations formed the basis of software models which attempt to imbue Portal with some of the decision-making quirks we would normally attribute to human creativity.
“We wanted to create a hands-free video calling experience that removes feelings of physical distance and is more like hanging out together,” Eric Hwang, one of the engineers behind Portal, explained to Digital Trends.
The resulting system — which Facebook says took it “under two years” to create from scratch — allows Portal to make decisions designed to improve the flow of a conversation. In a newly published blog post, it details some of the illustrations of why this might be necessary. For example, if you’re in a crowded room, full of people interacting with one another, it must choose when to follow an individual out of frame or when to zoom out to accommodate new subjects.
Similarly, it must learn to deal with changing light situations in real time. What do you do if your subject is lying down in a dark room, half covered by a blanket, but there are kids running around in the background causing motion blur? Portal weighs all of this information in less than the blink of an eye and tries to determine the best outcome. (If you want to manually control who it focuses on, that’s now possible too.)
Technical challenges
From a technical perspective, a a couple of things make Portal’s technology impressive. The first is that it can do all of this without the use of an actual moving camera. Early on in the development process, Portal’s engineers tried out prototypes which used a motorized camera, which swiveled to face subjects. However, this was decided against on the basis that it caused a lag and a point of potential mechanical failure. Instead, Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.
Second, the team working on Portal found a way to achieve its decision making processes without having to rely on cloud computing. According to Hwang, the computational firepower is all achieved in-device.
“Capturing everyone in a video frame isn’t a hard engineering problem, as many engineers can do that with today’s computer vision advancements,” he said. “The innovation is in capturing the relevant people or person in real-time, on-device, using just the small mobile chip inside Portal as processing power. Usually these types of A.I. tasks require dedicated, large servers. [We] overcame that obstacle by compressing complex computer vision models until they could fit on the chip we use for Portal and still run accurately and reliably.”
To do this, Portal draws on Facebook’s long-term investment in artificial intelligence. It uses a 2D pose-detection system which runs at 30 frames per second. The intentionality of these poses help Portal to make continuous decisions about what its subjects are doing — and when it might need to digitally pan or zoom as a result. It additionally utilizes research into depth cameras developed by Facebook Reality Labs as part of the social media giant’s virtual reality efforts.
A growing market
Facebook is convinced that it is onto a winner with Portal. It’s easy to see where its confidence comes from. Right now, the smart speaker market is booming. Although largely dominated by market leader Amazon, it is growing at more than 100 percent year-on-year. That’s good news for tech companies searching for the next big thing at a time of flattening smartphone sales.
While Facebook was the last of the big four tech giants (Amazon, Alphabet, Facebook and Apple) to jump on the bandwagon, it is still one of the first wave of smart speakers centered around the screen as a communication device.
“Portal is the only product on the market of its kind,” Hwang said. “Today, smart speakers and displays are built around information and commerce. Portal is built to make it easier to connect with the people that matter most: our closest friends and family. And Portal is focused on connecting people — part of Facebook’s mission — which is not currently served well by the home device market.”
Privacy challenges ahead?
So what’s stopping stopping Facebook? Well, potentially privacy. Users have proven surprisingly willing to embrace “always listening” gadgets from companies like Google with a vested interest in user data. But a device that both watches and listens you is more invasive still. Furthermore, Facebook’s reputation is still suffering after last year’s Cambridge Analytica scandal.
Just days before this very article was published, the Washington Post reported that Facebook is negotiating a record breaking, multi-billion dollar settlement with the FTC for its privacy misdemeanors. With a growing backlash from many former users, it’s yet to be revealed if Facebook has an Amazon Echo-style hit on its hands — or an Amazon Fire Phone-style flop.
Facebook assured us that it does not listen to, view, or keep the contents of Portal video calls, which are additionally encrypted to avoid eavesdropping. The fact that Portal’s A.I. smarts run locally on the device, and not on Facebook servers, also means that this information does not leave your home. Voice commands are sent to the company only after you say “Hey Portal,” and users can delete their voice history in Facebook’s Activity Log at any time.
But there’s no getting around the fact that there is still a degree of data collection taking place. “While we don’t listen to, view, or keep the contents of your Portal video calls, or use this information to target ads, we do process some device usage information to understand how Portal is being used and to improve the product,” Facebook notes. (Portal’s privacy policy can be read here.)
Portal offers some very smart technology with massive implications for the future of video chat. There’s no doubt that the company has managed to pull off something very impressive from a technological point of view. But whether it can convince potential customers that this is a solution they need in their lives will, ultimately, prove to be the real achievement.
