The prediction business
To achieve the ideal balance of user control and simplicity, it’s important to understand how the AI actually functions within the camera. With the goal of capturing moments the user will find interesting, the key factor isn’t deciding what’s interesting—it’s predicting.
My colleague Aseem Agarwala recently wrote about how Clips actually works. He describes what’s happening inside the camera and how a combination of factors were used to derive a predictive “interestingness” score. The score represents the degree of confidence that a given moment, captured by the camera, is interesting to the user. In designing the algorithm, we had to strike a balance between precision and recall. Do we include only the moments the algorithm is very confident are “interesting” (precision)? Or do we include everything that might be interesting, in order to provide full coverage (recall), knowing that some less accurate predictions will make it through?
Predicting interestingness in photographic content is an educated guessing game. It’s not a matter of right versus wrong, but a balancing act among all the features that make for interesting clips. In the end, the human and the machine collaborate to get the best content, rather than relying entirely on the camera’s predictions.