Skip to content Skip to navigation

Smart Automation

Nest testing Lab
While encouraging user's proactive involvement in reducing their energy use is an important goal, a complementary strategy looks at how automation can improve efficiency with minimal user input.

Nest Labs' thermostat—sensor-driven, Wi-Fi-enabled, self-learning and programmable—is one example. This approach circumvents persistence issues in user behavior. In order for these techniques to be effective, however, they need to consider behavior. The vast majority of current home automation systems operate without such consideration, using one of two insufficient paradigms. Either the systems operate using fixed rules that fail to account for individual differences, or they require that users specify the operation rules themselves which is time consuming, unintuitive and may be easily ignored.

This project built an automated light and TV control implementation using a network of wireless switches based on detecting the location of a user and their pose with a number of cameras. Investigators also developed a web-based user interface to capture the user's input about the automation setting and build a context-aware user profile, which was used to adapt the setting according to the user's preferences.

Nest testing Lab

The first task of this project was to create an integrated approach based on ubiquitous sensing of the environment and algorithms that predict user behavior and automatically adapt. The second task was to evaluate the system through in vivo testing. The project delivered a modeling system that uses real world data to build a model that predicts user behavior. Predictions from the system were integrated into existing interfaces to present predicted future behaviors in an accessible manner.

The automation was accomplished by using machine learning algorithms that address both user models and decision-making. Modeling algorithms use sensor data collected from a variety of sources. Such a model could learn from observations such that, for example, when a user is sitting in the living room in the evening, the TV and lights in that room are likely to be on. And it could also learn the probability that the user will transition to a different situation, like going to sleep. Researchers then applied decision-making algorithms to prescribe system changes that limit energy use. For example, the system could predict that computer use is typically low while the user is watching TV, and it could power down the computer in another room. The key element is that both the modeling and decision making processes are adaptive and will adjust to a user's behavior without requiring manual entry of preferences.

To achieve adaptivity in providing services to the user, the system was found to need to support two functions: sense the activity and state of the user, and customize service to the user's profile. To achieve this, three functional modules were developed and described in published papers. In the first module, behavior analysis of the user in a home environment was achieved based on multi-camera vision processing. In the second module, a user profile was defined and hierarchical reinforcement learning was employed as a technique to learn the user profile dynamically. The third module is a decision maker which employs the user profile to control services to maximize user comfort and utility.

Publications and Presentations

Towards adaptive and user-centric smart home applications
In Behavior Monitoring and Interpretation, part of the "Ambient Intelligence and Smart Environments" book series (IOS Press)
Khalili, A.H., Wu, C., Aghajan, H. (2011)

Learning Human Behaviour Patterns in Work Environments
Presented at IEEE's Workshop on CVPR4BH, in conjunction with CVPR 2011
Chen, C., Aztiria, A., Aghajan, H. (2011)

Discovering Social Interactions in Real Work Environments
Presented at IEEE's International Workshop on Social Behavior Analysis, in conjunction with FG 2011.
Chen, C., Cilla, R., Wu, C., Aghajan, H. (2011)

Multiview Social Behavior Analysis in Work Environments
Presented at IEEE's International Conference on Distributed Smart Cameras
Chen, C., Aghajan, H. (2011).

Hierarchical Preference Learning for Light Control from user Feedback
Presented at IEEE's Computer Vision & Pattern Recognition
Khalili, A., Wu, C., Aghajan, H. (2010)

Future work

On further development effort, algorithms can be created to detect complex but unobvious schedule regularities and automatically control devices in a home area network. The behavior of the network can change to adapt to underlying changes in the users' behavior or in preferences or with seasonal changes. Other algorithms can be developed to track actual and expected energy use of appliances and suggest when appliances or electronics should be repaired or replaced. On commercialization efforts, researchers can share their findings with commercial entities involved in home automation services and collaborate with them on developing test cases for actual user deployment settings.