We are exploring interactions, algorithms and architectures that will allow us to interact programmatically with objects in our physical environments in interesting ways via new forms of affordance aware computing. If we could map the entire world in terms of affordances, what kind of interactions could we create on top of that map? By building context-aware architectures and defining affordances that can be detected by computers, we hope to open the door for building new classes of applications that can transform the way we interact with the physical world on a daily basis. This can range from simple examples like pointing out fun puddles or leaf piles to jump in to more complex ideas like creating parkour games or obstacle courses using our everyday physical world.