I remember the first time I saw the Microsoft Surface. I was on the Tablet PC team and visited a design symposium where all the designers around the company came and shared their latest. Sabrina Boler was there demoing the Surface and we played a game of scrabble using privacy screens placed on the surface that would keep my pieces hidden from view. It was unlike anything I'd seen before. I knew I needed to be part of this team. I transferred over and was assigned to the Shell team, also known as the operating system.
Shell 1.0 & 1.5
Cross-Functional: I transferred over and was assigned to the Shell / OS feature team. It was a collaborative group of design, research, PM, dev, and test. I took a pretty rough user experience, and cleaned up what I could before the version shipped. Then we went right into designing version 1.5, which consisted of a lot of new features, and a visual refresh to align with the new brand.
Visual Design: I was really good at Flash at the time, so I ended up creating the animations in Flash and exporting the animations for the UI as png sequences. I got to work with the brand team to create the loading animations, and Surface logo animation. And I loved partnering with our audio designer to create the sonic experience. I revamped the main transition in and out of applications using a zooming approach. I also solved the math for the fisheye zoom for the carousal when our engineers couldn't figure it out.
Interaction Design: Some of the problems we were solving at the time were around the home buttons: preventing accidental activation and aiding discovery. How to make the interface work from any angle, or for multiple users. And how it works with physical objects.
The object routing feature allowed developers to associate their tagged objects with one or more applications on the Surface. We made a unique layout system inspired by magnetic forces. It kept the UI close to the associated object, but away from the edge of the display as well as other objects.
v2 redesign
After shipping version 1.5 the team decided to go back to the drawing board and rethink what Surface could be. I proposed this as a possible direction. On the left is my Flash Wizard of Oz prototype. On the right is the interactive version we created in collaboration with engineering.
Shortly after this, the Metro design language became all the rage and redesigned this circular model to be more rectilinear. But most of the core interaction concepts I proposed became our V2 direction: modular applications that could be extracted, moved around on the canvas, and recombined into new data types.
Media PLAYER
I moved from the Shell team over to the experiences team for version 2 of Surface. I helped design the web browser, map, media players, and Movie Maker. Here are a couple of examples of the minimal design aesthetic we were going for as we applied the Metro Design Language with our multi-touch Surface. For context, this was 2008, two years before the first iPad was released.
Movie Maker
One of my favorite applications I have ever designed was Movie Maker for Surface. Back then my favorite method to do interaction design was physical paper prototypes. Here are a few images of my process.
I just found this video after 14 years. We were making some serious magic back then. It broke my heart when half the team was laid off a year before. And the final blow was when they cut the release and re-orged us into Windows. Thus began my transition to Artefact, 8 months later.
Interaction primatives
At some point, I was working on the interaction primitives and input for Surface. I explored a wide variety of touch-based and tangible computing methods.
Here is a marking menu that I designed and worked with a prototyper to create. I was hoping Buxton would be proud if we had shipped this thing. Unfortunately, we were re-orged into Windows before we could apply the visual design to this.
I was working with Daniel Wigdor at the time and he was hot on Buxton's Magic Lenses and we wanted to see how we might be able to apply them to the Surface. We ended up making this way too complex and moved away from it afterward. But fun at the time.
Here was the version the engineers put together. We got fairly far on the marking menus and lens interactions.
Long before the v2 redesign, Jesse Graupmann and I were experimenting with how to rotate things on Surface. Before the iPhone came out people didn't understand the multi-touch rotation gesture. When asked how to rotate an object many of them would pivot their finger to rotate, then rotate from the edge, then multi-touch rotation. We figured out a method to combine all three methods of rotation into a single model: 1) multi-touch rotation, 2) single-finger drag, and 3) single-finger pivot. We weighted the single finger drag based on distance from the center of the object. And we weighted the single-finger pivot based on the angle of the finger. Fingers that were pointed directly down couldn't rotate the object, fingers that are flatter could.
Tangible computing
One of the unique aspects of the Surface was the tangible objects that could be tracked when placed down on the display. I explored a series of phycons (physical icons) that would allow the user to perform actions by just picking up objects and placing them. These were the initial set of phycons we tested with users.
Menu dial. Was eventually shipped on the Surface Studio.
Media player phycon
Delete phycon
New and recycle phycons
Mail phycon
We eventually created a toolkit of these phycons and sent them to all our partners.
Samsung Sur40
The entire software team was re-orged into Windows and the contractors were left to ship the product with Samsung. I never had seen our tangible computing kit used until this video I found today. There's the clear dial physon in use!
We did retain some of the spirit and ship some of our features thanks to the incredible design contractor, Jon Garn. The designs that made it through shipping: 1) the home screen with the home button in the center of the water, 2) the stacking scrolling mechanism from Movie Maker, 3) the zooming transitions, 4) the rotatable home menu. It was too bad we couldn't get the modular OS working. I tried that model again on Magic Leap and that time was able to get it shipped.