Microsoft Rich Interactive Narratives
Microsoft Rich Interactive Narratives (RIN) research project combines traditional forms of storytelling with Videos as well as multiple interactive visualizations such as Maps, Images, Panoramas, Photosynths, Deep Zoom images to create compelling interactive digital narratives. About The RIN Project, Microsoft Research
About the project
While multiple types of rich visualization technologies like Streaming Video, Gigapixel Panoramas, Photosynths, Deep Zoom Images and Terapixel Maps are widely available on the Internet, each of these media types have their own custom experiences, websites and mobile applications, as well as their own proprietary rendering logic, data formats, and back-end services. The research team at Microsoft Research India in collaboration with the Interactive Visual Experience group in Microsoft Research Redmond and the Microsoft Research Connections, were looking to create new ways of combining narrative storytelling and interactive experiences utilizing new kinds of media (Expression Streams), while also providing a cohesive and consistent experience across these various types of new visual content on the web.
RIN's core technology had its origins in the development of The Sri Andal Temple project. This demo featured an experience that led users through an interactive, narrated walk-through that included Photosynth and HD View stitched images of a temple in Tamil Nadu. While The Sri Andal Temple project introduced RIN's technology concept of Experience Streams, the end user experience hadn't yet been fully optimized for the immersive experience that it was intended to be.
My role
I came on board the research team in 2010 as Senior lead designer to spearhead the user experience effort. I drove the design of a unified experience that ties together various types of media and makes interacting and viewing content seamless from one experience to the next and helped refine an experience that would be a blend between cinematic and interactive. I was responsible for the evolution and refinement of the RIN user experience and interaction model from its initial demo version, provided guidelines for creating and viewing content, as well as created content to help drive the UX. I also engaged with external content partners to help create content and experience framework to showcase RIN technology.
Process
Identifying opportunities for improving the experience
I researched and evaluated several award winning interactive content available online at the time. On comparing these with The Sri Andal Temple demo, I found that the Andal demo while providing an initial proof of concept of the technology's capabilities didn't quite stand up to other experiences that were being created on the web using ubiquitous content creation tools like Flash. The experience was not consistent across media types (Experience Streams). The interactive UI that appeared for each Experience Stream were controls native to that particular media (Photosynth controls, Map controls etc.) and appeared visually disconnected which made for a somewhat confusing end to end experience. Navigating between segments was not intuitive. Interactive content was not easy to discover during the cinematic experience. Related information was difficult to discover and overall the experience lacked the kind of finesse that people expect with interactive rich multimedia experiences on the web.
Designing an extensible user experience system and defining guidelines for creating compelling experiences
Common UI elements that define the basic structure of an experience
Rich interactive immersive experiences require high quality, consistent end-to-end user experiences that engage and delight the end-user. The first step was to provide a simple framework for all Narratives as a template that could be easily customized.
I proposed an extensible user experience model that could accommodate various types of complex interactive Experience Streams (media) with intuitive and predictable UI throughout.
Affordances would be provided for common UI elements and controls such as a toolbar with play/pause, volume, progress bar, title and branding.
Creating a cohesive UX system across multiple media types (Experience Streams)
Predictable controls irrespective of media type
As with maps, videos and images several of the new visualization technologies such as Photosynth, Deep Zoom, Pivot, Relationship Graph and Chronozoom also came with native controls that were all uniquely styled, appeared in unpredictable ways and were not always intuitive.
Types of Experience Streams with native controls
The challenge was to design a user experience that would be extensible and accommodate the (sometimes complex) native controls while ensuring that the interaction model is seamless.
I addressed this issue of a disconnected experience across disparate visualizations by first collating an inventory of controls across the multiple Experience Streams. Wire frame mock-ups were created to define an interaction model that was flexible and could easily accommodate additional controls for various kinds of Experience Streams. It was also important to provide a common aesthetic style across the experiences to create a familiar environment throughout.
The interface was designed to be scalable and accommodate natural input.
When this model was tested among a few internal users, I found that discovery of controls improved with predictable placement of controls. Switching controls for experiences such as Panoramas and Deep Zoom images was less ambiguous since the UI for these experiences is familiar, while some of the more complex experiences such as Photosynths required a certain amount of knowledge of the visualizations to recognize the controls. Overall this seemed like a workable model that people would build familiarity with over time.
Extensible toolbar
Designing immersive, fluid end-user experiences
My intent was to make the UI transparent - Focus on content not on interface. The cinematic (narrative) experience needed to be seamless and uninterrupted by UI with appropriate cues at various points to discover and enable interaction.
I provided guidelines for introducing interactive "moments" within the narrative. There could be a visual cue or a pause in the narrative along with voice-over guiding the user to interact with content. At other times the user could pause the narrative or even by clicking/tapping, the narrative would pause and UI would be enabled to allow for interactivity.
During both the cinematic as well as the interactive experience, transitions would allow the user to move within and between content types (Experience Streams) without experiencing lags, delays or changes in the overall user experience.
While interacting with content within a narrative the user would always know where they were and affordances would be available to easily return to the cinematic experience.
The experience allowed fluid transition between a cinematic scripted experience and interactive exploration. A narrative overview or table of contents acted as a hub and anchored the experience.
Narrative overview/Table of contents - Launches the cinematic and intractive experiences (mock up)
Narrative overview as hub for interactivity (mock up)
Designing an extensible and customizable user experience
A default visual style based on the Windows Metro style that could be themed
I provided art direction and helped maintain the aesthetic quality bar for the default visual theme which was based on the Windows Metro style with a Zune-like navigation model for the Narrative overview. While this fit the requirements of a clean, modern, recognizable aesthetic, it also provided us with a palette of options without needing to invest in a huge amount of design resource.
The user experience (including visual style) was designed to be themed by content creators. I created several mock-up explorations to help drive code requirements for themes. (While there was no explicit user interface to easily theme an experience completely, hooks would be in place for content creators to modify the themed elements in code).
Re-usable components
Interactive components such as embedded artifacts, hyperlinks, pop-ups, slide shows, content browsers, text overlays, navigation legends and so on were designed as re-usable building blocks.
A review of common interactive elements found in various multimedia experiences, provided a basic palette of interactive components. While I defined the default behavior and properties for these elements, flexibility was provided in the code structure to accommodate variations and themes.
While rudimentary UI would be provided for creating some of these components such as the content browser and slideshows, the limitations of a research initiative meant that much of the experience including interactive components would have to be hand crafted in code.
Interactive elements as re-usable building blocks
Guidelines for authoring and creating Rich Interactive Narratives
Given the complexity of creating multimedia experiences that utilize complex visualization technologies and the inherent challenges of creating these rich narratives with partial tools it was necessary to provide fairly detailed guidelines for various aspects of content creation.
Although the team had built an authoring tool that allowed for a certain amount of assembly of a narrative, various aspects needed to be hand crafted. I put together a authoring workflow document that provided guidance on planning the narrative - creating storyboards and writing scripts, creating content - processing media, guidelines for making use of re-usable components, tips for assembling the narrative on a timeline in the RIN Authoring Tool and scripting the cinematic experience by defining key frames in code and testing the experience. At various stages I also provided input and feedback to the technology team on Narrative assembly and the basic authoring experience.
Creating compelling interactive narratives to evolve and showcase user experience and technology capabilities
Hand in hand with designing the user experience and interaction model, it was also necessary to create actual narrative content to refine the user experience and technology as well as highlight it's capabilities. I created a few test narratives to enable and drive technology and showcase the user experience model. Early test narratives such as Ladakh would later be used as experimental platforms to enable Microsoft Kinect based gesture interactions. Video demo of Kinect enabled RINs
GlacierWorks, Everest Kiosk
I worked closely with third party content expert and Mountaineer David Braeshears and his team at GlacierWorks to design and create a custom-tailored, hand-crafted Everest Kiosk experience to be deployed in the form of Rich Narrative Media at the MIT Museum. This interactive chronicle was tailored specially for the Kiosk environment where the user can interact with the narratives on a touch screen with touch flicks and gestures. Video demo of The Everest Kiosk
A complete kiosk experience was designed and built to showcase all of RIN's capabilities. This project required hands-on engagement at many levels. In the beginning it involved interfacing with our content partners to create and refine a storyboard and script. And as with some of the other test narratives that were created during the evolution of the project, I designed the overall interaction and aesthetic direction as well as actual handcrafting of some of the interactive elements. The project was developed in close partnership with the technology team.
Everest Kiosk
GlacierWorks: "Rivers of Ice"
Another key engagement with the GlacierWorks team was the creation of the narrative GlacierWorks: “Everest: Rivers of Ice”. This was launched as a marquee multimedia HTML5 site for Internet Explorer 10. Pixel Lab an external design agency was chartered to create the experience.
Although Pixel Lab was responsible for building the experience and for the development of the site, The research team having already built the Kiosk experience was on point for various aspects of content creation and assembly. Several elements that had been created for the Kiosk would be recreated for the HTML 5 website version. (Embedded artifacts and "Swipe"- a comparison of glaciers over the years).
Once again, I worked closely with the editors at GlacierWorks to help develop a storyboard and script that would work elegantly within the RIN parameters. Given my prior involvement with content creation and assembly I was again called upon to help with various aspects of creating and managing content production. This included keeping track of media, processing content for the final narrative, procuring and editing audio, helping with the hand-crafting of embedded content and finally testing to ensure that the experience worked smoothly and was glitch free in all respects.
Evolution from the Kiosk experience to the re-designed HTML 5 version
Storyboard and script with markers guiding placement of embedded content
