Category Archives: Social

Anticipating More from Cortana – A Look At : The Future of The Windows Phone

Microsoft Research – April 17, 2014 

 

Most of us can only dream of having the perfect personal assistant, one who is always there when needed, anticipating our every request and unobtrusively organizing our lives. Cortana, the new digital personal assistant powered by Bing that comes with Windows Phone 8.1, brings users closer to that dream.

Image

 

For Larry Heck, a distinguished engineer in Microsoft Research, this first release offers a taste of what he has in mind. Over time, Heck wants Cortana to interact in an increasingly anticipatory, natural manner.

Cortana already offers some of this behavior. Rather than just performing voice-activated commands, Cortana continually learns about its user and becomes increasingly personalized, with the goal of proactively carrying out the right tasks at the right time. If its user asks about outside temperatures every afternoon before leaving the office, Cortana will learn to offer that information without being asked.

Furthermore, if given permission to access phone data, Cortana can read calendars, contacts, and email to improve its knowledge of context and connections. Heck, who plays classical trumpet in a local orchestra, might receive a calendar update about a change in rehearsal time. Cortana would let him know about the change and alert him if the new time conflicts with another appointment.

Research Depth and Breadth an Advantage

While many people would categorize such logical associations and humanlike behaviors under the term ”artificial intelligence” (AI), Heck points to the diversity of research areas that have contributed to Cortana’s underlying technologies. He views Cortana as a specific expression of Microsoft Research’s work on different areas of personal-assistant technology.

“The base technologies for a virtual personal assistant include speech recognition, semantic/natural language processing, dialogue modeling between human and machines, and spoken-language generation,” he says. “Each area has in it a number of research problems that Microsoft Research has addressed over the years. In fact, we’ve pioneered efforts in each of those areas.”

The Cortana user interface
The Cortana user interface.

Cortana’s design philosophy is therefore entrenched in state-of-the-art machine-learning and data-mining algorithms. Furthermore both developers and researchers are able to use Microsoft’s broad assets across commercial and enterprise products, including strong ties to Bing web search and Microsoft speech algorithms and data.

If Heck has set the bar high for Cortana’s future, it’s because of the deep, varied expertise within Microsoft Research.

“Microsoft Research has a long and broad history in AI,” he says. “There are leading scientists and pioneers in the AI field who work here. The underlying vision for this work and where it can go was derived from Eric Horvitz’s work on conversational interactions and understanding, which go as far back as the early ’90s. Speech and natural language processing are research areas of long standing, and so is machine learning. Plus, Microsoft Research is a leader in deep-learning and deep-neural-network research.”

From Foundational Technology to Overall Experience

In 2009, Heck started what was then called the conversational-understanding (CU) personal-assistant effort at Microsoft.

“I was in the Bing research-and-development team reporting to Satya Nadella,” Heck says, “working on a technology vision for virtual personal assistants. Steve Ballmer had recently tapped Zig Serafin to unify Microsoft’s various speech efforts across the company, and Zig reached out to me to join the team as chief scientist. In this role and working with Zig, we began to detail out a plan to build what is now called Cortana.”

Researchers who made contributions to Cortana
Researchers who worked on the Cortana product (from left): top row, Malcolm Slaney, Lisa Stifelman, and Larry Heck; bottom row, Gokhan Tur, Dilek Hakkani-Tür, and Andreas Stolcke.

Heck and Serafin established the vision, mission, and long-range plan for Microsoft’s digital-personal-assistant technology, based on scaling conversations to the breadth of the web, and they built a team with the expertise to create the initial prototypes for Cortana. As the effort got off the ground, Heck’s team hired and trained several Ph.D.-level engineers for the product team to develop the work.

“Because the combination of search and speech skills is unique,” Heck says, “we needed to make sure that Microsoft had the right people with the right combination of skills to deliver, and we hired the best to do it.”

After the team was in place, Heck and his colleagues joined Microsoft Research to continue to think long-term, working on next-generation personal-assistant technology.

Some of the key researchers in these early efforts included Microsoft Research senior researchers Dilek Hakkani-Tür and Gokhan Tur, and principal researcher Andreas Stolcke. Other early members of Heck’s team included principal research software developer Madhu Chinthakunta, and principal user-experience designer Lisa Stifelman.

“We started out working on the low-level, foundational technology,” Heck recalls. “Then, near the end of the project, our team was doing high-level, all-encompassing usability studies that provided guidance to the product group. It was kind of like climbing up to the crow’s nest of a ship to look over the entire experience.

“Research manager Geoff Zweig led usability studies in Microsoft Research. He brought people in, had them try out the prototype, and just let them go at it. Then we would learn from that. Microsoft Research was in a good position to study usability, because we understood the base technology as well as the long-term vision and how things should work.”

The Long-Term View

Heck has been integral to Cortana since its inception, but even before coming to Microsoft in 2009, he already had contributed to early research on CU personal assistants. While at SRI International in the 1990s, his tenure included some of the earliest work on deep-learning and deep-neural-network technology.

Heck was also part of an SRI team whose efforts laid the groundwork for the CALO AI project funded by the U.S. government’s Defense Advanced Research Projects Agency. The project aimed to build a new generation of cognitive assistants that could learn from experience and reason intelligently under ambiguous circumstances. Later roles at Nuance Communications and Yahoo! added expertise in research areas vital to contributing to making Cortana robust.

The Cortana notebook menu
The notebook menu for Cortana.

Not surprisingly, Heck’s perspectives extend to a distant horizon.

“I believe the personal-assistant technology that’s out there right now is comparable to the early days of search,” he says, “in the sense that we still need to grow the breadth of domains that digital personal assistants can cover. In the mid-’90s, before search, there was the Yahoo! directory. It organized information, it was popular, but as the web grew, the directory model became unwieldy. That’s where search came in, and now you can search for anything that’s on the web.”

He sees personal-assistant technology traveling along a similar trajectory. Current implementations target the most common functions, such as reminders and calendars, but as technology matures, the personal assistant has to extend to other domains so that users can get any information and conduct any transaction anytime and anywhere.

“Microsoft has intentionally built Cortana to scale out to all the different domains,” Heck says. “Having a long-term vision means we have a long-term architecture. The goal is to support all types of human interaction—whether it’s speech, text, or gestures—across domains of information and function and make it as easy as a natural conversation.”

Advertisements

OneDrive and Yammer takes Social Collaboration to a new level on SharePoint Online

Yammer brings conversations to your OneDrive and SharePoint Online files

Christophe Fiessinger is a group product manager on the enterprise social team.

5810.image-1[1]

At SharePoint Conference 2014 we announced new enterprise social experiences across Office 365 helping businesses work more like networks by leveraging the power of the cloud to bring people together, gain quicker access to relevant insights and help make smarter decisions, faster.

Today we’re announcing the release of one of those features–document conversations–which essentially embeds the social collaboration capabilities of Yammer into the Office apps you use to get work done every day. Get ready for a new, simple way to collaborate on the content you produce with Office Online and store in the cloud in SharePoint Online or OneDrive for Business.

 

Document conversations enable people to share their ideas and expertise around Office documents, images and videos right from within the content they are editing or reviewing. Imagine being able to ask questions, find expertise and offer feedback about content without having to leave the application you’re working in!

 

Because it’s Yammer you can also view and participate in conversations outside your document, on your mobile device, in Microsoft Dynamics CRM or any app where a Yammer feed is embedded! Get ready for a totally new way to produce incredible content!

Here’s how document conversations work. When you open a file in your browser from your cloud store, you see the file on the left with a contextual Yammer conversation in a pane on the right. You can collapse and expand the Yammer pane as needed.

You can do more than join in a conversation from the Yammer pane. You can also post a message, @mention your coworkers, and publish to a Yammer group—either public or private.

Document conversations are easy to join in Yammer as well. If you’re working in Yammer, you’ll see a threaded conversation in the group the post was published in with an icon that enables you to open the file from the cloud location where it lives. The Yammer conversations about files are visible to users in the group but only users who have permission to view or edit the file can open it.

Document conversations are progressively being rolled out to our customers during the course of this summer where it will then be available across all sites within a tenant. To leverage document conversations, you will need to enable Yammer as your default social network.

 

For additional information, see this post: Make Yammer your default social network in Office 365.Get started today by storing your files in the cloud on OneDrive for Business or SharePoint Online, and harness social collaboration across your company with Yammer. In the coming weeks Document conversations will then be activated in your organization and ready to use! Because we continue to innovate and integrate, subscribe to this blog to get the latest updates across Office 365. And don’t hesitate to reach out to us on Twitter or Facebook with your questions or suggestions.

Christophe Fiessinger @cfiessinger

Frequently asked questions

Q: What file types can be used for document conversations?

A: Document Conversations supports over 30 common file types, including .doc, .xls, .ppt, .pdf, .png, .gif, .mp4, .avi, and more.

Q: Can I see conversations in Office desktop or when I send a document as a mail attachment?

A: No. Currently Yammer threads are visible only in files stored in a SharePoint Online document library or OneDrive for Business in Office 365.

Q: What happens if a file is renamed?

A: Document Conversations uses Yammer’s Open Graph protocol, so when a post is published it also contains a link to the file. This link serves as the glue between the file and its associated conversations. Because the link changes according to the file name, when a file is renamed, the link changes, causing the Yammer conversations to become disassociated from the new file name.

Q: Can I start conversations in a Yammer external network?

A: No. We set our initial goal to build Document Conversations to help teams work better internally. While the new document conversations cannot be started in Yammer external networks today, we are exploring ways to extend collaboration around content to beyond your firewall.