Telemetry, Planning and Customer Obsession

As the newest PM in the Visual Studio team, it's an exciting time to be joining a project of this size and scale. We've just shipped Visual Studio 2015 and are already rapidly shifting towards planning the next couple of releases in detail. Even as we complete the last few elements of shipping Visual Studio 2015, we're triaging all the work items that didn't get into this version, identifying the big themes and scenarios we want to tackle for future releases, and initiating user research, prototyping and value testing of various concepts.

Telemetry has always played a large part in product planning and beta testing for Microsoft: for those who've opted in to share feedback through the Visual Studio Experience Improvement Program (Help / Customer Feedback Options), we receive certain anonymized data on which options and templates they are using most frequently. (Incidentally, there's nothing secret about what we collect - with feedback enabled, turn on Fiddler and enable HTTPS capturing and you'll occasionally see POSTs of the JSON data.

But that kind of background telemetry doesn't tell the whole story - it doesn't capture the emotional response you might have when using our product. So we look out for lots of other signals that can help us understand how our work is landing. Some examples of those signals include Twitter, UserVoice, and the "Send a Smile" / "Send a Frown" icons in Visual Studio itself. All this gets aggregated together in Azure, and we apply machine learning to help us categorize this data in different ways.

The result of this harvesting process is fed into a unique site called vs-feedback that lets us group and prioritize this feedback and turn it into work items or bugs if appropriate. Right now as people install the RTM build of Visual Studio 2015 for the first time, we're literally scanning through hundreds of these items an hour; in the past this quantity of feedback would have been unmanageable, but now we're able to get a better measure of what's working and what we need to fix. It's really exciting to see.

Triaging feedback from Visual Studio 2015

The screenshot above shows a couple of anonymized 'frowns', and you can see that even in their unreviewed state, these have automatically been put in the right area path in VSO and that there's telemetry available to help us understand the context of the problem. We can group multiple comments together, add comments or even respond directly via email where we need more information and you've shared those details with us.

So - thank you for your feedback. We read it all - the good, bad and the ugly - and it's making us more responsive and better able to anticipate the needs you have. Keep letting us know how we can make Visual Studio the best developer toolset for all your needs. We're listening!