Imagine how much more efficient your developers would be if they didn’t have to guess which features were being used and which could safely be deprecated?
What would the impact on user satisfaction be if exception data replete with usage context were delivered before your users had a chance to complain?
How would the quality of your software improve if your test plans were aligned with actual usage patterns and user preferences in production?
Application analytics is the branch of analytics purpose-built to make these scenarios a reality; to satisfy the “selfish interests” application stakeholders, e.g. development, test, product owners, operations, etc.
Figure 1: Application
analytics enhances both development and operations by providing deep insight into application adoption and user behavior within established development and operations platforms.
Application analytics integrates application usage data, app-centric analytics software and heuristics integrated into development and operations.
The diversity of today’s analytics solutions is (as it should be) customer driven. For example, web analytics’ principle customer is marketing and sales resulting in a focus on page views, clicks and conversions. Web analytics solutions share common:
- Objectives: monetization of web properties,
- Requirements: analysis of visitors, impressions, clicks and conversions, and
- Restrictions: meeting privacy and performance obligations.
In agile parlance, application analytics encompass analytics solutions where the primary customer is one or more application development “personas” who share common objectives, requirements and restrictions.
analytics provides empirical evidence of application usage and end-user behavior that, when properly integrated into a development process, provides:
- Insight into user requirements,
- Validation of development priorities and an
- Objective measure of test plan accuracy and completeness
Examples include:
- The Microsoft customer experience improvement program (CEIP) was “created to give all Microsoft customers the ability to contribute to the design and development of Microsoft products.” CEIP collects information about how Microsoft programs are being used “in the wild”.
- Application adoption and usage metrics within a specific operations framework,
- Production incident alerts from application exceptions, and
- Organizational adoption and productivity analysis connecting application investment to enterprise ROI.
Examples include:
- PreEmptive Analytics Community Edition that gives developers using Microsoft Visual Studio 2012 Professional the ability to create their own CEIP by allowing development and operations to identify and quickly respond to application exceptions that occur in production.
analytics seem obvious, but the details can make it difficult. Collecting, analyzing and acting on application runtime data poses unique challenges both in terms of the kinds of data that need to be gathered and the metrics that measure success.
Effective application analytics implementations must accommodate the diversity of today’s applications and the emergence of cloud, mobile and distributed computing platforms. The following application analytics requirements make plain why narrower analytics technologies should never be expected to fully satisfy development objectives.
is typically far more complex and heterogeneous than what is typically streamed from a web page or portal.
| Data types | Runtime telemetry: the variety, semantics and location of application runtime data |
|---|---|
| Feature | An application feature is not a click. A feature can span one or more methods, incorporate multiple components, run across runtime surfaces and even be implemented multiple times in different languages, e.g. Windows Presentation Foundation (WPF), Microsoft Silverlight and HTML5. Measuring usage and performance of an arbitrarily defined scope is required for monitoring across devices and platforms. |
| Application Data | Many of today’s applications are data-driven where the actual behavior itself is encoded in the data. Knowing what templates, workflows and other “modern” content is being processed can be more valuable than knowing which workflow or rendering engine processed that data. |
| Session | Session information can be defined differently within an app server, a mobile session, within a browser or distributed across all of the above serviced by a cloud-based service. |
| Event | Unhandled exceptions, caught and thrown exceptions, unexpected performance or suspicious user behavior can all constitute a “production event.” |
| Application | Applications are often comprised of multiple components – some on-premises and some service-based. These applications (and their components) are versioned at unpredictable cadences. Calculating the workflow across distributed applications and then reconciling this activity over time and across versions is an applications analytics requirement. |
| Stack | While many applications are run inside a “sandbox”, e.g. mobile, Windows Runtime, Windows Azure – many other applications have full access to the underlying OS and computing metal. Tracking screen resolution, chip manufacturers and hardware availability is often essential to understanding user experience and application behavior. |
| Identity | A users’ identity can be defined and tracked by device ID, IP address, user credentials, software license and more. Application analytics solutions must have the capacity to enforce privacy and security policies at both client and aggregate levels. Ensuring effective data governance is a precursor to effectively analyzing the resulting runtime data. |
| Categories | Runtime architectures and technologies: existing and emerging languages and platforms |
|---|---|
| Architectures and surfaces | Applications are much more than a simple presentation layer and a sequence of user actions. Instrumentation must span client-server, cloud services (public and private), web servlet and mobile platforms, architectures and surfaces. |
| Languages and runtimes | Today’s applications will incorporate managed, native and scripted components including the Microsoft .NET Framework, C++, Java and JavaScript. |
Figure 2: Five functional phases of application analytics implementation.
| DevOps phase | IDE & application lifecycle management (ALM) integration: role and use case driven scenarios |
| 1. Instrumentation | Instrumentation is the logic inside an application that creates the runtime data to be analyzed. Instrumentation can be coded via an API (required for native and scripted apps) or injected post-compile into managed assemblies. |
| 2. Build and deploy | Apps can be built manually, as part of a continuous build process and automated to support cloud platforms. Support for the various manufacturing processes and payload formats is required to ensure efficient and scalable deployments. |
| 3. Runtime data management | Managing runtime data will require scale, governance, and security controls. Application runtime data management requirements shift dramatically across industry, use case and jurisdictional boundaries. |
| 4. Runtime data publishing | Different stakeholders require distinct presentations and analysis; developers, architects, product owners and line of business management bring different perspectives and priorities to the same underlying runtime data. Reports, dashboards, export and programmatic access are all typically required when use cases span sprint planning, customer support and business performance monitoring. |
| 5. Integration | Integration of application analytics into development platforms, e.g. Visual Studio and TFS, operations, e.g. operations manager, and customer relationship management, e.g. Microsoft Dynamics through reports and event scheduling delivers the “last mile” of application analytics’ productivity. |
| Risk management | Restrictions: performance, stability, privacy, and complexity |
|---|---|
| Performance and stability | Collecting, caching, and transmitting runtime data efficiently across devices without performance penalties (when working properly) while not impacting application stability or user experience if/when one or more aspects of the analytics solution should fail. This can be especially challenging when considering special dependencies such as battery life, data plans, network characteristics… |
| Security and privacy | Consumer, business-to-business, and line-of-business applications each come with their own security and privacy obligations. These obligations are further fragmented by industry and by jurisdiction. Application analytics instrumentation, transport and content management must be extensible and able to enforce these requirements on an application-by-application basis. |
| Complexity | Complexity introduces waste and risk – and ultimately resistance to adoption. Integration into existing platforms, processes and methodologies is a requisite to effective application analytics implementations. |
analytics solution that monitors exceptions and creates or updates work items inside Team Foundation Server (TFS) based upon user-defined thresholds.
PA for TFS CE is designed to track unhandled exceptions on applications running on the .NET Framework and Java runtimes. Support for the five phases of application analytics as follows:
| DevOps phase | PreEmptive Analytics for TFS Community Edition |
|---|---|
| 1. Instrumentation | Instrumentation is accomplished with Dotfuscator Community Edition. Unhandled exception monitoring with optional opt-in and user-feedback is supported for the .NET Framework, Silverlight, Microsoft Windows Phone and XNA applications. An API for Java applications is also available as a free download that includes support for Android. Support for native code, JavaScript and Java is provided by PreEmptive Solutions. |
| 2. Build and deploy | Dotfuscator Community Edition is interactive. The command line interface and MSBuild support is available with Dotfuscator Professional from PreEmptive Solutions. |
| 3. Runtime data management | A server-side data collector is included withVisual Studio Team Foundation Server 2012. The collector endpoint is referenced via a URL embedded into the application being monitored as part of the instrumentation phase above. The collector can sit next to the TFS server, on an entirely different server, and even inside Microsoft Windows Azure. |
| 4. Runtime data publishing | An aggregator service is also included with TFS in Visual Studio 2012 that polls the collector endpoint. When user-defined thresholds are met, the aggregator will create (or update) a Production Incident work item inside TFS in Visual Studio 2012. |
| 5. Integration | In Visual Studio 2012, TFS work items created by PA for TFS CE are tracked, assigned, prioritized and reported against as any other first class work item type. |
Figure 3: Finding PreEmptive Analytics inside Visual Studio 2012 off of the tools menu
Figure 4: Instrumentation. Inside Dotfuscator CE, adding the setup attribute identifies the collector endpoint for runtime data. The endpoint can be on-premises next to a TFS server or hosted on Windows Azure remotely.
Figure 5: Visual Studio 2012 integration. Production Incident work items are surfaced inside Visual Studio automatically when volume thresholds are met. In this “All Items” query, you can see the type of exception, how many exceptions of this type have been detected and on how many machines. You can see more detail on this work item below including a stack track as well as the work item’s assignment, prioritization and classification.
Figure 6: Reporting. One example of summary charts included with PA for TFS CE shows the status of all open incidents.
.
| Use case | Community Edition | Professional |
|---|---|---|
| Track unhandled on .NET Framework and Java applications | Yes | Yes |
| Automatically create and update TFS work items in Visual Studio 2012 | Yes | Yes |
| Provide opt-in and user-feedback options at runtime | Yes | Yes |
| Support Visual Studio 2010 | Yes | |
| Track caught and thrown exceptions | Yes | |
| Support custom data and extensible rule and work item definitions | Yes | |
| Support JavaScript and native application monitoring |
Yes | |
| Measure feature and session usage | Yes |
- Development has unique requirements that are not being met by web, BI or other non-development centered analytics solutions.
- Application analytics offers specific capabilities designed to meet development and operation’s needs.
- Visual Studio 2012 offers integrated application analytics “out of the box” with opportunities to extend these capabilities through integration and partner options.
You can visit these sites to learn more:
SharePoint Samurai 