In Depth : Top 5 DevOps Best Practices for Achieving Security, Scalability and Performance

 

 

As Companies and Consultants like me continue its look into how best to invest DevOps related time and money, the focus is now changing in order to relate back some of the various best practices experts in the industry have suggested will help create scalable, secure and high performance deployments.

Here are five tips and best practices that have emerged from the hands on experiences of the industry’s foremost experts that I use with every DevOps Consulting project :

 

1. Be vigilant of overall security risk

Reuven Harrison, CTO and Co-Founder of Tufin emphasizes the growing complexities of networks. He says that increased adoption of virtualization, cloud, BYOD and emerging technologies like software defined networks (SDNs) means that networks are becoming more complex and heterogeneous, and so do the security risks.

“As SDN and network virtualization continues to mature, the only way to manage these networks with any degree of efficiency and security is to automate key management functions,” he says. “That is the premise of DevOps,.

 

But DevOps must include security as a key component because without it, the volume and pace of network change that technologies such as SDN and virtualization introduce will skyrocket the level of IT risk in the environment.”

The big challenge is that to date, security has been considered an afterthought, and security organizations are considered to be business inhibitors, telling organizations what can’t be done, instead of how to do things securely. 

It is a cultural issue that requires security, developers, and operations teams to foster a level of trust and collaboration that does not yet exist. The only way to do this is incrementally, and with vigilance.

 

2. Watch changes in security risk

Torsten Volk, VP Product Management, Cloud for ASG Software Solutions says that it is important to think of DevOps as a collaborative mindset and process that leads developers and IT operations to a faster and more efficient way of deploying, operating and upgrading applications.

“Each new release comes with the same set of security considerations as it did the time before DevOps,” he says. “However, when new releases are delivered at a much higher cadence, security has to also be an ongoing point of focus.”

DevOps tools help in this regard by proactively ensuring consistent configuration of infrastructure and software components. Even more, these tools can be used to automatically remediate security concerns by constantly validating the proper application of security best practices and taking automatic countermeasures.

While this latter scenario might sound advanced, it is the endpoint that every DevOps team should aspire to reach.

 

3. Pay attention to scalability

According to Aaron M. Lee, Managing Principal of DevOps at Pythian, there are two kinds of scalability that DevOps engineers tend to address: application and organization.

“An app’s scalability is really a question of how long it takes and how much it costs to build and operate a system that successfully delivers a certain level of concurrency; one that matches or exceeds user demand over some time period,” said Lee.

“Estimating answers to these questions is a critical success factor for many companies, and the ability to do so often goes unrecognized until it’s too late.”

Lee says that scalability is everyone’s problem. Business and technology folks have to agree on the right balance of functionality, time to market, cost, and risk tolerance.

You need the right measurable objectives, including how many users, and how many concurrent requests over those endpoints for a demand pattern.

 

4. Strive for ease of use 

– DevOps is about automation and repeatability. Dr. Andy Piper, CTO of London based Push Technology says this requires configurable virtual environments, and lots of them. “To scale, you need to automate,” he says.

“So, make sure you are using tools such as Puppet and Chef to automate the building and configuration of VMs. Similarly, make sure you have the horsepower to back this up either in-house, which is more tricky to dynamically scale, or in the cloud if your product is amenable to that.”

At the end of the day, making a product easy to install, configure and run will make the whole DevOps process much easier.

5.Manage your gateways

Susan Sparks, Director, Program Management for InfoZen’s Cloud Practice says that while the new goal is to build the best culture between development and operations teams, it is still good to keep some gates between the functions to ensure the production environments remain stable.

“Our teams are structured such that we have operations personnel included in development discussions and daily scrums so the operations teams understand what will be changing in the various future releases,” she says. “The operations team maintains responsibility for the stability of the production operation. We found that this approach has worked well for us.

We recommend using automation in both testing and operations. Our integration testing has allowed us to find issues prior to them reaching production, and our operations automation allows for cost efficiencies and better quality operations.

With automation, fewer people touch the production environment, which significantly reduces human errors. This also helps with security posture, as less people have a need to touch the production environments.”

DevOps isn’t hard. What is hard is tackling the challenges that arise when an organization is not taking a DevOps approach to integration, development and deployment , and I think its very difficult to try and argue this point with me, especially in the SA ICT Industry.

By adopting a DevOps approach, and heeding these five tips, a successful DevOps environment is just an implementation or two away.

 

 

How To : Use Azure BizTalk Services to Integrate with an On-Premises SAP Server

Microsoft Azure BizTalk Services provides a rich set of integration capabilities enabling organizations to create hybrid solutions such that their customer or partner facing applications are hosted on Azure, while the data related to customers or partners is stored on-premises using LOB applications.

To demonstrate how to integrate applications with an on-premises LOB application using BizTalk Services, let us consider a scenario involving two business partners, Fabrikam and Contoso.

Business Scenario

Contoso sends a purchase order (PO) message to Fabrikam in an X12 Electronic Data Interchange (EDI) format using the PO (X12 850) schema. Fabrikam (that uses an SAP Server to manage partner data), accepts PO from its partners using the ORDERS05 IDOCS.

To enable Contoso to send a PO directly to Fabrikam’s on-premises SAP Server, Fabrikam decides to use Azure’s integration offering, BizTalk Services, to set up a hybrid integration scenario where the integration layer is hosted on and the SAP Server is within the organization’s firewall. Fabrikam uses BizTalk Services in the following ways to enable this hybrid integration scenario:

1. Fabrikam uses Microsoft Azure BizTalk Services SDK to create a BizTalk Service project. The project includes a XML One-Way Bridge to send messages to a relay endpoint, which in turns sends the message to the on-premise SAP system.
2. Fabrikam uses the BizTalk Adapter Service component available with BizTalk Services to expose the Send operation on ORDERS05 IDOC as an operation using Service Bus relay endpoint. The XML One-Way Bridge sends messages to this relay endpoint. Fabrikam also creates the schema for Send operation using BizTalk Adapter Service and includes the schema as part of the BizTalk Service project.

   Note
   A Send operation on an IDOC is an operation that is exposed by the BizTalk Adapter Pack on any IDOC to send the IDOC to the SAP Server. BizTalk Adapter Service uses BizTalk Adapter Pack to connect to an SAP Server.

3. Fabrikam uses the Transform component available with BizTalk Services to create a map to transform the PO message in X12 format into the schema required by the SAP Server to invoke the Send operation on the ORDERS05 IDOC.
4. Fabrikam uses the Microsoft Azure BizTalk Services Portal available with BizTalk Services to create and deploy an EDI agreement under the BizTalk Services subscription that processes the X12 850 PO message. As part of the message processing, the agreement also does the following:
   1. Receives an X12 850 PO message over FTP.
   2. Transforms the X12 PO message into the schema required by the SAP Server using the transform created earlier.
   3. Routes the transformed message to the XML One-Way Bridge that eventually routes the message to a relay endpoint created for sending a PO message to an SAP Server. Fabrikam earlier exposed (as explained in bullet 1 above) the Send operation on ORDERS05 IDOC as a relay endpoint, to enable partners to send PO messages using BizTalk Adapter Service.

Once this is set up, Contoso drops an X12 850 PO message to the FTP location. This message is consumed by the EDI receive pipeline, which processes the message, transforms it to an ORDERS05 IDOC, and routes it to the intermediary XML bridge. The bridge then routes the message to the relay endpoint on Service Bus, which is then sent to the on-premises SAP Server. The following illustration represents the same scenario.

How to Use This Article

 

This tutorial is written around a sample, SAPIntegration, which is available as part of the download (SAPIntegration.zip) from the MSDN Code Gallery. You could either use the SAPIntegration sample and go through this tutorial to understand how the sample was built or just use this tutorial to create your own application.

This tutorial is targeted towards the second approach so that you get to understand how this application was built. Also, to be consistent with the sample, the names of artifacts (e.g. schemas, transforms, etc.) used in this tutorial are same as that in the sample.

The sample available from the MSDN code gallery contains only half the solution, which can be developed at design-time on your computer. The sample cannot include the configuration that you must do on the BizTalk Services Portal on Azure.

For that, you must follow the steps in this tutorial to set up your EDI bridge. Even though Microsoft recommends that you follow the tutorial to best understand the concepts and procedures, if you really wish to use the sample, this is what you should do:

• Download the SAPIntegration.zip package, extract the SAPIntegration sample and make relevant changes like providing your service namespace, issuer name, issuer key, SAP Server details, etc. After changing the sample, deploy the application to get the endpoint URL at which the XML One-Way Bridge is deployed.
• Use the BizTalk Services Portal to configure the Receive settings as described at Step 5: Create and Deploy the EDI Receive Pipeline and follow the procedures to route messages from the EDI Receive bridge to the XML One-Way Bridge you already deployed.
• Drop a test message at the FTP location configured as part of the agreement and verify that the application works as expected.
  • If the message is successfully processed, it will be routed to the SAP Server and you can verify the ORDERS IDOC using the SAP GUI.
  • If the EDI agreement fails to process the message, the failure/error messages are routed to a relay endpoint on Service Bus. To receive such messages, you must set up a relay receiver service that receives any message that comes to that specific relay endpoint. More details on why you need this service and how to use it are available at Step 6: Test the Solution.

Steps in the Solution :

 

• Step 1: Set up Your Computer
• Step 2: Expose a Relay Endpoint to Invoke Operations on ORDERS05 IDOC
• Step 3: Transform the X12 850 PO Message to the ORDERS05 Message
• Step 4: Create and Deploy the XML Bridge
• Step 5: Create and Deploy the EDI Receive Pipeline
• Step 6: Test the Solution

Step 1: Set up Your Computer

This topic provides you with instruction and pointers to set up your computer on which you will perform the steps to set up the hybrid integration scenario described in Tutorial: Using Azure BizTalk Services to Integrate with an On-Premises SAP Server. You must do the following to set up your computer:

• Install Microsoft Azure BizTalk Services SDK. You can download the installer from http://go.microsoft.com/fwlink/?LinkId=235057. You use this SDK to configure and deploy the XML One-Way Bridge that sits between the EDI agreement and the relay endpoint.
• Install BizTalk Adapter Service. You use this to expose the Send operation on an IDOC as a relay endpoint on Service Bus.You can download the installer from http://go.microsoft.com/fwlink/?LinkId=235057. Refer to the BizTalk Services installation guide at http://go.microsoft.com/fwlink/?LinkId=237092 to install the software prerequisites for BizTalk Services SDK and BizTalk Adapter Service.
• Install the WCF LOB Adapter SDK. This is required for installing the Adapter Pack on the computer.
• Install the Adapter Pack. This contains the SAP adapter that is required to establish connectivity to an SAP Server and for exposing SAP artifacts as operations.
• Install the SAP client libraries. The SAP adapter needs these libraries to connect to an SAP Server. For information on where to install the SAP client libraries from, refer to the Adapter Pack installation guide (BizTalkAdapterPack_InstallationGuide.htm) at http://go.microsoft.com/fwlink/?LinkId=240161.
• Download and extract the EDI message schemas (MicrosoftEdiXSDTemplates.zip) available at http://go.microsoft.com/fwlink/?LinkId=235057. This contains the X12 850 Purchase Order message schema that is required for the business scenario we use for this article.

After you have finished installing and downloading these components, you are ready to start setting up the business scenario.

Step 2: Expose a Relay Endpoint to Invoke Operations on ORDERS05 IDOC

This topic has not yet been rated – Rate this topic

Updated: November 21, 2013

There are two main steps required to expose an SAP artifact as an operation that can be invoked by sending a message over Service Bus – create an LOB Target and an LOB Relay.

• An LOB Target defines how an Azure application communicates to the Line-of-Business (LOB) system. The LOB Target controls the LOB system connection URI, the operation to perform, and the connection credentials.
• An LOB Relay is a WCF service running within an organizations firewall and listens to a relay endpoint on the Service Bus. As the name suggests, the LOB Relay acts as a relay between the Service Bus relay endpoint and the LOB system. It receives the message at the Service Bus relay endpoint and passes it on to the relevant LOB system using the LOB Target configuration.

For more information, see BizTalk Adapter Service Architecture. In this topic, we will create an LOB Target and an LOB Relay to expose the Send operation on the ORDERS05 IDOC.

To create an LOB Target and LOB Relay

1. Open Visual Studio (as an administrator), create a new BizTalk Service project, and name it SAPIntegration.
2. You first start with adding a BizTalk Adapter Service server. This is the server where you installed the Runtime component of BizTalk Adapter Service. To add a BizTalk Adapter Service server, from the Server Explorer in Visual Studio, right-click BizTalk Adapter Services, and select Add BizTalk Adapter Service. In the Add BizTalk Adapter Service dialog box, enter the URL of the WCF service that monitors that Service Bus relay service, and then click OK.

   Because you have all the components of BizTalk Adapter Service installed on the same computer, the URL for that service will be http://localhost:8080/BAService/ManagementService.svc/.

   Note
   If you had installed BizTalk Adapter Service Runtime component on a separate computer, you would have replaced ‘localhost’ in the above URL with the name of that computer.

3. In this tutorial we are creating an application to integrate with SAP, so we must add an SAP target. Expand the newly added server, expand LOB Types, right-click SAP, and select Add SAP Target.

   The Add a Target wizard starts. Perform the following steps to create an LOB Target.

   1. Read the information on the Before You Begin page, and then click Next.
   2. On the Connection Parameters page, specify the details for the SAP Server to connect to and the credentials to use for the connection. Click Next.
   3. On the Operations page, expand the ORDERSO5 IDOC category (under \IDOC\ORDERS\). There are several versions of the IDOC available. For this tutorial, we’ll select ORDERS05.V3(700). Expand this IDOC, select Send, and then click the right arrow to add it to the Selected Operations box.

      Click Next.

   4. In the Runtime Security page, specify the security mechanism to be used by the LOB Server to authenticate the target resource when a message arrives from a client. For this tutorial, select Fixed Username and specify the credentials to connect to the SAP server.
   5. On the Deployment page, you create an LOB Relay and an LOB Target to provide connectivity to your on-premise LOB applications from the cloud.

      Select the Create new option to create a new relay and provide the following values:

      Name	Description
      Namespace	Specify the Service Bus namespace on which the LOB relay endpoint is created.
      Issuer name	Specify the issuer name for the Service Bus namespace
      Issuer secret	Specify the issuer secret for the Service Bus namespace
      Relay path	Specify a name for the relay. For this tutorial, enter sapintegration01.
      Target sub-path	Enter a sub-path to make this target unique. For this tutorial, enter orders.

      The Target runtime URL read-only property displays the URL where the relay is deployed on Service Bus. This is the path where you could send a message to be inserted into the on-premises SAP Server. In our scenario, this is where the bridge sends the message.

      Click Next.

   6. On the Summary page, review the values you specified in the previous steps, and then click Create.
   7. When the wizard completes, click Finish.

      In Visual Studio Server Explorer, you now have an entry under the SAP node. This represents the relay endpoint created in Service Bus to relay PO messages coming from the cloud to the on-premises SAP system.

To add schemas

1. After adding the relay endpoint to an SAP system, you must add schemas that to send ORDERS05 PO messages to the SAP server. To add the schemas, right-click the relay endpoint and select Add schemas to SAPIntegration. In the dialog box, do the following:
   • Enter a filename prefix that will be included in the name of each schema file that is generated. For this tutorial, specify this as SAPIntegration_.
   • Enter a folder name that will be added to your solution under which all the schemas will be added. For this tutorial, specify the folder name as LOB Schemas.
   • Enter the credentials to connect to an SAP system.

   Click OK. The schemas are added to the project under an LOB Schemas folder.

To use the LOB Target

1. Right-click anywhere on the BizTalk Service project design surface, select Properties and update the BizTalk Service URL property to include your BizTalk Services name. This is the name that you provided in Azure Management Portal while provisioning the BizTalk Services.
2. Set the security property for the relay endpoint.
   1. Right-click the LOB Target in Server Explorer and select Properties.
   2. In the Properties grid, click the ellipsis (…) against the Runtime Security property.
   3. In the Edit Security dialog box, select Fixed Username and specify username and password to connect to the SAP Server.
   4. Click OK.
3. Drag and drop the LOB Target onto the design surface. Note the Entity Name property of the LOB Target. The default value is Relay-Path_target-sub-path. If using the examples above, it will be sapintegration01_orders.
4. Open the .config file for the LOB Target, which typically has the naming convention as YourRelayPath_target-sub-path.config. Specify the Service Bus issuer name and issuer secret, as shown below:
     <sharedSecret issuerName="owner" issuerSecret="issuer_secret" />
   
   

   Save changes to the config file.

 

Step 3: Transform the X12 850 PO Message to the ORDERS05 Message

Both the X12 850 schemas and ORDERS05 schemas are pretty complex and require functional expertise in the respective domains to understand and create maps between the two schemas.

While you already generated the schema for ORDERS05 IDOC, you can get the schema for X12 PO message (X12_00401_850.xsd) from the MicrosoftEdiXSDTemplates.zip that you must have downloaded and extracted before. You must add the X12_00401_850.xsd schema as well to the SAPIntegration project.

Creating a transform between the X12 850 PO and ORDERS05 PO requires functional domain knowledge of both the X12 schema and the ORDERS05 schema.

Only then one can identify which field in the X12 schema maps to which field in the ORDERS05 schema. In this tutorial, we do not get into such details and instead use an existing transform (AzureTransformations.trfm) between these two schemas. This transform is available as part of the SAPIntegration project that you can download from the MSDN Code Gallery.

To include the transform in the BizTalk Service project, right-click the project name, click Add, click Existing Items, and then navigate to the location where you downloaded the SAPIntegration sample from the MSDN Code Gallery. Select the AzureTransformations.trfm and then click Add.

Step 4: Create and Deploy the XML Bridge

In this topic, you create an XML One-Way Bridge that will act as a connector between the EDI Receive bridge and the relay endpoint for the ORDERS05 IDOC in SAP. After configuring the bridge, you connect it to the SAP relay endpoint, and then deploy the solution.

To configure the XML Bridge

1. In the SAPIntegration project, from the Solution Explorer, double-click the MessageFlowItinerary.bcs file to open the bridge configuration surface.
2. Right-click anywhere on the BizTalk Service project design surface, select Properties, and update the BizTalk Service URL property to include your BizTalk Services name. This is the name that you provided in Azure Management Portal while provisioning the BizTalk Services.
3. From the Toolbox, drag and drop the XML One-Way Bridge component to the bridge design surface.
4. Right-click the XML One-Way Bridge, select Properties, and change the value for Entity Name and Relative Address properties to B2BConnector. As a result, the complete endpoint URL where the bridge is deployed, which is shown in the Runtime Address property, will resemble https://<mybiztalkservicename>.biztalk.windows.net/default/B2BConnector. This is where the EDI Receive bridge sends the ORDERS05 PO message.
5. Double-click the XML One-Way Bridge to open the Bridge Configuration design surface. Because this bridge only routes the message from the EDI Receive bridge to the relay endpoint, there’s not much configuration required for each stage in the bridge stage other than specifying the message types of the message that this bridge routes. To specify the message type, on the XML One-Way Bridge design surface, within the Message Types box, click the add icon [ ] to open the Message Type Picker dialog box.
6. In the Message Type Picker dialog box, from the Available message types box, select the schema for the request message and then click the right arrow icon [ ], and then click OK. For this tutorial, select the Send schema (http://Microsoft.LobServices.Sap/2007/03/Idoc/3/ORDERS05//700/Send). The selected schema should now be listed under the Request Message Type box.
7. Save the bridge configuration.

To connect the bridge to the relay endpoint

1. In the SAPIntegration project, from the Toolbox, select the Connection component, and connect the XML One-Way Bridge component with the SAP relay endpoint you already added in Step 2: Expose a Relay Endpoint to Invoke Operations on ORDERS05 IDOC.
2. Set the filter condition on the connection. The routing condition for this scenario is to route all messages to the LOB Target. To do so, select the connecting line, and from the Properties grid, click the ellipsis (…) against the Filter Condition property, and then select Match All. This ensures that all messages that come to the bridge are routed to the relay endpoint.
3. Set the Route Action property on the connection. Before you set the route action, we must understand why it is required. The message sent from the EDI receive bridge to the relay endpoint must have the Action SOAP header set on it. This header defines what operation must be performed on the SAP system. The message that comes from the EDI receive pipeline does not have this header set. Hence, in this intermediary XML bridge, you set the route action on the message before it is sent the relay endpoint. As part of the route action, you add the required header on the message. Perform the following steps to set the route action.
   1. Find out the value that will be set for the Action SOAP header message. To do so, right-click the SAP relay endpoint from the Server explorer, and from the Properties grid, expand Operations, and copy the value. For this tutorial, the value is http://Microsoft.LobServices.Sap/2007/03/Idoc/3/ORDERS05//700/Send.

      

   2. Go back to the bridge configuration surface, select the connection between the bridge and the SAP relay, and from the Properties grid, click the ellipsis (…) against the Route Action property. In the Route Actions dialog box, click Add to open the Add Route Action dialog box. In the Add Route Action dialog box, do the following:
      • Under Property (Read From) section, select Expression and specify the value that you copied earlier.

        Important
        Make sure you specify the value for Expression within single quotes.

      • Under Destination (Write-To) section, set the Type to SOAP and the Identifier to Action.

        

      • Click OK in the Add Route Action dialog box to add the route action. Click OK in the Route Actions dialog box and then click Save to save changes to an Enterprise Application Integration project.
4. Save the project. The final bridge configuration resembles the following:

   

To deploy the solution

1. In Visual Studio, right click the SAPIntegration solution, and then click Build Solution.
2. Once the build succeeds, right click the SAPIntegration solution, and then click Deploy Solution.
3. In the deployment window, the Deployment Endpoint is a read-only property and the value is derived from the BizTalk Service URL/Namespace set in the message flow surface. However, you must provide the ACS Namespace for BizTalk Services, Issuer Name, and Shared Secret.
4. Click Deploy. The Visual Studio Output pane displays the deployment progress and result. The URL where the bridge is deployed is also displayed in the Output pane. For this tutorial, the bridge is deployed at http://<mybiztalkservicename>.biztalk.windows.net/default/B2BConnector.

 

Name(required)

Email(required)

Agency / Company(required)

Job Spec(required)

Position

Company Position is at

Submit

Δ