One of the primary goals for a CloudScape based discovery project is to organize all of the discovered (and licensed) workloads into the various Application Stacks that they support. Once this organizational work has been completed, the platform is better positioned to provide dependency, Cloud cost, resource requirements, and other information from this application centric point of view. This supports more detail planning and analysis for Strategic IT Planning, Cloud and Datacenter migration, Security reviews, and other Discovery use cases.
Application Review, in this context, is the process of reviewing and refining the automatically generated application stacks (AutoApp_xxx), taking system dependencies, input from application teams, best practices, and other factors into account.
There is not necessarily one way to approach this process, however this document will outline some best practices for application review and refinement. The end result of this effort should be a set of application stacks that most closely represent the source environment and provide the vantage point needed to support the desired outcome(s). In other words, the dependency, Cloud cost, resource requirements, and other insights will be easily viewed from the perspective of these application stacks.
Viewing Connectivity in CloudScape 2.0
All of the processes outlined below are completed using the following two pages of the CloudScape 2.0 portal:
- Add Intelligence >> All Applications
- Add Intelligence >> View Individual Application Stacks
Connectivity can be visualized by working in and expanding the right side frame on both of these pages. This includes providing the ability to visualize critical or all connectivity as well as viewing connectivity internal to the application stack, external to the application stack, or both (all). When viewing the Add Intelligence >> View Individual Application Stacks page you also have two additional options for viewing and navigating connectivity information:
- View All Connections (top left) – Provides a table summary of all connectivity and includes three levels of drill down
- Dependent Stacks (from visualization control) – Provides a visualization of all stack-to-stack connectivity relative to the individual stack you are currently viewing.
The View All Connections can be an important way to view connectivity because it is NOT limited to connectivity between licensed nodes. The visualizations are limited to connectivity between licensed nodes with the exception of the RSG-Out Of Scope Services
In any visualization you can select a segment to view the details specific to that relationship whether it be workload to workload, or application stack to application stack.
Verifying Scope with RSG-Out Of Scope Services
The initial and primary mechanism of setting the scope for the CloudScape analysis is the discovery phase. During this phase you will select the subnets to include in discovery and ensure you have access to the workloads contained within the boundaries of those subnets. However, additional scope validation can be performed post-licensing through a review of a special auto-generated application stack called RSG-Out Of Scope Services.
The following includes the steps needed for a detailed review of the RSG-Out Of Scope Services
group. The level to which you will need to perform this is dependent on the need for true “discovery” of the environment. In other words, if you are leveraging the discovery aspects of the platform to find systems you may not know about or have documentation on, then you would perform this validation in detail. If you are running CloudScape against a known subset of the environment, then a more cursory review of RSG-Out Of Scope Services
may be sufficient.
Iterate through the RSG-Out Of Scope Services
group and verify whether or not the hosts in this group should be brought into licensing/scope. IP addresses and / or device names will appear in this group if they are offering one of the following services: MSSQL, MySQL, oracle, NFS, or iSCSI, but are either not licensed, or their IP is out of scope (not included in the subnet ranges requested to be scanned). Our default advice is that everything in the environment should be licensed during the burst, so working through this group can be an important first step in ensure a complete scope. Once you license the devices in the RSG-Out Of Scope Services
Out of Scope Services group (or decide that they are to remain out of scope), you should re-run auto-grouping to make sure that the group is empty – or that all hosts can be safely ignored. *Please note that you should routinely re-run auto-grouping and check this group as new services may come online at any time. This step successfully resolves any devices that are OFFERING
those 5 services but are not licensed or not in scope.
Make sure that there are not any CONSUMERS of services that are not licensed. (FYI, nothing can go into an application stack unless it is licensed). This means going through the individual application stacks and checking their connectivity to “locations”. A “location” is a set of subnets that are bound together in a physical location. RISC Networks’ patent-pending algorithm determines which subnets should be grouped together into a location. A location will appear in the connectivity table as “LocationX”. This algorithm depends on data from the network devices (routers and switches) to make a determination. In the absence of this specific network data from routers/switches, all devices are placed into our default locations:
- Risc-multicast (any multicast addresses)
- Risc-unknown-private (RFC 1918 addresses that are not already assigned a location – which would be all RFC 1918 addresses if there are no routers/switches in scope)
- Risc-unknown-internet (all Class A,B, and C addresses that are non-RFC1918 and are not part of a location – again likely all – although we do have a few defined internet locations – such as Amazon EC2 IP Ranges)
To check consumption of important services, click “View All Connections” when viewing the RSG-Out Of Scope Services
stack. This will bring up a table of all group- to-group and group-to-location connectivity. Sort this table by the destination group such that the RSG-Out Of Scope Services
group is the destination. Once sorted, on the far right of the table you will see a “View Details” link, you should click this link for the first location listed in the source column.
This will bring up another table that you should then click the button in the top left of the table to show a protocol view. Review the listed protocols to understand connectivity and if there are important services being consumed outside of this group. You want to verify that the traffic is not on a well-known port or protocol name, such as MSSQL, oracle, MySQL, etc…
To understand particular IPs that are communicating on a protocol select the “View Details” link on the far right of the table, this will bring up another table that lists out the specific IP addresses. You should verify this list to understand if those items currently not licensed should be brought into scope. Repeat these steps for every location group within the first group to group/location table.
If you have a large environment where going through all the AutoApp stacks would be too time consuming, then the best way to determine consumers of services out of scope is to run the auto-grouping with the 95th group and view this information against the RSG-95th Highly Connected Stack
. The RSG-95th Highly Connected Stack
is the top 5% of hosts by connectivity in the network. In other words, it is the list of servers that talk the MOST. The “hosts with the most” we like to say.
Verify whether or not the out of scope IPs that are CONSUMERs or are OFFERING important services described above should be licensed. The group of the service host (the one offering the service) is inconsequential. The important thing is to determine if all of those hosts should be brought into scope and licensed as they may be front end servers or other applications that are accessing servers you are focused on. Once this process has been repeated, and there are not major connections in or out of the 3 major groups (Isolated Devices
, RSG-95th Highly Connected Stack
, and RSG-Out Of Scope Services
), you can move on to application review.
Application Stack Review and Refinement
First, it is important to understand what your starting or initial reference point will be. For most organizations, this will be the set of Application Stacks that are automatically generated by the platform using the Build App Stack
function (found in Add Intelligence >> All Applications).
The CloudScape platform will automatically generate a set of Application Stacks based on the inventory, process, and connectivity data collected from each workload. This includes a set RISC Service Groups (RSG-<ServiceName>)
that consist of infrastructure services such as Active Directory, AntiVirus, and Configuration Management. RISC Service Groups are based on the applications and processes that are running on the workloads as opposed to connectivity data. This also includes a set of Application Stacks that are automatically created based on observed dependency (connectivity) data. These groups are named AutoApp_XXX where XXX is an arbitrary identification number that is appended to each group. The number of AutoApp Application Stacks will vary depending on the size of the environment.
Only licensed workloads can be placed in an Application Stack.
Some organizations may prefer to use internally documented Application-to-Server mappings as the baseline to begin this process (e.g. CMDB). In this case the Application Stacks must be manually created and workloads are assigned to them as documented. The CloudScape 2.0 portal has several UI features that allow for bulk manipulation of server workloads and their assignment to specific application stacks. This includes the ability to filter server lists based on naming convention, shift select, and right click to assign to an existing or new Application Stack.
Use the Graph in Add Intelligence >> All Applications to switch the view from Application Stacks to servers. You can sort and filter on any column in order to facilitate bulk movement of workloads to Application Stacks. Since Application Stack
is one of the available columns this can also help to locate systems in the platform.
No matter which methodology you use as your initial reference point, you will likely use a combination of connectivity data, internal documentation (CMDB), and “tribal knowledge” during the Application Review and Refinement process.
Application Review Process (using Build App Stacks)
There are two techniques that can be used to review and refine automatically generated Application Stacks: 95th Resolution
and Visualization with Movement
. We recommend having the application owners participate in either approach as they can more rapidly iterate over known hosts based on their knowledge of the environment.
The easy way to determine which approach would work best for you is to first Build App Stacks
without the 95th group. This will, generally, generate several application stack groups (AutoApp-xxx). Review these groups by selecting them and viewing the number of hosts in a group. If a group appears to have over 10 hosts, then visualize the group. This will visualize connectivity for that group, if the visualization is unworkable because there are too many connections then you should proceed with the 95th Resolution approach, otherwise the initial auto-grouping was sufficient and you can continue with the Visualization with Movement approach.
95th Resolution Approach
This approach utilizes the RSG-95th Highly Connected Stack
. By running the auto-generation of application stacks with the 95th Percentile option selected, you will generate a group called RSG-95th Highly Connected Stack
– which will be a small subset of heavily connected nodes.
Review and disposition each host in the 95th Group…”What is this host used for?”. Typically, you will have knowledge of these hosts since they are so heavily connected. You should disposition each host within the group into one of two categories:
These would be hosts that are offering a service to the rest of your environment. We have attempted to capture some of these in our RISC Service Group(s) such as exchange, Citrix, NFS, etc, but we have not scripted for every service that can be offered. Something like antivirus, authentication, logging, etc. would be examples of services. To disposition a Service, create a new application stack (antivirus for example). Once this is complete move that host by right clicking the host and selecting Edit Stack Membership.
These would be hosts that are a part of an application stack. These could be things like a shared DB. There are two ways to disposition a member of an application stack. For environments with heavily shared architectures (think large jboss layers or large shared DBs) proceed to Step Two, otherwise disposition Application Stacks by simply leaving them in the 95th Group and proceeding to step two.
This is only for environments with heavily shared architectures as noted above, otherwise skip this step. Create a new application stack (Shared DB for example). Move the server(s) that are part of that cluster (or it might just be a single db server) into the new application stack you just created. You can immediately view the connections of that application stack and see where that server or cluster is communicating. You would likely see other AutoApp-xxx stacks. If this communication is important (such as DB traffic), it is an indication that these application stacks are likely front ends running off of the DB. If the DB is shared, you can leave it there, and rename the front end groups (e.g. AutoApp-xxx becomes Main SAP FrontEnd) or you can move the servers from AutoApp-xxx into the new group you built for the server(s) you pulled from the 95th group.
You repeat this process until the 95th group is empty. If you want to do a second round, you can re-run the auto-grouping and select the 95th group again. Since the original top 5% were grouped into Application Stacks, a new set of top 5% hosts will be identified and included in the 95th
group, and can now be rationalized and grouped using the same methods as above. This is a very effective approach in environments with heavily shared architectures (think large jboss layers or shared DBs)
Re-run the Auto Generate Groups algorithm without the 95th group. This will place any remaining workloads, not in a confirmed Application Stack, into an AutoApp_XXX stack based on connectivity. Proceed to the Visualization with Movement
Any application stacks not confirmed upon re-execution of the algorithm will be deleted and rebuilt. Only those Application Stacks that have been confirmed will remain intact.
Visualization and Movement Approach
This approach works well in environments that are not heavily shared (no large DB farms or jboss/tomcat layers). We recommend having the application owners participate in this process as they can more rapidly disposition hosts based on their knowledge of the environment.
Run auto-grouping without the 95th group. This can sometimes result in large vertical stacks that may join together multiple stacks if they use a shared resource or an unmapped RSG (an infrastructure service that we have not identified yet).
Once you begin to go through AutoApp-xxx stacks, you can visualize them. In many cases you will see a single server or a small group of servers that link all of the hosts together. You can use the table below the visualization to see the actual application processes in use on the connections (if netstat data was collected). Our auto generated groupings may group single applications or multiple if they share DBs for example. Your objective is to decide how to and if application stacks should be broken into smaller ones.
You can then make a decision as to whether that host (or small group of hosts) should be allowed to link those groups together. (We have already decided YES, but you can overrule us!) If not, right click on the host, select Edit Stack Membership and create a new application stack to move the host into. You can then immediately close the visualization and re-open it, to see the result of your move. The visualization may now show multiple clusters of servers. What you are seeing is the result that you will get the NEXT time you run Build App Stacks
. You must re-run the auto-grouping to have the system break that stack apart, into two or more application stacks.
Once this has been completed you should disposition that group by giving it a friendly name that is meaningful to you and the business (e.g. AutoApp-xxx should be renamed to ApplicationName). Repeat this process for each AutoApp_XXX Application Stack.
All of this should be done with the application owners input. So while you may iterate a few times before you engage the application owner(s), they are critical to making the ultimate decisions on validating groups.
Remember to view connectivity data to check the inter-stack relationships and make sure that there are not any critical dependencies that would reuire further investigation or stack membership changes. This is also where you can check and verify that you don’t have unlicensed hosts using a service on a licensed node.
Review for Critical Dependencies
Once you have refined and confirmed an Application Stack, it is important to review the connectivity data for critical dependencies. These are dependencies that would dictate a change in Application Stack membership (e.g. a workload that needs to be moved into the stack you are reviewing).
For instances where there is connectivity that is critical and noteworthy but does not dictate a change in stack membership, you may choose to document this information for planning purposes. This can include Reporting, BI, or Data Transformation services that leverage database connectivity into these Application Stacks.
There are several options available to view connectivity for this purpose.
- Workload based visualizations
- Application Stack based visualizations
- Using View All Connections
Workload Based Visualizations
When you select an Application Stack in Add Intelligence >> All Applications or when you are in the View Individual Application Stack view, the bottom right hand section of the page provides a workload based visualization. Expand the visualization to provide more working area and use the visualization controls to show All
connectivity that is Critical
This provides an easy way to see workload to workload connectivity. You can click on any segment to view the details specific to that relationship. Make sure to use the column chooser to make sure you include application and process information. Review the connectivity to determine if any of it requires a change in Application Stack membership or if something that is noteworthy and must be somehow accounted for in your migration plan.
You are typically looking for “critical” connectivity such as Oracle, MySQL, MSSQL or known application traffic such as SAP.
Application Stack Based Visualizations
When viewing the Add Intelligence >> View Individual Application Stacks page you have the option to view Dependent Stacks
(from visualization control). This provides a visualization of all stack-to-stack connectivity relative to the individual stack you are currently viewing. Use the visualization controls to filter out infrastructure services such as Active Directory, Antivirus, etc.. This allows you to focus on true application to application connectivity. From here you can select any Application Stack to Application Stack segment that is interesting or that requires further investigation. By default, this will return a listing of the host to host connectivity that exists between these two stacks. You can switch this to a Protocol view to review from this basis. Again you are typically looking for “critical” connectivity such as Oracle, MySQL, MSSQL or known application traffic such as SAP. If you do see a protocol that is critical or interesting, you can View Details to drill into this and see the specific host to host connectivity for this protocol. This will also list the application and processes associated with the connectivity adding additional context to your review. Continue reviewing these Application Stack to Application Stack relationships until you are satisfied that there is nothing that would dictate a change in Application Stack membership.
View All Connections
When viewing the Add Intelligence >> View Individual Application Stacks page you have the option to View All Connections
(top left). This provides a table summary of all connectivity specific to that Application Stack and includes three levels of drill down.
The View All Connections
option can be an important way to view connectivity because it is NOT limited to connectivity between licensed nodes. The visualizations are limited to connectivity between licensed nodes with the exception of the RSG-Out Of Scope Services
Use this option to view connectivity coming into or out of the licensed environment. For example, you can use this dialog to determine if an Application Stack is internet facing. In this case you will see connectivity coming in from the RISC-unknown-internet