Tag Archive

Below you'll find a list of all posts that have been tagged as "storage"

High Performance Computing Storage – Hybrid Cloud, Parallel File Systems, Key Challenges, and Top Vendors’ Products

The toughest Terminator, T-1000 can demonstrate rapid shapeshifting, near-perfect mimicry, and recovery from damage. This is because it is made of mimetic polyalloy with robust mechanical properties. T-1000s naturally require top of the world speed, hi-tech communication system, razor-sharp analytical speed, and most powerful connectors and processors. Neural networks are also critical to the functioning of terminators. It stacks an incredible amount of data in nodes, which then communicates with the outer world depending on the input received. We infer one important thing – these Terminators produce an arduous amount of data. Therefore, it must require a sleek data storage system that scales and carry capabilities to compute massive datasets. Which, rings a bell – just like the case of terminators, High Performance Computing (HPC) also require equally robust storage to maintain compute performance. Well, HPC has been the nodal force to path defining innovations and scientific discoveries. This is because HPC enables processing of data and powering highly complex calculations at the speed of light. To give it a perspective, HPC leverages compute to deliver high performance. The rise of AI/ML, deep learning, edge computing and IoT created a need to store and process incredible amount of data. Therefore, HPC became the key enabler to bring digital technologies within the realm of daily use. In layman’s term, HPC can be referred as the supercomputers. The Continual Coming of the age of HPC The first supercomputer – CDC 6600 reigned for five years from its inception in 1964. CDC 6000 was paramount to the critical operations of the US government and the US military. It was considered 10 times faster to its nearest competitor – IBM 7030 Stretch. Well, it worked with a speed of up to 3 million floating-point operations per second (flops). The need for complex computer modeling and simulation never stopped over the decades. Likewise, we also witnessed evolution of high-performance computers. These supercomputer were made of core-components, which had more power and vast memories to handle complex workloads and analyze datasets. Any new release of supercomputers would make its predecessors obsolete. Just like new robots from the Terminator series. The latest report by Hyperion Research states that iterative simulation workloads and new workloads such as Al and other Big Data jobs would be driving the adoption of HPC Storage. Understanding Data Storage as an Enabler for HPC Investing in HPC is exorbitant. Therefore, one must bear in mind that it is essential to have a robust and equally proficient data storage system that runs concurrently with the HPC environment. Further some, HPC workloads differ based on its use cases. For example, HPC at the government & military secret agency consumes heavier workloads versus HPC at a national research facility. This means HPC storage require heavy customization for differential storage architecture, based on its application. Hybrid Cloud – An Optimal Solution for Data-Intensive HPC Storage Thinking about just the perfect HPC storage will not help. There has to an optimal solution that scales based on HPC needs. Ideally, it has to the right mix of best of the both – traditional storage (on-prem disk drives) and cloud (SSDs and HDDs). Complex, data-intensive IOPS can be channeled to SSDs, while usual streaming data can be handled by disk drives. An efficient combination of Hybrid Cloud – software defined storage and hardware configuration ultimately helps scale performance, while eliminating the need to have a storage tier separately. The software-defined storage must come with key characteristics – write back, read persistence performance statistics, dynamic flush, and I/O histogram. Finally, the HPC storage should support parallel file systems by handling complex sequential I/O. Long Term Solution (LTS) Lustre for Parallel File System More than 50 percent of the global storage architecture prefer Lustre – an open-source parallel file system to support HPC clusters. Well, for starters it offers free installation. Further, it provides massive data storage capabilities along with unified configuration, centralized management, simple installation, and powerful scalability. It is built on LTS community release allowing parallel I/O spanning multiple servers, clients, and storage devices. It offers open APIs for deep integration. The throughput is more than 1 terabyte/second. It also offers integrated support for an application built on Hadoop MapReduce applications. Challenges of Data Management in Hybrid HPC Storage Inefficient Data Handling The key challenge in implementing hybrid HPC storage is inefficient data handling. Dealing with the large and complex dataset and accessing it over WAN is time-consuming and tedious. Security Security is an another complex affair for HPC storage. The hybrid cloud file system also must include in-built data security. These small files must not be vulnerable to external threats. Providing SMBv3 encryption for files moving within the environment could be a great deal. Further, building the feature of snapshot replication can deliver integrated protection to the data in a seamless manner. Right HPC product End users usually find it difficult to choose the right product relevant to their services and industry. Hyperion Research presents an important fact. It states, “Although a large majority (82%) of respondents were relatively satisfied with their current HPC storage vendors, a substantial minority said they are likely to switch storage vendors the next time they upgrade their primary HPC system. The implication here is that a fair number of HPC storage buyers are scrutinizing vendors for competencies as well as price.” Top HPC Storage products Let’s briefly understand the top varied HPC Storage products in the market. ClusterStor E1000 All Flash – By Cray (A HPE Company) ClusterStor E1000 enables handling of the data at the speed of exascale. Its core is a combination of SSD and HDD. ClusterStor 1000 is a policy-driven architecture enabling you to move data intelligently. ClusterStor E1000 HDD-based configuration offers up to 50% more performance with the same number of drives than its closest competitors. This all-flash configuration is perfect for mainly small files, random access, and terabytes to single-digit PB capacity requirements. Source: Cray Website HPE Apollo 2000 System – By HPE The HPE Apollo 2000 Gen10 system is designed as an enterprise-level, density-optimized, 2U shared infrastructure chassis for up to four HPE ProLiant Gen10 hot-plug servers with the entire traditional data center attributes—standard racks and cabling and rear-aisle serviceability access. A 42U rack fits up to 20 HPE Apollo 2000 system chassis, accommodating up to 80 servers per rack. It delivers the flexibility to tailor the system to the precise needs of your workload with the right compute, flexible I/O, and storage options. The servers can be “mixed and matched” within a single chassis to support different applications, and it can even be deployed with a single server, leaving room to scale as customer’s needs grow. Source: HPE Website PRIMERGY RX2530 M5 – By Fujitsu The FUJITSU Server PRIMERGY RX2530 M5 is a dual-socket rack server that provides high performance of the new Intel® Xeon® Processor Scalable Family CPUs, expandability of up to 3TB of DDR4 memory and the capability to use Intel® Optane™ DC Persistent Memory, and up to 10x 2.5-inch storage devices – all in a 1U space saving housing. The system can also be equipped with the new 2nd generation processors of the Intel® Xeon® Scalable Family (CLX-R) delivering industry-leading frequencies. Accordingly, the PRIMERGY RX2530 M5 is the optimal system for large virtualization and scale-out scenarios, databases and for high-performance computing. Source: Fujitsu Website PowerSwitch Z9332F-ON – By Dell EMC The Z9332F-ON 100/400GbE fixed switch comprises Dell EMC’s latest disaggregated hardware and software data center networking solutions, providing state-of-the-art, high-density 100/400 GbE ports and a broad range of functionality to meet the growing demands of today’s data center environment. These innovative, next-generation open networking high-density aggregation switches offer optimum flexibility and costeffectiveness for the web 2.0, enterprise, mid-market and cloud service provider with demanding compute and storage traffic environments. The compact PowerSwitch Z9332F-ON provides industry-leading density of either 32 ports of 400GbE in QSFP56-DD form factor or 128 ports of 100 or up to 144 ports of 10/25/50 (via breakout), in a 1RU design. Source: Dell EMC Website E5700 – By NetApp E5700 hybrid-flash storage systems deliver high IOPS with low latency and high bandwidth for your mixed workload apps. Requiring just 2U of rack space, the E5700 hybrid array combines extreme IOPS, sub-100 microsecond response times, and up to 21GBps of read bandwidth and 14GBps of write bandwidth. With fully redundant I/O paths, advanced data protection features, and extensive diagnostic capabilities, the E5700 storage systems enable you to achieve greater than 99.9999% availability and provide data integrity and security. Source: NetApp Website ScaTeFS – By NEC Corporation The NEC Scalable Technology File System (ScaTeFS) is a distributed and parallel file system designed for large-scale HPC systems requiring large capacity. To realize load balancing and scale-out, all typical basic functions of a file system (read/write operation, file/directory generation, etc.) are distributed to multiple IO servers uniformly since ScaTeFS does not need a master server for managing the entire file system such as a metadata server. Therefore, the throughput of the entire system increases, and parallel I/O processing can be used for large files. Source: NEC Website HPC-X ScalableHPC – By Mellanox Mellanox HPC-X ScalableHPC toolkit is a comprehensive software package that includes MPI and SHMEM/PGAS communications libraries. HPC-X ScalableHPC also includes various acceleration packages to improve both the performance and scalability of high performance computing applications running on top of these libraries, including UCX (Unified Communication X) which accelerates point-to-point operations, and FCA (Fabric Collectives Accelerations) which accelerates collective operations used by the MPI/PGAS languages. This full-featured, tested and packaged toolkit enables MPI and SHMEM/PGAS programming languages to achieve high performance, scalability and efficiency, and to assure that the communication libraries are fully optimized of the Mellanox interconnect solutions. Source: Mellanox Website Panasas ActiveStor-18 – By Mircorway Panasas® is the performance leader in hybrid scale-out NAS for unstructured data, driving industry and research innovation by accelerating workflows and simplifying data management. ActiveStor® appliances leverage the patented PanFS® storage operating system and DirectFlow® protocol to deliver high performance and reliability at scale from an appliance that is as easy to manage as it is fast to deploy. With flash technology speeding small file and metadata performance, ActiveStor provides significantly improved file system responsiveness while accelerating time-to-results. Based on a fifth-generation storage blade architecture and the proven Panasas PanFS storage operating system, ActiveStor offers an attractive low total cost of ownership for the energy, government, life sciences, manufacturing, media, and university research markets. Source: Mircoway Website Future Ahead Dataset is growing enormously. And, there will be no end to it. HPC storage must be able to process data at the speed of the light to maintain compute efficiency at peak levels. HPC storage should climb to exascale from petascale. It must have robust in-built security, be fault-tolerant, be modular in design and most importantly, scale seamlessly. HPC storage based on hybrid cloud technology is a sensible path ahead; however, the efforts must be geared to control its components at runtime. Further, focus should also be on dynamic marshaling via the applet provisioning and in-built automation engine. This will improve compute performance and reduce costs.

Aziro Marketing

High Performance Computing HPC Hybrid-Cloud storage

Federated Data Services through Storage Virtualization

When one talks about virtualization, the immediate thought that comes to mind is about server/ host virtualization otherwise understood from the virtualization offerings from the likes of VMware, Citrix, Microsoft, etc. However, there is a not-so-explored & not much known data center technology that can contribute significantly to a modern (future) data center. When we talk of real time cloud application deployment (access anywhere) with enterprise workloads, there should be something more that the Infrastructure should support, to enable effective consolidation and management of storage/ host infrastructure across a data center.This article aims to introduce Storage Virtualization (SV) as a technology and the role this can play in enabling federated data services use cases. Aziro (formerly MSys Technologies) also has been a leading virtualization services provider working on the same technology.The Need for Storage VirtualizationTraditional data centers are largely FC-SAN based, where monoliths of huge enterprise storage arrays are hosted, deployed, configured, and managed but with niche expertise. Most of mission critical applications of the world run on such data centers (DC). EMC (Dell EMC), NetApp, IBM, HP (HPE), etc. are a few major players in this arena. The appliances these companies have built are tested and proven on the field for the reliability, efficiency and availability across various workloads.However, the major constraint of an IT investor of the modern times is related to the DC/ DR manageability and upgradability, more in the context of upcoming products with alternate technologies such as hyper converged storage; than defy the storage array based implementations. With vendor lock-in’s, rigid & propriety storage management API’s/ UI’s, it is a cumbersome process to think of an idea of having heterogeneous storage arrays with various vendors in a DC. Also, it poses the challenge of having skilled administrators who are well-versed on all different product implementations and management.Before it was a hyper-converged storage, the storage majors ventured to innovate an idea that could possibly solve this problem. This is how Storage Virtualization was born – where a way was envisaged to have heterogeneous storage arrays in a DC but still could seamlessly migrate data/ applications between them through a unified management interface. Not just that, the thrust was to see a bigger picture of application continuity to data center business continuity scaling up the scope of the high availability picture.What is Storage Virtualization?Storage virtualization (SV) is the pooling of physical storage from multiple storage arrays or appliances into what appears to be a single storage appliance that can be managed from a central console/ unified storage management application. Storage Virtualization could be an appliance hosted between the host and the target storage or could be just a software VM.Some popular SV SAN solutions available in the market are IBM SVC, EMC VPlex, NetApp V-series, etc.Use case & Implementation – How does it work?Let’s look at a practical data center use case of a heterogeneous data center, where there are 9 enterprise storage arrays, say 2 nos. of Dell EMC VMAX, 1 nos. of HPE 3PAR, 1 nos. of IBM V7000 & 5 nos. of EMC Clariion CX300. Consider that all legacy applications are currently hosted in EMC Clariion array and all the mission critical applications are hosted independently in EMC/ HPE/ IBM arrays. Let’s assume that the total data center storage requirements are already met and with the current infrastructure, it can easily support the requirement for the next 5 years. Consider, just between HPE, EMC and IBM arrays, we have sufficient storage space to accommodate the legacy applications as well. However, there isn’t a way yet to manage such a migration or a consolidated management of all different storage devices.Now, let’s look at some of the use case requirements/ consolidation challenges that a storage consultant should solve:Fully phase out Legacy CX300 Arrays and migrate all the legacy applications to one of enterprise arrays say, IBM V7000, but with minimum down time.Setting up a new data center, DC2 about 15 miles away and moving 2 of the enterprise arrays, say 2* EMC VMAX arrays to the new site and host this as an active-active data center/ disaster recovery site for former DC (DC1).The current site, DC1 should become DR site for the new DC, DC2 however should actively engage I/O and business should continue. (Synchronous use case)Management overhead of using products from 3 different vendors should reduce and should be simplified.The entire cycle of change should happen with minimum downtime except for the case of physical movement/ configuration of VMAX arrays to the new site.The architecture should be scalable for data requirements of next 5 years in such a way that new storage arrays from existing or new vendors can be added with no downtime/ disruption.The DC & DR sites are mutually responsive to each other during an unforeseen disaster and are highly available.Solution IllustrationThis is a classic case for Storage Virtualization Solution. An SV solution is typically an appliance with software & intelligence that is sandwiched between the initiator (hosts) and the target (heterogeneous storage arrays). For the initiator, the SV is the target and for the target, the SV becomes the initiator. All the storage disks from the target (with/ without data) appear as a bunch of unclaimed volumes in the SV. As far as hosts are concerned, they appear to the SV as unmapped initiators unregistered. Storage- Initiator groups are created (registered) in the SV which can be modified/ changed on the fly giving flexible host migration at the time of server disaster.There are different SV solutions available from vendors such as EMC VPlex that can help cases of local DC migration as well as migration between sites / DC’s. Let’s see how the solution unfolds to our use case requirements.Storage from both legacy array and the new array once configured to access the hosts through an SV solution, the storage disks/ LUNs appear as pool of storage at the DV interface. The SV solutions encapsulates the storage so that data migration between both the arrays can happen non-disruptively. Vendor1- Vendor2 replications are challenging and often disruptive.SV solutions are configured in a fully HA configuration providing fault tolerance at every level (device, storage, array, switch, etc.).Across site SV solution such as EMC VPlex Metro can perform a site-site data mirroring (synchronous) that too which both the sites are fully in active-active IO configuration.The entire configuration done through HA Switches provides option to scale to add existing/ new vendor storage arrays as well new Hosts/ Initiators with zero down time.The entire solution be it at local DC level or multi-site would be fully manageable through a common management UI/ Interface reducing the dependence on skilled storage administrators who are vendor specific.A SV solution consolidates the entire storage and host infrastructure to a common platform simplifying the deployment and management. Also, this sets a new dimension to hyper-converged storage infrastructure to be scaled across sites.A SV solution is agnostic, to the host and storage giving diversity of deployment options. For e.g. various host hardware, operating systems, etc.All the features of a storage array are complimented to its full potential along with superior consolidation across Storage/ sites with additional availability/ reliability features.Solutions like VMware vMotion does help in site- site migration, however, an SV solution provides the infrastructure support for that happen at the storage device level that too across sites.ConclusionIt’s just a matter of time, when we will see more efficiently packaged & effectively deployed SV solutions. Perhaps, it could be called software defined SV solution that can be hosted on a VM instead of an appliance. Storage consolidation is a persistent problem, more so in the modern days, due to the diversity of Sever Virtualization/ SDS Solutions, varieties of Backup and recovery applications/ options available to an IT Administrator. There should be a point where DC should become truly converged where best of every vendor can co-exist in its own space complimenting each other. However, there is a business problem to that wish. For now, we can only explore more on what SV can offer us.

Aziro Marketing

datacenter storage storage virtualization virtualization

How to use Log Analytics to Detect Log Anomaly?

INTRODUCTIONWe’ll focus on the problem of detecting anomalies in application run-time behaviors from their execution logs.Log Template usage can be broadly classified to determine:The log occurrence counts [include error, info, debug and others] from the specific software components or software package or modules.The cause for the application anomalies, which includes a certain software component(s), actual hardware resource or its associated tasks.The software components or software package or modules, which are “most utilized “or “least utilized”. This helps to tweak the application performance, by focusing on the most utilized modules.This new technique helps to:Overcome the instrumentation requirements or application specific assumptions made in prior log mining approaches.Improve by orders of magnitude the capability of the log mining process in terms of volume of log data that can be processed per day.BENEFITS OF THIS SOLUTION:Product Engineering Team can effectively utilize this solution across several of its products for monitoring and improving the product functional stability and performance.This solution will help detect the application abnormalities in advance and alert the administrator to take corrective actions and prevent application outage.This solution tries to preserve the application logs and anomalies. This can be effectively utilized for improving the operation efficiency by,System IntegratorApplication Administrator(s)Site Reliability Engineer(s)Quality Assurance Ops Engineer(s)SOLUTION ARCHITECTURE:ELK Stack (Elasticsearch, Logstash, and Kibana) is the most popular open source log analysis platform. ELK is quickly overtaking existing proprietary solutions and has become the first choice for companies shipping for log analysis and management solutions.ELK stack is comprised of three separate yet alike open-source products:Elasticsearch, which is based on Apache Lucene is a full-text search engine to perform full-text and other complex searches.Logstash processes the data before sending it to Elasticsearch for indexing and storage.Kibana is the visualization tool that enables you to view log messages and create graphs and visualizations.FilebeatInstalled on a client that will continue to push their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash. ELK Stack along with Filebeat preserves the application logs as long as we want. These pre-served application logs can be used for log template mining, further triaging to find evidence of the application malfunctioning or anomalies observed.TECHNOLOGIES:Python 3.6, NumPy, Matplotlib, Plotly, and Pandas modules.HIGH LEVEL APPROACHES:Log Transformation Phase: Transformation of logs [unstructured data] into structured data to categorize the log message into the below-mentioned dimensions and facts.Dimension:Time Dimension [ Year, Month, Date, Hour, Min, Second]Application Dimension [ Thread Name, Class Name, Log Template, Dynamic Parameter, and its combinations.Fact: Custom Log MessageLog Template Mining Phase: Log mining process consumes the “Custom Log Message” to discover the Log Templates and enable the analytics in any or all of the dimensions as mentioned above.Log Template Prediction Phase: In addition to the discovery of Log Template pattern, log mining process also helps to predict the relevant log template for the received “Custom Log Message”.LOG TRANSFORMATION PHASE:LOG PARSING:Unstructured Record — Structured Record :Creation of Dimension Table for preserving Time and Application dimension details.LOG TEMPLATE MINING PHASE:Individual log lines are compared against other log lines to identify the common as well as unique word appearance etc.Log Template Generations are accomplished by following the below mentioned steps:Log Lines Clustering: Clustering the log lines, which are closely matching w.r.t common words and its ordering.Unique Word Collection: identifying and collecting unique words within clustersUnique Word Masking: Masking one of the randomly selected log line and using the result log line as Log Template.Log Template Validation: Applying log template to all the clustered log lines to extract the unique words and ensuring that those words are unique.Dynamic Parameter Extraction: Applying log template to all the clustered log lines and extracting and persisting dynamic parameter(s) are against each log lines.LOG TEMPLATE PREDICTION PHASE:Log Lines Cluster Identification: Identifying log line common words and unique word b/w Log Template and Custom Log Message.Log Template Identification: Selecting a closely matched Log Template as Log Template and extracting unique or dynamic parameters using the selected Log Template.Log Template Generation: Triggering process, in case if no log template are closely matching.Dynamic Parameter Extraction: Extracting applied log template to all the clustered log lines and dynamic parameter(s) and persisting them against each log lines.Log Template Persistence:Processing the received real-time log line and found the log template match from the Log Template Inventory.Processing and updating the Inventory Log Template based on the received real-time log line.Processing and creating the new Log Template from the received real-time log line and updating the Inventory Log Template.ANOMALIES DETECTION PHASE:Identifying application anomalies through theDetection of the spike in total log records, error records received on the particular moment [date & time scale].Detection of the spike in processing time.i.e. time difference between the subsequent log records on the particular moment [date & time scale].Detection of the spike in few application threads emitting large number of log records for the particular moment [date & time scale].Registering of the administrator with the system to receive asynchronous notification about the anomalies either through E-Mail or SMS etc.Persisting anomalies details in a distributed database like Cassandra Database with the aggregated information likeSpike in total log records, error records count on the specific timeSpike in processing time on the specific timeApplication threads which emitted a large number of log records on the specificANOMALIES DETECTION USING LOGS COUNT:Plott the line graph using the time-scale to depict the number of log line occurrencesGenerate the same report for error log records too.ANOMALIES SPOT LOG RECORD COUNT:Bar Graph can be used to show the significant contribution by the several log templates, which causes the anomalies.This graph can be launched by clicking on the anomalies point presented from the logs count report.ROOT CAUSE [ACTUAL RESOURCE, SOFTWARE COMPONENT] FOR ANOMALIES POINT:The report will be generated for the selected Log Template.This report can be launched by clicking on the Log Template Occurrence Report for a particular Log Template, where the significant contribution found for anomalies.ANOMALIES DETECTION BASED ON THREADLine Graph can be used to show the significant contribution by the different threads, which causes the anomalies.ANOMALIES DETECTION BASED ON PROCESSING TIME B/W LOGS RECORD ENTRY TIME:Line Graph can be used to depict the cumulative processing time b/w log line[includes regular logs as well as error logs]ANOMALIES ROOT CAUSE ANALYSIS BY SEARCHING & FILTERING FROM RAW LOG RECORD:GUI presents about the list of unique words [which represents the actual resources used by the application] extracted from the Log Record to construct the Log Template.Searching the log record b/w specific time frame for the specific keyword or set of keywords [must be one among the unique words found during the Log Template Mining Phase] with AND or OR condition.Log Record Search result presents the table with the following sortable columns [ single or multiple column sorting ] :Date TimeLog Sequence IDThreadCustom Log Message [ with the highlighted search keywords]SEARCH FORM:SEARCH RESULT:CONCLUSIONSo far, this solution presents the various steps, which can be collectively used to analyze the logs and identify the anomalies in the application, as well as the resource(s) causing those anomalies.Detection of following cases can be considered as an anomaly for an applicationRequest timeout or zero requests processing time i.e. application hung or deadlock.Prolonged, consistent increase in processing time.Heavy and constant increase in application memory usage.5.1 DIRECTIONS FOR FUTURE DEVELOPMENTThis solution can be further extended to analyze the control flow as a whole, using control flow graph mining. This control flow mining helps to detect or determine the application anomalies by detecting the following cases:Deviation from the recorded functional flow.Most and least accessed or utilized functions and the resource associated.Cumulative processing time per control flow, by associated resources.The number of active control flow for a given moment of time on a real-time basis.Control flow graph classification based on the cumulative processing time.REFERENCESAnomaly Detection Using Program Control Flow Graph Mining from Execution Logs by Animesh Nandi, Atri Mandal, Shubham Atreja, Gargi B. Dasgupta, Subhrajit Bhattacharya, IBM Research, IIT Kanpur, 2016.An Evaluation Study on Log Parsing and Its Use in Log Mining by Pinjia He, Jieming Zhu, Shilin He, Jian Li, and Michael R. Lyu, Department of Computer Science and Engineering, 2016

Aziro Marketing

Anomaly Detection Anomaly monitor log analytics Log mining storage

SSD and Enterprise Storage: 4 effective parameters for performance measurement

A solid-state drive (SSD) is a data storage device that uses solid-state memory to store persistent data. An SSD emulates a hard disk drive interface, thus easily replacing it in most applications. Unlike mechanical hard disk drives, solid state disks are made up of silicon memory chips and have no moving parts. As with hard disks, the data is persistent in SSDs when they are powered down. A common method is to keep as much application data as possible in server memory, thereby reducing the frequency with which the application must retrieve data from the physical HDDs, as this process has much longer read or write latency than server memory.Understanding parameters for measuring Oracle DB BenchmarkDeploying SSD in place of hard disk drives can result in immediate performance gains and can eliminate the bottlenecks caused by mechanical hard disk I/O latency. Oracle Performance Tuning with Solid State Disk provides a comprehensive guide that enables DBAs to make the transition to SSD successfully. By accelerating Oracle databases, applications can handle more transactions, more concurrent users and deliver higher profits and productivity gains. SSD is especially useful for Oracle undo logs, redo logs and the TEMP tablespace, but it can be used with any Oracle data file for unbelievable access speed.The blog discusses the process of identifying the I/O subsystem problems, analyzing what to put on the SSD and understanding the Automatic workload repository (AWR).1. Identifying I/O Subsystem ProblemsThe I/O subsystem is a vital component of an Oracle database. Oracle database is designed so that if an application is well written, its performance should not be limited by I/O. Tuning I/O can enhance the performance of an application if the I/O system is operating at or near capacity and is not able to service the I/O requests within an acceptable time.If your system is experiencing I/O subsystem problems, the next step is to determine which components of your Oracle database are experiencing the highest I/O and in turn causing I/O wait time. Better performance can be achieved by isolating these hot data objects to an SSD file system.2. Analyzing What to Put on Solid State DiskThere are two types of operations in Oracle that utilize the high speed disk subsystem: database reads and database writes. Oracle database reads should be as fast as possible and allow for maximum simultaneous access. In order to support the highest possible read speed the disk assets must provide for low latency access to data from multiple processes.With an SSD, latency is virtually eliminated and data access is immediate. SSD architecture also allows for many high-bandwidths I/O ports, each supporting simultaneous random access without performance degradation. The speed of a Solid State Disk cannot be slowed down by mechanical limitations and its access latency is improved by several orders of magnitude.3. Analyzing Oracle AWR ReportWe can analyze the IO and wait interface statistics by:Oracle Enterprise ManagerAWR and STATSPACK reports.Custom ScriptsThe Oracle Enterprise Manager provides excess of data and reports for Oracle database activity. There are custom scripts also available for identifying the hot data objects. AWR (Automatic Workload Repository) and STATSPACK reports allow us to take a focused look at specific time intervals.Reading the AWR ReportThis section contains detailed guidance for evaluating each section of an AWR report. The key segments in an AWR report include:Report Summary Section: This gives an overall summary of the instance throughout the snapshot period, and it contains important aggregate summary information.Cache Sizes: This report displays the size of each SGA region after AMM has changed them. This information can be compared to the original init.ora parameters at the end of the AWR report.Load Profile: This section shows important rates expressed in units of per second and transactions per second.Shared Pool Statistics: This is a good summary of changes to the shared pool during the snapshot period.Top 5 Timed Events: This is the most important section in the AWR report. It shows the top wait events and can quickly show the overall database bottleneck.Custom Scripts utilize the V$ series of views to generate reports showing I/O distribution, timing data and wait statistics. For data and temp file-related statistics, the v$filestat and v$tempstat tables are utilized. For wait interface information, the v$waitstat, v$sysstat and v$sesstat tables can be utilized.A look at the OS IOSTAT command confirms that the I/O subsystem is undergoing an extreme amount of stress.We had used IOSTAT and VMSTAT for collecting 5 second intervals during the entire duration of the testing. This gave us real-time data on RAM paging and CPU enqueues. In this benchmark, we used AWR reports to identify the performance metrics by referring various sections available.4. Identifying the Most Frequently Accessed TablesThe I/O Stats Section in AWR reports shows all the important I/O activity for the instance and shows I/O activity by tablespace, data file, and includes buffer pool statistics.From the AWR report generated during the initial Benchmark TPC-C test, the segment I/O statistics section reported the information used to isolate specific objects that would benefit from being placed on SSDs. The segments with the most logical reads and physical reads are presented in below tables. These segments should be considered as possible candidates to be placed on SSDs.Table 1: Segments by Logical ReadsTable 2: Segments by Physical ReadsFor our specific example, the user indexes such as C_ORDER_LINE_I1, C_ORDER_I1, and C_STOCK_I1 indexes and tables such as C_CUSTOMER, C_ORDER_LINE and C_STOCK involving the largest number of reads were selected to move to SSD.The Oracle database stores data on the files that are accessed in the V$FILESTAT table. This table starts gathering information as soon as a database instance is started. When a database instance is stopped, the data in the V$FILESTAT table is cleared. Therefore, if the database instance is routinely stopped, it is important to capture the data from the V$FILESTAT table before the data is cleared. It is possible to create a program to gather this data and move it to a permanent table.The following fields are available from V$FILESTAT:FILE#: Number of the filePHYRDS: Number of physical reads donePHYBLKRD: Number of physical blocks readPHYWRTS: Number of physical writes donePHYBLKWRT: Number of physical blocks writtenA simple query and report from the V$FILESTAT table will indicate which Oracle database files are frequently accessed. Adding PHYRDS and PHYWRTS gives the total I/O for a single file. By sorting the files by total I/O, it is possible to quickly identify the files that are most frequently accessed. The most frequently accessed files are good candidates for moving to SSD.

Aziro Marketing

storage

« Previous 1 2

EXPLORE ALL TAGS

Ansible Test Automation

application containerization

applications

Application Security

application testing

artificial intelligence

asynchronous replication

business intelligence

Cloud Cost Optimization

cloud devops

Cloud Infrastructure

Cloud Interoperability

Cloud Native Solution

Cloud Storage Security

Codeless Automation

Cognitive analytics

Configuration Management

container world conference

continuous-delivery

continuous deployment

continuous integration

data backup and recovery

Descriptive analytics

Descriptive analytics tools

Digital Transformation

dockercon 2019 san francisco

end-to-end-test-automation

High Performance Computing

icinga for monitoring

Image Recognition 2024

kubernetesday bangalore

Low-Code No-Code Platforms

mobile-application-testing

mobile-automation-testing

Predictive analytics tools

prescriptive analysis

Rapid Application Development

raspberry pi

RDMA

real time analytics

realtime analytics platforms

Real-time data analytics

Recovery

Recovery as a service

recovery as service

rsa

rsa 2019

rsa 2019 san francisco

Selenium Test Automation

selenium testng

serverless

Serverless Computing

Site Reliability Engineering

software defined storage

software-testing

software testing trends

software testing trends 2019

storage virtualization

support

Synchronous Replication

testing automation tools

thought leadership articles

trends

tutorials

ui automation testing

ui testing

ui testing automation

vCenter Operations Manager

vmworld 2019 san francisco

VMworld 2019 US

vROM

Web Automation Testing

web test automation

WFH

LET'S ENGINEER

Your Next Product Breakthrough

Book a Free 30-minute Meeting with our technology experts.

Aziro has been a true engineering partner in our digital transformation journey. Their AI-native approach and deep technical expertise helped us modernize our infrastructure and accelerate product delivery without compromising quality. The collaboration has been seamless, efficient, and outcome-driven.

CTO

Fortune 500 company