Tag Archive

Below you'll find a list of all posts that have been tagged as "devops"
blogImage

Aziro (formerly MSys Technologies) 2019 Tech Predictions: Smart Storage, Cloud’s Bull Run, Ubiquitous DevOps, and Glass-Box AI

2019 brings us to the second-last leg of this decade. From the last few years, IT professionals have been propagating rhetoric. They state that the technology landscape is seeing a revolutionary change. But, most of the “REVOLUTIONARY” changes, has, over the time lost their gullibility. Thanks to the awe-inspiring technologies like AI, Robotics, and upcoming 5G networks most tech pundits consider this decade to be a game changer in the technology sector.As we make headway into 2019, the internet is bombarded with numerous tech prophecies. Aziro (formerly MSys Technologies) presents to you the 2019 tech predictions based on our Storage, Cloud, DevOps and digital transformation expertise.1. Software Defined Storage (SDS)Definitely, 2019 looks promising for Software Defined Storage. It’ll be driven by changes in Autonomous Storage, Object Storage, Self-Managed DRaaS and NVMes. But, SDS will also be required to push the envelope to acclimatize and evolve. Let’s understand why so.1.1 Autonomous Storage to Garner MomentumBacked by users’ demand, we’ll witness the growth of self-healing storage in 2019. Here, Artificial Intelligence powered by intelligent algorithms will play a pivotal role. Consequently, companies will strive to ensure uninterrupted application performance, round the clock.1.2 Self-Managed Disaster Recovery as a Service (DRaaS) will be ProminentSelf-Managed DRaaS reduces human interference and proactively recovers business-critical data. It then duplicates the data in the Cloud. This brings relief during an unforeseen event. Ultimately, it cuts costs. In 2019, this’ll strike chords with enterprises, globally, and we’ll witness DRaaS gaining prominence.1.3 The Pendulum will Swing Back to Object Storage as a Service (STaaS)Object Storage makes a perfect case for cost-effective storage. Its flat structure creates a scale-out architecture and induces Cloud compatibility. It also assigns unique Metadata and ID for each object within storage. This accelerates the data retrieval and recovery process. Thus, in 2019, we expect companies to embrace Object Storage to support their Big data needs.1.4 NMVes Adoption to Register TractionIn 2019, Software Defined Storage will accelerate the adoption rate of NVMes. It rubs off glitches associated with traditional storage to ensure smooth data migration while adopting NVMes. With SDS, enterprises need not worry about the ‘Rip and Replace’ hardware procedure. We’ll see vendors design storage platforms that append to NVMes protocol. For 2019, NMVes growth will mostly be led by FC-NVME and NVMe-oF.2. Hyperconverged Infrastructure (HCI)In 2019, HCI will remain the trump card to create a multi-layer infrastructure with centralized management. We’ll see more companies utilize HCI to deploy applications quickly. This’ll circle around a policy-based and data-centric architecture.3. Hybridconverged Infrastructure will Mark its FootprintHybridconverged Infrastructure (HCI.2) comes with all the features of its big brother – Hyperconverged Infrastructure (HCI.1). But, one extended functionality makes the latter smarter. Unlike HCI.1, it allows connecting with an external host. This’ll help HCI.2 mark its footprint in 2019.4. VirtualizationIn 2019, Virtualization’s growth will be centered around Software Defined Data Centers and Containers.4.1 ContainersContainer technology is ace in the hole to deliver promises of multi-cloud – cost efficacy, operational simplicity, and team productivity. Per IDC, 76 percent of users’ leverage containers for mission-critical applications.4.1.1 Persistent Storage will be a Key ConcernIn 2019, Containers’ users will envision a cloud-ready persistent storage platform with flash arrays. They’ll expect their storage service providers to implement synchronous mirroring, CDP – continuous data protection and auto-tiering.4.1.2 Kubernetes Explosion is ImminentThe upcoming Kubernetes version is rumored to include a pre-defined configuration template. If true, it’ll enable an easier Kubernetes deployment and use. This year, we are also expecting a higher number of Kubernetes and containers synchronization. This’ll make Kubernetes’ security a burgeoning concern. So, in 2019, we should expect stringent security protocols around Kubernetes deployment. It can be multi-step authentication or encryption at the cluster level.4.1.3 Istio to Ease Kubernetes Deployment HeadacheIstio is an open source service mesh. It addresses the Microservices’ application deployment challenges like failure recovery, load balancing, rate limiting, A/B testing, and canary testing. In 2019, companies might combine Istio and Kubernetes. This can facilitate a smooth Container orchestration, resulting in an effortless application and data migration.4.2 Software Defined Data CentersMore companies will embark on their journey to Multi-Cloud and Hybrid-Cloud. They’ll expect a seamless migration of existing applications to a heterogeneous Cloud environment. As a result, SDDC will undergo a strategic bent to accommodate the new Cloud requirements.In 2019, companies will start cobbling DevOps and SDDC. The pursuit of DevOps in SDDC will be to instigate a revamp of COBIT and ITIL practice. Frankly, without wielding DevOps, cloud-based SDDC will remain in a vacuum.5. DevOpsIn 2019, companies will implement a programmatic DevOps approach to accelerate the development and deployment of software products. Per this survey, DevOps enabled 46x code deployment. It also skyrocketed the deploy lead time by 2556x. This year, AI/ML, Automation, and FaaS will orchestrate changes to DevOps.5.1 DevOps Practice Will Experience a Spur with AI/MLIn 2019, AI/ML centric applications will experience an upsurge. Data science teams will leverage DevOps to unify complex operations across the application lifecycle. They’ll also look to automate the workflow pipeline – to rebuild, retest and redeploy, concurrently.5.2 DevOps will Add Value to Functions as a Service (FaaS)Functions as a Service aims to achieve serverless architecture. It leads to a hassle-free application development without perturbing companies to handle the monolithic REST server. It is like a panacea moment for developers.Hitherto, FaaS hasn’t achieved a full-fledged status. Although FaaS is inherently scalable, selecting wrong user cases will increase the bills. Thus, in 2019, we’ll see companies leveraging DevOps to fathom productive user cases and bring down costs drastically.5.3 Automation will be the Mainstream in DevOpsManual DevOps is time-consuming, less efficient, and error-prone. As a result, in 2019, CI/CD automation will become central in the DevOps practice. Consequently, Infrastructure as a Code to be in the driving seat.6. Cloud’s Bull Run to ContinueIn 2019, organizations will reimagine the use of Cloud. There will be a new class of ‘born-in-cloud’ start-ups, that will extract more value by intelligent Cloud operations. This will be centered around Multi-Cloud, Cloud Interoperability, and High Performance Computing. More companies will look to establish a Cloud Center of Excellence (CoE). Per RightScale survey, 57 percent of enterprises already have a Cloud Center of Excellence.6.1 Companies will Drift from “One-Cloud Approach.”In 2018, companies realized that having a ‘One-Cloud Approach’ encumbers their competitiveness. In 2019, Cloud leadership teams will bask upon the Hybrid-Cloud Architecture. Hybrid-Cloud will be the new normal within Cloud Computing in 2019.6.2 Cloud Interoperability will be a Major ConcernIn 2019, companies will start addressing the issues of interoperability by standardizing Cloud architecture. The use of the Application Programming Interface (APIs) will also accelerate. APIs will be the key to instill the capability of language neutrality, which augments system portability.6.3 High Performance Computing (HPC) will Get its Place on CloudIndustries such as Finance, Deep Learning, Semiconductors or Genomics are facing the brunt of competition. They’ll envision to deliver high-end compute-intensive applications with high performance. To entice such industries, Cloud providers will start imparting HPC capabilities in their platform. We’ll also witness large scale automation in Cloud.7. Artificial IntelligenceFor 2019 AI/ML will come out of the research and development model to be widely implemented in organizations. Customer engagements, infrastructure optimization, and Glass-Box AI, will be in the forefront.7.1 AI to Revive Customer EngagementsBusinesses (startups or enterprise) will leverage AI/ML to enable a rich end-user experience. Per Adobe, enterprises using AI will more than double in 2019. Tech and non-tech companies, alike, will strive to offer personalized services leveraging Natural Language Processing. The focus will remain to create a cognitive customer persona to generate tangible business impacts.7.2 AI for Infrastructure OptimizationIn 2019, there will a spur in the development of AI embedded monitoring tools. This’ll help companies to create a nimble infrastructure to respond to the changing workload. With such AI-driven machines, they’ll aim to cut down the infrastructure latency, infuse robustness in applications, enhance performances, and amplify outputs.7.3 Glass-Box AI will be crucial in Retail, Finance, and HealthcareThis is where Explainable AI will play its role. Glass-Box AI will create key customers’ insights with underlying methods, errors or biases. In this way, retailers don’t necessarily follow every suggestion. They can sort out responses that fit rights in that present scenario. The bottom-line will be to avoid customer altercations and bring out fairness in the process.

Aziro Marketing

blogImage

Building Package Using Omnibus

Omnibus is a tool for creating full-stack installers for multiple platforms. In general, it simplifies the installation of any software by including all of the dependencies for that piece of software. It was written by the people at Chef, who use it to package Chef.Omnibus consists of two pieces- omnibus and omnibus software. omnibus – the framework, created by Chef Software, by which we create full-stack, cross-platform installers for software. The project is on GitHub at chef/omnibus.omnibus-software – Chef Software’s open source collection of software definitions that are used to build the Chef Client, the Chef Server, and other Chef Software products. The software definitions can be found on GitHub at chef/omnibus-software.Omnibus provides both, a DSL for defining Omnibus projects for your software, as well as a command-line tool for generating installer artifacts from that definition.Omnibus has minimal prerequisites. It requires Ruby 2.0.0+ and Bundler.Getting StartedTo get started install omnibus > gem install omnibus You can now create an omnibus project inside your current directory using project generator feature> omnibus new demo This will generate a complete project skeleton in the directory as following:create omnibus-demo/Gemfile create omnibus-demo/.gitignore create omnibus-demo/README.md create omnibus-demo/omnibus.rb create omnibus-demo/config/projects/demo.rb create omnibus-demo/config/software/demo-zlib.rb create omnibus-demo/.kitchen.local.yml create omnibus-demo/.kitchen.yml create omnibus-demo/Berksfile create omnibus-demo/package-scripts/demo/preinst chmod omnibus-demo/package-scripts/demo/preinst create omnibus-demo/package-scripts/demo/prerm chmod omnibus-demo/package-scripts/demo/prerm create omnibus-demo/package-scripts/demo/postinst chmod omnibus-demo/package-scripts/demo/postinst create omnibus-demo/package-scripts/demo/postrm chmod omnibus-demo/package-scripts/demo/postrm It creates the omnibus-demo directory inside your current directory and this directory has all omnibus package build related files. It is easy to build an empty project without doing any change run> bundle install --binstubs bundle install installs all Omnibus dependenciesbundle install installs all Omnibus dependencies The above command will create the installer inside pkg directory. Omnibus determines the platform for which to build an installer based on the platform it is currently running on. That is, you can only generate a .deb file on a Debian-based system. To alleviate this caveat, the generated project includes a Test Kitchen setup suitable for generating a series of Omnibus projects.Back to the Omnibus DSL. Though bin/omnibus build demo will build the package for you, it will not do anything exciting. For that, you need to use the Omnibus DSL to define the specifics of your application.1) ConfigIf present, Omnibus will use a top-level configuration file name omnibus.rb at the root of your repository. This file is loaded at runtime and includes number of configurations. For e.g.-omnibus.rb # Build locally (instead of /var) # ------------------------------- base_dir './local' # Disable git caching # ------------------------------ use_git_caching false # Enable S3 asset caching # ------------------------------ use_s3_caching true s3_access_key  ENV['S3_ACCESS_KEY'] s3_secret_key  ENV['S3_SECRET_KEY'] s3_bucket      ENV['S3_BUCKET'] Please see config doc for more information. You can use different configuration file by using –config option using command line$ bin/omnibus --config /path/to/config.rb 2) Project DSLWhen you create an omnibus project, it creates a project DSL file inside config/project with the name which you used for creating project for above example it will create config/project/demo.rb. It provides means to define the dependencies of the project and metadata of the project. We will look at some contents of project DSL filename            "demo" maintainer      "YOUR NAME" homepage        "http://yoursite.com" install_dir     "/opt/demo" build_version   "0.1.0" # Creates required build directories dependency "preparation" # demo dependencies/components dependency "harvester" ‘install_dir’ option is the location where package will be installed. There are more DSL methods available which you can use in this file. Please see the Project Doc for more information.3) Software DSLSoftware DSL defines individual software components that go into making your overall package. The Software DSL provides a way to define where to retrieve the software sources, how to build them, and what dependencies they have. Now let’s edit a config/software/demo.rbname "demo" default_version "1.0.0" dependency "ruby" dependency "rubygems" build do  #vendor the gems required by the app  bundle “install –path vendor/bundle” end In the above example, consider that we are building a package for Ruby on Rails application, hence we need to include ruby and rubygems dependency. The definition for ruby and rubygems dependency comes from the omnibus-software. Chef has introduced omnibus-software, it is a collection of software definitions used by chef while building their products. To use omnibus-software definitions you need to include the repo path in Gemfile. You can also write your own software definitions. Inside build block you can define how to build your installer. Omnibus provide Build DSL which you can use inside build block to define your build essentials. You can run ruby script and copy and delete files using Build DSL inside build block.Apart from all these DSL file omnibus also created ‘package-script’ directory which consist of pre install and post install script files. You can write a script which you want to run before and after the installation of package and also before and after the removal of the package inside these files.You can use the following references for more exampleshttps://github.com/chef/omnibushttps://www.chef.io/blog/2014/06/30/omnibus-a-look-forward/

Aziro Marketing

blogImage

Beginners Guide to a Career in DevOps

ABSTRACTThe software development lifecycles moved from waterfall to agile models. These improvements are moving toward IT operations with evolution of Devops.DevOps primarily focuses on collaboration, communication, integration between developers and operations.AGILE EVOLUTION TO DEVOPSWaterfall model was based on a sequence starting with requirements stage, while development stage was under progress. This approach is inflexible and monolithic. In the agile process, both verification and validation execute at the same time. As developers become productive, business become more agile and respond to their customer requests more quickly and efficient.WHAT IS DEVOPSIt is a software development strategy which bridges the gap between the developers and IT Staff. It includes continuous development, continuous testing, continuous integration, continuous deployment, continuous monitoring throughout the development lifecycle.WHY DEVOPS IS IMPORTANT1.Short development cycle, faster innovation2.Reduced deployment failures, rollback and time to recover3.Improved communication4.Increased efficiencies5.Reduced costsWHAT ARE THE TECHNOLOGIES BEHIND DEVOPS?Collabration, Code Planning, Code Repository, Configuration Management, Continuous integration, Test Automation, Issue Tracking, Security, MonitoringHOW DOES DEVOPS WORKSDevOps uses a CAMS approachC=Culture, A=Automation, M=Measurement, S=SharingDEVOPS TOOLSTOP DEVOPS TESTING TOOLS IN 20191.Tricentis 2. Zephyr 3.Ranorex 4.Jenkins 5.Bamboo 6.Jmeter 7.Selenium 8.Appium 9.Soapui 10.CruiseControl 11.Vagrant 12.PagerDuty 13.Snort 14.Docker 15.Stackify Retrace 16.Puppet Enterprise 17.UpGuard 18.AppVerifyDEVOPS JOB ROLES AND RESPONSIBILITIESDevOps Evangelist – The principal officer (leader) responsible for implementingDevOps Release Manager – The one releasing new features & ensuring post-release product stabilityAutomation Expert – The guy responsible for achieving automation & orchestration of toolsSoftware Developer/ Tester – The one who develops the code and tests itQuality Assurance – The one who ensures the quality of the product confirms to its requirementSecurity Engineer – The one always monitoring the product’s security & healthDEVOPS CERITIFICATIONRet hat offers five courses with examDeveloping Containerized Applications, OpenShift Enterprise Administration, Cloud Automation with Ansible, Managing Docker Containers with RHEL Atomic Host, Configuration Management with PuppetAmazon web services offers the AWS certified DevOps EngineerSKILL THAT EVERY DEVOPS ENGINEER NEEDS FOR SUCCESS1.Soft Skills2.Broad understanding of tools and technologies2.1 Source Control (like Git, Bitbucket, Svn, VSTS etc)2.2 Continuous Integration (like Jenkins, Bamboo, VSTS )2.3 Infrastructure Automation (like Puppet, Chef, Ansible)2.4 Deployment Automation & Orchestration (like Jenkins, VSTS, Octopus Deploy)2.5 Container Concepts (LXD, Docker)2.6 Orchestration (Kubernetes, Mesos, Swarm)2.7 Cloud (like AWS, Azure, GoogleCloud, Openstack)3.Security Testing4.Experience with infrastructure automation tools5.Testing6.Customer-first mindset7.Collabration8.Flexibility9.Network awareness10.Big Picture thinking on technologiesLINKS:https://www.quora.com/How-are-DevOps-and-Agile-differenthttps://www.altencalsoftlabs.com/blog/2017/07/understanding-continuous-devops-lifecycle/https://jenkins.io/download/https://www.atlassian.com/software/bamboohttp://jmeter.apache.org/download_jmeter.cgihttp://appium.io/https://www.soapui.org/downloads/download-soapui-pro-trial.htmlhttp://cruisecontrol.sourceforge.net/download.htmlhttps://www.vagrantup.com/downloads.htmlhttps://www.pagerduty.com/https://www.snort.org/downloadshttps://store.docker.com/editions/enterprise/docker-ee-trialhttps://puppet.com/download-puppet-enterprisehttps://www.upguard.com/demohttps://www.nrgglobal.com/regression-testing-appverify-download

Aziro Marketing

blogImage

How to Make MS Azure Compatible with Chef

Let’s get an overview of Microsoft Azure cloud and, how the popular configuration management tool Chef can be installed on Azure for making them work together.Chef IntroductionChef is a configuration management tool that turns infrastructure into code. You can easily configure your server with the help of Chef. Chef will help you automate, build, deploy, and manage the infrastructure process.If you want to know more about Chef, please refer to https://docs.chef.io/index.html.In order to know how you can create a user account on hosted Chef, please refer to:https://manage.chef.io/signup; alternatively, use open-source Chef by referring to:https://docs.chef.io/install_server.html.Microsoft AzureMicrosoft Azure is a cloud computing platform and infrastructure created by Microsoft for building, deploying, and managing applications and services through a global network of Microsoft-managed datacenters.It provides both PaaS and IaaS services and supports many different programming languages, tools, and frameworks, including both Microsoft-specific and third-party software and systems.For more details refer to: http://azure.microsoft.com/en-in/ andhttp://msdn.microsoft.com/en-us/library/azure/dd163896.aspx.There are three ways to install Chef extension on Azure cloud:By using Azure Portal2. Azure PowerShell CLI3. Knife-azure – The Chef’s CLI tool for Azure providerPrerequisites  Active account on the Azure cloud; https://manage.windowsazure.com or https://portal.azure.comActive account on hosted Chef; See https://manage.chef.io/signup.We need your Chef’s account organization_validation key, rb and run_list.Sample format for client.rb file­:log_location: STDOUT chef_server_url  "https://api.opscode.com/organizations/" validation_client_name "-validator" +36+91. From Azure portal log into your azure account at https://portal.azure.com1.1 List existing virtual machines:1.2 Select existing VM:1.3 Click the Extensions section:1.4 Select Add Extension:1.5 Select the Chef extension:1.6 Click the Create button:1.7 Upload Chef configuration files:1.8 You can now see the Chef extension for VM: 2. Azure PowerShell CLI ToolAzure PowerShell is a command line tool used to manage Azure cloud resources. You can use cmdlets to perform the same tasks that you can perform from the Azure portal.Refer- http://msdn.microsoft.com/en-us/library/azure/jj156055.aspxPrerequisitesInstall Azure PowerShell Tool; refer to http://azure.microsoft.com/en-in/documentation/articles/install-configure-powershell/Azure user accounts publish settings file.We are going to use Azure PowerShell cmdlets to install the Chef extension on Azure VM.2.1 Import your Azure user account into your PowerShell Session. Download subscription credentials for accessing Azure. This can be done by executing a cmdlet.PS C:\> Get-AzurePublishSettingsFile It will launch your browser and download the credentials file. PS C:\> Import-AzurePublishSettingsFile PS C:\> Select-AzureSubscription -SubscriptionName "" PS C:\> Set-AzureSubscription -SubscriptionName "" -CurrentStorageAccountName ""2.2 Create a new Azure VM and install the Chef extension# Set VM and Cloud Service names PS C:\> $vm1 = "azurechef" PS C:\> $svc = "azurechef" PS C:\> $username = 'azure' PS C:\> $password = 'azure@123' PS C:\> $img = #Note- Try Get-AzureVMImage cmdlet to list images PS C:\> $vmObj1 = New-AzureVMConfig -Name $vm1 -InstanceSize Small -ImageName $img #Add-AzureProvisioningConfig  for Windows OR Linux VM # For Windows VM     PS C:\> $vmObj1 = Add-AzureProvisioningConfig -VM $vmObj1 -Password $password -AdminUsername $username –Windowsor# For Linux VM     PS C:\> $vmObj1 = Add-AzureProvisioningConfig -VM $vmObj1 -Password $password -LinuxUser $username -Linux # Set AzureVMChefExtension for Windows OR Linux VM # For Windows VM     PS C:\> $vmObj1 = Set-AzureVMChefExtension -VM $vmObj1 -ValidationPem "C:\\users\\azure\ \msazurechef-validator.pem" -ClientRb "C:\\users\\azure\\client.rb" -RunList "getting-started" -Windowsor# For Linux VM      PS C:\> $vmObj1 = Set-AzureVMChefExtension -VM $vmObj1 -ValidationPem "C:\\users\\azure\ \msazurechef-validator.pem" -ClientRb "C:\\users\\azure\\client.rb" -RunList "getting-started" -Linux # Create VM     PS C:\> New-AzureVM -Location 'West US' -ServiceName $svc -VM $vObj12.3 Install SetAzureVMChefExtension on existing azure VM:# Get existing azure VM     PS C:\> $vmObj1 = Get-AzureVM -ServiceName  -Name # Set AzureVMChefExtension for Windows OR Linux VM # For Windows VM     PS C:\> $vmObj1 = Set-AzureVMChefExtension -VM $vmObj1 -ValidationPem "C:\\users\\azure\ \msazurechef-validator.pem" -ClientRb "C:\\users\\azure\\client.rb" -RunList "getting-started" -Windowsor# For Linux VM     PS C:\> $vmObj1 = Set-AzureVMChefExtension -VM $vmObj1 -ValidationPem "C:\\users\\azure\ \msazurechef-validator.pem" -ClientRb "C:\\users\\azure\\client.rb" -RunList "getting-started" -Linux2.4 You can use following cmdlets to Remove Chef Extension from VM# Get existing azure VM PS C:\> $vmObj1 = Get-AzureVM -ServiceName  -Name # Remove Chef Extension from VM PS C:\> Remove-AzureVMChefExtension -VM $vmObj1 # Update VM PS C:\> Update-AzureVM -ServiceName $vmName -Name $vmName -VM $vmObj12.5 You can get current state of Chef Extension by using following cmdlet:# Get existing azure VM PS C:\> $vmObj1 = Get-AzureVM -ServiceName  -Name # Get Chef Extension details from VM PS C:\> Set-AzureVMChefExtension -VM $vmObj13. Knife-Azure Chef’s Plugin: A knife plugin to create, delete, and enumerate Windows Azure resources to be managed by Chef. The knife-azure plugin (v1.4.0.rc.0) gives features to create VM and install chef extension on Windows Azure Cloud.For more details refer to https://docs.chef.io/plugin_knife_azure.html orhttps://github.com/opscode/knife-azurePrerequisites: ruby v1.9.3 +chef v11.0 +knife-azure v1.4.0.rc.0 pluginAzure user account publishsettings fileChef user accounts configuration filesInstall Ruby:On Windows- http://rubyinstaller.org/On Linux- https://rvm.io/Install Chef:$ gem install chefInstall knife-azure plugin:$ gem install knife-azure --preDownload Chef’s Starter Kit:This starter kit includes Chef’s user/organization related configuration details. i.e., user.pem,organization-validator.pem and knife.rb files.Please refer to: https://learn.chef.io/legacy/get-started/#installthestarterkit orhttps://manage.chef.io/starter-kit.Run knife azure command to create VM and install chef extension-Create Windows VM command:$ knife azure server create  --azure-source-image   --azure-dns-name   --azure-service-location " " --winrm-user  --winrm-password   --azure-publish-settings-file  -c  --bootstrap-protocol "cloud-api"Create Linux VM command:$ knife azure server create -I  -x  -P  --bootstrap-protocol "cloud-api" -c  --azure-service-location" " --azure-publish-settings-fileNote- To get$ knife azure image list -c   --azure-publish-settings-fileMicrosoft Azure is the leading public cloud platform out there, and Chef is one of the most sought after continuous integration and delivery tool. When they come together, ramifications can be great. Please share your comments and questions below. 

Aziro Marketing

blogImage

Learn How to orchestrate Your Infrastructure Fleet with Chef Provisioning

Chef Provisioning is a relatively new member in the Chef family. It can be used to build infrastructure topologies using the new machine resource. This blog post shows how this is done. You bring up and configure individual nodes with Chef all the time. Your standard workflow would be to bootstrap a node, register the node to a Chef server, and then run Chef client to install software and configure the node. You would rinse and repeat this step for every node that you want in your fleet. Maybe you have written a nice wrapper over Chef and Knife to manage your clusters using Chef. Until recently, Chef did not have any way to understand the concept of cluster or fleet. So if you were running a web application with some decent traffic,there would be a bunch of cookbooks and recipes to install and configure: web servers, DB server, background processor, load balancer, etc. Sometimes, you might have additional nodes for Redis or RabbitMQ. So let us say, your cluster consists of three web servers, one DB server, one server that does all the background processing, like generate PDFs or send emails etc., and one load balancer for the three web servers. Now if you wanted to bring such a cluster for multiple environments, say “testing”, “staging,” and “production,” you would have to repeat the steps for each environment; not to mention, your environments could possibly be powered by different providers–production and staging on AWS, Azure, etc. But testing could possible be on local infrastructure, maybe in VMs. This is not difficult, but it definitely makes you wonder whether you could do it better–if only you could describe your infrastructure as code that comes up with just one command. That is exactly what Chef Provisioning does. Chef Provisioning was introduced in Chef version 12. This helps you describe your cluster as code and build it at will as many times as you want and on various types of clouds, virtual machines, or even on bare metal. The Concepts Chef provisioning depends on two main pillars–machine resource and drivers. Machine Resource “machine” is an abstract concept of a node from your infrastructure topology. It could be an AWS EC2 instance or a node on some other cloud provider. It could be a Vagrant-based virtual machine, a Linux container, or a Docker instance. It could even be a real, physical bare-metal machine. “machine” and other related resources (like machine_batch, machine_image, etc.,) can be used to describe your cluster infrastructure. Each “machine” resource describes whatever it does using standard Chef recipes. General convention is to describe your fleet and its topologies using “machine” and other resources in a separate file. We will see this in detail soon, but for now here is how a machine is described. #setup-cluster.rb machine 'server' do recipe 'nginx' end machine 'db' do recipe 'mysql' end A recipe is one of a “machine” resource’s attributes. Later we will see a few more of these along with their examples. Drivers As mentioned earlier, with Chef Provisioning you can describe your clusters and their topologies and then deploy them across a variety of clouds, VMs, bare metal, etc. For each such cloud or machine that you would like to provision, there are drivers that do the actual heavy lifting. Drivers convert the abstract “machine” descriptions into physical reality. Drivers are responsible for acquiring the node data, connecting to them via required protocol, bootstrapping them with Chef, and running the recipes described in the “machine” resource. Provisioning drivers need to be installed separately as gems. Following shows how to install and use AWS driver via environment variables in your system. $ gem install chef-provisioning-aws $ export CHEF_DRIVER=aws Running Chef-client on the above recipe will create two instances in your AWS account referenced by your settings in “~/.aws/config.” We will see an example run later in the post. Driver can be set in your knife.rb if you so prefer. Here, we set the chef-provisioning-fog driver for AWS. driver 'fog:AWS' It is possible to set driver inline in the cluster recipe code. require 'chef/provisioning/aws_driver' with_driver 'aws' machine 'server' do recipe ‘web-server-app' end In the following example, Vagrant driver is given the driver attribute and a driver URL as the value. “/opt/vagrantfiles” will be looked up for Vagrantfiles in the following case. machine 'server' do driver 'vagrant:/opt/vagrantfiles' recipe 'web-server-app' end It’s a good practice to keep driver details and cluster code separate as it lets you use the same cluster descriptions with different provisioners by just changing the driver in the environment. It is possible to write your own custom provisioning drivers. But that is beyond the scope of this blog post. The Provisioner Node An interesting concept you need to understand is that Chef Provisioner needs a “provisioner-node” to provision all machines. This node could be a node in your infrastructure or simply your workstation. chef-client (or chef-solo / chef-zero) runs on this “provisioner node” against a recipe that defines your cluster recipe. Chef Provisioner then takes care of acquiring a node in your infrastructure, bootstrapping it with Chef, and then running the required recipes on the node. Thus, you will see that chef-client runs twice–once on the provisioner node and then on the node that is being provisioned. The Real Thing Let us dig a deeper now. Let us first bring up a single DB server. Using Chef knife you can upload your cookbooks to the Chef server (you could do it with chef-zero as well). Here I have put all my required recipes in a cookbook called “cluster” and uploaded it to a Chef server and set the “chef_server_url” in my “client.rb” and “knife.rb”. You can find all the examples here. Machine #recipes/webapp.rb require 'chef/provisioning' machine 'db' do recipe ‘database-server’ end machine 'webapp' do recipe 'web-app-stack' end To run the above recipe: sudo CHEF_DRIVER=aws chef-client -r "recipe["cluster::webapp"]" This should bring up two nodes in your infrastructure — a DB server and a web application server as defined by the web-app-stack recipe. The above command assumes that you have uploaded the cluster cookbook consisting of the required recipes to the Chef server. More Machine Goodies Like any other Chef resource, machine can have multiple actions and attributes that can be used to achieve different results. A “machine” can have a “chef_server” attribute, which means different machines can talk to different Chef servers. “from_image” attribute can be used to set a machine image that can be used to create a machine. You can read more about machine resource here. Parallelisation Using machine_batch Now if you would like to have more than one web application instances in your cluster and you need more web app servers, say 5 instances, what do you do? Run a loop over your machine resource. 1.upto(5) do |i| machine "webapp-#{i}" do recipe 'web-app-stack' end end The above code snippet, when run, should bring up and configure five instances in parallel. “machine” resource parallelizes by default. If you describe multiple “machine” resources consecutively with same actions, then Chef Provisioning combines them into a single (“machine_batch”, more about this later) resource and runs it in parallel. This is great because it saves a lot of time. The following will not parallelize because the actions are different. machine 'webapp' do action :setup end machine 'db' do action :destroy end Note: if you put other resources between “machine” resources, the automatic parallelization does not happen. machine 'webapp' do action :setup end remote_file 'somefile.tar.gz' do url 'https://example.com/somefile.tar.gz' end machine 'db' do action :setup end Also, you can explicitly turn off parallelization by setting “auto_batch_machines = false” in Chef config (knife.rb or client.rb). Using “machine_batch” explicitly, we can parallelize and speed up provisioning for multiple machines. machine_batch do action :setup machines 'web-app-stack', 'db' Machine Image It is even possible to define machine images using a “machine_image” resource which can be used to build machines by the “machine” resource. machine_image 'web_stack_image' do recipe ‘web-app-stack’ end The above code will launch a machine using your chosen driver, install and configure the node as per the given recipes, create an image from this machine, and finally destroy the machine. This is quite similar to how Packer tool launches a node, configures it, and then freezes it as image before destroying the node. machine 'web-app-stack' do from_image 'web_stack_image' end Here a machine “web-app-stack” when launched will already have everything in the recipe “web-app-stack”. This saves a lot of time when you want to spin up machines, which have common base recipes. Think of a situation where team members need machines with some common stuff installed, and different people install their own specific things as per requirement. In such a case, one could create an image with the basic packages e.g., build-essential, ruby, vim, etc., and that base image could use a source machine image for further work. Load Balancer A very common scenario is to put a bunch of machines, say web-application-servers, behind a load balancer thus achieving redundancy. Chef Provisioning has a resource specifically for load balancers, aptly called “load_balancer”. All you need to do is create the machine nodes and then pass the machines to a “load_balancer” as below. 1.upto(2) do |node_id| machine “web-app-stack-#{node_id}” end load_balancer "web-app-load-balancer" do machines %w(web-app-stack-1 web-app-stack-2) end The above code will bring up two nodes–webapp-stack-1 and webapp-stack-2 and put a load balancer in front of them. Final Thoughts If you are using the AWS driver, you can set machine_options as below. This is important if you want to use customized AMIs, users, security groups, etc. with_machine_options :ssh_username => '', :bootstrap_options => { :key_name => '', :image_id => ‘', :instance_type => ‘’, :security_group_ids => '' } If you don’t provide the AMI ID, the AWS driver defaults to a certain AMI per region. Whatever AMI you use, you have to use the correct ssh username for the respective AMI. [3] One very important thing to note would be that there exists a Fog driver (chef-provisioning-fog) for various cloud services including EC2. So, there are often different names for the parameters that you might want to use. For example, the chef-provisioning-aws driver that depends on AWS Ruby SDK uses “instance_type” where as the Fog driver uses “flavor_id”. Security Groups use the key “security_groups_ids” in the AWS driver and takes ID as value, but the Fog driver uses “groups” and takes the name of the Security Group as its value. This can at times lead to confusion if you are moving from one driver to another. At the time of writing this article, I could use the help of the documentation of various drivers. The best way to understand them would be to check the examples provided, run them and learn from them–maybe even read the source code of various drivers to understand how they work. Chef Provisioning recently got bumped to 1.0.0. I would highly recommend to keep an eye on the GitHub issues in case you face some trouble. References [1] https://docs.chef.io/provisioning.html [2] https://github.com/pradeepto/chef-provisioning-playground [3] http://alestic.com/2014/01/ec2-ssh-username [4] https://github.com/chef/chef-provisioning/issues  

Aziro Marketing

EXPLORE ALL TAGS
2019 dockercon
Advanced analytics
Agentic AI
agile
AI
AI ML
AIOps
Amazon Aws
Amazon EC2
Analytics
Analytics tools
AndroidThings
Anomaly Detection
Anomaly monitor
Ansible Test Automation
apache
apache8
Apache Spark RDD
app containerization
application containerization
applications
Application Security
application testing
artificial intelligence
asynchronous replication
automate
automation
automation testing
Autonomous Storage
AWS Lambda
Aziro
Aziro Technologies
big data
Big Data Analytics
big data pipeline
Big Data QA
Big Data Tester
Big Data Testing
bitcoin
blockchain
blog
bluetooth
buildroot
business intelligence
busybox
chef
ci/cd
CI/CD security
cloud
Cloud Analytics
cloud computing
Cloud Cost Optimization
cloud devops
Cloud Infrastructure
Cloud Interoperability
Cloud Native Solution
Cloud Security
cloudstack
cloud storage
Cloud Storage Data
Cloud Storage Security
Codeless Automation
Cognitive analytics
Configuration Management
connected homes
container
Containers
container world 2019
container world conference
continuous-delivery
continuous deployment
continuous integration
Coronavirus
Covid-19
cryptocurrency
cyber security
data-analytics
data backup and recovery
datacenter
data protection
data replication
data-security
data-storage
deep learning
demo
Descriptive analytics
Descriptive analytics tools
development
devops
devops agile
devops automation
DEVOPS CERTIFICATION
devops monitoring
DevOps QA
DevOps Security
DevOps testing
DevSecOps
Digital Transformation
disaster recovery
DMA
docker
dockercon
dockercon 2019
dockercon 2019 san francisco
dockercon usa 2019
docker swarm
DRaaS
edge computing
Embedded AI
embedded-systems
end-to-end-test-automation
FaaS
finance
fintech
FIrebase
flash memory
flash memory summit
FMS2017
GDPR faqs
Glass-Box AI
golang
GraphQL
graphql vs rest
gui testing
habitat
hadoop
hardware-providers
healthcare
Heartfullness
High Performance Computing
Holistic Life
HPC
Hybrid-Cloud
hyper-converged
hyper-v
IaaS
IaaS Security
icinga
icinga for monitoring
Image Recognition 2024
infographic
InSpec
internet-of-things
investing
iot
iot application
iot testing
java 8 streams
javascript
jenkins
KubeCon
kubernetes
kubernetesday
kubernetesday bangalore
libstorage
linux
litecoin
log analytics
Log mining
Low-Code
Low-Code No-Code Platforms
Loyalty
machine-learning
Meditation
Microservices
migration
Mindfulness
ML
mobile-application-testing
mobile-automation-testing
monitoring tools
Mutli-Cloud
network
network file storage
new features
NFS
NVMe
NVMEof
NVMes
Online Education
opensource
openstack
opscode-2
OSS
others
Paas
PDLC
Positivty
predictive analytics
Predictive analytics tools
prescriptive analysis
private-cloud
product sustenance
programming language
public cloud
qa
qa automation
quality-assurance
Rapid Application Development
raspberry pi
RDMA
real time analytics
realtime analytics platforms
Real-time data analytics
Recovery
Recovery as a service
recovery as service
Retail
rsa
rsa 2019
rsa 2019 san francisco
rsac 2018
rsa conference
rsa conference 2019
rsa usa 2019
SaaS Security
san francisco
SDC India 2019
SDDC
security
Security Monitoring
Selenium Test Automation
selenium testng
serverless
Serverless Computing
Site Reliability Engineering
smart homes
smart mirror
SNIA
snia india 2019
SNIA SDC 2019
SNIA SDC INDIA
SNIA SDC USA
software
software defined storage
software-testing
software testing trends
software testing trends 2019
SRE
STaaS
storage
storage events
storage replication
Storage Trends 2018
storage virtualization
support
Synchronous Replication
technology
tech support
test-automation
Testing
testing automation tools
thought leadership articles
trends
tutorials
ui automation testing
ui testing
ui testing automation
vCenter Operations Manager
vCOPS
virtualization
VMware
vmworld
VMworld 2019
vmworld 2019 san francisco
VMworld 2019 US
vROM
Web Automation Testing
web test automation
WFH

Real People, Real Replies.

That's how it usually goes, you reach out with a question or a rough idea, a real person at Aziro responds, and a conversation turns into something big.
Start yours with us.

Phone

Talk to us

+1 227 232 3176

Email

Drop us a line at

info@aziro.com

Got a Tech Challenge? Let’s Talk

Country code

Got a Tech Challenge? Let’s Talk

Country code