Tag Archive

Below you'll find a list of all posts that have been tagged as "opscode-2"

10 Steps to Setup and Manage a Hadoop Cluster Using Ironfan

Recently, we faced a unique challenge – setup DevOps and management for a relatively complex Hadoop cluster on the Amazon EC2 Cloud. The obvious choice was to use a configuration management tool. Having extensively used Opscode’s Chef and given the flexibility and extensibility Chef provides; it was an obvious choice. While looking around for the best practices to manage a hadoop cluster using Chef, we stumbled upon: Ironfan What is Ironfan? In short Ironfan, open-souced by InfoChimps provides an abstraction on top of Chef, allowing users to easily provision, deploy and manage a cluster of servers – be it a simple web application or a complex Hadoop cluster. After a few experiments, we were convinced that Ironfan was the right thing to use as it simplifies a lot of configuration avoiding repetition while retaining the goodness of Chef. This blog shows how easy it is to setup and manage a Hadoop cluster using Ironfan. Pre-requisties: Chef Account (Hosted or Private) with knife.rb setup correctly on your client machine. Ruby setup (using RVM or otherwise) Installation: Now you can install IronFan on your machine using the steps mentioned here. Once you have all the packages setup correctly, perform these sanity checks: Ensure that the environment variable CHEF_USERNAME is your Chef Server username (unless your USER environment variable is the same as your Chef username) Ensure the the environment variable CHEF_HOMEBASE points to the location which contains the expanded out knife.rb ~/.chef should be a symbolic link to your knife directory in the CHEF_HOMEBASE Your knife/knife.rb file is not modified. Your Chef user PEM file should be in knife/credentials/{username}.pem Your organization’s Chef validator PEM file should be in knife/credentials/{organization}-validator.pem Your knife/credentials/knife-{organization}.rb file Should contain your Chef organization Should contain the chef_server_url Should contain the validation_client_name Should contain path to validation_key Should contain the aws_access_key_id/ aws_secret_access_key Should contain an AMI ID of an AMI you’d like to be able to boot in ec2_image_info Finally in the homebase rename the example_clusters directory to clusters. These are sample clusters than comes with Ironfan. Perform a knife cluster list command : $ knife cluster list Cluster Path: /.../homebase/clusters +----------------+-------------------------------------------------------+ | cluster | path | +----------------+-------------------------------------------------------+ | big_hadoop | /.../homebase/clusters/big_hadoop.rb | | burninator | /.../homebase/clusters/burninator.rb | ... Defining Cluster: Now lets define a cluster. A Cluster in IronFan is defined by a single file which describes all the configurations essential for a cluster. You can customize your cluster spec as follows: Define cloud provider settings Define base roles Define various facets Defining facet specific roles and recipes. Override properties of a particular facet server instance. Defining cloud provider settings: IronFan currently supports AWS and Rackspace Cloud providers. We will take an example of AWS cloud provider. For AWS you can provide config information like: Region, in which the servers will be deployed. Availibility zone to be used. EBS backed or Instance-Store backed servers Base Image(AMIs) to be used to spawn servers Security zone with the allowed port range. Defining Base Roles: You can define the global roles for a cluster. These roles will be applied to all servers unless explicitly overridden for any particular facet or server. All the available roles are defined in $CHEF_HOMEBASE/roles directory. You can create a custom role and use it in your cluster config. Defining Environment: Environments in Chef provide a mechanism for managing different environments such as production, staging, development, and testing, etc with one Chef setup (or one organization on Hosted Chef). With environments, you can specify per environment run lists in roles, per environment cookbook versions, and environment attributes. The available environments can be found in $CHEF_HOMEBASE/environments directory. Custom environments can be created and used. Ironfan.cluster 'my_first_cluster' do # Enviornment under which chef nodes will be placed environment :dev # Global roles for all servers role :systemwide role :ssh # Global ec2 cloud settings cloud(:ec2) do permanent true region 'us-east-1' availability_zones ['us-east-1c', 'us-east-1d'] flavor 't1.micro' backing 'ebs' image_name 'ironfan-natty' chef_client_script 'client.rb' security_group(:ssh).authorize_port_range(22..22) mount_ephemerals end Defining Facets: Facets are group of servers within a cluster. Facets share common attributes and roles. For example, in your cluster you have 2 app servers and 2 database servers then you can group the app servers under the app_server facet and the database servers under the database facet. Defining Facet specific roles and recipes: You can define roles and recipes particular to a facet. Even the global cloud settings can be overridden for a particular facet. facet :master do instances 1 recipe ‘nginx’ cloud(:ec2) do flavor ‘m1.small’ security_group(:web) do authorize_port_range(80..80) authorize_port_range(443..443) role :hadoop_namenode role :hadoop_secondarynn role :hadoop_jobtracker role :hadoop_datanode role :hadoop_tasktracker end facet :worker do instances 2 role :hadoop_datanode role :hadoop_tasktracker end In the above example we have defined a facet for Hadoop master node and a facet for worker node. The number of instances of master is set to 1 and that of worker is set to 2. Each master and worker facets have been assigned a set of roles. For master facet we have overridden the ec2 flavor settings as m1.medium. Also the security group for the master node is set to accept incoming traffic on port 80 and 443. Cluster Management: Now that we are ready with the cluster configuration lets get a hands on cluster management. All the cluster configuration files are placed under the $CHEF_HOMEBASE/clusters directory. We will place our new config file as hadoop_job001_cluster.rb. Now our new cluster should be listed in the cluster list. List Clusters: $ knife cluster list Cluster Path: /.../homebase/clusters +-------------+-------------------------+ | cluster | path | +-------------+-------------------------+ hadoop_job001 HOMEBASE/clusters/hadoop_job001_cluster.rb +-------------+-------------------------+ Show Cluster Configuration: $ knife cluster show hadoop_job001 Inventorying servers in hadoop_job001 cluster, all facets, all servers my_first_cluster: Loading chef my_first_cluster: Loading ec2 my_first_cluster: Reconciling DSL and provider information +-----------------------------+-------+-------------+----------+------------+-----+ | Name | Chef? | State | Flavor | AZ | Env | +-----------------------------+-------+-------------+----------+------------+-----+ | hadoop_job001-master-0 | no | not running | m1.small | us-east-1c | dev | | hadoop_job001-client-0 | no | not running | t1.micro | us-east-1c | dev | | hadoop_job001-client-1 | no | not running | t1.micro | us-east-1c | dev | +-----------------------------+-------+-------------+----------+------------+-----+ Launch Cluster: Launch Whole Cluster: $ knife cluster launch hadoop_job001 Loaded information for 3 computer(s) in cluster my_first_cluster +-----------------------------+-------+---------+----------+------------+-----+------------+--------- -------+----------------+------------+ | Name | Chef? | State | Flavor | AZ | Env | MachineID | Public IP | Private IP | Created On | +-----------------------------+-------+---------+----------+------------+-----+------------+----------------+----------------+------------+ | hadoop_job001-master-0 | yes | running | m1.small | us-east-1c | dev | i-c9e117b5 | 101.23.157.51 | 10.106.57.77 | 2012-12-10 | | hadoop_job001-client-0 | yes | running | t1.micro | us-east-1c | dev | i-cfe117b3 | 101.23.157.52 | 10.106.57.78 | 2012-12-10 | | hadoop_job001-client-1 | yes | running | t1.micro | us-east-1c | dev | i-cbe117b7 | 101.23.157.52 | 10.106.57.79 | 2012-12-10 | +-----------------------------+-------+---------+----------+------------+-----+------------+----------------+----------------+------------+ Launch a single instance of a facet: $ knife cluster launch hadoop_job001 master 0 Launch all instances of a facet: $ knife cluster launch hadoop_job001 worker Stop Whole Cluster: $ knife cluster stop hadoop_job001 Stop a single instance of a facet: $ knife cluster stop hadoop_job001 master 0 Stop all instances of a facet: $ knife cluster stop hadoop_job001 Setting up a Hadoop cluster and managing it cannot get easier than this! Just to re-cap, Ironfan, open-souced by InfoChimps, is a systems provisioning and deployment tool which automates entire systems configuration to enable the entire Big Data stack, including tools for data ingestion, scraping, storage, computation, and monitoring. There is another tool that we are exploring for Hadoop cluster management – Apache Ambari. We will post our findings and comparisons soon, stay tuned!

Aziro Marketing

Amazon EC2 big data cloud computing hadoop opscode-2 others

Building Package Using Omnibus

Omnibus is a tool for creating full-stack installers for multiple platforms. In general, it simplifies the installation of any software by including all of the dependencies for that piece of software. It was written by the people at Chef, who use it to package Chef. Omnibus consists of two pieces- omnibus and omnibus software. omnibus – the framework, created by Chef Software, by which we create full-stack, cross-platform installers for software. The project is on GitHub at chef/omnibus. omnibus-software – Chef Software’s open source collection of software definitions that are used to build the Chef Client, the Chef Server, and other Chef Software products. The software definitions can be found on GitHub at chef/omnibus-software. Omnibus provides both, a DSL for defining Omnibus projects for your software, as well as a command-line tool for generating installer artifacts from that definition. Omnibus has minimal prerequisites. It requires Ruby 2.0.0+ and Bundler. Getting Started To get started install omnibus > gem install omnibus You can now create an omnibus project inside your current directory using project generator feature > omnibus new demo This will generate a complete project skeleton in the directory as following: create omnibus-demo/Gemfile create omnibus-demo/.gitignore create omnibus-demo/README.md create omnibus-demo/omnibus.rb create omnibus-demo/config/projects/demo.rb create omnibus-demo/config/software/demo-zlib.rb create omnibus-demo/.kitchen.local.yml create omnibus-demo/.kitchen.yml create omnibus-demo/Berksfile create omnibus-demo/package-scripts/demo/preinst chmod omnibus-demo/package-scripts/demo/preinst create omnibus-demo/package-scripts/demo/prerm chmod omnibus-demo/package-scripts/demo/prerm create omnibus-demo/package-scripts/demo/postinst chmod omnibus-demo/package-scripts/demo/postinst create omnibus-demo/package-scripts/demo/postrm chmod omnibus-demo/package-scripts/demo/postrm It creates the omnibus-demo directory inside your current directory and this directory has all omnibus package build related files. It is easy to build an empty project without doing any change run > bundle install --binstubs bundle install installs all Omnibus dependencies bundle install installs all Omnibus dependencies The above command will create the installer inside pkg directory. Omnibus determines the platform for which to build an installer based on the platform it is currently running on. That is, you can only generate a .deb file on a Debian-based system. To alleviate this caveat, the generated project includes a Test Kitchen setup suitable for generating a series of Omnibus projects. Back to the Omnibus DSL. Though bin/omnibus build demo will build the package for you, it will not do anything exciting. For that, you need to use the Omnibus DSL to define the specifics of your application. 1) Config If present, Omnibus will use a top-level configuration file name omnibus.rb at the root of your repository. This file is loaded at runtime and includes number of configurations. For e.g.- omnibus.rb # Build locally (instead of /var) # ------------------------------- base_dir './local' # Disable git caching # ------------------------------ use_git_caching false # Enable S3 asset caching # ------------------------------ use_s3_caching true s3_access_key ENV['S3_ACCESS_KEY'] s3_secret_key ENV['S3_SECRET_KEY'] s3_bucket ENV['S3_BUCKET'] Please see config doc for more information. You can use different configuration file by using –config option using command line $ bin/omnibus --config /path/to/config.rb 2) Project DSL When you create an omnibus project, it creates a project DSL file inside config/project with the name which you used for creating project for above example it will create config/project/demo.rb. It provides means to define the dependencies of the project and metadata of the project. We will look at some contents of project DSL file name "demo" maintainer "YOUR NAME" homepage "http://yoursite.com" install_dir "/opt/demo" build_version "0.1.0" # Creates required build directories dependency "preparation" # demo dependencies/components dependency "harvester" ‘install_dir’ option is the location where package will be installed. There are more DSL methods available which you can use in this file. Please see the Project Doc for more information. 3) Software DSL Software DSL defines individual software components that go into making your overall package. The Software DSL provides a way to define where to retrieve the software sources, how to build them, and what dependencies they have. Now let’s edit a config/software/demo.rb name "demo" default_version "1.0.0" dependency "ruby" dependency "rubygems" build do #vendor the gems required by the app bundle “install –path vendor/bundle” end In the above example, consider that we are building a package for Ruby on Rails application, hence we need to include ruby and rubygems dependency. The definition for ruby and rubygems dependency comes from the omnibus-software. Chef has introduced omnibus-software, it is a collection of software definitions used by chef while building their products. To use omnibus-software definitions you need to include the repo path in Gemfile. You can also write your own software definitions. Inside build block you can define how to build your installer. Omnibus provide Build DSL which you can use inside build block to define your build essentials. You can run ruby script and copy and delete files using Build DSL inside build block. Apart from all these DSL file omnibus also created ‘package-script’ directory which consist of pre install and post install script files. You can write a script which you want to run before and after the installation of package and also before and after the removal of the package inside these files. You can use the following references for more examples https://github.com/chef/omnibus https://www.chef.io/blog/2014/06/30/omnibus-a-look-forward/

Aziro Marketing

chef devops opscode-2

Chef Knife Plugin for Windows Azure (IAAS)

Chef is an open-source systems management and cloud infrastructure automation framework created by Opscode. It helps in managing your IT infrastructure and applications as code. It gives you a way to automate your infrastructure and processes. Knife is a CLI to create, update, search and delete the entities or manage actions on entities in your infrastructure like node (hosts), cloud resources, metadata (roles, environments) and code for infrastructure (recipes, cookbooks), etc. A Knife plug-in is a set of one (or more) subcommands that can be added to Knife to support additional functionality that is not built-in to the base set of Knife subcommands. The knife azure is a knife plugin which helps you automate virtual machine provisioning in Windows Azure and bootstrapping it. This article talks about using Chef and knife-azure plugin to provision Windows/Linux virtual machines in Windows Azure and bootstrapping the virtual machine. Understanding Windows Azure (IaaS): To deploy a Virtual Machine in a region (or service location) in Azure, all the components shown described above have to be created; A Virtual Machine is associated with a DNS (or cloud service). Multiple Virtual Machines can be associated with a single DNS with load-balancing enabled on certain ports (eg. 80, 443 etc). A Virtual Machine has a storage account associated with it which storages OS and Data disks A X509 certificate is required for password-less SSH authentication on Linux VMs and HTTPS-based WinRM authentication for Windows VMs. A service location is a geographic region in which to create the VMs, Storage accounts etc The Storage Account The storage account holds all the disks (OS as well as data). It is recommended that you create a storage account in a region and use it for the VMs in that region. If you provide the option –azure-storage-account, knife-azure plugin creates a new storage account with that name if it doesnt already exist. It uses this storage account to create your VM. If you do not specify the option, then the plugin checks for an existing storage account in the service location you have mentioned (using option –service-location). If no storage account exists in your location, then it creates a new storage with name prefixed with the azure-dns-name and suffixed with a 10 char random string. Azure Virtual Machine This is also called as Role(specified using option –azure-vm-name). If you do not specify the VM name, the default VM name is taken from the DNS name( specified using option –azure-dns-name). The VM name should be unique within a deployment. An Azure VM is analogous to the Amazon EC2 instance. Like an instance in Amazon is created from an AMI, you can create an Azure VM from the stock images provided by Azure. You can also create your own images and save them against your subscription. Azure DNS This is also called as Hosted Service or Cloud Service. It is a container for your application deployments in Azure( specified using option –azure-dns-name). A cloud service is created for each azure deployment. You can have multiple VMs(Roles) within a deployment with certain ports configured as load-balanced. OS Disk A disk is a VHD that you can boot and mount as a running version of an operating system. After an image is provisioned, it becomes a disk. A disk is always created when you use an image to create a virtual machine. Any VHD that is attached to virtualized hardware and that is running as part of a service is a disk. An existing OS Disk can be used (specified using option –azure-os-disk-name ) to create a VM as well. Certificates For SSH login without password, an X509 Certificate needs to be uploaded to the Azure DNS/Hosted service. As an end user, simply specify your private RSA key using –identity-file option and the knife plugin takes care of generating a X509 certificate. The virtual machine which is spawned then contains the required SSH thumbprint. I am text block. Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo. Gem Install Run the command gem install knife-azure Install from Source Code To get the latest changes in the knife azure plugin, download the source code, build and install the plugin: 1. Uninstall any existing versions $ gem uninstall knife-azure Successfully uninstalled knife-azure-1.2.0 2. Clone the git repo and build the code $ git clone https://github.com/opscode/knife-azure $ cd knife-azure $ gem build knife-azure.gemspec WARNING: description and summary are identical Successfully built RubyGem Name: knife-azure Version: 1.2.0 File: knife-azure-1.2.0.gem 3. Install the gem $ gem install knife-azure-1.2.0.gem Successfully installed knife-azure-1.2.0 1 gem installed Installing ri documentation for knife-azure-1.2.0... Building YARD (yri) index for knife-azure-1.2.0... Installing RDoc documentation for knife-azure-1.2.0... 4. Verify your installation $ gem list | grep azure knife-azure (1.2.0) To provision a VM in Windows Azure and bootstrap using knife, Firstly, create a new windows azure account: at this link and secondly, download the publish settings file fromhttps://manage.windowsazure.com/publishsettings The publish settings file contains certificates used to sign all the HTTP requests (REST APIs). Azure supports two modes to create virtual machines – quick create and advanced. Azure VM Quick Create You can create a server with minimal configuration. On the Azure Management Portal, this corresponds to the “Quick Create – Virtual Machine” workflow. The corresponding sample command for quick create for a small Windows instance is: knife azure server create --azure-publish-settings-file '/path/to/your/cert.publishsettingsfile' --azure-dns-name 'myservice' --azure-source-image 'windows-image-name' --winrm-password 'jetstream@123' --template-file 'windows-chef-client-msi.erb' --azure-service-location "West US" Azure VM Advanced Create You can set various other options in the advanced create including service location or region, storage-account, VM name etc. The corresponding command to create a Linux instance with advanced options is: knife azure server create --azure-publish-settings-file "path/to/your/publish/settings/file" --azure-vm-size Medium --azure-dns-name "HelloAzureDNS" --azure-service-location "West US" --azure-vm-name 'myvm01' --azure-source-image "b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-13_04-amd64-server-20130423-en-us-30GB" --azure-storage-account "helloazurestorage1" --ssh-user "helloazure" --identity-file "path/to/your/rsa/pvt/key" To create a VM and connect it to an existing DNS/service, you can use a command as below: knife azure server create --azure-publish-settings-file "path/to/your/publish/settings/file" --azure-connect-to-existing-dns --azure-dns-name 'myservice' --azure-vm-name 'myvm02' --azure-service-location 'West US' --azure-source-image 'source-image-name' --ssh-user 'jetstream' --ssh-password 'jetstream@123' List available Images: knife azure image list List currently available Virtual Machines: knife azure server list Delete and Clean up a Virtual Machine: knife azure server delete --azure-dns-name myvm02 'myservice' --chef-node-name 'myvm02' --purge This post is meant to explain the basics and usage for knife-azure.

Aziro Marketing

chef devops opscode-2 others tutorials

Learn about the Latest Enterprisy Updates to knife-cloudstack!

Opscode’s Chef is open-source systems integration framework built specifically for automating the cloud. Knife is a powerful CLI that is used by administrators to interact with Chef. It is easily extensible to support provisioning of cloud resources. There is currently support for over 15 cloud providers including Amazon EC2, Rackspace, Openstack and Cloudstack. Ever since the acquisition of Cloud.com by Citrix, Cloudstack (now re-christened as Citrix CloudPlatform) is being actively morphed into a more enterprise-focused product with support for Production-grade networking appliances like the Netscalar suite, F5 Big IP, Cisco Nexus 1000V and networking features like InterVLAN communication and Site-to-Site VPN. Continuing in the spirit, the Knife Cloudstack plugin has recently received major updates that are targeted towards enterprises using Cloudstack/Cloudplatform in private environments: Microsoft Windows Server bootstrapping: Microsoft Windows Server is widely used across Enterprises to host a variety of critical internal and external applications including Microsoft Exchange, Sharepoint, CRM. We have added support to easily bootstrap provision and bootstrap Windows machines via the WinRM protocol with ability to use both Basic and Kerberos modes of Authentication. Support for Projects: Cloudstack Projects is one of the widely used feature in Enterprises allowing BUs to isolate their compute, networking and storage resources for better chargeback, billing and management of resources. The plugin now supports the ability to spawn servers, choose networks and allocate IP addresses in specific projects. Choose between Source NAT and Static NAT: Enterprises host certain Applications for their customers, partners or employees on public IP addresses. Hence they prefer to use static NAT (IP forwarding, EC2 Style) rather than Source NAT (Port Forwarding) for increased security and control. Enabling static NAT is as simple as setting a flag. Ability to choose networks: Typically enterprises prefer isolating different types of traffic on different networks. eg. VoIP traffic on a higher QoS networks, separate storage/backup networks and so on. The plugin now adds the ability spawn virtual machines as well as allocate public IP addresses from specific networks. Sample Examples: Windows Bootstrapping knife cs server create --cloudstack-service 'Medium Instance' --cloudstack-template 'w2k8-basic' --winrm-user 'Administrator --winrm-password 'xxxx' --winrm-port 5985 --port-rules "3389:3389:TCP" --bootstrap-protocol winrm --template-file windows-chef-client-msi.erb knife cs server create --cloudstack-service "Medium Instance" --cloudstack-template "w2k8-with-AD" --kerberos-realm "ORG_WIDE_AD_DOMAIN" --winrm-port 5985 --port-rules "3389:3389:TCP" --bootstrap- protocol winrm --template-file windows-chef-client-msi.erb Support for Projects and Static NAT knife cs server create --cloudstack-service 'Medium Instance' --cloudstack-template 'w2k8-basic' --cloudstack-project 'Engg-Dev' --winrm-user 'Administrator --winrm-password 'Fr3sca21!' -- static-nat --port-rules "3389:TCP" --bootstrap-protocol winrm Choose specific networks: knife cs server create "rhel-node-1" --node-name "rhel-node-1" -T "RHEL 5.7-x86" --bootstrap-protocol ssh --ssh-user root --ssh-password **** --service "Small Instance" --networks "Admin-Default" --port-rules '22:tcp' The plugin is available to download from the source at: knife-cloudstack Update: knife-cloudstack-0.0.13 has released to rubygems.org with these changes. gem install knife-cloudstack for the latest

Aziro Marketing

cloud computing cloudstack opscode-2 private-cloud

How to write Ohai plugin for the Windows Azure IaaS cloud

Aziro Marketing

cloud computing devops opscode-2 others tutorials

EXPLORE ALL TAGS

Ansible Test Automation

application containerization

applications

Application Security

application testing

artificial intelligence

asynchronous replication

business intelligence

Cloud Cost Optimization

cloud devops

Cloud Infrastructure

Cloud Interoperability

Cloud Native Solution

Cloud Storage Security

Codeless Automation

Cognitive analytics

Configuration Management

container world conference

continuous-delivery

continuous deployment

continuous integration

data backup and recovery

Descriptive analytics

Descriptive analytics tools

Digital Transformation

dockercon 2019 san francisco

end-to-end-test-automation

High Performance Computing

icinga for monitoring

Image Recognition 2024

kubernetesday bangalore

Low-Code No-Code Platforms

mobile-application-testing

mobile-automation-testing

Predictive analytics tools

prescriptive analysis

Rapid Application Development

raspberry pi

RDMA

real time analytics

realtime analytics platforms

Real-time data analytics

Recovery

Recovery as a service

recovery as service

rsa

rsa 2019

rsa 2019 san francisco

Selenium Test Automation

selenium testng

serverless

Serverless Computing

Site Reliability Engineering

software defined storage

software-testing

software testing trends

software testing trends 2019

storage virtualization

support

Synchronous Replication

testing automation tools

thought leadership articles

trends

tutorials

ui automation testing

ui testing

ui testing automation

vCenter Operations Manager

vmworld 2019 san francisco

VMworld 2019 US

vROM

Web Automation Testing

web test automation

WFH

LET'S ENGINEER

Your Next Product Breakthrough

Book a Free 30-minute Meeting with our technology experts.

Aziro has been a true engineering partner in our digital transformation journey. Their AI-native approach and deep technical expertise helped us modernize our infrastructure and accelerate product delivery without compromising quality. The collaboration has been seamless, efficient, and outcome-driven.

CTO

Fortune 500 company