Sunday, October 25, 2015

Puppet; the Continuous Delivery Tool

 Puppet; a tool that supports to automate application deployment.  Puppet enable you to practice continuous delivery. In this post I provide an an overview of Puppet Open Source continuous delivery tool, and outline it's necessary configurations and installations instructions specific to a Linux CentOS environment with recommended best practices. At the end of this post I have mentioned how to deploy a war file to JBoss Wildfly via it’s command line  tool.

Puppet is an automation software for IT system administrators and consultants. It allows you to automate repetitive tasks such as the installation of applications and services, patch management, and deployments. Configuration for all resources are stored in so called "manifests", that can be applied to multiple machines or just a single server. 

Puppet Open Source Tool have two major components; Puppet Master and Puppet Agent. Those are intended to host in two separate locations where Puppet Master keeps all manifest scripts related to deployment automation while puppet agent's are intended to frequently (in every 30mins of time) communicate with Puppet Masters to detect any updates to configurations and deployment artifacts, and pull them to agent's environment to finish the deployment.



Puppet Master is responsible for keeping agent specific deployment scripts while Puppet Agent is responsible for accessing Puppet Master and automate the deployment. First of all, Puppet Master's 8140 port must be enable to access via Puppet Agent and also both Puppet Master and Puppet Agent hosted servers needs to have their FQDNs registered with a DNS.

Master Configuration

On CentOS/RHEL 6, where iptables is used as firewall, add following line into section ":OUTPUT ACCEPT" of /etc/sysconfig/iptables.

#vim /etc/sysconfig/iptables

Add the following line to iptables to open port 8140.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8140 -j ACCEPT

Close the file after saving it.

Restart the iptables service.

# service iptables restart

Open hosts file to add FQDN of Puppet Master.

#vim /etc/hosts

Add FQDNs to the file.

10.101.15.190 nexus-jenkins.abc.lk
10.101.15.197 dev-179.abc.lk

Close the file after saving it.

Agent Configuration

Puppet client nodes have to know where the Puppet master server is located. The best practice for this is to use a DNS server, where Puppet domain name can be configured. If a DNS server is not available, /etc/hosts file can be modified as follows.

#vim /etc/hosts

Add FQDN of Puppet Master to the file.

10.101.15.197 nexus-jenkins.abc.lk

Close the file after saving it.

Installing Puppet Master

Since Puppet is not in basic CentOS or RHEL distribution repositories, add a custom repository provided by Puppet Labs

# rpm -ivh https://yum.puppetlabs.com/el/6.5/products/x86_64/puppetlabs-release-6-10.noarch.rpm

Install the "puppet-server" module in master server.

#   yum install puppet-server

When the installation is done, set the Puppet server to automatically start on boot and turn it on.

#   chkconfig puppetmaster on
#   service puppetmaster start


Installing Puppet Client

Since Puppet is not in basic CentOS or RHEL distribution repositories, add a custom repository provided by Puppet Labs

# rpm -ivh https://yum.puppetlabs.com/el/6.5/products/x86_64/puppetlabs-release-6-10.noarch.rpm

Install the puppet agent service in agent server.

#   yum install puppet

When the installation is done, set the Puppet server to automatically start on boot and turn it on.

#   yum chkconfig puppet on

Specify the Puppet master servers FQDN in /etc/sysconfig/puppet file.

#   vim /etc/sysconfig/puppet

Add the following line to specify the FQDN of the puppet master.

PUPPET_SERVER=nexus-jenkins.abc.lk

The master server name also has to be defined in the section of agent's puppet configuration file.

# vim /etc/puppet/puppet.conf

Add the following line to specify the master server.

server=nexus-jenkins.abc.lk

Start the puppet client.

# service puppet start

Certificate Verification

Execute the below command in puppet agent to generate a certificate request.

# puppet agent --test

Following error message will be appear in the terminal.

Exiting; no certificate found and waitforcert is disabled

Go back to puppet master server and list all certificate requests by executing the following command.

#   puppet cert list

Sign the certificate by executing the following command in puppet master's terminal.

#   puppet cert sign dev-86.abc.lk

Note: puppet agent's FQDN

Deployment Orchestration

For deployment automations, make sure site.pp file exist in /etc/puppet/manifests directory.

Following instructions are targeted to be placed in Puppet-Master node.

Create the following directory structure using mkdir command.
/etc/puppet/modules/[project_name]/files/
Example:
/etc/puppet/modules/xyz/files/

Open the /etc/puppet/manifests/site.pp file to configure the deployment plan.
# vim /etc/ puppet/manifests/site.pp

Add the following content to the file.

node 'pqr.abc.lk' {
                file { "/tmp/xyz/portal.war":
                                ensure => 'present',
                                mode => 0755,
                                owner => abc,
                                group => abc,
                                source => "puppet:///modules/xyz/portal.war"
                }
                exec { "deploy_portal":
                                command => "/home/abc/wildfly/bin/jboss-cli.sh --connect --command=\"deploy --force /tmp/xyz/portal.war\" "
                }
}


 References:
Installing Puppet: Red Hat Enterprise Linux (and Derivatives) — Documentation — Puppet Labs. 2015. Installing Puppet: Red Hat Enterprise Linux (and Derivatives) — Documentation — Puppet Labs. [ONLINE] Available at: https://docs.puppetlabs.com/guides/install_puppet/install_el.html. [Accessed 05 October 2015].

Installing Puppet: Post-Install Tasks — Documentation — Puppet Labs. 2015. Installing Puppet: Post-Install Tasks — Documentation — Puppet Labs. [ONLINE] Available at: https://docs.puppetlabs.com/guides/install_puppet/post_install.html. [Accessed 05 October 2015].

Language: Node Definitions — Documentation — Puppet Labs. 2015. Language: Node Definitions — Documentation — Puppet Labs. [ONLINE] Available at: https://docs.puppetlabs.com/puppet/3.8/reference/lang_node_definitions.html. [Accessed 05 October 2015].

How to install Puppet server and client on CentOS and RHEL - Xmodulo. 2015. How to install Puppet server and client on CentOS and RHEL - Xmodulo. [ONLINE] Available at: http://xmodulo.com/install-puppet-server-client-centos-rhel.html. [Accessed 05 October 2015].

Saturday, October 24, 2015

Continuous Delivery

Continuous Delivery; A quite new term I found very recently. I was working on a deployment automation task few weeks ago and here I summarize what I learnt about this practice and how it is beneficial to developers.

Continuous deployment is deploying every change that passes automated tests to production; simply it is the practice of releasing every good build to user. It is all about putting the release schedule in the hands of the business rather than in the hands of development team.

Introducing continuous delivery to a project means making sure the software is always production ready throughout its entire lifecycle and ability to interchange among release versions using a fully automated process in a matter of seconds or minutes.

According to Martin Fowler, continuous delivery is when,
  • Software is deployable throughout its lifecycle.
  • Team prioritizes keeping the software deployable over working on features.
  • Anybody can get fast, automated feedback on the production readiness of their systems any time somebody makes a change to them.
  • Perform push-button deployments of any version of the software to any environment on demand.

Incorporation of automation, frequent code releases, testing at every stage of the process, and a pull-based architecture that permits only successful releases to move to the next stage reduces errors and make it easier to improve the software delivery process.

Automation allows making successful processes repeatable. When introducing a new feature, make a change to a service underlying system or infrastructure, automation let us make the change quickly and safely without introducing errors that would result from repeating the process manually.

Releasing code frequently rather than releasing big releases once or twice means testing the product more often. There’s less change in each release, so it’s easier to isolate and fix problems. It’s also easier to roll back when needed.

Pull based architecture prevents passing code that fails automated tests to the next stage of development. This prevents errors propagating and making them harder to diagnose.

Software developers are rewarded for delivering quality software that addresses business needs, on schedule. Continuous delivery practices give software developers the ability to provision themselves with production like environment and automated deployment so they can run automated tests. Instead of standing in their way, the operations team helps developers get their work done. Continuous delivery depends on continuous integration, which means every change is merged into and tested against the main code base, reducing the opportunity for long-standing feature branches and large merge windows that can lead to serious errors. Deployment becomes much less stressful when changes are small and tested at every step.



Above video contains a speech given by Martin Fowler about Continuous Delivery.

Solving "org.jboss.as.cli.CliInitializationException: Failed to connect to the controller" Exception

I had a requirement of deploying an application to a JBoss Wildfly server via its command line tool, which is known as "jboss-cli". I executed below command to deploy the war file, which gave me a "CliInitializationException".
jboss-cli.bat --connect --command="deploy --force E:\Projects\CD&CI\portal.war"

The exception thrown from the jboss-cli tool:

org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:278)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:253)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.modules.Module.run(Module.java:292)
        at org.jboss.modules.Main.main(Main.java:455)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at localhost:9990
        at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:1020)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:832)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:811)
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:276)
        ... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to http-remoting://localhost:9990. The connection timed out
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
        at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:997)
        ... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to http-remoting://localhost:9990. The connection timed out
        at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:120)
        at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
        at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:204)
        at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:169)
        at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:129)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:117)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:92)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
        ... 13 more

Problem is, I was using a native-interface instead of a http-interface so it won't use the correct protocol.

Solution is simple, I had to change the standalone-full.xml file as below to specify the correct interface.

<management-interfaces>
    <!--<http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">
        <socket-binding http="management-http" />
    </http-interface>-->
    <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">
        <socket interface="management" port="${jboss.management.http.port:9990}" secure-port="${jboss.management.http.port:9993}"/>
    </http-interface>
</management-interfaces>

So that, jboss-cli tool will select the correct protocol in order to initialize it's execution.