Linux add disk space using LVM ec2 AWS instances when launching an image with larger storage

fdisk /dev/xvda
n (create new partition – use defaults)
p (primary partition)
3 (partition number)
w (write partition table to disk)

reboot server

mkfs.ext3 /dev/xvda3 (write file system to partition)
pvcreate /dev/xvda3 (Initialize the partition for use as a physical volume in lvm and add it to VolGroup)
vgextend VolGroup /dev/xvda3
lvextend -L +70G /dev/mapper/VolGroup-lv_root
resize2fs /dev/mapper/VolGroup-lv_root

Alternate way Automated node.js Deployments to Bluemix via Jenkins

After having used the CloudFoundry plugin for Jenkins for a bit we found that the BM API was periodically erroring out and returning such things as:

400 Bad Request
and
Java.net.SocketTimeoutException: Read timed out
and
502 Bad Gateway

So I installed the cf tools on the jenkins server and was many times able to get successful build pushed to bluemix by running an Excecute Shell in the Build Step with the following commands:

cd public;
npm install;
gulp build;
cd ../;
sudo cf api https://api.ng.bluemix.net;
sudo cf login -u [email protected] -o organization.com -s staging -p password;
sudo cf push MyNodeApp-staging;

If you are binding to services, like mongodb, the trick is to have a manifest.yml file in the root of the project (should be in .gitignore if you pushing multiple branches) In this case the manifest looks like:

applications:
- services:
  - mongodb-H
  disk_quota: 1024M
  host: MyNodeApp-staging
  name: MyNodeApp-staging
  path: .
  domain: mybluemix.net
  instances: 1
  memory: 256M
env:
  NODE_ENV: staging

Where the service name to bind to is specified and the environment variable NODE_ENV is passed through for the application to handle environmental case switching.

php55 Warning on startup

PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/5.5/modules/libevent.so’ – /usr/lib64/php/5.5/modules/libevent.so: undefined symbol: php_sockets_le_socket in Unknown on line 0

Warnings like this are typically due to certain module load order dependencies and can be fixed by ensuring the dependant module loads after the required module. In this case I reordered the module loads by renaming /etc/php-5.5.d/libevent.ini to x-libevent.ini (sockets.ini would then load before it.)

Automated node.js Deployments to Bluemix via Jenkins

One of our new clients is a partner of IBM and we are building a node.js-based application for them. Through their IBM partnership they were encouraged to leverage IBM’s Bluemix Cloud Foundry-based PaaS for development and deployment. We have plenty of experience with virtualized environments and are always interested in exploring new technologies and platforms. However, with the client’s best interest and budget taken to heart, we decided to try and use as much of our existing workflow and process as possible rather than spending a ton of hours getting ramped on the bluemix DevOps platform.

So we have our codebase checked into bitbucket just like every other project and we are running jenkins hooks to our CI server to trigger builds on check in. There a wonderful Jenkins plugin for Cloud Foundry code pushes, which works perfectly for pushing to Bluemix – https://wiki.jenkins-ci.org/display/JENKINS/Cloud+Foundry+Plugin

Once the plugin is installed via Jenkins plugin manager, it should appear as a “Post Build Action” in the project configuration screen. (If you install the plugin and that option does not show up, Jenkins is likely pointed at a java version prior to 1.7 – you can confirm this by looking in the jenkins logfile for an error about “Unsupported major.minor version 51.0″)

Once you get the option, you simply add your bluemix credentials via the credential manager, then set up the Cloud Foundry post build action with the correct values for the parameters:

Capture

And the web app gets pushed to bluemix, which takes care of all the vendor dependencies and deploys the app!

Checking out Revision d846a37fc9b234f37713005dc88a7ba78855f3d6 (origin/staging)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d846a37fc9b234f37713005dc88a7ba78855f3d6
 > git rev-list d846a37fc9b234f37713005dc88a7ba78855f3d6 # timeout=10
Cloud Foundry Plugin:
Pushing SDStaging app to https://api.ng.bluemix.net
App already exists, resetting.
App deleted.
Creating new app.
Pushing app bits.
Starting application.
       Installing IBM SDK for Node.js (0.12.2) from cache
       Using default npm version: 2.7.4
-----> Checking and configuring service extensions before installing dependencies
-----> Building dependencies
       Installing node modules
       > (node-gyp rebuild 2> builderror.log) || (exit 0)
       > [email protected] install /tmp/staged/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/node_modules/kerberos
       make: Entering directory `/tmp/staged/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/node_modules/kerberos/build'
         CXX(target) Release/obj.target/kerberos/lib/kerberos.o
         CXX(target) Release/obj.target/kerberos/lib/worker.o
         CC(target) Release/obj.target/kerberos/lib/kerberosgss.o
         CC(target) Release/obj.target/kerberos/lib/base64.o
         CXX(target) Release/obj.target/kerberos/lib/kerberos_context.o
         SOLINK_MODULE(target) Release/obj.target/kerberos.node
         SOLINK_MODULE(target) Release/obj.target/kerberos.node: Finished
         COPY Release/kerberos.node
       make: Leaving directory `/tmp/staged/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/node_modules/kerberos/build'
       > (node-pre-gyp install --fallback-to-build) || (node-gyp rebuild 2> builderror.log) || (exit 0)
       > [email protected] install /tmp/staged/app/node_modules/mongoose/node_modules/bson/node_modules/bson-ext
       make: Entering directory `/tmp/staged/app/node_modules/mongoose/node_modules/bson/node_modules/bson-ext/build'
         CXX(target) Release/obj.target/bson/ext/bson.o
         SOLINK_MODULE(target) Release/obj.target/bson.node
         SOLINK_MODULE(target) Release/obj.target/bson.node: Finished
         COPY Release/bson.node
       make: Leaving directory `/tmp/staged/app/node_modules/mongoose/node_modules/bson/node_modules/bson-ext/build'
       [email protected] node_modules/vhost
       [email protected] node_modules/debug
       └── [email protected]
       [email protected] node_modules/cookie-parser
       └── [email protected]
       ├── [email protected]
       └── [email protected]
       ├── [email protected]
       ├── [email protected]
       └── [email protected]
       [email protected] node_modules/serve-favicon
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       [email protected] node_modules/passport
       [email protected] node_modules/body-parser
       └── [email protected] ([email protected])
       [email protected] node_modules/morgan
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected] ([email protected])
       ├── [email protected] ([email protected])
       ├── [email protected] ([email protected], [email protected])
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       [email protected] node_modules/express
       └── [email protected] ([email protected], [email protected])
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected] ([email protected], [email protected])
       ├── [email protected]
       ├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
       ├── [email protected]
       ├── [email protected] ([email protected])
       [email protected] node_modules/hbs
       ├── [email protected] ([email protected])
       [email protected] node_modules/mongoose
       ├── [email protected] ([email protected], [email protected])
       ├── [email protected]
       ├── [email protected]
       ├── [email protected] ([email protected])
       └── [email protected] ([email protected], [email protected], [email protected])
       ├── [email protected]
       └── [email protected] ([email protected], [email protected])
       ├── [email protected]
       ├── [email protected]
       └── [email protected] ([email protected])
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected] ([email protected])
       ├── [email protected]
       ├── [email protected] ([email protected], [email protected])
-----> Checking startup method
       No Procfile; Adding 'web: npm start' to new Procfile
-----> Checking and configuring service extensions after installing dependencies
-----> Installing App Management
       Creating runtime environment
-----> Finalizing build
-----> Building runtime environment
       Caching results for future builds
       Cleaning previous cache
-----> Build succeeded!
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       ├── [email protected]
       [email protected] /tmp/staged/app
       ├── [email protected]
       └── [email protected]
       ├── [email protected]
       ├── [email protected]
       WARNING: Node version not specified in package.json

DevOps Automation – Ansible+Semaphore is Indispensable!

UPDATE
Semaphore 2.x is in current release and much of the information regarding the install process below is no longer relevant as the package has been significantly refactored. The developers have made it much easier to install: See the install directions here: https://github.com/ansible-semaphore/semaphore/wiki/Installation

I deviated a bit, but got it up and running in less than 5 minutes. Here’s how:

As root:

#cd /usr/bin
#wget https://github.com/ansible-semaphore/semaphore/releases/download/v2.0.2/semaphore_linux_amd64
#mv semaphore_linux_amd64 semaphore
#chmod a+x semaphore
#semaphore -setup

semaphorescreen1
semaphorescreen2
semaphorescreen3

Note that I already had mysqld running on the server.

As Spire Digital delves deeper in DevOps, I have had the opportunity to test drive a number of automation technologies – Puppet and Chef, of course.  But the open-source, python-based ansible project is far and away my preferred technology for automating server configuration and initial web app deployments and configuration.  It even handles could-based instance provisioning!  One of it’s best features is that it needs no client agent running on the target server – everything is done through ssh using key-based authentication.  We run redhat-flavored VMs (amazon-linux-ami) so installing ansible goes like this:

#sudo yum install epel-release
#sudo yum-config-manager --enable epel
#sudo yum install ansible

And then you have all of ansible’s cli tools available to issue ad-hoc commands via ansible or run playbooks using ansible-playbook.  It gets even better with a GUI.  And while I think that the ansible tower GUI is reasonably priced (especially when compared to puppet’s GUI), I found the semaphore project on github to be lean and very useful – it’s not yet well documented, so you do have to deal with some guesswork or hope the developers will answer questions on the github board for the project, but once you get everything configured and understand how it works, it’s great!

The semaphore application is a node.js app that normally runs under vagrant or docker – I skipped the vagrant/docker, and just set it up as a node.js app directly on a fresh dedicated server.  Two gotchas:

1. need to npm install some modules for dependencies:
async body-parser connect-redis express jade logtrail morgan passport.socketio socket.io bcrypt bugsnag cookie-parser express-session less-middleware mongoose passport serve-static validator

2. need to copy lib/credentials.default.json to lib/credentials.json

Then spin it up running node bin/semaphore.js  -  it will bind to port 80 by default, but you can change that in lib/config.js by changing the exports.port variable.  Point a browser at your install and log in!

What does semaphore do?  So basically it runs playbooks for you.  And here’s how:  Put your playbooks in a git repository (we use bitbucket)  There are only 2 top-level navigation items in semaphore:  Playbooks (which point to the git project)  and Identities (which are private/public ssh keypairs)  You enter keys for the git repository as an identity, then you enter the URL to your repo containing your playbooks and associate it with the right identity.

playbook

Semaphore does not use ansible’s default hosts file, so you’ll need to enter hosts and host groups directly into semaphore.

Now you need to enter a Job.  A job is a playbook in your repo – for example if you want to run a playbook from your repository that lives in a subdirectory of the project called “project-deploy” you would enter project-deploy/software-deploy.yaml  as the “Play” (“Name” can be whatever.)

 

jobjobs

When you click on run for that job, semaphore pulls down the repo into a temp folder and runs ansible-playbook on the .yaml file that you specified – you can monitor the output of the job by going to tasks > see outputtasksCapture

*Protip – semaphore will use the same ssh keypair for fetching the git repo as it will for connecting to the remote host, so you need to be sure that the public key for your git account is also in the authorized_keys file on your target host. You may also need to add an ssh config file to /root/.ssh/config that looks like this:

Host *
  StrictHostKeyChecking no
  CheckHostIp no
  PasswordAuthentication no
  PreferredAuthentications publickey

*Protip2 – semaphore downloads git repos into /root/ as configured in vagrant or docker that works fine, but if you’re working on an install outside of a VM, you need to run semaphore as root.

Here’s an example playbook that ensures a server has everything we need to run our app:

---
- hosts: testservers
# set the php version that needs to run on the server here
# and ansible will pull in the relevant vars file for that version
# it does not matter if php is currently on the image as this playbook
# completely uninstalls any version and replaces it with the version
# specified here.
# [email protected] - 05/13/15
vars:
phpversion: 55
vars_files:
- software-vars-php{{ phpversion }}.yaml
gather_facts: true
remote_user: ec2-user
become: yes

tasks:
- name: Install EPEL repo.
yum: name={{ repo_package }} state=latest

- name: enable EPEL.
shell: yum-config-manager --enable {{ repo_name }}

- name: clean up any existing php.
yum: name=php* state=absent

- name: clean up any existing php54.
yum: name=php54* state=absent

- name: clean up any existing php55.
yum: name=php55* state=absent

- name: clean up any existing php56.
yum: name=php56* state=absent

- name: Next - update all core packages
yum: name=* state=latest

- name: Then, install required system packages.
yum: name={{ item }} state=latest
with_items: "{{ system_packages }}"

- name: Now, install required npm packages.
npm: name={{ item }} global=yes
with_items: "{{ npm_packages }}"

- name: Install apache security configuration file.
copy: src=conf/x-secure.conf dest=/etc/httpd/conf.d/x-secure.conf

- name: And, restart httpd services.
service: name=httpd state=restarted

- name: Almost done - now enable httpd service at boot.
shell: chkconfig httpd on

- name: Finally, enable mysqld service at boot.
shell: chkconfig mysqld on

And here is the associated variables file:

---
repo_package: epel-release
repo_name: epel
system_packages:
- git
- httpd24
- httpd24-tools
- httpd24-devel
- mod24_ssl
- php55
- php55-bcmath
- php55-cli
- php55-common
- php55-dba
- php55-devel
- php55-gd
- php55-imap
- php55-intl
- php55-ldap
- php55-mbstring
- php55-mcrypt
- php55-mysqlnd
- php55-odbc
- php55-pdo
- php55-pecl-jsonc
- php55-pecl-jsonc-devel
- php55-pecl-memcached
- php55-pecl-oauth
- php55-pecl-redis
- php55-pecl-ssh2
- php55-pgsql
- php55-pspell
- php55-soap
- php55-tidy
- php55-xml
- php55-xmlrpc
- mysql55-server
- redis
- npm
- nodejs
- mongodb
npm_packages:
- bower
- npmlog
- node-uuid

 

Git yourself some semaphore here: https://github.com/ansible-semaphore/semaphore

SpireMedia Seeks DevOps Engineer (Denver-RiNo)

Based on fit. Consistently ranked as one of Denvers Best Places to Work, SpireMedia (http://www.spiremedia.com) is a Denver-based digital product development firm that transforms business through the use of web and mobile technologies. The firm serves an international clientele from start-ups to Fortune 500 companies and has been in business for more than 16 years. Were hiring for a full-time position in DevOps Engineering.

Job Description:
The DevOps Engineer will primarily be responsible for effective provisioning, installation/configuration, operation, monitoring, maintenance and deployment of infrastructures and applications used to support SpireMedia’s Digital Product Development, Staging, QA, and Client Production environments. This includes streamlining and automating deployments, configuring and optimizing virtual instances, configuration of supporting environments and engineering robust toolsets to support an always available, highly responsive environment. The successful candidate will be a hands-on contributor and team player with a programming background and passion for operations, automation, scalability, security, and quality.

Responsibilities:
• System monitoring, troubleshooting and tuning in Development, Staging and production environments
• Work with internal and external teams to break down complex infrastructure problems and drive resolution
• Develop tools to assist in automation of deployments and monitoring of production environments
• Troubleshoot high transaction web-based operations and environmental issues
• Application of common Linux and Windows system administration tools and tasks
• Apply programming/scripting knowledge in latest technologies contributing to operational efficiencies
• Leverage scripting and programming skills to develop custom tool integrations
• Apply practical experience in deployment and software configuration management methodologies
• Work with Product Owners, Development and QA teams to achieve high quality release management
• Clearly write technical specifications and effectively communicate across teams and roles
• Apply knowledge of best practices and IT operations in an always-up, always-available service
• Other assigned tasks

Qualifications/Requirements:
• Bachelor’s degree in Computer Science/Engineering or equivalent work experience
• 2+ years of technical expertise in Full Stack Programming
• 3+ years of experience in scripting languages, such as Python, Visual Basic, perl, etc.
• 3+ years of experience in systems integration within linux-based environments
• Experience in LAMP technologies
• Experience in Windows Server stack administration
• Experience with automation/configuration management tools such as Puppet/Chef/Jenkins
• Experience with Amazon Web Services included EC2, RDS, S3, CloudFront, IAM, etc.
• Experience with highly responsive / high availability systems
• Experience with continuous integration systems such as Hudson/Jenkins/TeamCity/Bamboo
• Knowledgeable in version control such as CVS and Git
• Knowledgeable in common SCM practices, such as branching and code merges
• Familiarity with MS SQL and MySQL (NoSQL / MongoDB experience is a plus, also)
• Experience with mobile application deployment tools including FlightTest, Google Play, and Apple Store
• Team player with strong analytical, problem-solving, and communication skills

Apply through the Contact Us link on our website at http://www.spiremedia.com or via email [email protected]

Please include resume and salary requirements.

Steps to install apache-solr under tomcat6 on Amazon Ec2 Linux AMI

Ensure Java 1.7 is installed and is default selected JAVA_HOME or install

Coldfusion CFIDE bitcoin mining exploit – URL attack vectors