Trivial bash Script to restart services in AWS

Too many aws servers? Been there and I hate it. Following is a simple script that I use to restart services running on EC2 instances. I am using AWS Cli to get the ip address based on tag names and then ssh into the box and to the command. Note that the tag names are the same as service names in my case.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

#!/bin/bash   set -e   function usage {   echo >&2 "Usage: $0 [ -e environment -n service_name -a action ]"   exit 1 }   while getopts ":n:e:a:" FLAG; do   case $FLAG in     n) NAME=${OPTARG};;  e) ENV=${OPTARG};;  a) ACTION=${OPTARG};;  [?]) usage;;   esac done   if [[ -z $NAME || -z $ENV || -z $ACTION ]]; then   usage fi   for fid in $(aws ec2 describe-instances --filters "Name=tag:ServerEnv,Values=${ENV}" "Name=tag:SubSystem,Values=${NAME}" --query 'Reservations[*].Instances[*].PrivateIpAddress' --output text) do  ssh -n $fid "sudo ${ACTION} ${NAME}; exit;" done

RabbitMQ Upgrade - Bash Script

An easy to read, trivial bash script for upgrading RabbitMQ service.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

function check_rmq_version {     rmq_version=$(sudo rabbitmqctl status | grep "rabbit," | cut -d',' -f3 | cut -d'}' -f1 | tr -d '"')     echo && echo "    Version = $rmq_version" && echo }   function stop_rmq {     echo "Stopping RabbitMQ..."     sudo service rabbitmq-server stop }   function kill_erlang {     echo "Killing stray RMQ/erlang processes..."     #pids=$(ps -fe | grep erlang | grep rabbitmq | awk '{ print $2 }')     #echo $pids     pgrep -u rabbitmq -x beam | xargs kill -9 || echo && echo "    RabbitMQ already stopped"     echo }   function upgrade_rmq_version {     echo "Changing directory to /tmp..."     cd /tmp     echo       echo "wgetting RabbitMQ .rpm file from official website..."     url="http://www.rabbitmq.com/releases/rabbitmq-server/v3.4.3/rabbitmq-server-3.4.3-1.noarch.rpm"     wget $url     echo       echo "Validating signature..."     url="http://www.rabbitmq.com/rabbitmq-signing-key-public.asc"     sudo rpm --import $url     echo       echo "Upgrading RabbitMQ version..."     file="rabbitmq-server-3.4.3-1.noarch.rpm"     sudo yum install $file     echo }   function start_rmq {     echo "Starting RabbitMQ"     sudo service rabbitmq-server start }   function main {     check_rmq_version   # Checking the current version of RabbitMQ     stop_rmq            # Stopping the rabbitmq-server service     kill_erlang         # Killing erlang to ensure RMQ is stopped     upgrade_rmq_version # Upgrading RabbitMQ     start_rmq           # Starting the rabbitmq-server service     check_rmq_version   # Checking the current version of RabbitMQ }   main

ReIndexing Elasticsearch in Scala

The following scala script reads from one index and writes to another script using Scan and scroll method. The script also takes in a partial function where the values from one index can be manipulated before saving into another index. This script assumes you have a field called "id" and an field called "submitDate" so it can continually perform scan and scroll once the preliminary index copy is done, so keep the index's in sync.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77

import org.json4s._ import org.json4s.JsonDSL._ import spray.client.pipelining._ import spray.http._ import spray.httpx._ import spray.httpx.encoding._ import scala.concurrent._ import scala.concurrent.duration._ import akka.actor._ import scala.collection.mutable.ListBuffer import scala.annotation.tailrec   class ElasticsearchReIndexerActor(esInputHost: String,   esOutputHost: String,    inputIndex: String,   outputIndex: String,   indexType: String,   processData: (JValue) => JValue) extends Actor {     import context.dispatcher       val ouputIndexClient = new ESClient(s"http://$esOutputHost", outputIndex, indexType, context)   val pipeline = addHeader("Accept-Encoding", "gzip") ~> sendReceive ~> decode(Gzip) ~> unmarshal[HttpResponse]   var lastUpdateDateTime: String = "1900-01-01"     def receive = {  case "init" => {    val scanId: String = (Await.result(getScanId(lastUpdateDateTime), 60 seconds) \\ "_scroll_id").extract[String]    self ! scanId  }  case scanId: String => iterateData(scanId)  case ReceiveTimeout => self ! "init"   }     def getScanId(startDate: String): Future[JValue] = {  println("Query data with date gte: " + lastUpdateDateTime)    val esQuery = "{\"query\":{\"bool\":{\"must\":[{\"range\":{\"submitData\":{\"gte\":\"" + lastUpdateDateTime + "\"}}}]}}}"  val esURI = s"http://$esInputHost/$inputIndex/$indexType/_search?search_type=scan&scroll=5m&size=50"  val esResponse: Future[HttpResponse] = pipeline(Post(esURI, esQuery))     esResponse.map(r => { parse(r.entity.asString) })   }     def iterateData(scanId: String) = {  val scrollData = ("scroll_id" -> scanId)  val esURI = Uri(s"http://$esInputHost/_search/scroll?scroll=5m")  val esResponse: HttpResponse = Await.result(pipeline(Post(esURI, scanId)), 60 seconds)  val responseData: JValue = ModelJsonHelper.toJValue(esResponse.entity.asString)  val bulkList = new ListBuffer[JValue]()    val bulkData: ListBuffer[JValue] = (responseData \ "hits" \ "hits" \ "_source") match {    case JNothing | JNull => throw new Exception("Result set is empty")    case JArray(dataList) => {   dataList.foreach { data =>     val id = (data \ "id").extract[String]     val bulkIndexType = ("index" -> (("_index" -> outputIndex) ~    ("_type" -> indexType) ~ ("_id" -> id)))     bulkList += bulkIndexType     bulkList += processData(data)   }   bulkList    }    case x => throw new Exception("UNKNWON TYPE: " + x);  }  val bulkResponse: SearchQueryResponse = Await.result(ouputIndexClient.bulk(bulkList.toList), 60 seconds)     (responseData \\ "_scroll_id") match {    case JNothing | JNull => {   lastUpdateDateTime = DateTime.now.toString   context.setReceiveTimeout(1.minute)   println("Paused at: " + lastUpdateDateTime)    }    case x => self ! x.extract[String]  }   }  }

Notes:

• The ESClient is an extension of on wabisabi Library for elasticsearch
• The Actor initially performs a scan-scroll with submit date gte 1900
• Once the initial scan-scroll is done, it pauses for a minute and performs a scan-scroll again with the submitDate of previous endTime (dateTime.now minus 1 minute)
• This way every minute after the previous run it will continually keep the index in sync
• The partial function "processData" provides a way to manipulate the original data, manipulate it and save it to the new index
• Bulk-indexing is used for saving to the new index, hence a the "id" field is required to determine the "id" of the new document

Usage:

?

1 2 3 4 5 6 7 8 9 10 11 12

val esReindexActor = system.actorOf(Props(new ElasticsearchReIndexerActor(    "localhost:9200",    "localhost:9200",    "inputIndex",    "outputIndex",    "someType",    doNothing)), name = "esReIndexer")   esReindexActor ! "init"  }     def doNothing(data: JValue): JValue = data

Calling SOAP Service in Scala

Scala out of the box has limited capability to call SOAP service neither does libraries such as http-dispatch or spray client. SOAP service is reality is just a xml request/response service and lo and behold, XML is a first class citizen in Scala.

One of the widely used library is ScalaXB which helps in generating case classes given a xsd or wsdl file. Scalaxb is an XML data-binding tool for Scala that supports W3C XML Schema (xsd) and Web Services Description Language (wsdl) as the input file. This is great but it's quite hard to maintain and the code readability goes down the drain as the code is dynamically generated. For example, the following screen shot is what scalaxb generates when either a wsdl or xsd is provided


But what we really need is a trivial way to call a webservice using our existing http clients. Following is one way of doing so.
In this below example, i am calling a service that returns back a list of keywords given a list of keywordid's.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

import spray.client.pipelining._ import akka.actor.{ ActorRefFactory } import spray.http._ import spray.httpx._ import scala.concurrent.Future import scala.xml._   class KeywordService(keywordServiceURL: String, implicit val actorRefFactory: ActorRefFactory) {   import actorRefFactory.dispatcher     def sendAndReceive = sendReceive     def fetchKeywords(keywordIds: List[Int], language: String = "en-us"): Future[Elem] = {  if (keywordIds.isEmpty) {    Future { <xml/> }  } else {    val requestDetail = new GetKeywordDetailsRequest("test", 0, Terms(termIds = keywordIds), DesiredDetails(languageCodes = language))    doRequest(keywordServiceURL, wrap(requestDetail.toXML), Some(keywordLookupMetric))  }   }     private val mapErrors = (response: HttpResponse) => {  response.status.isSuccess match {    case true  => response    case false => throw new Exception(response.entity.asString)  }   }     private def doRequest(uri: String, data: Elem, timer: Option[Timer] = None): Future[Elem] = {  val kwdServiceURI = Uri(uri)  val pipeline = addHeader("SOAPAction", "http://xxx.com/GetKeywordDetails") ~> sendAndReceive ~> mapErrors ~> unmarshal[HttpResponse]  val kwdServiceResponse: Future[HttpResponse] = pipeline(Post(kwdServiceURI, data))  kwdServiceResponse map {    r => XML.loadString(r.entity.asString(spray.http.HttpCharsets.`UTF-8`).replaceAll("[^\\x20-\\x7e]", ""))  } recover {    case any: UnsuccessfulResponseException => throw any;  }   }     def wrap(xml: Elem): Elem = {  val buf = new StringBuilder  buf.append("<s:Envelope xmlns:s=\"http://schemas.xmlsoap.org/soap/envelope/\">")  buf.append("<s:Body xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">")  buf.append(xml.toString.split('\n').map(_.trim.filter(_ >= ' ')).mkString)  buf.append("</s:Body>")  buf.append("</s:Envelope>")  XML.loadString(buf.toString)   } }

In the above script, all that i am doing is constructing the soap request headers manually and performing a POST operation. There are couple of things to be noted:

• "SOAPAction" header is manually added to let the service know which service operation it is intended for
• Setting the charset to UTF-8 and removing unicode characters "[^\\x20-\\x7e]"
• Removing the unicode characters are necessary as scala fails to parse the response. This mostly seems to happens when calling .NET WCF services

GetKeywordDetailsRequest is a class that has the input parameters and has a function that generates the formatted xml for the soap request

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

case class Terms(   termIds: List[Int],   status:  Int       = 0 )   case class DesiredTermDetails(   ancestors:              Boolean = false,   category:               Boolean = false,   children:               Boolean = false,   translations:           Boolean = true,   mappingSynonyms:        Boolean = false,   searchSynonyms:         Boolean = false,   requiredRelationships:  Boolean = false,   suggestedRelationships: Boolean = false,   languageCodes:          String  = "en-us" )   class GetKeywordDetailsRequest(user: String = "test", mode: Int = 0, terms: Terms, desiredTermDetails: DesiredTermDetails) {     def toXML = {  val requestXML = <GetKeywordDetails xmlns="http://xxx.com/">         <GetKeywordDetailsRequest xmlns="http://xxxx.com/zzzz.xsd">        <User>{ user }</User>        <Mode>{ mode }</Mode>        <Terms>          {         for { tID <- terms.termIds } yield <TermID>{ tID }</TermID>          }          <Status>{ terms.status }</Status>        </Terms>        <DesiredTermDetails>          <Ancestors>{ desiredTermDetails.ancestors }</Ancestors>          <Category>{ desiredTermDetails.category }</Category>          <Children>{ desiredTermDetails.children }</Children>          <Translations>{ desiredTermDetails.translations }</Translations>          <MappingSynonyms>{ desiredTermDetails.mappingSynonyms }</MappingSynonyms>          <SearchSynonyms>{ desiredTermDetails.searchSynonyms }</SearchSynonyms>          <RequiredRelationships>{ desiredTermDetails.requiredRelationships }</RequiredRelationships>          <SuggestedRelationships>{ desiredTermDetails.suggestedRelationships }</SuggestedRelationships>          <LanguageCodes>{ desiredTermDetails.languageCodes }</LanguageCodes>        </DesiredTermDetails>         </GetKeywordDetailsRequest>       </GetKeywordDetails>    requestXML   } }

Scala Parser Combinators - SQL Parser Example

Scala Parser Combinators: https://github.com/scala/scala-parser-combinators

Scala Parser Combinators is basically a parsing framework for extracting data when there is a pattern in the given input. This framework provides a more statically typed, functional way of extracting instead of using regex expression which can get hard to read.

In this post, lets build a SQL parser where given a valid sql statement we can identify the "table" name, "column" names and other sql properties. Following are some fundamental functions, operations that Scala combinator provides which would help in parsing:

• " | ": says “succeed if either the left or right operand parse successfully”
• " ~ ": says “succeed if the left operand parses successfully, and then the right parses successfully on the remaining input”
• " ~> ": says “succeed if the left operand parses successfully followed by the right, but do not include the left content in the result”
• " <~ ": is the reverse, “succeed if the left operand is parsed successfully followed by the right, but do not include the right content in the result”
• " ^^ ": says “if the left operand parses successfully, transform the result using the function on the right”
• " ^^^ ": says “if the left operand parses successfully, ignore the result and use the value from the right”
• " rep(fn) ": says "parse the given input using the parser function fn"
• " repsep(ident, char) ": says "parse the given input and split the input using the given 'char'"

Lets start out with a set of SQL statements and it's associated Parser code

select * from users

?

1 2 3 4 5 6 7 8

import scala.util.parsing.combinator._ import scala.util.parsing.combinator.syntactical._   case class Select(val fields: String*) case class From(val table: String)   def selectAll: Parser[Select] = "select" ~ "*" ^^^ (Select("*")) //output: Select("*") def from: Parser[From] = "from" ~> ident ^^ (From(_)) //output: From("users")

select name,age from users

?

1 2 3 4

def select: Parser[Select] = "select" ~ repsep(ident, ",") ^^ {     case "select" ~ f => Select(f: _*)   } //output: Select(List[String]("name", "age"))

select count(name) from users

?

1 2 3 4 5 6

case class Count(val field: String)     def count: Parser[Count] = "select" ~ "count" ~> "(" ~> ident <~ ")" ^^ {      case exp => Count(exp)   } //output: Count("name")

select * from users order by age desc

?

1 2 3 4 5 6 7 8 9 10 11

abstract class Direction   case class Asc(field: String*) extends Direction   case class Desc(field: String*) extends Direction     def order: Parser[Direction] = {     "order" ~> "by" ~> ident ~ ("asc" | "desc") ^^ {       case f ~ "asc" => Asc(f)       case f ~ "desc" => Desc(f)     }   } //output: Desc("age")

select * from users order by name, age desc

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14

abstract class Direction   case class Asc(field: String*) extends Direction   case class Desc(field: String*) extends Direction     def order: Parser[Direction] = {     ("order" ~> "by" ~> ident ~ ("asc" | "desc") ^^ {       case f ~ "asc" => Asc(f)       case f ~ "desc" => Desc(f)     }) | ("order" ~> "by" ~> repsep(ident, ",") ~ ("asc" | "desc") ^^ {       case f ~ "asc" => Asc(f: _*)       case f ~ "desc" => Desc(f: _*)     })   } //output: Desc("name", "age")

select age from users where age>30

?

1 2 3 4 5 6 7 8

def where: Parser[Where] = "where" ~> rep(predicate) ^^ (Where(_: _*))       def predicate = (      ident ~ "=" ~ wholeNumber ^^ { case f ~ "=" ~ i => NumberEquals(f, i.toInt) }      | ident ~ "<" ~ wholeNumber ^^ { case f ~ "<" ~ i => LessThan(f, i.toInt) }      | ident ~ ">" ~ wholeNumber ^^ { case f ~ ">" ~ i => GreaterThan(f, i.toInt) })   //output: GreaterThan("age", 30)