Tuesday, June 5, 2018

Bash - get flags and arbitrary strings in no particular order


#!/bin/bash

# Get arguments and separate them to flags and arbitrary strings
# Execution example: ./get_opts.sh Something -w 10 somethingElse -c20 whatever

# For debug purpose, show everything we received:
# echo $@

# Save original arguments for later use / debug purpose
original_arguments="$@"

# Initialize variable
strings=""

# While there are arguments, keep looping
while (( "$#" )); do
case "$1" in
# if the next argument begins with '-w'
-w*)
# Put all the digits in this or the next argument in  the variable 'warn' and shift the argument array
warn=$(echo $1|tr -cd  '[[:digit:]]')
if [ "$warn" ]; then shift 1;
else warn=$(echo $2|tr -cd  '[[:digit:]]'); shift 2;
fi
;;
# if the next argument beings with '-c'
-c*)
# put all the digits in this or the next argument in the variable 'crit' and shift the argument array
crit=$(echo $1|tr -cd  '[[:digit:]]')
if [ "$crit" ]; then shift 1;
else crit=$(echo $2|tr -cd  '[[:digit:]]'); shift 2;
fi
;;
*)
# Arbitrary strings go here
strings="${strings} $1"
shift
;;
esac
done
echo Warn=${warn} Crit=${crit} Arbitary strings=${strings}
exit

Sunday, November 15, 2015

Best inventory command like tool for linux!

I've been looking around for a nice tool to give me information about my machine. Yes, I know there's lspci etc etc etc, but these are pretty spamy, ugly and take a while to decipher. To the rescue comes inxi. To me it's exactly what I've been looking for. I run it like this:
# inxi -Fi
System:    Host: XXXXXXXXX Kernel: 3.16.0-30-generic x86_64 (64 bit) Console: tty 0 Distro: Ubuntu 14.04 trusty
Machine:   System: Xen product: HVM domU version: 4.1.5 serial: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
           Mobo: N/A model: N/A Bios: Xen version: 4.1.5 date: 06/14/2013
CPU:       Quad core Intel Xeon CPU E5645 (-HT-MCP-) cache: 12288 KB flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3)
           Clock Speeds: 1: 2400.146 MHz 2: 2400.146 MHz 3: 2400.146 MHz 4: 2400.146 MHz
Graphics:  Card: Cirrus Logic GD 5446 X-Vendor: N/A driver: N/A tty size: 164x47 Advanced Data: N/A for root out of X
Network:   Card: Failed to Detect Network Card!
           WAN IP: 62.0.100.191 IF: eth0 ip: 192.168.3.202
Drives:    HDD Total Size: 64.4GB (2.5% used) 1: id: /dev/xvda model: N/A size: 64.4GB
Partition: ID: / size: 51G used: 1.5G (4%) fs: ext4 ID: swap-1 size: 8.85GB used: 0.00GB (0%) fs: swap
RAID:      No RAID devices detected - /proc/mdstat and md_mod kernel raid module present
Sensors:   None detected - is lm-sensors installed and configured?
Info:      Processes: 128 Uptime: 8 min Memory: 172.1/7979.7MB Runlevel: 2 Client: Shell (bash) inxi: 1.9.17

Thursday, October 29, 2015

Setting up multiple hard drives at once on Ubuntu

Sometimes you need to set up multiple hard drives sequentially, say for example for a Hadoop data node.
Here's how you do it quick and easy:
# for hdd in /dev/sd{c..h};do echo -e "n\n\n\n\n\nw\n\n"|fdisk $hdd;mkfs.ext4 ${hdd}1;done

In order to automount, use:
# blkid

Add/replace the UUID of the new HDD in /etc/fstab like the following examples:
UUID=b8923549-5794-4cad-89b3-0813fafb9566 /            ext4    errors=remount-ro 0       1
UUID=f0fc9526-0579-4116-994b-c79878f7f393 /hdfs/10    ext4    noatime 0    1

Changing UUID of an existing partition:
# tune2fs /dev/sdl1  -U d81ce3c1-27fa-4a37-8d61-3bebc7ac4e92

Extra options:
  • Enable writeback mode. This mode will typically provide the best ext4 performance.
# tune2fs -o journal_data_writeback /dev/sda10
  • Delete has_journal option
# tune2fs -O ^has_journal /dev/sda10

Sunday, April 13, 2014

Hadoop zookeeper connections CDH4

I'm currently running two hadoop clusters  (Cloudera CDH4) and both suffer from the same problem, stale zookeeper connections that choke it to death.
After digging around, I've located two culprits. One is the thrift hive server we run manually for Tableau connection and the other is Hue.
To make a long story short, here's my solution:

#!/bin/bash
# Function check thrift process id
check_pid() {
        pid=`cat ~/thrift.pid`
    #echo $pid
    ps -A|grep $pid|grep java > /dev/null 2>&1
        result=$?
}


# Function restart Hue service
restart_hue() {
    curl -X POST -u 'admin:admin' http://localhost:7180/api/v2/clusters/TEST%20-%20CDH4/services/hue1/commands/restart  >/dev/null 2>&1
}


# Function kill thrift server
kill_thrift() {
        kill $pid
        sleep 2
        check_pid
        if [ "$result" == 0 ]; then {
                kill -9 $pid
                sleep 2
        }
        fi
    check_pid
    if [ "$result" == 0 ]; then {
        echo "Error: Failed to kill server"
        exit 1
        }
    fi
}


# Sanity
if [ ! -f ~/thrift.pid ]; then {
    echo "Error: thrift process id file ~/thrift.pid not found."
    exit 1
}
fi


if [ `whoami` != "admin" ]; then {
    echo "Error: you are not logged in or executing as user admin, please do so."
    exit 1
}
fi


# Main
restart_hue
result=$?
if [ "$result" -gt 0 ]; then {
        echo Error restarting Hue service, please check cluster integrity.
        exit 1
}
fi
zookeeper_connections=`echo srvr|nc localhost 2181|grep -i connections`
result=$?
if [ "$result" -gt 0 ]; then {
    echo Error: no reply from Zookeeper.
    exit 1
}
fi
zookeeper_connections=${zookeeper_connections##*: }
#echo $zookeeper_connections
if [ $zookeeper_connections -gt 100 ]; then {
    echo "Zookeeper has $zookeeper_connections open conenctions."
    check_pid
    if [ "$result" -gt 0 ]; then {
        echo "Error: can not find thrift server PID"
        exit 1
    }
    fi   
    echo "Found Thrift server, restarting"
    kill_thrift
    export HIVE_PORT=10001
    hive --service hiveserver &
    pid=$!
    echo $pid> ~/thrift.pid
    return=$?
}
else {
    echo "Zookeeper has $zookeeper_connections open conenctions."
}
fi
exit $return
This script is executed every morning via crontab to keep zookeeper from choking. It's been running happily for a few months now :)

Emailing from Bash CLI or script on ubuntu

Basically what you might be looking for is a command line or shell script email sender that knows how to work with gmail. If you are, search no more!

$ sudo apt-get install msmtp openssl ca-certificates
$ vim ~/.msmtprc
account default
host smtp.gmail.com
port 587
from GMAIL_ACCOUNT
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on
user GMAIL_ACCOUNT
password GMAIL_PASSWORD
logfile ~/.msmtp
$ chmod 600  ~/.msmtp

Prepare a file starting wit:
to: DESTINATION
from: SENDER
subject: SUBJECT

And finally:
msmtp -t < YOUR_FILE

Tuesday, April 8, 2014

Forcing Wacom Intous to work with a single monitor on dual monitor setup

So your Wacom Intous refuses to stay locked with a single monitor in your dual + monitor setup no matter what you tell it? There's a quick fix, in the Wacom driver properties, use manual resolution!
For example, to lock it to your left most screen on a FullHD setup put: 0x0 - 1919x1079 and voilĂ !

How to determine mysql table storage engine

So you're trying to find out which storage engine is being used by tables in your mysql database? Thank to Ronald Bradford's blog entry, the best answer I found is this:
mysql> use database_name;
mysql> SELECT table_name,engine FROM INFORMATION_SCHEMA.TABLES WHERE table_schema=DATABASE();