Build application with bash scripts and schedule the build script

In this article, I will describe my build script, which will build application when there are new commits, it will send notification to telegram with bot.

Goal

• Check new commits from git repository
• Send message to Telegram with bot
• Save build number
• Check the build is running or not
• Check the script is running correctly or not
• Schedule the script

Check new commits from git repository

We use gitlab to manage codes, and enable code review from the gitlab, so there are some merge commits, I need to check whether there are new commits or not.

The scripts to check new commits are like below:

Fetch remote changesgit fetch origin

Get the count of no-merge commitscommits=`git rev-list --no-merges HEAD..origin/develop --count`
if [ $? -ne 0 ]; then { sendFailedMessage "Failed to get commits."; } fi
if [ ! $commits -gt 0 ]; then
 echo "No new commits, skip this build"
 exit
fi

Get new commit messagesIFS=''  message=`git log --no-merges --pretty=format:"- *%s by %an*" HEAD...origin/develop`
if [ $? -ne 0 ]; then { sendFailedMessage "Failed to commit messages."; } fi
echo "The message is $message"

Send message to Telegram with bot

We can create a bot in Telegram, and send message with bot, we can also add the bot to a group, so the bot can send message to a group.

I create a simple service, which can receive the chatId and message, so on the bash script, I can send message to Telegram with curl command, just like below:sendMessage() {
if [ -z "$1" ]; then
   echo "-The message does not specified"
 else
   curl -X POST -G --data-urlencode "chat_id=$CHAT_ID" --data-urlencode "parse_mode=Markdown" --data-urlencode "message=$1" https://notify.y9i.net/bot/telegram
 fi
}

I also add another method to send failed message, so I can add some emoj to the message like below:sendFailedMessage() {
 if [ -z "$1" ]; then
   sendMessage "❌ Build $BUILD_NAME failed"
 else
   sendMessage "❌ $1"
 fi
 cleanPid
 exit 1
}

Save build number

In the bash script, I also save the build number to local disk, so we can save increate the build number every time when the script runs.

Get the script dir

I need to save the build number on the place where the bash script located, the script I used to get the script directory is like below, the code is got from github:# taken from: https://github.com/arne-cl/RSTTool-Linux/blob/a8532cbd3ed72b522c35b51c547f8bd1f04a9206/rsttool.sh
SOURCE="${BASH_SOURCE[0]}"
while [ -L "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
 DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
 SOURCE="$(readlink "$SOURCE")"
 [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done
DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
echo "The DIR is $DIR"

Increase the build number

The following code is I used to save and increate the build number:BUILD_NUMBER_FILE=$DIR/<filename>
if [ -f $BUILD_NUMBER_FILE ]; then
 build_number=$(cat $BUILD_NUMBER_FILE)
else
 build_number=0
fi
build_number=$(($build_number + 1))
echo $build_number > $BUILD_NUMBER_FILE

Check build is running or not

Sometimes the script might be called multiple times, in order make sure that only one job is running, we need to check whether the build is running or not.

Solution: Check with pid

The final solution is the pid solution, I will check the pid file, if the pid file already exists, then get the pid and check whether the process with pid exists or not, if the process exists, then print the message and skip the job.

If the process not exists or pid file does not exists, then save the pid then run the job.

The pid related functions are like below:pid="$$"
pidPath="$DIR/.job.pid"checkPid() {
 if [ -f $pidPath ]; then
   oldPid=$(cat $pidPath)
   processCount=`ps -o pid= -p $oldPid | wc -l`
   if [ $processCount -gt 0 ]; then
     echo "Another job already running with pid $oldPid, skip this job"
     exit
   fi
 fi
}# Save the pid in .pid under local directory
savePid() {
 echo $pid > "$pidPath"
}cleanPid() {
 rm "$pidPath"
}

The ps -p command will output with a header, with parameter -o pid= can omit the header, so we can use wc -l to check the process count.

Before return from the job, we need to clean the pid file, the clean function just delete the pid file.

Solution: Check with process name

This is my first solution, but I think it’s not good enough, so I prefer the pid solution, but I also keep it here.

Check process with the following commands:ps aux | grep $scriptName | grep -v | grep '$$'

use wc -l to counter process count:jobCount=`ps aux | grep $scriptName | grep -v "$$" | grep -v grep | wc -l`
if [ $? -ne 0 ]; then { echo "Failed to get process count."; exit 1; } fi
echo "The job count is $jobCount"
if [ $jobCount -gt 1 ]; then
 echo "There are running job running, skip this job"
 exit 1
fi

The script above use the command substitution, the command substitution will run on a subshell, so even add grep -v "$$", there still has one process.

But with this solution, it might cause a problem when we use symlink with another file name, such as ln -s script.sh test.sh , if I run the test.sh, then check with script name might not correct.

Check the script is running correctly or not

After every command we can check the return value of the command with $? with 0, if the value is 0, that means the command is running without error.

We will terminate the script if there is an error, at the end of the script, we can send a success notification.

Schedule the script

Schedule with crontab

The easiest method is use the crontab, for example, we can run the command every hour, if there are some new commits, then the build will start, otherwise, just skip the job.

We can add a script to run with crontab, and this script will save the build log to a file, the following content is a sample script, for example save the content on file ~/build.sh.#!/bin/bashnohup <path fo the build script> > ~/logs/log_build.log 2>&1 &

The crontab script for every hour is like below:0 * * * * ~/build.sh

Schedule with webhook

We can connect with webhook for some environment, for example on github and gitlab, we can configure the webhook to our server, when get the events, we can trigger the build script.

Full example