Converting selinium code to headless tests with PhantomJS

The word test here is used ambiguously as I use selenium for more than just UI tests, it makes a great tool for browser automation which I use it for this purpose a lot. While writing your browser automation most of the time it’s easier to do it in browser mode by using the firefox or chrome driver so that you can visually inspect the HTML. Once you are done writing the code and finished testing sometimes you would prefer to switch it to headless mode so that it can be run without having a UI, at which point you are likely to tryout PhantomJS driver and your fully tested code starts throwing all sorts of errors like “Element not found” or “Stale Element Exception”. If all this errors go away if you switch back to chrome or firefox driver then the likeliest cause of your troubles is you might need to add delays in most places where you have the browser loading new data compared to the other browser drivers.
To me this seemed a bit counter intuitive at first as I though headless mode should run faster therefore requiring even less time to load UI changes, but I guess it might actually take a little longer since all the browser rendering is being done is software only. Just thought to put this out there as I have run into the issue a few times.

Automate letsencrypt certificate renewal

I recently switched from self signed certs to free SSL certs from letsencrypt and for the first time I could load my https url without getting the annoying prompt from chrome due to self signed certificates. The only problem is the certs expire pretty fast in about 90 days as of this writing, while this is nothing to complain about since the certs are free handling the renewal each time manually would be a pain and also leave me in a bind in case I forgot to do it.
I decided to automate the renewal process to save myself the hassle of having to do it manually and found two resources here and here on how to do it, I went with a combination of the two methods as my requirements were different.
I wanted the renewal to be run from a script to support email notification on success or failures which is similar to the first source and use the webroot plugin to perform renewal as it has lesser steps to perform renewal reducing any failure points during the process like the second source. The script needed to be able to run everyday and check cert expiration I didn’t want to hard code the cron job to run based on how long the certs are valid that way if letsencrypt changes the life of the certs no change is required on my side.
Let’s get started I won’t cover the install as that’s covered by letsencrypt site, I would advise you to read the different install methods and choose the one that best fits your needs.
After performing the install
Create your config file which will contain the arguments submitted to letsencrypt api I named mine “muthii.com.ini”

rsa-key-size = 4096
server = https://acme-v01.api.letsencrypt.org/directory
text = True
authenticator = webroot
agree-tos = True
renew-by-default = True
email = root@domain.com
webroot-path = /your/webserver/path

Run the command used to create/renew your certs, which creates the certs for you and shows you the path to find them.
[cc lang=”bash”]/root/.local/share/letsencrypt/bin/letsencrypt -c /path/muthii.com.ini -d muthii.com -d www.muthii.com auth[/cc]

Only run the above command if you haven’t created your certs or are ready to renew your current certs, otherwise just grab the script file and add it to your cron. Make sure to change the emails and file paths based on your setup. I have commented out the echo statements and only enable then for testing

For someone doing this for the first time locate your ssl.conf file used by your server and set the paths to the new certs

SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/fullchain.pem

Once you are done setting up head over to SSLLabs and test your certificate is recognized as expected, then setup a cron job to run the script daily .

0 2 * * * sh /path/SSLRenew.sh

CIFS VFS: cifs_mount failed w/return code = -2

I hit this error while adding a samba mount to my fstab, but mounting the same end point would work when executed from command line. For my scenario it turns out that it might be an issue with cifs-utils or kernel if your mount point is under more than one sub-directory. My solution was to go with option 3 and have my target as a share


"//host.IPAddress/share/subdir/subdir/target" - This failed with error "CIFS VFS: cifs_mount failed w/return code = -2"
"//host.IPAddress/share/target" - This worked
"//host.IPAddress/target" - This worked

Owncloud Error: Console has to be executed with the same user as the web server is operated

After a recent Owncloud 8.o.x update I started getting this error being logged whenever the owncloud cron job run. To resolve the issue I had to change the cron job to be run as the user apache.

su -s /bin/sh apache -c "php -f /path/to/owncloud/cron.php"

The webserver on CentOS is run under the user apache, on other linux flavous it’s www-data to find out what it is on your system just check the error being logged it will log the user running the webserver.


Console has to be executed with the same user as the web server is operated
Current user: someuser
Web server user: apache <- This is the user you want. Unexpected error!

Getting java JDK as a zip instead of exe

Sometimes you may need to have the java JDK available on your system without having to run the installer exe/msi for different reasons. This solution worked for me with JDK8u45, I haven’t tried it with other JDK versions so your mileage might vary.
Thanks to @Marc T for this solution.


create destination folder (c:\jdk8)
download JDK exe from Oracle
7zip -> unzip into destination folder
Open command prompt and enter cmd [cd c:\jdk8]
Enter this command to unpack the contents of the folder [for /r %x in (*.pack) do .\bin\unpack200 -r "%x" "%~dx%~px%~nx.jar"]

FreeNX – 1004 Error: NX Agent exited with exit status 1

This error is very generic and while googling I found different issues can cause it. I was able to resolve it this particular instance by creating the folders


/tmp/.X11-unix - as root
/tmp/.ICE-unix - as user logging in

and file
/tmp/.X0-lock - as root

Which had been deleted while manually cleaning up a previous session. The statement below was also logged when this error occured but it to appears to be a generic error logged for different cases whenever a session fails.

596 Session startup failed

VP8/9 vs h264/5 for personal videos

Recently I needed to encode some clips taken with my phone to a web optimized format since the videos were already in mp4 my initial thought was to keep them in mp4 format h264 video/AAC Audio.
First things first to get the videos to a web optimized format a re-encode would be needed so I had to accept that some drop in quality was going to happen. That was the easy part. A little googling and I found that libfdk_aac is the best AAC codec currently to use with ffmpeg, so I uninstalled my RPM installed ffmpeg and tried to compile my own. Getting sources for all the codecs I needed to have in my ffmpeg was pretty straight forward, but the build failed with some make error gcc was having issues building for the platform. This left me with 2 options wipe the OS fedora 20 and upgrade to the latest 21 or spin up a VM and use that to do the compile. The second option seemed like the way to go but I still wasn’t sold on the idea of having to do all this extra effort just to encode some phone clips.

During this thinking period when I was trying to get some time to go with option two I came across an article about vp9 and it hit me I was willing to jump through all this hoops to use a proprietary codec while there was a comparable opensource codec I could use with ffmpeg out of the box. So I re-installed ffmpeg from RPM spent about an hour tweaking some scripts to do the conversion and rotation of the clips.
Played back the newly encoded webm files and I could barely notice a difference with the original files, they were also 50% – 70% the size of the original files. Getting my video hosting application to work with webm files took about another hour as it was not on the list of supported extensions this was uprising since it’s an open source application.

Right now am feeling pretty happy with my choice as expected safari and IE don’t support webm so for those needing to watch the clips am just telling them to use Chrome or firefox. While the video hosting application allows me to share the clips with outsiders for my internal consumption of the clips I use Plex Media Server once they add support for webm to their native clients I will be converting the whole collection of clips from mp4 to webm.

Update: I received some feedback from family members and they are able to view the videos without a problem, so looks like I will be sticking with VP8/9.

Useful Git commands

Sometimes when using apps checked out from Git an update might break something so it’s good to know a few git commands. Move to root of the app folder you had checked out to perform git commands.

git log
This command will list all the commits so you can see at which commit you are and pick the one before yours and copy the commit hashcode it will be a long string.

git checkout hashcode
With ‘hashcode’ being the commit hashcode copied from the log. This command will move/revert your app/code to the commit you specified. Just restart your app after that to run it with the new code.

git checkout master
This command will move your code to the master branch from development. Do this if you want your app to be more stable Alpha/Developement branches break now and then as people checkin new code. They have the benefit of getting new features before the stable branch if you don’t mind having something break now and then which is ok with me 🙂

Python IMAP search and SMTP with Gmail

Recently I had a need to search through my gmail and fwd some emails on a daily basis. I checked out Gmails rest interface but it was a bit of an overkill for what I was doing. So I settled on just plain IMAP and SMTP. The main idea was search through my inbox using several search parameters, get all the emails that match the pattern and fwd them to a list of emails. So I wrote a script to do this and had cron call the script at the end of the day. I put up the source in github for anyone interested.

KVM usage tips

I have been playing around with KVM lately, just to see how it stacks up against VirtualBox and am loving it. I have found the following commands interesting while working with VMs.

List all VMs
virsh list --all

Start stop a VM
virsh start/stop vmName

Clone VM
virt-clone --original vmName --name newVmName --file /some/path/newVmName.qcow2

Clone VM and assign Random Mac address to new image
virt-clone --original vmName --name newVmName --file /some/path/newVmName.qcow2 --mac=RANDOM
Copy over the xml for the new clone which you can use to define your clone if moving the image to another host
cp /etc/libvirt/qemu/newVmName.xml /some/path/newVmName.xml

Shrink/Sparcify VM image
virt-sparsify --format qcow2 --compress vmName.bkp.qcow2 vmName.qcow2 --tmp /some/path
This command assumes you renamed the original VM image file to “vmName.bkp.qcow2” incase the new image has problems you can revert to original image. You can skip the –tmp argument if you have more than double the size of the VM image available in your /tmp directory. You will need to make sure you have package “libguestfs-tools” installed which supplies the sparsify command.

Delete VM
virsh undefine vmName
This removes the XMl configuration for qemu you might have to delete the VM image yourself.

List storage pools
virsh pool-list

Refresh pool data
virsh pool-refresh poolName
You will need to run this command if you delete a VM and you need to reuse the VM name you had used, so make sure you refresh the pool where your deleted image was located.

Disable autostarting of a VM

virsh autostart vmName --disable

Move VM to another system Offline
Step 1 from original host shutdown your VM, copy xml and image to backup location.

virsh dumpxml vmName > /bkpLocation/vmName.xml
cp /locationofVMs/vmname.qcow2 /bkpLocation/vmName.qcow2

Bye default if you didn’t change image storage location it will be “/var/lib/libvirt/images”. I prefer to store them in a separate partition from the OS.

Step 2 on new host get VM xml and image and enjoy.

virsh define /bkpLocation/vmName.xml
Domain vmName defined from /bkpLocation/vmName.xml
virsh start vmName

This all works if you are moving from host1 and host2 with similar cpus, for my purpose host2 had a different cpu type so I had to first create a new vm in host2 and compare the xml with my backed up copy then changed a few lines specific to CPU on host2.