Xylil

Script for generating STIR/SHAKEN certificates (including self-signed)

2024-03-25T00:00:00-05:00

While working on my company’s implemenation of STIR/SHAKEN, I ran into a very basic issue – how do I easily create test self-signed SHAKEN certificates?

While I did find a couple of posts on how to do this, I wanted a nice script that would accomplish the following:

Generate either a self-signed certificate or a key/CSR pair for use with an official certificate authority
Allow easy passing of the certificate subject information (Country, State, Locality, etc.)
Create certificate filenames with a unique identifier, to prevent name collisions (by default, the identifier is a UNIX timestamp)

Below is the script I came up with:

true

Credit to the original source posts, here and here.

FreeSWITCH Dbh 'fetch' convenience functions for Lua

2020-07-01T00:00:00-05:00

freeswitch.Dbh is the method I’ve chosen for connecting to a database via a Lua script in FreeSWITCH.

The interface is simple to use, and provides all the basics needed to communicate with a database via a configured ODBC driver.

One thing I found lacking, however, were ‘fetch’ type helper functions, that:

Returned a multi-row result in a table of tables
Returned a single-row result in a table
Returned a single-row, single-column result directly.

Below are the helper functions I came up with. You might also be able to extend the dbh object directly with methods – I just chose to pass the dbh object into a standalone function call.

true

Pacemaker NFSv4 resource agent for Debian

2020-06-05T00:00:00-05:00

If you’re interested in running an NFSv4-only server in a Pacemaker cluster on Debian, you’ll probably find the default nfsserver RA in the Debian-packaged resource agents to be lacking.

It suffers from serveral issues:

The default behavior is to use the NFS init script to handle starting/stopping/monitoring the NFS server. This is broken in the monitor case as the service is reported as active even if it’s been killed.
Starting/stopping the service is dog slow. I’d venture to say at least half of the total cluster failover time in my setup with the default RA was consumed by starting/stopping the NFS daemon and all the related v2/v3 services.

So if you’re willing to go with NFSv4 only, here’s a much better approach:

Follow the Debian NFS server setup, including the adjustments under NFSv4 only.
Drop the below RA in a valid path for Pacemaker to find it, and make it executable.
Configure your resource(s).

CAVEATS:

Does not work for v2/v3, and will not manage any of the services that are v2/v3 only.
Only tested on Debian Buster, YMMV.

true

Fencing/STONITH IAM configuration on Google Cloud Platform (GCP)

2020-05-24T00:00:00-05:00

UPDATE 2024-01-28: The fence_gce agent now additionally requires the compute.zoneOperations.list permission in the list.instances IAM role. I have updated the code to reflect this change.

Having spent the better part of a day figuring this out, I thought it might be nice to save others some time.

My specific need was to configure STONITH for a Pacemaker cluster deployed on Google Cloud Platform, using fence_gce.

By far the trickiest part was figuring out Google’s extremely robust permissions system to meet this case in the most secure manner. The goal: to allow each node in a two-node cluster to fence the other node (by reset or poweroff), while restricting each node to only be able to fence its peer, and nothing else.

Below I’ll post some psuedo-code that should give you an idea about how to execute this for your particular needs. I deployed this solution using Terraform, but the concepts should be easily adaptable to other use cases.

In a nutshell:

Create two custom IAM Roles (I called them fencing and list.instances):
- fencing: permissions to reset or stop a server and get access to its state
- list.instances: Permissions to list the available instances in a project (this was a small security sacrific, I’d prefer that a node can’t see other nodes this way, but the fencing agent seems to require it, and as of this writing, there’s no clean way on GCP to restrict the nodes listed via the API)
For each instance in a cluster, create a Service Account, and configure the instance run as that service account, granting all the default scopes and the ‘compute-rw’ scope’
At the project level, add an IAM policy binding that binds all the instance service accounts to the list.instances role.
At the instance level, add an IAM policy binding that binds the fencing role to the instance service account for the instance that will fence the instance in question. For example, if you have data1 and data2 instances Then for the data1 instance you’ll bind data2’s service account to the fencing role, and vice-versa. This allows each node to fence its peer, but to have no other permissions for fencing, not even for itself.

Psuedo-code:

true

Bash script to play a sound file every X seconds

2019-02-02T00:00:00-06:00

As part of my continued efforts to improve awareness of my physical body while I’m hacking away at a keyboard, I got the idea that it would be nice if a pleasant sound played every X seconds to remind me stop slouching, relax, etc.

I wanted it to be simple, easy to start and stop, and easy to configure both the time interval between plays and the file being played.

Finding nothing of that ilk in some quick Google searches, I present the below for your usage and hopefully increased awareness :)

Run without arguments for the help.

true

Debian start a ramdisk on boot

2017-03-13T00:00:00-05:00

I run some servers that use a ramdisk to increase performance for recording audio files. Since ramdisks evaporate on a reboot, and I have services that depend on the ramdisk to operate properly, I needed a way to:

Automatically create the ramdisk on boot
Ensure it’s set up prior to the services that depend on it
Perform cleanup on the ramdisk when the server is shut down

The set of scripts below accomplish this nicely by leveraging systemd. Here’s a quick rundown of what each does:

ramdisk: The script is written as a System V init script, and would be placed at /etc/init/ramdisk. Does the work of:
- Creating the ramdisk
- Calling an optionally configured script to perform cleanup prior to tearing down the ramdisk
customize: This is an optional file to override the default settings of the first script. Place it at /etc/default/ramdisk with any customizations you like.
ramdisk.service: systemd service file that calls the start/stop functions of ramdisk. Place this anywhere systemd recognizes service scripts, and run systemctl daemon-reload.

The nice thing about this setup through systemd is that any other systemd services that depend on the ramdisk being set up can simply include this in their service description:

Requires=ramdisk.service
After=ramdisk.service

true

ICE/CBP regulations summary for domestic travelers

2017-02-28T00:00:00-06:00

The election of Donald Trump has roused me from my political slumber, to be sure…

I’m trying to stay informed, and also signed up for the excellent Daily Action alerts, which keeps me plugged in to things I can do every day to make my voice heard.

Today’s alert was in reference to ICE/CBP’s request to see the ID of all passengers on a US domestic flight.

According to the expert cited in that article, it’s highly unlikely that these authorities have a legal right to demand you produce ID under these circumstances.

So today my little action of resistance was to print up a convenient half-page summary of the regulations the expert referred to in that article. Two copies per page, should be small enough to fold up and fit in a wallet, purse, or back pocket.

I’ll be carrying mine around, at least for awhile, to remind myself that I don’t want to live in a police state.

HUGE DISCLAIMER:

What I’m sharing is for informational purposes only. I’m not an attorney, and did not deeply vet this information, but simply looked up the US codes referenced in the article and made them a bit easier to carry around.

FreeSWITCH Kickstart, an automated server configuration tool

2016-04-06T00:00:00-05:00

Project background

Several years ago I began the journey to better systematize my server management workflow. At the time I was managing over 20 server instances, almost entirely ‘by hand’, meaning I did things like keeping checklists for the steps to rebuild critical servers, running software updates on each individually, etc. Then, I finally took the time to research server configuration management software, selected Salt as my tool, and completely re-oriented my relationship to dealing with software updates, new server builds, etc.

Managing a server this way means that you don’t change anything about the server directly, but instead write code that makes the changes for you. While this does involve a learning curve (it’s almost always faster to just run a command straight from the command line than write code to run the command for you), the benefits of this approach far outweigh the costs in the long run:

Automatically rebuild any server from scratch: As long as you have data backups, it’s no longer a big deal if your server tanks. Rebuilding becomes a glorified version of rebooting your computer when it’s acting weird
Consistency in dev -> staging -> production environments: You can be much more sure that the server you’re doing development on is consistent with the server that will run in production. With tools like Vagrant, combined with server configuration management, you can run virtually the identical server on your laptop that you do for your customers.
Clear documentation: This cannot be underestimated. The exact steps to re-create your mission critical infrastructure is written in code, which means both certainty about how it works, and ease of transfer of this knowledge to other maintainers.

Even though this approach is well known by hard core sysadmins, I suspect many people who could profit from it still have not taken the leap, either through ignorance or the barrier of the learning curve.

Fast forward to last year, and I’m working on CircleAnywhere, an online group meditation platform, using FreeSWITCH for some of the video conferencing portions. Naturally the server, being mission-critical to the project, was built using a Salt configuration for all the reasons mentioned above. It was a fair amount of work to hammer out the deployment, and it occurred to me that a lot of the heavy lifting I had done was the same heavy lifting that anyone rolling out a new FreeSWITCH server (whether for local development or in production) had to do. So I decided to do just a little bit more work, and repackage my efforts for the good of others. :)

FreeSWITCH Kickstart

Thus was born FreeSWITCH Kickstart, a Vagrant/Salt configuration for automatically deploying a FreeSWITCH server for either development or production. Installation locally via Vagrant or remotely involves only a few simple steps (the most difficult of which is acquiring valid SSL certificates for production builds).

The project strives to provide these three things:

An easy way to get up and running with a local FreeSWITCH development environment
A sane/convenient starting point for rolling a production server.
A nice starter template for admins who want their FreeSWITCH server under Salt server configuration management.

A ton of stuff is done automagically for you – even if you don’t like/need it all, it’s still an awesome way to get going!

I’ve written up a fairly comprehensive features list, so you know what you’re getting.

I hope the community will benefit from this work, and doubly hope that others will offer refinements on this first effort. My intention is to maintain this project for the FreeSWITCH community, keeping it up to date with the installation procedure for the latest code and recommended operating system. If anyone would like to join me in maintainership of the project, I would certainly welcome it. :)

Jester, FreeSWITCH, Lua, and my quest for an awesome scripting toolkit

2016-01-20T00:00:00-06:00

Back in 2010, in collaboration with Star2Star Communications, I wrote a Lua scripting library for FreeSWITCH called Jester. Its most marketable highlight was a profile that offered a complete drop-in replacement for Asterisk’s Comedian Mail.

My deeper motivation for writing it was to try and bring some sanity and consistency to the process of implementing more advanced voice workflows. This motiviation arose from my own experience of the mishmash of dialplan XML and one-off Lua scripts that still to this day forms the foundation of my first major VoIP system. I began to believe that there must be a better way – if the goal was to do any number of fairly common things, like grab data from a database, send an email, talk to a webservice – can’t we basically standardize those into reusable units? So, Jester…

Fast forward to 2016, and my toolkit, through almost complete lack of maintenance, had become legacy code. In addition, the years had revealed both the strengths and weaknesses of my original efforts, and I wanted to do something about it.

In the last few weeks, I took the first big steps to revive and reshape Jester for a new and improved release. A lot of the hard work is now done:

Updated the entire codebase to be compatible with all 5.x versions of Lua, which means it will now run on anything from the terrifically old FreeSWITCH 1.0.7 to the very latest code.
Complete refactoring/update of the user documentation, now available online and nicely formatted.
Preserved the original Asterisk Comedian Mail replica, so it can still be used as a drop in replacement for those moving from Asterisk to FreeSWITCH
Good progress on handling some of the original architectural issues

Going forward, I’d love to get some other people on board with the project, particularly those interested in helping to turn Jester into a solid, well-used library for the land beyond the dialplan. I think the biggest reason for the initial flaws was that I didn’t have any other eyes on my designs.

Please do contact me if you’re interested in teaming up.

New release of Luchia, Lua API for CouchDB

2015-12-31T00:00:00-06:00

Many years ago, I wrote Luchia, a Lua API for Apache CouchDB.

The library worked great, and had fair unit/integration test coverage, but after many years it was no longer compatible with the newer versions of Lua.

So, I took a few days recently to revive it and roll an updated release. It’s now compatible with Lua versions 5.1, 5.2, and 5.3, and also sports 100% unit test coverage.

Installation via LuaRocks, Lua’s package manager, is a breeze, just do:

luarocks install luchia

You can check out a summary and API usage in the online documentation.

Happy Luchia’ing. :)

Using Salt with services managed by Monit

2015-09-29T00:00:00-05:00

I use Salt for configuration management on all my newer servers. I’m particularly fond of its watch requisite, and use it thoroughly to handle restarting all my Salt managed services if need be.

This generally works without much fuss. However, when a service that’s managed by Salt is also being managed by Monit, there’s a chicken and egg problem: when Salt automatically restarts a service using its standard method, Monit thinks the service has died, and problems ensue.

I really wanted my Salt service management goodness and Monit, so after a bit of thinking I came up the the idea of creating a psuedo-service. Salt manages the psuedo-service instead of the regular service, and the psuedo-service simply wraps the Monit commands for starting, stopping, and restarting the service. This allows Salt to trigger service actions via Monit.

This approach solved the issue beautifully without much code or effort. Below I’m including the RHEL/CentOS version of the psuedo-service – other platforms would require some minor adjustments, but the idea is the same.

As seen from the line defining the service name, I manage this script in Salt as a template, so that I can install it for multiple services.

Hope this helps anybody who gets stuck on this same issue!

true

Vagrant crash course for the busy developer

2015-05-12T00:00:00-05:00

Today a friend remarked on his interest to learn Vagrant, and lack of time to do so. It occurred to me that the script I’ve been building to quickly knock up and down Vagrant VMs could be a handy aid along the shortest path for developers to begin their journey into this amazing toolset.

So without further ado, the shortest path I know of for a busy dev to start playing with Vagrant:

Install VirtualBox and Vagrant

…or for Homebrew Cask users, the even easier brew cask install virtualbox vagrant…
Run the following from CLI:

vagrant box add bento/centos-7.1 --provider virtualbox vagrant box add bento/debian-8.2 --provider virtualbox

…or pick some other box you like here.
Download the script below, stick it in your PATH, make it executable
Execute without args for the well-written help

…or run quick-vagrant.sh -c to spin up your first box…
Enjoy :)

I also made the script a bit like a cheat sheet for Vagrant commands – if you search it for ‘vagrant’, you’ll see the first time any command is used, there’s a comment describing what it does. Or, here’s a more neatly formatted cheat sheet.

If you find this helpful getting up to speed, please pass it along to your busy developer friends! If you’d like to suggest improvements to the script, feel free to contact me.

true

General node init script for RHEL/CentOS 5.x/6.x

2015-03-16T00:00:00-05:00

I wanted to have a general RHEL/CentOS init script that I could drop onto servers to start node processes. The features I wanted were:

Ability to reuse it for multiple node processes on the same server.
Ability to run as non-root user.
The classic service start/stop/restart/status functionality.

I had previously written a node init script using forever, but I ended up not liking it. This script still isn’t perfect, but it seems more generally useful to me.

Drop the script in /etc/init.d/[name] and make it executable, the custom configuration in /etc/sysconfig/[name], and use service [name] start to fire it up.

true

Installing Jekyll on RHEL/CentOS 5.x and 6.x with modern Ruby and Python versions

2015-01-20T00:00:00-06:00

Jekyll can be run fairly easily on older (5.x and 6.x) versions of RHEL/CentOS, even though the stock versions of the necessary software are too old. The key is to use some simple supporting software to install what you need in a clean way.

The first handy tool is Ruby Version Manager, or RVM. Here’s a handy cheat sheet – use that to install RVM and the latest version of Ruby, set the installed Ruby as the default, and install the Jekyll gem.

Once that’s done, you may run into this error when trying to use Jekyll:

Liquid Exception: Failed to get header

This Jekyll issue indicates that Python greater than 2.6 and less the 3.x needs to be installed in order for syntax highlighting to work. Never fear, pyenv to the rescue!

Use the pyenv installer to get pyenv set up and running on your server, and have a look at the pyenv command reference to install an appropriate version of Python and set it as the default. Note: If you’re running RHEL/CentOS 5.x, you’ll have one more hurdle to cross before using pyenv, see this post for detailed instructions.

If you’d also like to set up your Jekyll site to update from a git push, I’ve detailed one workable approach in this post – if you do use it, just make sure you put the necessary RVM/pyenv environment set up stuff in the .bashrc file of the user the git push command runs as on the server.

Happy Jekyll’ing!

Installing a working pyenv on RHEL/CentOS 5.x

2015-01-19T00:00:00-06:00

RHEL/CentOS 5.x has long since lost its freshness, but some of us are still running servers with it, and can do so for several more years before its end of life.

Perhaps, like me, you have a need to run a more modern version of Python than 5.x installs by default. I recently found pyenv, and it looked to fit my needs perfectly, as I didn’t want to mess with the system version or build custom RPM’s.

Once I installed the build requirements, I used the handy pyenv-installer project to get it up and running, then ran the simple command to install Python 2.7.8. Unfortunately, things hit a bump with this error:

Error message:
subprocess.CalledProcessError: Command '['wget', 'https://pypi.python.org/packages/source/s/setuptools/setuptools-7.0.zip', '--quiet', '--output-document', '/tmp/python-build.20141210170309.3741/Python-2.7.8/setuptools-7.0.zip']' returned non-zero exit status 1

After a bit of digging, I found that this was actually due to a bug in version 1.11 of wget. As far as I could tell, this issue was not fixed upstream until the 1.12 release, and CentOS 5.x is frozen at 1.11.

I decided it was worth building a custom wget RPM package for my 5.x servers to get past this issue. After setting up my RPM build environment, I headed over to rpm.pbone.net to locate a suitable source RPM. wget-1.12-4.fc14.src.rpm ended up suiting my needs – the newer versions of wget RPMs had some build dependencies that were a bit awkward to fulfill, and 1.12 would solve my problem, so…

Per http://wiki.centos.org/HowTos/RebuildSRPM, the –nomd5 switch is needed when installing newer fedora source RPMs:

rpm --nomd5 -ivv wget-1.12-4.fc14.src.rpm

From there it was simply a matter of building the RPM:

cd /usr/src/redhat/SPECS
rpmbuild -bb wget.spec

Then installing it, by finding the wget-1.12-4 RPM file in one of the subdirectories of /usr/src/redhat/RPMS, changing to that directory, and running:

yum --nogpgcheck localinstall wget-1.14-3.x86_64.rpm

After this, pyenv install [version] should work as advertized…Python for the modern age!

Automating sysctl deployments using sysctl.d on RHEL/CentOS 5.x, 6.x, 7.x

2015-01-08T00:00:00-06:00

I’ve fallen in love with automated server deployments in the last year, with my primary weapon being Salt.

One of the corner cases I’ve run into is adding sysctl settings specific to a feature set. For example, when a server needs Redis installed, I want to add the following kernel optimization via sysctl:

    vm.overcommit_memory = 1

It’s sloppy to add this to /etc/sysctl.conf – too hard to maintain in a modular fashion. Wouldn’t it be nice if there was a place we could drop a file with that sysctl setting in it, which would be automatically read on boot? This would enable adding and removing multiple sysctl settings a breeze to automate.

Well, it turns out that RHEL/CentOS does have this support via /etc/sysctl.d. While only RHEL/CentOS 7.x sports the directory out of the box, all three versions provide access to it via init scripts, and anything placed in /etc/sysctl.d will be read on boot, provided that the networking init script’s start action is called (it’s enabled by default).

Unfortunately, this is a bit of an odd placement for triggering a reload of the sysctl settings. I also wanted the ability to only reload the sysctl settings as part of a feature installation on a running server.

The path to get this feature turned out to be pretty short. /etc/init.d/functions contains an apply_sysctl function which handles all the dirty work of completely reloading all sysctl settings, including those placed in /etc/sysctl.d. This extremely short wrapper script does the job:

Armed with that script, I simply use Salt to automatically install it to /usr/local/bin on all servers, and call it any time a file in /etc/sysctl.d is added, removed, or modified.

Nginx logs in JSON format

2012-07-14T00:00:00-05:00

I’ve recently decided that it’s a good idea to output server logs in JSON format. To this end, today I took some time to figure out how to do this for Nginx. The log_format parameter is the one you want to use, I simply added another named format to the http section of nginx.conf, which then allows the named format to be used in any other config file. Here’s what I whipped up – this is just the default main format ported to JSON:

log_format  json  '{'
                    '"remote_addr": "$remote_addr",'
                    '"remote_user": "$remote_user",'
                    '"time_local": "$time_local",'
                    '"request": "$request",'
                    '"status": $status,'
                    '"body_bytes_sent": $body_bytes_sent,'
                    '"http_referer": "$http_referer",'
                    '"http_user_agent": "$http_user_agent",'
                    '"http_x_forwarded_for": "$http_x_forwarded_for"'
                  '}';

I formatted it one row per parameter in the config file, as it’s easier for me to read, but Nginx will concatenate all those separate strings into one line in the log file. Once this is done, use the format in any of places where it’s accepted, for example:

access_log /path/to/file/access.log json;

For more details on all this stuff, check out the online documentation for Nginx’s logging module.

Resend Postfix messages stuck in mail queue to another address

2012-07-06T00:00:00-05:00

Update 2015-02-19: Based on suggestions from David Keegel, I’ve tweaked the script to properly hold/unhold messages, and include the sender’s email address in the message envelope.

Ever had somebody give you a bad email address, and the messages pile up in your Postfix mail queue? Even if you have the right address to send the stuck emails to, Postfix (at least on CentOS 5.x based on all my research) provides no easy way to:

Resend all those messages to the new address
Remove the old stuck messages

We needed this functionality badly for our business, where we unfortunately get bad email addresses given to us all the time. So, using my marginal bash skills and several hours of my time, I whipped up the script below to automate this process.

Caveat: I’ve tested this script extensively with Postfix on CentOS 5.x, but nowhere else, really. So test first, use at your own risk!

#!/bin/bash # This script allows sending of messages stuck in the Postfix queue to another # email address. Useful if the original email address is bad. if [ $# -ne 2 ]; then echo "Usage: reroute-queued-email-messages.sh <oldaddress> <newaddress>" exit 1 fi OLD_ADDRESS="$1" NEW_ADDRESS="$2" MAILQ=`which mailq` POSTCAT=`which postcat` POSTSUPER=`which postsuper` SENDMAIL=`which sendmail` TAIL=`which tail` GREP=`which grep` AWK=`which awk` SED=`which sed` # Pulls queue IDs for all messages sent to the original address. function get_ids_by_address { $MAILQ | $TAIL -n +2 | $GREP -v '^ *(' | $AWK -v address="$1" 'BEGIN { RS = "" } { if ($8 == address) print $1 }' } MESSAGE_IDS=`get_ids_by_address $OLD_ADDRESS` # Loop through each message ID, output the message, and pipe it through # sendmail to the new address. for message_id in $MESSAGE_IDS; do # Puts the message on hold. $POSTSUPER -h $message_id # Extract the sender. sender=`$POSTCAT -q $message_id | $GREP -m 1 '^From: ' | $SED '/^From: */!d; s///; q'` # sed gets rid of the leading queue metadata, grep -v gets rid of the # metadata markers. $POSTCAT -q $message_id | $SED -n '/MESSAGE CONTENTS/,$p' | $GREP -v "^\*\*\*.*\*\*\*$" | sendmail -f "$sender" $NEW_ADDRESS echo "Delivered message ID $message_id to $NEW_ADDRESS" done echo -n "Remove old messages from mail queue? [y/N]: " read delete if [ "$delete" = "y" ]; then for message_id in $MESSAGE_IDS; do # Deletes the message from the mail queue by queue ID. $POSTSUPER -d $message_id echo "Deleted message ID $message_id from queue" done else for message_id in $MESSAGE_IDS; do # Takes the message off hold. $POSTSUPER -H $message_id done echo "The following messages were left in the mail queue:" echo "$MESSAGE_IDS" fi exit 0

db-query-assistant package released

2012-04-28T00:00:00-05:00

Today I’m happy to announce the official public release of db-query-assistant.

For those who haven’t read my initial post on the module, here is a quick summary:

High-level library for node database drivers.
Configurable connection pooling.
Issue multiple simultaneous queries, and get all results back in a callback when the last query completes.
Issue queries in series, getting the results for each previous query back before executing the next one.
Issue transactional queries, with automatic rollback on query failure.

This release includes a fairly comprehensive unit test suite for the core library, using the awesome mocha test framework. I decided against unit tests for the drivers, as I feel those would be better served by integration tests (which I may add in the future).

The module has been published to the npm registry, and a simple npm install db-query-assistant will now do the trick.

Sampler API 6.x 1.1 released

2012-03-25T00:00:00-05:00

The 6.x-1.1 release of Sampler API is now out and ready for download. This has a number of important bugfixes, as well as the addition of locking support so the same metric cannot be run again while already running.

Most of the fixes came out of the work I’m doing to get metrics collection fully deployed on drupal.org.