It’s no secret that inter-networking is not for the faint of heart. It’s often a nightmarish task only the brave and/or insane are willing to accomplish. (Yes, I know you’ve been wondering; that picture does have 3 switches in there somewhere, and is real. Find more on /r/cablefail.) I felt the desire to share a recent tail of some of my adventures in my current line of work as a wears-all-the-hats (like.. all of them) info-tech professional. But first… a little back story.. apparently… since I set out to write a little story about a networking problem I had recently at work for the sake of putting content onto my blog but instead went into a good chunk of my life’s history… so fuck it, I’m leaving it, deal with it.

Deal with it glasses

… the abyss looks back into you.

My journey into the pits of hell started when I was young, naive, and too smart for my own good. Lured by the promise of no homework and getting to play with technology under the guise of learning (and getting credits!), I was suckered into taking the Cisco Certified Networking Associates certification course through my high school, offered by the local junior college (I was a Sophomore at the time).

OSI model and Cisco branding jammed down my throat day after day, my instructor likening it to Cisco beating the concepts into you with a shovel, and so it was. But it was fun, it was something new. Learning binary for addressing and subnet math, switching, routing, WAN, LAN, EGRP, BGP, more and more, on and on it went. I even went so far as to take up the ultimate nerd honor and don a red jacket and compete in regional and state-level SkillsUSA VICA inter-networking competitions for three years running. There I was, the geeky kid who took computers way too seriously, in a special club where I got to have days off from school to drive down to Southern California and stay in a hotel for a few nights to compete at who was the best at plugging in ethernet and writing configurations for Cisco routers and switches. If it sounds retarded and boring… retrospectively it was. But it beat being stuck in a podunk piece of shit school hating everyone and everything around me.

Where that lead me

I soon thereafter left that school just before my senior year, the fine California public school system decided it was going to lose my grade history back through halfway through my sophomore year and proclaim I would then be ineligible to graduate. Though they had my three folders worth of behavioral track record and lacking attendance that they saw so fit to throw in my face at every opportunity. None thought to make the correlation that if I had been suspended a few times and had not been serving my detentions that I had actually attended school during that time period, fancy that logic.

My only saving grace was my top percentile state standardized test scores that I was able to take to a neighboring school district in order to get them to sign off on the needed credits for the middle portion of my high school career and finish my senior year by way of independent study. A blessing this was for my, and a boon for my skills in info-tech. A once-every-two-week meeting with a teacher to get handed a semesters worth of a few classes or another; taking these home to cram in a few hours and spend two weeks doing whatever I wanted until the next meeting! Yes I could have taken more work to do during that period, or scheduled to go in more often since I was blowing through the work with narry a care. (Who could imagine that reading a chapter and taking a test was all that was necessary for some students, rather than the piles of busywork drawing out the process of learning a subject for the sake of extracting the funds the state pays the school for said student to warm a chair day after day). But I felt content to have what felt to me to be an extended summer vacation.


What did I fill the time with you ask? (Or probably didn’t, because let’s be honest, who’s still reading this bullshit anyway… ) Oh.. setting up a room full of so many second hand computers and networking equipment as to nearly touch the ceiling and attract the attention of the local federal authorities seeking out Marijuana grow operations in the area due to the power usage and heat generated; causing them to stake out my poor parents house for a few days before realizing we weren’t the next Botwin family.

Running test networks, servers, writing code, whatever I could do in my bedroom and my home broadband connection (thanks mom!) to learn and explore the technology and do things that I felt would be fun. This involved a lot of Linux and open source technology, IRC, forums, and a whole lot of screwing things up. I learned early on about network security, hanging with  a bunch of hacker types… my first lesson was never to give someone root, and never use the same password for everything. That was a wonderful few hours of horror while my email and forum accounts were rifled through in the name of fun and teaching a young noob a lesson. But it was all returned to me with good pat on the back.

I’m almost getting to the point!

Fast forward a number of years and I’ve found myself at my current job. I do everything from repair your grandmas Dell Dimension 8400 to ensure her AOL desktop still works, up to managing a small multi-homed network and datacenter of Linux/BSD servers. VoIP engineering, network engineering, software development, systems administration, and retail level technical support (told you I wear all the hats..). It’s a great little local business staffed by some cool chaps in a very lax working environment with hardly any management headache. Having next to no qualifications other than I took a class once about Cisco networking, and spent way too much time in my bedroom pretending to be a datacenter, I get to do what I’m the best at doing and get paid to do it. Sure I don’t make as much as I could be making in a larger establishment that might have much more strict working conditions and experience requirements… I’m happy to have the freedom to work while also learning, breaking things, and flying by the seat of my jeans.

Finally.. what I actually meant to write about…

So to completely switch topics entirely after some needless back story (suck it!) onward to my joyous technical clusterfuckery!

We here at work have a FreePBX server we use for our day-to-day business phone junk what such. Based on our setup of some 13 extensions, it’s cheaper for us to just buy the SIP trunks from a provider and run our own server. Otherwise, (barring setting up a special deal with a VoIP provider company which basically is never going to happen in a million years) we’d have to pay a monthly between 20 and 25 dollars a month PER extension. We instead pay for a 4-channel SIP trunk with a flat allotment minutes (metered outbound I believe).

So in the interest of not using shithole analog phones with no nifty features like IVRs, holiday recordings, voice mails, and the like… we have to deal with routing SIP and RTP traffic through our NAT. Which is not a feat that is easily accomplished in a stable fashion.

So many routers out there want to be HELPFUL (HAHA) with RTP sessions, and the VoIP provider we use who shall not be named does not make things easy. Instead of using a SIP registration string like everyone else on the planet.. we have to give them our external IP address.. and they just flood us with VoIP-laced UDP packets. No handshake, just… OH HEY WE HAVE A CALL HERE YOU GO BLARRGGHHH to every IP address that we give them, which at that point was two for both of the uplinks.

Dutifully, our pfsense router would take the traffic from the provider and merrily NAT forward it to the PBX. Great… but it would induce a race condition where the SIP session would initialize and return out whichever path to the WAN opened NAT first, and the RTP media stream would find it’s way out the default gateway. Sometimes these were not the same path… YAY! Oh the joy of random dropped calls and one-way audio.

The Solution!

So.. being the industrious networking nerd that I am… thought to fix this solution once and for all, and maintain the ability for seamless failover to our backup uplink without having to activate/deactivate rules manually. VPN!!!! OpenVPN to be exact. A wonderful open source project renowned the world over for secure, platform agnostic privately encrypted tunneling. So I turned to my favorite VPS company DigitalOcean (I’m not sponsored by them, but if you’re curious check them out at that link with my referral code for $10 free credit to try them out, which is 2 months of their lowest tier VPS which has been more than enough for my purposes over the last couple years), host of this very site, to set up an OpenVPN server in which to tunnel out my PBX to the WAN and have a single IP address the provider can flood me with their packets, and not care which uplink I’m using to hit the open net. Here’s kind of what it looks like:

Funny topology map

And it’s working GREAT! Our pfSense router is built out of a very reliable SuperMicro Haswell-family motherboard with a Xeon E3. 4 Built-in Intel Server NICs and a 5th that’s a dedicated IPMI port. Intel AES acceleration to boot.. we’re getting some 10-16ms of latency from our office to the San Francisco DigitalOcean datacenter, and an overall throughput of about 75% of our available bandwidth (based on speed tests to various servers) through this little 512MB RAM, single core VPS running Ubuntu 14.04. They let you crank through the 1Gbit uplink with a Terrabyte of transfer (only metered outbound) to boot! It’s astonishing.. next to no overhead (same ping time to the VPS over unencrypted, regular traffic). Best part is.. we’re actually REDUCING our latency to the SIP trunk provider. Some weird route that our main link was taking us on to get to our provider added about 10-15ms compared to going through the VPN. The path to DigitalOcean does not include this route. DigitalOcean is peered to top-tier Level3, which our SIP provider in LA also has as peering to in whatever datacenter they’re hosted at. It’s a win-win! All for an extra $5 a month.


So that’s it, way too much back history for a dumb post about doing the obvious, but hey.. I had fun, and it’s my corner of the internet to do as I please with! Hope you enjoyed! If not.. fuck you and leave a comment telling me how much you didn’t enjoy it and I can ignore it like 99.999999% of the world ignores this blog!