Author Archives: Nathan

Exchange Free/Busy & Out of Office Issues

Once again, I’ve went many months between posts.  Sorry.  Life has just been crazy!  Since the last post we’ve moved to a hybrid config with Exchange 2010 and Office 365.

An issue I ran into a while back was with an on-prem user not able to see free/busy calendar info for anyone else, on-prem or 365.  At first I thought it was a problem with the hybrid configuration because it was initially reported that the on-prem user couldn’t see free/busy info for people already moved to O365 (and I had just setup the Hybrid configuration).  However after digging into the details, it turns out they couldn’t see free/busy info for anyone in the Org regardless of the location of their mailbox.  AND, they could access that info from OWA!  It was only broken in Outlook.  And the final head scratcher, the user hadn’t changed their password recently and wasn’t routinely getting locked out of AD.

It took some time to track down, but eventually it was found to be an old saved credential in the Window Credential Manager.  Deleted that, relaunched Outlook, and *poof*!  Free/busy info works!  I have no idea why it simply broke that part of Outlook and didn’t generate any failed login attempts in Exchange, nor did it lock out the AD account.  It simply broke Free/Busy info (and only free/busy).

Twitter For Mac

Twitter just announced (via a tweet of course) that the native Mac app is officially dead and they will be moving an “experience that’s consistent across platforms”.

Well that’s great. The Twitter for Mac app has been a crap app for years, not really supported (slow up to be updated and I would say a lack of meaningful updates when it is actually updated. It’s basically a piece of crap. I’ve been a loyal Tweetbot user since almost day of using Twitter. Twitterific is pretty awesome too. I highly recommend either one. Both don’t display the nasty in-line ads in the recently deceased Mac app and they both sync timeline position with their iOS counterpart (if you care about such things; I do 😀).

Try either one out, I’m sure you’ll find yourself actually liking Twitter again.

Update:  Twitterific is on sale.  Get it now!

https://itunes.apple.com/us/app/twitterrific-5-for-twitter/id1289378661?mt=12

How To Stream Music & Video With Plex on FreeBSD

Introduction

Plex is a great piece of software, if you’ve never heard it before think of it as an easy to use service that runs on a computer that streams just about any format of audio/video to a Smart TV, Apple TV, Roku, or modern console. You can even easily configure it so that your iOS device can stream content from your media server across the Internet. Perfect!

If you are running a small blog on a VPS instance, chances are you have way more bandwidth, disk space, RAM, and CPU than you need for the blog. I’ve put together this short tutorial on how to install & configure Plex on your VPS to stream your music/video collection from that instead of your home computer (for example, if you want to stream your music library without hitting your residential data cap at home!).

Prerequisites

Before you begin this guide you’ll need the following:

Freebsd 10.x Droplet

A non-root user with sudo privileges (https://www.digitalocean.com/community/tutorial_series/getting-started-with-freebsd) explains how to set this up.)

Up To Date Ports Tree

Step 1 — Updating Ports Tree

The Ports Tree is updated constantly, you should update your system’s local copy on a regular basis. To do this manually, execute the following command:

sudo portsnap fetch update

Step 2 – Install Plex

Plex doesn’t have a lot of prerequisites, the easiest way to install is to run:

sudo pkg install plexmediaserver

After the package has been installed, the installation process tells you how to set the plexmediaserver service to start automatically at boot as well as how to manually start it. We’ll perform both commands now:

sudo sysrc plexmediaserver_enable=YES

and

sudo service plexmediaserver start

Step 3 – Configure Plex

root@freebsd:~ # pkg install plexmediaserver
Updating FreeBSD repository catalogue…
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
plexmediaserver: 1.0.0.2261
compat9x-amd64: 9.3.903000.20160608

Number of packages to be installed: 2

The process will require 170 MiB more space.
65 MiB to be downloaded.

Proceed with this action? [y/N]: y

After downloading and installing, it will prompt you with the following information:

Message from plexmediaserver-1.0.0.2261:
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

multimedia/plexmediaserver includes an RC script:
/usr/local/etc/rc.d/plexmediaserver

TO START PLEXMEDIASERVER ON BOOT:
sysrc plexmediaserver_enable=YES

START MANUALLY:
service plexmediaserver start

Once started, visit the following to configure:
http://localhost:32400/web

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

NOTES: If there is an issue getting to the initial setup screen of the server after adding your IP to the allowed network setting, try a different browser or close all tabs and windows and empty cookies.

If there is a problem “Claiming” the new server, log out of plex and back in again.

If there is a problem with the new server showing up on iOS, sign out of your account in the plex app and force kill it. Re-launch and log back in.

WWDC 2016 Prediction

Everyone these days has a WWDC prediction. New hardware, Xcode for IOS, OS X getting renamed to mac OS, the list goes on and on.

So here’s my quick prediction.

Apple recently announced subscription pricing across all of the App Stores. So what better way to lead the change than to move all of thier professional apps over to a subscription? Logic & Final Cut on a subscription from the Mac App Store! You heard it here first!

Desktop Central Forwarding Agent

Desktop Central has the ability to manage smartphones via the standard MDM APIs. It can do much of the same things any MDM solution can offer, and if you already have it in place for employee computers you might be interested in using it for your MDM solution as well.

One of the optional components is simply called the Desktop Central Forwarding Server. You install this on a server in your DMZ, open a few ports between it and the internal Desktop Central server, a few ports between it and the Internet, and your mobile devices can be managed when not on the internal network. All without exposing your Desktop Central server to the Internet. However, there is one key step that isn’t clearly explained in the documentation.

There is a step when you install the Fowarding Server that you need to to copy over a couple encrpytion keys from Desktop Central for the installer to import. You also need to generate Apple MDM certificates from apple.com and import those into Desktop Central. The first step is so the traffic between the Forwarding Server and Desktop Central is encrypted and you don’t run into any issues with the Forwarding Server complaining about not trusting the Self Signed Certs on Desktop Central. The certs from Apple that get imported into Desktop Central are to allow Desktop Central to be able to manage iOS devices (send push notifications, remotely lock and wipe the device, etc). However, if you import the files from Desktop Central to the Forwarding Server and then import the Apple certs you will break the connection between Desktop Central and the Forwarding Server. It is critical that you import the Apple certs to Desktop Central first, and then copy Desktop Central’s keys over to the Forwarding Server. Otherwise you end up stuck on an extremely unhelpful error message when trying to enroll an iOS device remotely. The exact error you get is:

** PROFILE INSTALLATION FAILED**

Profile Failed To Install

With no explanation as to why that is happening.

Make sure you do the certs in the order specificed above. Your day will go a lot smoother if you do.

Exchange & UPN Suffixes

I recently ran into a problem where we reactivated ActiveSync after having it disabled for years because we used a solution for mobile email that didn’t rely on ActiveSync (Good for Enterprise). We decided to re-evaluate ActiveSync, it’s improved a lot since Exchange 2003 and early versions of iOS. Re-enabling the proper settings for a couple users to test with and we were good to go (note, Outlook for iOS is awesome….check it out if you haven’t already). After a bit of testing we decided to increase the number of users using it. Enabling ActiveSync for those mailboxes is all we need to do, right?

Not so much.

I ran into a problem where none of them could setup thier Exchange account in either Outlook for iOS or Mail.app. It failed at the authenication setup. To make things more confusing my user account worked just fine, even on the same phones thier user account failed on.

A newly created test account failed in the same way, so off to https://testconnectivity.microsoft.com I went. The test the website performed an Exchange ActiveSync test and said it worked fine. Connected, authenicated, logged in, etc It’s all good, except no one can connect using thier phone, outside of a few people it worked for orginally (and continues to work for). OWA and Outlook on a PC has always worked and continues to work. It seems to be limited just to ActiveSync. At this point I’m completely out of ideas.

Check the firewall, nothing seems wrong there. Trying a more complex password, no difference. Trying a simpler password, no difference. Finally I think to try a Mac OS X based mail app that talks to Exchange via ActiveSync. Boom, it works! What in the world…..? So it isn’t an ActiveSync problem after all, or least not 100% an ActiveSync problem.

Eventually I found an error when trying one of the Autodiscover tests at https://testconnectivity.microsoft.com/ with the test account. In the past this same test had worked fine with my account, but I decided to try it with the test account. And it failed! Finally, something to dig into.

What I found was that the Auto Discover test reported:

An HTTP 401 Unauthorized response was received from the remote Unknown server. This is usually the result of an incorrect username or password. If you are attempting to log onto an Office 365 service, ensure you are using your full User Principal Name (UPN).

Digging into that error this was helpful, I found out the UPN suffix for my test account was set to domain.local instead of domain.com (which is what our email address is). Changed that and the test account was able to be setup on phones and devices as expected!

Release Notes & Patches

It’s been a while since I’ve posted to the blog. I had (and have) aspirations of writing here on a regular basis, not every day but certainly more often than I have been lately. I don’t have time to post every day (or multiple times a day) about news happening in the Sysadmin part of the world. There are better sites out there for that type of thing, this site doesn’t need to replicate work that already being done better elsewhere.

I want to focus more on longer/better but less frequent articles. I want to continue writing posts more like the Unifi post. This one is about the importance of reading release notes for all the bits of software sysadmins are responsible for in a modern datacenter.

I just finished a major software upgrade for my company’s production VMware cluster. It was running vSphere 5.5 xxxx and needed to be upgraded to 5.5 update 3, both to address a bug we were experiencing at the version we were at but to also get the wide range of security fixes that had been patched between the two builds. Seems simple enough, right? I mean just login to the vSphere client, connect to the vSphere Update Manager and go to town.

Not so much. I’ve got an approved maintenance window of 3 hours a week, same 3 hours every Thursday. The business knows that’s the time upgrades happen, but everything needs to be back in a running state before 10 PM. I can’t get all of this done in one 3 hour block, so things need to be kept happy and running between maintenance windows.

Besides vSphere, I also needed to account for the following:

  • Trend Micro Deep Security
    • Has various hooks into each host in order to be able to inspect and product the guest VMs. Needs to support both the existing ESXi build as well update 3. Also needed to confirm that the new version of DSM would work with the existing appliances since they could only be upgraded as each host was upgraded in turn.
      -vShield Networking and Security
    • Needed upgraded to address bugs, etc but also needs to be upgraded to a version that is supported by Deep Security, the version of ESXi I was currently running, as well as the version of ESXi I would be going to.
  • Nutanix Controller VMs (NOS)
    • Although there were no known issues at the time of update 3a’s release, I waited approximately 2 weeks for Nutanix to do internal QA with their code and Update 3a to ensure there were no tricky gotchas waiting for me. That’s great because that’s one less thing I need to worry about and it isn’t like I didn’t have a couple maintenance windows worth of other updates that needed to be applied for prior to rolling out the update hypervisor anyway.
  • Horizon View Desktops
    • Needed to upgrade to a version of Horizon View that supported both the current build of ESXi I was on as well as the Update 3a. The VMware Product Interoperability listed no such version. I had to open a ticket with VMware support to verify which build of View I should go to. The matrix has since been updated to show version 6.1.1 was the magic build for me.

After a lot of checking, double checking, and note taking I had a comprehensive set of steps in Omnifocus that would result in an updated cluster that could be completed in chunks spread across several weeks with no downtime outside of the Thursday night maintenance window.

That process was:

  • Upgrade vShield Network
  • Update Deep Security Manager
  • Upgrade vCenter Server Appliance
  • Upgrade Horizon View Connection Server
  • Upgrade Nutanix Controller software
  • Begin updating the hypervisor on each host, one at a time.
    • Pick first host
      • Put host in maintenance mode
        • Upgrade vShield Endpoint Driver
        • Upgrade Trend Micro Filter Driver
        • Upgrade physical NIC drivers for ESXi (update needed)
        • Reboot
        • Remove old Trend Micro appliance
        • Provision new Trend Micro appliance
        • Apply vSphere updates
        • Reboot
        • Exit maintenance mode
    • Verify Nutanix Controller services restarted and rejoined the cluster
  • Repeat for additional hosts

I was lucky. I managed to just barely squeak by without needing to do multiple updates of a single product to get up to date. If I had waited much longer, I’d have had to upgrade vSphere partway, upgrade View, then upgraded vSphere the rest of the way, then finish updating View.

I’ve got resources in the cluster such that we can continue to run at 100% load with one host out of the cluster. I could power off test VMs and other non-critical servers to free up resources so that more than one host could be down at a time. But at the end of the day, I decided that the time savings from jumping through all the hoops to be able to reboot multiple hosts at once would likely be the same as if I just took down one host a time and vMotion’d everything around. In the end, I just did it one host a time. To get everything updated and make it through two reboots of a physical server (rebooting a VM has us all so spoiled, such a fast reboot cycle versus booting a physical server) took about an hour each. I ended up doing two hosts (back to back) in a maintenance window, so it took a few weeks to get everything done.

In news that will come as a shock to absolutely no one who reads a Sysadmin blog, before I got all my hosts upgraded to the latest and greatest build…….a new round of patches was released. Don’t get me wrong, bugs need fixed and security holes need patched. I’m glad to receive improvements and updates. I just need to not let it go so long between update cycles. It makes it a real pain to get it all sorted out.

Release Notes

It’s been a while since I’ve posted to the blog. I had (and have) aspirations of writing here on a regular basis, not every day but certainly more often than I have been lately. I don’t have time to post every day (or multiple times a day) about news happening in the Sysadmin part of the world. There are better sites out there for that type of thing, this site doesn’t need to replicate work that already being done better elsewhere.

I want to focus more on longer/better but less frequent articles. I want to continue writing posts more like the Unifi post. This one is about the importance of reading release notes for all the bits of software sysadmins are responsible for in a modern datacenter.

I just finished a major software upgrade for my company’s production VMware cluster. It was running vSphere 5.5 xxxx and needed to be upgraded to 5.5 update 3, both to address a bug we were experiencing at the version we were at but to also get the wide range of security fixes that had been patched between the two builds. Seems simple enough, right? I mean just login to the vSphere client, connect to the vSphere Update Manager and go to town.

Not so much. I’ve got an approved maintenance window of 3 hours a week, same 3 hours every Thursday. The business knows that’s the time upgrades happen, but everything needs to be back in a running state before 10 PM. I can’t get all of this done in one 3 hour block, so things need to be kept happy and running between maintenance windows.

Besides vSphere, I also needed to account for the following:

Trend Micro Deep SecurityHas various hooks into each host in order to be able to inspect and product the guest VMs. Needs to support both the existing ESXi build as well update 3. Also needed to confirm that the new version of DSM would work with the existing appliances since they could only be upgraded as each host was upgraded in turn. -vShield Networking and Security
Needed upgraded to address bugs, etc but also needs to be upgraded to a version that is supported by Deep Security, the version of ESXi I was currently running, as well as the version of ESXi I would be going to.
Nutanix Controller VMs (NOS)Although there were no known issues at the time of update 3a’s release, I waited approximately 2 weeks for Nutanix to do internal QA with their code and Update 3a to ensure there were no tricky gotchas waiting for me. That’s great because that’s one less thing I need to worry about and it isn’t like I didn’t have a couple maintenance windows worth of other updates that needed to be applied for prior to rolling out the update hypervisor anyway.
Horizon View DesktopsNeeded to upgrade to a version of Horizon View that supported both the current build of ESXi I was on as well as the Update 3a. The VMware Product Interoperability listed no such version. I had to open a ticket with VMware support to verify which build of View I should go to. The matrix has since been updated to show version 6.1.1 was the magic build for me.
After a lot of checking, double checking, and note taking I had a comprehensive set of steps in Omnifocus that would result in an updated cluster that could be completed in chunks spread across several weeks with no downtime outside of the Thursday night maintenance window.

That process was:

Upgrade vShield Network
Update Deep Security Manager
Upgrade vCenter Server Appliance
Upgrade Horizon View Connection Server
Upgrade Nutanix Controller software
Begin updating the hypervisor on each host, one at a time.Pick first hostPut host in maintenance modeUpgrade vShield Endpoint Driver
Upgrade Trend Micro Filter Driver
Upgrade physical NIC drivers for ESXi (update needed)
Reboot
Remove old Trend Micro appliance
Provision new Trend Micro appliance
Apply vSphere updates
Reboot
Exit maintenance mode
Verify Nutanix Controller services restarted and rejoined the cluster
Repeat for additional hosts

I was lucky. I managed to just barely squeak by without needing to do multiple updates of a single product to get up to date. If I had waited much longer, I’d have had to upgrade vSphere partway, upgrade View, then upgraded vSphere the rest of the way, then finish updating View.

I’ve got resources in the cluster such that we can continue to run at 100% load with one host out of the cluster. I could power off test VMs and other non-critical servers to free up resources so that more than one host could be down at a time. But at the end of the day, I decided that the time savings from jumping through all the hoops to be able to reboot multiple hosts at once would likely be the same as if I just took down one host a time and vMotion’d everything around. In the end, I just did it one host a time. To get everything updated and make it through two reboots of a physical server (rebooting a VM has us all so spoiled, such a fast reboot cycle versus booting a physical server) took about an hour each. I ended up doing two hosts (back to back) in a maintenance window, so it took a few weeks to get everything done.

In news that will come as a shock to absolutely no one who reads a Sysadmin blog, before I got all my hosts upgraded to the latest and greatest build…….a new round of patches was released. Don’t get me wrong, bugs need fixed and security holes need patched. I’m glad to receive improvements and updates. I just need to not let it go so long between update cycles. It makes it a real pain to get it all sorted out.

Website Hosting

So, if anyone has visited the site recently you likely noticed two things immediately. A distinct lack of posts and an ever changing roulette wheel of Content Management Systems (and if you dug a little deeper, an ever changing hosting provider as well). Why? Why would I do that?

I wanted to try a few different setups. I started out on Squarespace, moved to WordPress on Linode, and then moved from there to Ghost on Digital Ocean. Now I’m back on Squarespace. Between each jump I had to export and convert the posts of the blog. I also had to figure out themes, the look and feel of the site, setup an SSL cert (or not). I actually wrote blog posts about each jump, why I was moving from Squarespace to WordPress, why I was moving from WordPress to Ghost, now I’m writing about why I ended right back where I started. I mean that literally. I logged into Squarespace to fire up a two week trial and found my old site was just there, just inactive. I re-upped with Squarespace, manually copied over the pitiful 3 or 4 posts I’ve made in the few months since I moved to WordPress and was good to go.

Part of the reason I moved from Squarespace to WordPress was because this is a Systems Administration blog. It seemed (and in some ways still does) a little lame for me to not roll up my sleeves and ssh into my server and keep things running in tip top shape. But I do that all day every day at my 9 to 5 job. Do I really want to sign up to do that in my off hours as well? If I’m honest with myself, no……I really don’t. I like certain aspects of it, sure. But all in all, I get more than enough of that at work. After being on WordPress for a while I realized that all I got done with my “free” time is screw around with the underlying OS of the blog and tweaking bits of WordPress instead of actually writing for the blog. I also realized that at $10 a month, Linode is at the very high end of what a small site like this would cost to host. So between my $10 a month Linode instance, worrying about WordPress exploits, and in general feeling a bit “bleh” about the whole thing I moved to Ghost on Digital Ocean.

Ghost doesn’t use a traditional SQL database like WordPress. Without MySQL, I didn’t really need a VPS with 1 GB of RAM. The smallest Droplet at Digital Ocean would work fine (cutting the hosting cost by a whooping $5/mo or a much more impressive 50%). So I setup Nginx and Ghost (actually I used the Digital Ocean Ghost template) and configured it to host multiple separate instances of ghost. One for this site and one for my personal site. My thinking was the droplet costs the same no matter how I use it and both sites will be very low traffic so why not. The personal site never got a single piece of content written for it or posted. I spent an evening or two making it all work together and be happy with the free SSL cert from Let’s Encrypt. I got that setup and working and the only blog post I ever wrote was a brief post explaining that I moved the site to Digital Ocean and Ghost and to stay tuned for new awesome posts!

Eventually what I realized is that once you pay for a whole year of Squarespace at once to get the 10% discount and then apply another 10% off discount from your favorite podcast it’s less than the $10 a month for Linode (and my uber awesome oh-so-cheap DO Droplet….was saving me literally $2.50 a month). I decided it was time to just admit it. I love screwing around with servers just a little too much. I can’t help myself. I’d rather do that than write blog posts. Plus none of the themes and tweaks I did to either Ghost or WordPress made it look half as good as this theme from Squarespace. So why not use Squarespace for my blog? It’s cheap. It looks great. And on the occasion I get mentioned by someone with a few thousand twitter followers I don’t need to worry about my site crumbling under the load.

In addition to that stunning realization, I discovered something incredible.

Migrating content between different web sites really sucks. Like really. Yeah, import/export features get you 95% of the way there. But man, that last 5% is awful. If only there was a way to write and save blog content in plain text while keeping the formatting, etc intact. That’s right! There’s this thing called Markdown and I’m an idiot for not using it sooner! Actually I started using it back when I wrote that one post while on Ghost. But YES. Starting with this post, and all posts going forward, they will be saved as Markdown formatted files saved on my computer. Where they can be easily backed up and easily manipulated if I ever move away from Squarespace (not anytime soon).

So here I am, here I’m staying. Maybe once Google reindexes my site this post will save some other sysadmin from thinking “I wonder where I should host my blog? I know! I’ll spin up an instance of WordPress on a VPS!”. Trust me. It costs just as much to host on Squarespace once you factor in your time, if you are like me (not an artistic person) the site will look better for it, and on the chance someone famous links to your site you don’t need to worry about the server falling over.

Ghost

It’s been a while since I’ve posted last, but things have been changing behind the scenes at the site. Yep, yet another CMS and hosting provider change.

I’ll have a more indepth post later, but for the time being if you get a SSL certificate error, it’s because of moving the site and I haven’t got everything updated yet (Yay for working from guest wifi while your car is being worked on). 🙂