Category Archives: sysadmin

Exchange Free/Busy & Out of Office Issues

Once again, I’ve went many months between posts.  Sorry.  Life has just been crazy!  Since the last post we’ve moved to a hybrid config with Exchange 2010 and Office 365.

An issue I ran into a while back was with an on-prem user not able to see free/busy calendar info for anyone else, on-prem or 365.  At first I thought it was a problem with the hybrid configuration because it was initially reported that the on-prem user couldn’t see free/busy info for people already moved to O365 (and I had just setup the Hybrid configuration).  However after digging into the details, it turns out they couldn’t see free/busy info for anyone in the Org regardless of the location of their mailbox.  AND, they could access that info from OWA!  It was only broken in Outlook.  And the final head scratcher, the user hadn’t changed their password recently and wasn’t routinely getting locked out of AD.

It took some time to track down, but eventually it was found to be an old saved credential in the Window Credential Manager.  Deleted that, relaunched Outlook, and *poof*!  Free/busy info works!  I have no idea why it simply broke that part of Outlook and didn’t generate any failed login attempts in Exchange, nor did it lock out the AD account.  It simply broke Free/Busy info (and only free/busy).

Desktop Central Forwarding Agent

Desktop Central has the ability to manage smartphones via the standard MDM APIs. It can do much of the same things any MDM solution can offer, and if you already have it in place for employee computers you might be interested in using it for your MDM solution as well.

One of the optional components is simply called the Desktop Central Forwarding Server. You install this on a server in your DMZ, open a few ports between it and the internal Desktop Central server, a few ports between it and the Internet, and your mobile devices can be managed when not on the internal network. All without exposing your Desktop Central server to the Internet. However, there is one key step that isn’t clearly explained in the documentation.

There is a step when you install the Fowarding Server that you need to to copy over a couple encrpytion keys from Desktop Central for the installer to import. You also need to generate Apple MDM certificates from apple.com and import those into Desktop Central. The first step is so the traffic between the Forwarding Server and Desktop Central is encrypted and you don’t run into any issues with the Forwarding Server complaining about not trusting the Self Signed Certs on Desktop Central. The certs from Apple that get imported into Desktop Central are to allow Desktop Central to be able to manage iOS devices (send push notifications, remotely lock and wipe the device, etc). However, if you import the files from Desktop Central to the Forwarding Server and then import the Apple certs you will break the connection between Desktop Central and the Forwarding Server. It is critical that you import the Apple certs to Desktop Central first, and then copy Desktop Central’s keys over to the Forwarding Server. Otherwise you end up stuck on an extremely unhelpful error message when trying to enroll an iOS device remotely. The exact error you get is:

** PROFILE INSTALLATION FAILED**

Profile Failed To Install

With no explanation as to why that is happening.

Make sure you do the certs in the order specificed above. Your day will go a lot smoother if you do.

Exchange & UPN Suffixes

I recently ran into a problem where we reactivated ActiveSync after having it disabled for years because we used a solution for mobile email that didn’t rely on ActiveSync (Good for Enterprise). We decided to re-evaluate ActiveSync, it’s improved a lot since Exchange 2003 and early versions of iOS. Re-enabling the proper settings for a couple users to test with and we were good to go (note, Outlook for iOS is awesome….check it out if you haven’t already). After a bit of testing we decided to increase the number of users using it. Enabling ActiveSync for those mailboxes is all we need to do, right?

Not so much.

I ran into a problem where none of them could setup thier Exchange account in either Outlook for iOS or Mail.app. It failed at the authenication setup. To make things more confusing my user account worked just fine, even on the same phones thier user account failed on.

A newly created test account failed in the same way, so off to https://testconnectivity.microsoft.com I went. The test the website performed an Exchange ActiveSync test and said it worked fine. Connected, authenicated, logged in, etc It’s all good, except no one can connect using thier phone, outside of a few people it worked for orginally (and continues to work for). OWA and Outlook on a PC has always worked and continues to work. It seems to be limited just to ActiveSync. At this point I’m completely out of ideas.

Check the firewall, nothing seems wrong there. Trying a more complex password, no difference. Trying a simpler password, no difference. Finally I think to try a Mac OS X based mail app that talks to Exchange via ActiveSync. Boom, it works! What in the world…..? So it isn’t an ActiveSync problem after all, or least not 100% an ActiveSync problem.

Eventually I found an error when trying one of the Autodiscover tests at https://testconnectivity.microsoft.com/ with the test account. In the past this same test had worked fine with my account, but I decided to try it with the test account. And it failed! Finally, something to dig into.

What I found was that the Auto Discover test reported:

An HTTP 401 Unauthorized response was received from the remote Unknown server. This is usually the result of an incorrect username or password. If you are attempting to log onto an Office 365 service, ensure you are using your full User Principal Name (UPN).

Digging into that error this was helpful, I found out the UPN suffix for my test account was set to domain.local instead of domain.com (which is what our email address is). Changed that and the test account was able to be setup on phones and devices as expected!

Release Notes & Patches

It’s been a while since I’ve posted to the blog. I had (and have) aspirations of writing here on a regular basis, not every day but certainly more often than I have been lately. I don’t have time to post every day (or multiple times a day) about news happening in the Sysadmin part of the world. There are better sites out there for that type of thing, this site doesn’t need to replicate work that already being done better elsewhere.

I want to focus more on longer/better but less frequent articles. I want to continue writing posts more like the Unifi post. This one is about the importance of reading release notes for all the bits of software sysadmins are responsible for in a modern datacenter.

I just finished a major software upgrade for my company’s production VMware cluster. It was running vSphere 5.5 xxxx and needed to be upgraded to 5.5 update 3, both to address a bug we were experiencing at the version we were at but to also get the wide range of security fixes that had been patched between the two builds. Seems simple enough, right? I mean just login to the vSphere client, connect to the vSphere Update Manager and go to town.

Not so much. I’ve got an approved maintenance window of 3 hours a week, same 3 hours every Thursday. The business knows that’s the time upgrades happen, but everything needs to be back in a running state before 10 PM. I can’t get all of this done in one 3 hour block, so things need to be kept happy and running between maintenance windows.

Besides vSphere, I also needed to account for the following:

  • Trend Micro Deep Security
    • Has various hooks into each host in order to be able to inspect and product the guest VMs. Needs to support both the existing ESXi build as well update 3. Also needed to confirm that the new version of DSM would work with the existing appliances since they could only be upgraded as each host was upgraded in turn.
      -vShield Networking and Security
    • Needed upgraded to address bugs, etc but also needs to be upgraded to a version that is supported by Deep Security, the version of ESXi I was currently running, as well as the version of ESXi I would be going to.
  • Nutanix Controller VMs (NOS)
    • Although there were no known issues at the time of update 3a’s release, I waited approximately 2 weeks for Nutanix to do internal QA with their code and Update 3a to ensure there were no tricky gotchas waiting for me. That’s great because that’s one less thing I need to worry about and it isn’t like I didn’t have a couple maintenance windows worth of other updates that needed to be applied for prior to rolling out the update hypervisor anyway.
  • Horizon View Desktops
    • Needed to upgrade to a version of Horizon View that supported both the current build of ESXi I was on as well as the Update 3a. The VMware Product Interoperability listed no such version. I had to open a ticket with VMware support to verify which build of View I should go to. The matrix has since been updated to show version 6.1.1 was the magic build for me.

After a lot of checking, double checking, and note taking I had a comprehensive set of steps in Omnifocus that would result in an updated cluster that could be completed in chunks spread across several weeks with no downtime outside of the Thursday night maintenance window.

That process was:

  • Upgrade vShield Network
  • Update Deep Security Manager
  • Upgrade vCenter Server Appliance
  • Upgrade Horizon View Connection Server
  • Upgrade Nutanix Controller software
  • Begin updating the hypervisor on each host, one at a time.
    • Pick first host
      • Put host in maintenance mode
        • Upgrade vShield Endpoint Driver
        • Upgrade Trend Micro Filter Driver
        • Upgrade physical NIC drivers for ESXi (update needed)
        • Reboot
        • Remove old Trend Micro appliance
        • Provision new Trend Micro appliance
        • Apply vSphere updates
        • Reboot
        • Exit maintenance mode
    • Verify Nutanix Controller services restarted and rejoined the cluster
  • Repeat for additional hosts

I was lucky. I managed to just barely squeak by without needing to do multiple updates of a single product to get up to date. If I had waited much longer, I’d have had to upgrade vSphere partway, upgrade View, then upgraded vSphere the rest of the way, then finish updating View.

I’ve got resources in the cluster such that we can continue to run at 100% load with one host out of the cluster. I could power off test VMs and other non-critical servers to free up resources so that more than one host could be down at a time. But at the end of the day, I decided that the time savings from jumping through all the hoops to be able to reboot multiple hosts at once would likely be the same as if I just took down one host a time and vMotion’d everything around. In the end, I just did it one host a time. To get everything updated and make it through two reboots of a physical server (rebooting a VM has us all so spoiled, such a fast reboot cycle versus booting a physical server) took about an hour each. I ended up doing two hosts (back to back) in a maintenance window, so it took a few weeks to get everything done.

In news that will come as a shock to absolutely no one who reads a Sysadmin blog, before I got all my hosts upgraded to the latest and greatest build…….a new round of patches was released. Don’t get me wrong, bugs need fixed and security holes need patched. I’m glad to receive improvements and updates. I just need to not let it go so long between update cycles. It makes it a real pain to get it all sorted out.

Release Notes

It’s been a while since I’ve posted to the blog. I had (and have) aspirations of writing here on a regular basis, not every day but certainly more often than I have been lately. I don’t have time to post every day (or multiple times a day) about news happening in the Sysadmin part of the world. There are better sites out there for that type of thing, this site doesn’t need to replicate work that already being done better elsewhere.

I want to focus more on longer/better but less frequent articles. I want to continue writing posts more like the Unifi post. This one is about the importance of reading release notes for all the bits of software sysadmins are responsible for in a modern datacenter.

I just finished a major software upgrade for my company’s production VMware cluster. It was running vSphere 5.5 xxxx and needed to be upgraded to 5.5 update 3, both to address a bug we were experiencing at the version we were at but to also get the wide range of security fixes that had been patched between the two builds. Seems simple enough, right? I mean just login to the vSphere client, connect to the vSphere Update Manager and go to town.

Not so much. I’ve got an approved maintenance window of 3 hours a week, same 3 hours every Thursday. The business knows that’s the time upgrades happen, but everything needs to be back in a running state before 10 PM. I can’t get all of this done in one 3 hour block, so things need to be kept happy and running between maintenance windows.

Besides vSphere, I also needed to account for the following:

Trend Micro Deep SecurityHas various hooks into each host in order to be able to inspect and product the guest VMs. Needs to support both the existing ESXi build as well update 3. Also needed to confirm that the new version of DSM would work with the existing appliances since they could only be upgraded as each host was upgraded in turn. -vShield Networking and Security
Needed upgraded to address bugs, etc but also needs to be upgraded to a version that is supported by Deep Security, the version of ESXi I was currently running, as well as the version of ESXi I would be going to.
Nutanix Controller VMs (NOS)Although there were no known issues at the time of update 3a’s release, I waited approximately 2 weeks for Nutanix to do internal QA with their code and Update 3a to ensure there were no tricky gotchas waiting for me. That’s great because that’s one less thing I need to worry about and it isn’t like I didn’t have a couple maintenance windows worth of other updates that needed to be applied for prior to rolling out the update hypervisor anyway.
Horizon View DesktopsNeeded to upgrade to a version of Horizon View that supported both the current build of ESXi I was on as well as the Update 3a. The VMware Product Interoperability listed no such version. I had to open a ticket with VMware support to verify which build of View I should go to. The matrix has since been updated to show version 6.1.1 was the magic build for me.
After a lot of checking, double checking, and note taking I had a comprehensive set of steps in Omnifocus that would result in an updated cluster that could be completed in chunks spread across several weeks with no downtime outside of the Thursday night maintenance window.

That process was:

Upgrade vShield Network
Update Deep Security Manager
Upgrade vCenter Server Appliance
Upgrade Horizon View Connection Server
Upgrade Nutanix Controller software
Begin updating the hypervisor on each host, one at a time.Pick first hostPut host in maintenance modeUpgrade vShield Endpoint Driver
Upgrade Trend Micro Filter Driver
Upgrade physical NIC drivers for ESXi (update needed)
Reboot
Remove old Trend Micro appliance
Provision new Trend Micro appliance
Apply vSphere updates
Reboot
Exit maintenance mode
Verify Nutanix Controller services restarted and rejoined the cluster
Repeat for additional hosts

I was lucky. I managed to just barely squeak by without needing to do multiple updates of a single product to get up to date. If I had waited much longer, I’d have had to upgrade vSphere partway, upgrade View, then upgraded vSphere the rest of the way, then finish updating View.

I’ve got resources in the cluster such that we can continue to run at 100% load with one host out of the cluster. I could power off test VMs and other non-critical servers to free up resources so that more than one host could be down at a time. But at the end of the day, I decided that the time savings from jumping through all the hoops to be able to reboot multiple hosts at once would likely be the same as if I just took down one host a time and vMotion’d everything around. In the end, I just did it one host a time. To get everything updated and make it through two reboots of a physical server (rebooting a VM has us all so spoiled, such a fast reboot cycle versus booting a physical server) took about an hour each. I ended up doing two hosts (back to back) in a maintenance window, so it took a few weeks to get everything done.

In news that will come as a shock to absolutely no one who reads a Sysadmin blog, before I got all my hosts upgraded to the latest and greatest build…….a new round of patches was released. Don’t get me wrong, bugs need fixed and security holes need patched. I’m glad to receive improvements and updates. I just need to not let it go so long between update cycles. It makes it a real pain to get it all sorted out.

Website Hosting

So, if anyone has visited the site recently you likely noticed two things immediately. A distinct lack of posts and an ever changing roulette wheel of Content Management Systems (and if you dug a little deeper, an ever changing hosting provider as well). Why? Why would I do that?

I wanted to try a few different setups. I started out on Squarespace, moved to WordPress on Linode, and then moved from there to Ghost on Digital Ocean. Now I’m back on Squarespace. Between each jump I had to export and convert the posts of the blog. I also had to figure out themes, the look and feel of the site, setup an SSL cert (or not). I actually wrote blog posts about each jump, why I was moving from Squarespace to WordPress, why I was moving from WordPress to Ghost, now I’m writing about why I ended right back where I started. I mean that literally. I logged into Squarespace to fire up a two week trial and found my old site was just there, just inactive. I re-upped with Squarespace, manually copied over the pitiful 3 or 4 posts I’ve made in the few months since I moved to WordPress and was good to go.

Part of the reason I moved from Squarespace to WordPress was because this is a Systems Administration blog. It seemed (and in some ways still does) a little lame for me to not roll up my sleeves and ssh into my server and keep things running in tip top shape. But I do that all day every day at my 9 to 5 job. Do I really want to sign up to do that in my off hours as well? If I’m honest with myself, no……I really don’t. I like certain aspects of it, sure. But all in all, I get more than enough of that at work. After being on WordPress for a while I realized that all I got done with my “free” time is screw around with the underlying OS of the blog and tweaking bits of WordPress instead of actually writing for the blog. I also realized that at $10 a month, Linode is at the very high end of what a small site like this would cost to host. So between my $10 a month Linode instance, worrying about WordPress exploits, and in general feeling a bit “bleh” about the whole thing I moved to Ghost on Digital Ocean.

Ghost doesn’t use a traditional SQL database like WordPress. Without MySQL, I didn’t really need a VPS with 1 GB of RAM. The smallest Droplet at Digital Ocean would work fine (cutting the hosting cost by a whooping $5/mo or a much more impressive 50%). So I setup Nginx and Ghost (actually I used the Digital Ocean Ghost template) and configured it to host multiple separate instances of ghost. One for this site and one for my personal site. My thinking was the droplet costs the same no matter how I use it and both sites will be very low traffic so why not. The personal site never got a single piece of content written for it or posted. I spent an evening or two making it all work together and be happy with the free SSL cert from Let’s Encrypt. I got that setup and working and the only blog post I ever wrote was a brief post explaining that I moved the site to Digital Ocean and Ghost and to stay tuned for new awesome posts!

Eventually what I realized is that once you pay for a whole year of Squarespace at once to get the 10% discount and then apply another 10% off discount from your favorite podcast it’s less than the $10 a month for Linode (and my uber awesome oh-so-cheap DO Droplet….was saving me literally $2.50 a month). I decided it was time to just admit it. I love screwing around with servers just a little too much. I can’t help myself. I’d rather do that than write blog posts. Plus none of the themes and tweaks I did to either Ghost or WordPress made it look half as good as this theme from Squarespace. So why not use Squarespace for my blog? It’s cheap. It looks great. And on the occasion I get mentioned by someone with a few thousand twitter followers I don’t need to worry about my site crumbling under the load.

In addition to that stunning realization, I discovered something incredible.

Migrating content between different web sites really sucks. Like really. Yeah, import/export features get you 95% of the way there. But man, that last 5% is awful. If only there was a way to write and save blog content in plain text while keeping the formatting, etc intact. That’s right! There’s this thing called Markdown and I’m an idiot for not using it sooner! Actually I started using it back when I wrote that one post while on Ghost. But YES. Starting with this post, and all posts going forward, they will be saved as Markdown formatted files saved on my computer. Where they can be easily backed up and easily manipulated if I ever move away from Squarespace (not anytime soon).

So here I am, here I’m staying. Maybe once Google reindexes my site this post will save some other sysadmin from thinking “I wonder where I should host my blog? I know! I’ll spin up an instance of WordPress on a VPS!”. Trust me. It costs just as much to host on Squarespace once you factor in your time, if you are like me (not an artistic person) the site will look better for it, and on the chance someone famous links to your site you don’t need to worry about the server falling over.

Installing Plex on a VPS

Plex is a great piece of software, if you’ve never heard it before think of it as an easy to use service that runs on a computer at home that streams just about any format of audio/video to a smart TV, Apple TV, Roku, or modern console. You can even easily configure it so that your iOS device can stream content from your media server across the Internet. Perfect!

However, maybe your home Internet upload speed is not very good. Or you have a data cap. Or you are trying to upload a massive amount of data to Amazon or Backblaze for backups and you don’t need to make that process even slower by using precious upload bandwidth for Plex. This site is hosted on a VPS instance with way more disk space than I need for a small blog, so I’ve plenty of disk space and bandwidth to stream my music from that instead of my from my home computer.

First of all, it’s as simple as downloading the .deb file from Plex’s site and following the simple install instructions to get the service installed. Really the one and only hiccup I ran into (and the reason I decided to write this blog post about it) is that once you’ve installed the service it is expecting you to configure it by visiting http://localhost:32400/web. However it’s a command line only Linux environment and Lynx doesn’t get the job done (I tried).

After much Googling, all I could find was references to using ssh to setup a tunnel and changing your browser’s proxy setting so that the Plex service thought you were accessing it from the local machine. That was, in my experience, a bunch of crap and never worked. Eventually I found a forum post that simply said to edit the Plex config file that restricted the initial setup to only happen from the local host. A quick trip to https://www.whatismyip.com and a quick edit in vi, and I was in business.

Here’s all you have to do:

  • Change into /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/
  • Edit the Preferences.xml file
  • There should be two lines, the second line is very long. It starts with Preferences in brackets.
    • After that tag, add the following:
      • allowedNetworks=”your.ip.address.here/your.subnet.mask.here”
  • For example, you’d put allowedNetworks=”1.2.3.4/255.255.255.255″
  • Save the file, restart the Plex service, and POOF! You can now login and configure the server via http://server-ip-address:32400/web

After you configure the service, be sure you remove the “allowedNetworks” tag from the XML file and restart the service.

EXIFTOOL

We’re having a fun snow day today in my part of the world. We’re expected to get somewhere between 12″ and 24″ of snow. As I type this, we’re at around 8″ of snow and it hasn’t stopped pouring the snow since around 9 AM this morning.

Now you might be wondering, just what does this have to do with the blog? Well I wanted a way to post some snow pictures to a forum, and didn’t want to use Imgur or Droplr. So I created a directory within /var/www/html on the server, set proper permissions, and started uploading photos from the camera roll on my iPhone. However I didn’t want to post photos with GPS data embedded in them. A quick trip to Google revealed a command line tool that’s perfect for the job. I had no idea exiftool existed, but it does and it does its job well. A quick run of the tool and my photos were metadata free.

Exiftool can be downloaded from your disto’s package system, and the quick example I used on how to use it can be found at Linux Magazine

Let’s Encrypt!

So you may have noticed that the blog now accepts HTTPS connections!  That’s right, https://www.thesysadminlife.com is now a working and valid URL.  I joined the beta of Let’s Encrypt, it took about 5 minutes to setup and couldn’t have been easier (especially considering what a pain in the ass SSL certs have typically been).

This site runs on Apache, which is supported web server for Let’s Encrypt client.  I got a copy of the latest code from Git, and ran the following command

./letsencrypt-auto --apache -d thesysadminlife.com -d www.thesysadminlife.com

It churned for a few minutes and then asked which Apache config file contains the virtual host settings for my site.  I am running Debian on a VPS that was provisioned from scripts, so there were three options to pick from and I wasn’t sure which one was correct.  My first attempt failed, so I re-ran the command above and picked the option to re-install the already provisioned cert.  With a different choice, it succeeded and everything worked fine.  I was also given the choice to redirect HTTP traffic to HTTPS traffic or to accept both.  Since this site is just a personal blog, I chose to accept both types (for now).

One thing I didn’t know before starting this was the certificates from Let’s Encrypt are only valid for 90 days.  I followed the instructions and easily setup a cron job that renews the cert every 60 days, giving me a month of buffer time in case something goes wrong.

It really was the best experience I’ve ever had when dealing with server certificates.  I’m not sure how it could have been easier.  I can completely recommend this service to anyone wanting to secure their site (though for an e-commerce site, perhaps a paid cert would be a better choice).

For setup instructions, check out the instructions over at Let’s Encrypt.

Takeown.exe

I ran into an issue the other day where a file on a network share ended up with its NTFS permissions being hosed in such a way that no one could edit, delete, or even take ownership of it.  I’m not sure how it happened, but it did and the ticket ended up with me to get it fixed.

Nothing I did in the GUI could fix the problem.  I could see the filesystem security attributes were hosed and nothing, not even taking ownership, would successfully complete.  After a quick visit to Google, I found the Technet page for takedown.exe.  It’s basically a tool for sysadmin’s to take ownership of a file with borked permissions.  Perfect!  That’s exactly what I need.

Unfortunately, it didn’t work and failed with a non-helpful generic error.  Turns out I was having a case of the stupids and the file was locked by a crashed application.  Killing the processes released the lock on the file and then I was able to delete the file and restore it from the previous days backup.  On the plus side, I found what looks to be a great tool to keep bookmarked for the future!