Monthly Archives: October 2015

Failed To Connect To VMWare Lookup Service

I ran into this just the other day with a new VCSA 5.5 Update 3A appliance.  No idea why it happened, I’ve installed the appliance few times now and I didn’t do anything differently this time that I didn’t do last time.  

 

Either way, regenerating the certs under the “admin” tab and rebooting the appliance fixed the issue.  Just be sure and uncheck that box once the VM boots back up. 

Dell Lifecycle Manager….Sucks!

 

Dell Lifecyle Manager………..it does a few different things but the most important thing I use it for?  A nice one stop shop for updating all of the various Dell fimware in one go.  It’s a lot faster than the silly ISO’s that take an hour or two to run and contain every firmware for every model from the past billion years or so.  And lifecycle manager should be a lot better than manually tracking down the firmware dates for your OS of choice and manually installing them.  Unfortunately for me (and any other fellow admins), it’s a massive pain in the butt to deal with.  Sometimes it just doesn’t work for mysterious reasons.  Sometimes it throws scary error messages in the middle of updating firmware on a spinning SAS disk.  And sometimes it downloads and installs firmware just fine, only for you to find out the new firmware is buggy and downgrading is not an option (luckily, in that case Dell was already working on the next version……and it was less buggy the second time around).

It’s been a long time since I last used Lifecycle manager.  Virtualization has done away with a lot of the fiddly stuff I used to deal with back when everything ran on bare metal.  But here I was, with fairly fast R910 that was getting flattened and repurposed.  Seemed like a great time to update all the firmware bits.  So I loaded up Lifecycle Manager, assigned an IP address, and clicked on the “Test” button.  It failed to ping itself, it failed to ping the DNS server, and it failed to ping the gateway.  But it was able to resolve ftp.dell.com via that same DNS setting.  And it was able to download new firmware just fine.  I have no idea what the error messages were for.  There’s zero reason it would’t have been able to ping its own gateway (or any of the other items).  But hey, all this from a hardware company widely known for their less than stellar track record with firmware and drivers.  I guess I was expecting too much, right?

OK, so that’s awesome.  It fails to connect to much of anything but can still access the ‘Net and download updates.  Whatever.

Partway though installing the firmware updates, I get this jewel:

 

I believe that error indicates I had a corrupt download (or someone at Dell fat fingered something in the system that says update X is for component X and it’s really for component 42, but whatever).  But great, there’s nothing like failed firmware updates to give you warm and fuzzy feelings.  At the end of the day, the system rebooted back into Lifecycle Manager, re-downloaded the failed update, and successfully installed it.  But dammit, it shouldn’t be such a bag of crap to do something “as simple” as apply a few firmware updates.  Get it together Dell, you need to be setting a better example now that you’re a parent company.

 

Note:  No hardware was harmed in the writing of this article 🙂 

Western Digital Buys SanDisk

A couple years ago we had a monster SQL Server database server that desperately needed fast storage to match the beefy CPU and RAM that accompanied the new PowerEdge we were planning on buying.  We eventually settled on a generation 1 Fusion IO card.  At the time Fusion IO was a new-ish company that had made a name for themselves producing some of the fastest PCIE SSD storage on the planet.  A few years later we refreshed the SQL Server hardware again and purchased a generation 2 Fusion IO card.  This was one was roughly the same price, but twice the speed and twice the capacity as the generation 1 card.  In the years we used the gen 1 card in production, we had absolutely zero issues with the card.  Not one hiccup, not one crash, not a single issue.  Spending the $$ for a Gen 2 card was a fairly simple decision to make.  Rarely have I dealt with hardware or software with literally zero issues, but those cards (and drivers, can’t forget the drivers!) literally gave us zero issues.

The Gen 2 card lived up to the marketing promise of twice the speed and the extra capacity meant we could keep both the database and log files on the card.  We’re now approaching the 3 year mark for the Gen 2 card.  It also has had zero issues.  Not a single hardware or software issue.  However a while after we bought the Gen 2 card, Fusion IO as a company was scooped up by SanDisk.  We’ve not needed to purchase additional cards since the Gen 2 card (thanks Nutanix!), so I really can’t say how Fusion IO as a product faired under SanDisk.  But now SanDisk has been bought by Western Digital.  Nowhere nearly as a massive a deal as Dell buying EMC, but it is a large acquisition nonetheless.  

Hopefully between SanDisk and WD the magic sauce that made Fusion IO so completely awesome is still there and still working

Failing As A Service

Ars has put up an article detailing a recently released paper that used an EC2 instance on Amazon’s cloud to break 512 bit encryption in just a couple hours for a grand total of less than $100.  Technically speaking this isn’t surprising.  Moore’s Law (not really a law) is thing, pretty much everyone who knows anything about computers has heard it and knows the gist of it.  And 512 bit encryption has’t been a recommend best practice in a long LONG time.  It’s no surprise that the computing horsepower needed to break encryption from the 1990’s is easy and cheap to acquire.

However as the article points out there are still servers on the Internet that use 512 bit keys for encryption.  I’d hazard a guess that any server running 512 bit keys probably isn’t getting patched either.  In fact I bet it’s been a long time since they have been touched by an admin.  The Internet in general would be better off without those servers.  Maybe it’s time for the web browsers of the world to start throwing up the scary red title bars and warnings you get if you go to a site with an improper or invalid SSL cert?  It should be easy enough for the browser to detect if the encryption is weak and accordingly inform the user.  Hopefully these are not e-commerce servers of any sort and the only people who are affected by the eventual “issues” will be the people who aren’t maintaining them (or paying someone to maintain them on their behalf)

And You Think You Have Legacy Systems To Support?

The NYC Metro system is upgrading the systems that controls the NYC subway system.  There’s a video over at Laughing Squid that details it all much better than I can in blog post, you should check it out.  Just be sure and remember this the next time you are feeling sorry for yourself with your last remaining Windows 2003 server.  🙂

 

 

 

Splunk Queries

I found a great little site (thanks the Splunk sub-Reddit) over at http://gosplunk.com.  As a mostly Windows shop running Splunk for log management, there are some real gems in there.  I restructured my Dashboard using some of the queries I found there.

Event IDs for Suspicious Behaviour

This can take a while to run if you have a large dataset, but it provides a look into a few different categories of Event IDs that I had previously overlooked.

source=WinEventLog:security User!=SYSTEM User!="LOCAL SERVICE" User!="NETWORK SERVICE" | eval Trigger=case(EventCode=516, "Audit Logs Modified",EventCode=517, "Audit Logs Modified",EventCode=612, "Audit Logs Modified",EventCode=623, "Audit Logs Modified",EventCode=806, "Audit Logs Modified",EventCode=807, "Audit Logs Modified",EventCode=1101, "Audit Logs Modified",EventCode=1102, "Audit Logs Modified",EventCode=4612, "Audit Logs Modified",EventCode=4621, "Audit Logs Modified",EventCode=4694, "Audit Logs Modified",EventCode=4695, "Audit Logs Modified",EventCode=4715, "Audit Logs Modified",EventCode=4719, "Audit Logs Modified",EventCode=4817, "Audit Logs Modified",EventCode=4885, "Audit Logs Modified",EventCode=4902, "Audit Logs Modified",EventCode=4906, "Audit Logs Modified",EventCode=4907, "Audit Logs Modified",EventCode=4912, "Audit Logs Modified", EventCode=642, "Account Modification",EventCode=646, "Account Modification",EventCode=685, "Account Modification",EventCode=4738, "Account Modification",EventCode=4742, "Account Modification",EventCode=4781, "Account Modification", EventCode=1102, "Audit Logs Cleared/Deleted",EventCode=517, "Audit Logs Cleared/Deleted", EventCode=628, "Passwords Changed",EventCode=627, "Passwords Changed",EventCode=4723, "Passwords Changed",EventCode=4724, "Passwords Changed", EventCode=528, "Successful Logons",EventCode=540, "Successful Logons",EventCode=4624, "Successful Logons", EventCode=4625, "Failed Logons",EventCode=529, "Failed Logons",EventCode=530, "Failed Logons",EventCode=531, "Failed Logons",EventCode=532, "Failed Logons",EventCode=533, "Failed Logons",EventCode=534, "Failed Logons",EventCode=535, "Failed Logons",EventCode=536, "Failed Logons",EventCode=537, "Failed Logons",EventCode=539, "Failed Logons", EventCode=576, "Escalation of Privileges",EventCode=4672, "Escalation of Privileges",EventCode=577, "Escalation of Privileges",EventCode=4673, "Escalation of Privileges",EventCode=578, "Escalation of Privileges",EventCode=4674, "Escalation of Privileges") | stats count by Trigger | sort - count

Windows Environment Logon Count is a great one that shows a stacked graph of network access to files/folders, Service Accounts, Local Console Access, Scheduled Tasks/Batch Files, Network Logins, and RDP Logins.

source="WinEventLog:security" | eval LogonType=case(Logon_Type="2", "Local Console Access", Logon_Type="3", "Accessing Network Folders or Files", Logon_Type="4", "Scheduled Task, Batch File, or Script", Logon_Type="5", "Service Account", Logon_Type="7", "Local Console Unlock", Logon_Type="8", "Network User Logon", Logon_Type="9", "Program launched with RunAs using /netonly switch", Logon_Type="10", "Remote Desktop via Terminal Services", Logon_Type="11", "Mobile Access or Network Domain Connection Resumed") | top limit=15 LogonType | eval percent = round(percent,2) . " %"

One of my favorite queries lists a table of geographical locations for IP addresses pulled from IIS access logs.  In particular, for Exchange OWA and Outlook Anywhere.

host="email.mydomain.com" sourcetype=iis NOT cs_method=POST | iplocation c_ip |stats count by City | sort count desc

I have 3 instances of this query on my dashboard, one each to show a different table for country, state, and city for our OWA users.  This pulls IIS logs form the Exchange server, filters out the “POST” entries and uses ‘iplocation’ to show what geographical location the GET requests for OWA and Outlook Anywhere are coming from.  It puts all of this in a table sorted by count, which makes it easy to see if we’re taking any hits from locations on the other side of the country or planet.

Spunk can be expensive, though a little less so since they introduced Splunk Light, however the alerting and reports that it enables is well worth it in my opinion.  Version 6.3 was recently released, it features quite a few new features! 

Dell, EMC, and VMware Walk Into A Bar….

Looks like Dell is buying EMC for $67 Billion, and that doesn’t include VMware.  At least not outright, Dell will become the majority stakeholder of VMware but not an outright owner.  VMware will remain an independent company after the Dell/EMC merger.

EMC did a pretty good job of being hands off with VMware.  In a way, that was the only option for them.  If EMC had created new features for their storage platform that only worked with vSphere or added features to the vSphere hypervisor that only worked with their hardware, the tech industry would have rightfully screamed foul play.  Early reports are indicating that Dell will keep their majority stock at arms reach.  That really is the best thing they could do in this particular situation.

$67 Billion is a LOT of money.  I can’t speak to EMCs storage hardware other than say several years ago it was a major pain in the butt to get a quote for a new SAN.  After we finally got it, it was very expensive and very complex to setup and maintain.  However by and large EMC’s gear is well regarded, at least in the “traditional” enterprise storage market.  I also can’t speak to Dell’s Compellent hardware.  But as a former Equal Logic customer and storage administrator, Holy Crap is that one platform that Dell needed to replace.  I expect the Equal Logic hardware to be phased out and replaced with EMC gear.  We may even see the Equal Logic name fade away, in some circles that name equals sub-par hardware and really buggy firmware.  It won’t be missed by this admin.

If the price for Dell to get back in the storage game (and get a nice shot in the arm with EMC’s other properties) is $67 Billion, so be it.  I hope it works out for them.  Knowing a good match for your company when you see it AND being able to purchase outright it is a rare thing at this level of the game.  That said, if I had $67 Billion to spend to revitalize my enterprise computing company, I’d have bought Nutanix or Pure.  Or both.  $67 Billion is a LOT of cash. 

Overcast 2.0

 

Overcast 2.0 was released earlier today.  Mac Stories wrote up a great review and interviewed the developer, Marco Armet, about the new release.

The two biggest features are support for streaming and moving away from free with IAP to completely free with a patronage model.

I can’t write a better review than Mac Stories, but that isn’t the intention of this post.  I just want to encourage anyone who previously tried Overcast but didn’t purchase the IAP to give it another try.  It was a day one purchase for me back when 1.0 was released.  As of this blog post, Smart Speed has saved me over 110 hours!  It really is the best podcast app (in my opinion).

I also want to give a quick shout out to the storage view feature that shows how much space is being used by each podcast subscription.  Also, for anyone with a 6S or 6S+, check out the 3D Touch menu from the spring board.  That is greatly appreciated!

I’d also like to encourage anyone who finds the app as useful as I do to look into Patronage screen under Settings.  It is not every day you see a truly great app drop the IAP and go completely free.  I’d like to send a few bucks Macro’s way. If you find Overcast half as useful as I do, you should to.

Tools of the Trade

I work as a Systems Administrator in a mostly Windows Server environment, plus our VMware cluster that runs on top of Nutanix hardware.  My laptop is a late 2013 15″ MacBook Pro, 256 GB SSD and 8 GB of RAM.  I’ve listed the software I’m using roughly in order of importance.  Up until I got the Mac, I had always used a laptop with whatever the current client version of Windows was at the time (XP or Windows 7).

Hardware

  • 2013 15″ MacBook
  • Dell 2415Q External Display
  • mStand Raindrop Stand (http://www.raindesigninc.com/mstand.html)
  • Apple Wired Full Sized Keyboard (the one that includes a full number pad)
  • USB Mouse

Software

  • Royal TSX (RDP and SSH)
  • Microsoft Office 2016
  • Virtual Box (Local Windows VM)
  • Chrome
  • vSphere Client (thick client as well as the web interface, which still requires Flash, hence running it on Chrome in a VM vs OS X)
  • Notepad++ for some powershell stuff
  • Dropbox
  • Omnifocus  (OS X and iOS)
  • Notability (OS X and iOS)
  • Text Wrangler
  • Super Duper!

As it turns out, at least with the way I work, this MacBook is more than capable of being used in my day to day job.  I have never been someone who installs snap-ins and other administrative tools on my local laptop.  Even when I had a Windows 7 laptop I would RDP into a “management” server that had all the tools, etc I needed.  I never liked the idea that the computer I use to browse the web (whether that be Safari on this Mac or IE on my previous Windows laptop) also being the machine that I use to connect to servers, SANs, and other bits of critical infrastructure.  That also gives an added benefit of not being tied to this piece of hardware.  I could drop this computer in the river and be up and running on a different computer VERY quickly.

There’s nothing in Outlook, all of that is stored in Exchange.  Royal TSX’s configuration is 1 file that’s saved on either Dropbox or my local documents folder.  My local instance of Windows is a VM that’s also a file location in my Documents folder.  Both of those items are simply files as far as OS X is concerned.  They are backed up regularly by multiple scheduled processes (Time Machine/Super Duper to a local USB drive, and we use Druva’s inSync to backup client machines to a locally hosted server).  Dropbox is Dropbox, what is there to backup for the purposes of this discussion?  Omnifocus is sync’d with Omni’s cloud service, and it also writes a backup of its entire database locally on a regular basis, which is then grabbed by Time Machine and Super Duper.  Notability is used for quick note taking and sync’s with iCloud, though it’s mostly used as a staging area for manipulating data or jotting down quick thoughts vs long term storage of documents (though it can be configured to backup to Dropbox if needed).  That rule applies even more so for Text Wrangler, but if there was something I wanted to keep permanently with Text Wrangler, everything is saved as a text document to the local drive and picked up by the backup methods discussed above.

Although some of these applications are Mac only, none of them exist in a genre exclusive to the Mac.  You could even make the point that if this were a Windows laptop, I wouldn’t need the Windows VM I use now.  Maybe, maybe not.  There is a lot to be said about the convenience of VMs simply being files on the host system.  It makes them easy to backup and easy to move around.

There are plenty of other applications that are used from time to time, but the list above covers the tools I use all day, every day.

Although I skipped Windows 8 and have not yet used Windows 10, I find that OS X’s implementation of Virtual Desktops (Spaces) as well as Window Management via Expose to be wonderful.  I’d legitimately have trouble moving to a computer that didn’t offer something like that.  Then there is the build quality of the actual laptop, battery life, thinness/lightness, and the great sleep/wake/sleep cycle for weeks on end that Windows was never able to pull off.  Top that off with having access to iMessage and FaceTime audio/video, being able to send/receive phone calls from my Mac, and other seamless connections between my iPhone and iOS apps and OS X, and I would be very hesitant to move back to Windows as my daily use machine.