It's a good time to be a Linux sysadmin!

Linux is everywhere now.  It's on my TV, on my Roku, it's the firmware running my SAN hardware, it's the OS behind all of our virtualization platforms. If you know me, you know my technology passions are for Linux, Perl and pretty much anything running open source.  I started using Linux about 1993/94 when I downloaded SLS from GEnie over my 2400 baud modem.  It was kernel 0.99p11.  It booted Linux and ran off of floppy.  Later, I got a larger copy from GEnie using work's 9600 baud connection.  I had to dialup Indy to get a speed that fast!

Later, when Windows 95 came out, I started running Linux as my desktop and I've stuck with it.  I remember that the guys in my office would always reboot their Windows 95 systems at lunch as a preventative measure to prevent accidental lockups.  One time, just prior to lunch, I distinctly remember Ken yelling "Oh  no!  It locked up and I've been working on that spreadsheet for an hour!" I typed "w" to get my uptime:

[rat@localhost Desktop]$ w
 16:39:19 up 99 days,  8:14,  4 users,  load average: 0.25, 0.25, 0.27


Ha!  They can't keep their desktop up half a day - mine's been up for 99 days and still going.  I think to myself: "Linux - I think I'll keep it."

It's been over 20 years.  I can remember telling someone I worked on Unix and Linux systems.  "Ah, a niche OS, eh?"  "Niche?"  Not really.  Unix/Linux was running the Internet before these guys knew there was an Internet.  It's always been the one to run DNS, web servers, DCHP servers, NTP (time protocol), etc.  It ran all the SMTP (mail) and SNMP (network and hardware monitoring).  Linux WAS the back office of any decent sized IT shop. 

Later, Apple forked BSD Unix into its operating system OSX.  Now Apple was Unix!  I was very much at home on the Apple OSX command line.  Everything was there: Bash shell, Perl, and all the core Unix commands.  Pretty cool. 

Next came Windows PowerShell.  So many of the commands were aliased to bash commands that I was once again at home.  I really like Windows 2012 R2 Core and PowerShell.  Each new version of PowerShell adds more and more bash-like aliases.

And now, Microsoft Azure Cloud Services has dozens of versions of Linux you can deploy in their cloud service.  One estimate was that over 75% of the virtual machines in Microsoft Azure Cloud Service were Linux.  Even better - Microsoft runs AZURE on Linux!!  What's more, the Azure services SDK client also runs on Linux!  It was pretty freaky seeing bash scripts with a Microsoft license boiler plate on them.  And now the Office 360 is coming to Linux!

It's not a niche!  It's not going away.  I'd say that, without fan fare, Linus' OS has finally won.  I saw the future 20 years ago.  It is Linux!!

Merry Christmas everyone!

P.S. Don't forget - Linux is the OS running Android phones and the Raspberry Pi.

References:
http://www.wired.com/2015/09/microsoft-using-linux-run-cloud/

The Joy of Modern Computing

1. Solaris Zones & Containers
2. Oracle VirtualBox
3. Xen 
4. VMware
5. KVM 
6. Heroku
7. Hyper-V 
8. Citrix XenServer 
9. Oracle VM Manager (OVM) 
10. Amazon Web Services (AWS) 
11. Microsoft Azure Cloud Services

These are a few of my favorite virtualization platforms that I've been using since 2006. I love virtualization!  I work mainly out of a virtualized desktop running ssh into virtualized systems.   Lately, I've been working a lot with Microsoft Azure Cloud Services.

It's funny how this is perceived by some people unfamiliar with it.  Today in a meeting, some people thought that moving systems to the "cloud" would cause a loss of IT positions.  I never thought so.  My least favorite part of the job is the part that takes .001% of your time - mounting the physical hardware in a rack and wiring it.  If that goes away, I will not miss it at all.

I love that I can clone, copy and snapshot VMs.  Using physical hardware now seems to me like working without a net. I would not want to go back.  Even better is having the services in the cloud.  No messy wires, no failing hard drives (I have an open ticket for one now).  Even backups are a breeze.

I have a 32 bit Windows 7 desktop VM that I migrated from a PC running CentOS 5 under Xen to a PC running Red Hat Linux 6 running KVM (that was a tough migration - but it worked!) and then I moved it again to a PC running Oracle Linux 7 under KVM (much easier move).  So, my Windows 7 has been with me through 3 PCs.  All of my stuff is there - all my files and software.  Virtualized PC's are really cool!

I also run additional VMs on my PC.  If a new distro comes out and I want to see what it looks like - I just load it from the ISO!  I don't have to have spare hardware.  It's great!  I have been running my desktop like this about 6 years and can't imagine doing it any other way.  (I started virtualizing my servers almost 10 years ago).

Lately, as I have built about a dozen various VMs in Azure.  I've been testing the cloning of VMs & filesystems and securing endpoints. (It took some time to figure out how to reserve static addresses.)

Now, I love Linux and my technology passion is learning pretty much anything OpenSource - BUT, even though I could do quite a bit with the cross platform kit for Azure under Linux - most of the work was easier in Powershell.  No problem - because I really like Powershell - BUT, my 32 bit Windows 7 system didn't seem to want to finish the install and hung twice.  I figured it was because it was 32 bit.  Again, thanks to virtualization - No problem - I just loaded up 64 bit Windows 10.  (I tried to like Windows 8 and 8.1 but eventually gave up. Skipping that one.)  Windows 10 is a nice mix and seems more intuitive then its predecessor and ran the Azure Powershell environment which made cloning much easier.

The cool thing is, I can still run my old Windows 7 for as long as it gets patches (and then I could just remove the network and keep it for nostalgia) BUT I also get to run Windows 10 and use the Azure Powershell features I need.

Anyway, Linux and KVM gives you a lot of possibilities and it's free (your VMs, may not be). 

Azure and Oracle Linux 6

Typical of my luck - the VERY FIRST Azure VM I install is the Oracle Linux 6.4.  Of course, the first thing I do after a new install is "yum upgrade".  For the first time ever, I get a weird error and cannot complete this:

...
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
kernel-headers is needed by glibc-headers-2.12-1.166.el6_7.3.x86_64
kernel-headers >= 2.2.1 is needed by glibc-headers-2.12-1.166.el6_7.3.x86_64
kernel-uek is needed by oracle-rdbms-server-11gR2-preinstall-1.0-12.el6.x86_64
kernel-uek is needed by oracle-rdbms-server-12cR1-preinstall-1.0-14.el6.x86_64
** Found 4 pre-existing rpmdb problem(s), 'yum check' output follows:
glibc-headers-2.12-1.107.el6_4.4.x86_64 has missing requires of kernel-headers
glibc-headers-2.12-1.107.el6_4.4.x86_64 has missing requires of kernel-headers >= ('0', '2.2.1', None)
oracle-rdbms-server-11gR2-preinstall-1.0-7.el6.x86_64 has missing requires of kernel-uek
oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64 has missing requires of kernel-uek
Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx-2015-11-17-08-39DucKzd.yumtx
...

Very strange.  I have never needed to install kernel headers to complete this task before.  So, I install kernel headers and still it insists on installing the UEK kernel - so I do.  Now it allows me to fully patch the system.

Installing : kernel-uek-firmware2.6.32-400.37.12.el6uek.noarch
Installing : kernel-uek-2.6.32-400.37.12.el6uek.x86_64
Installing : kernel-headers-2.6.32-573.8.1.el6.x86_64

I reboot and now my Azure appears to hang.  There is nothing I can do - with no console to see what's going on or choose a different kernel or single user mode.  I'm stuck.  Of course, there is always the procedure of detaching the drive, adding it to a rescue VM, editing, chroot'ing, re-running grub - yeah, fun.



I suspect it is trying unsuccessfully to boot the Unbreakable Linux Kernel (um, yeah, I've seen that break things before).  So, I scrap this new install and start again.  This time, taking it slow.  I remove the item requiring UEK:

yum remove oracle-rdbms-server-11gR2-preinstall oracle-rdbms-server-12cR1-preinstall

I do not need to run Oracle on this system (luckily - because this is really handy if you do) so now I can just install kernel-headers and re-run yum upgrade.  I do this - then reboot and it works.  (Strangely enough - I try Oracle Linux 7 and do not have this issue.  I need to review and see why [updated post later].)

Meanwhile, I run across this blog post from last month stating "UEK2 is not supported on Hyper-V and Azure as it does not include the required drivers".  yup - found that out the hard way.

I also noticed my favorite volume manager LVM is not installed by default.  I needed to add more disk space and would probably need to grow the data area in the future so I add a VG (volume group) for data and create a PV and LV (Physical and Logical Volume) for the data and logging areas.  Seem to work ok.  But the above blog post (from last month) says: "it is recommended that you use standard partitions rather than LVM". It complains mainly of issues with LVM name conflicts with cloned VMs.  I really think that was more of an issue with RHEL 5 (they named everything the same).  Anyway, my LVM group is, so far, unique, so I'm not worried (but I'll have to remember this).

That's all from me.  The documentation above covers a lot of Oracle Linux 6 issues and work-around.  Be sure to click on it.

Getting old browser history for new PC

I just got a new PC.  I have a pretty complicated setup.  I run Linux and use KVM to run Windows 7 for Outlook and Powershell.  I run a lot of VMs.  And, whenever I get a new PC, I migrate the VMs from the old system to the new.  Before I do, I'm in a transition period where I just use virt-manager to run the old VMs off the old system and display on my new desktop (cuts down on monitors & keyboards.

So, I have my shiny new PC and I want to check on my BackupPC servers (I have 3 of them).  I usually just click on them in my quick-launch but the browser history is missing.  I decided to just look at the old history instead of launching the remote (old) version of FireFox for some reason.  And, besides, I need to move that info anyway.

My boss thinks my setup is nuts but he generously donated an extra 1TB drive for my VMs.  I create a VG for this disk and start carving off space.

# lvcreate -L 165G -n lv_oldol6 vg_tank
# mkfs.ext4 /dev/mapper/vg_tank-lv_oldol6
# mkdir /oldol6
# vi /etc/fstab   (add new file system)
# mount /oldol6

Now I have a place to store my data from my old system. So I mount the old system via SSH.

# yum install sshfs
# mkdir /ssh
# cd /ssh
# sshfs me@old.system.ip:/ /ssh/
# tar -cf - . | (cd /oldol6/ && tar -xf - )
# sync   (old sysadmin good luck command)
# updatdb
# locate places.sqlite   (I look for the one in the old place)
# sqlite3 /oldol6/.mozilla/firefox/umw8efgh.default/places.sqlite
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .mode column
sqlite> .header on
sqlite> .schema
sqlite> select url FROM moz_places where title LIKE '%backuppc%';

Ta-dahhh!  That worked nicely.  I find the URL I'm looking for.

Booting Headless M3000 with XSCF and Wyse 55

The nice thing about Sun/Oracle Solaris Sparc systems is THEY JUST RUN!  It had been a while since I had rebooted the M3000 from a cold powered-off state.  We had some weird power issues over the holidays and lost some equipment and crashed others (despite UPS protection and generator backups).

The system was powered off - nothing but XSCF was on.  I couldn't remember the XSCF password.  I was trying root.  As it turns out, you are forbidden from creating an account called "root" on XSCF (it is one of the many reserved words).  I had to create a new user using the following procedure (thank you, Bikash!):

    login: default
Change the panel mode switch to Locked and press return…
Leave it in that position for at least 5 seconds. Change the panel mode switch to Service, and press return…

I did as directed...

 Create a user andrew

XSCF> adduser andrew

 Change the password for andrew

XSCF> password andrew
New XSCF password:
Retype new XSCF password:

 Grant andrew the following privileges, useradm, platadm, aplatop.

XSCF> setprivileges andrew useradm platadm platop

 After that, I was able to log in as my new user and view the errors.  There were none.

        XSCF> showlogs error 

Other options here: showlogs event, showlogs power, showlogs monitor, showlogs console and fmdump.

What I needed to do was boot.  Again, I struggled to remember the procedure for getting to the boot prompt.  I tried the T2 commands to start a console - no luck.  I tried simply console which got me close.  I needed a domain_id that I did not know.  Then I vaguely remembered something and tried this:

    XSCF>  console -d 0


I answered Y to "Do you really want to start console" and then it did nothing.  You need an extra ENTER here before the boot prompt showed up.  Then I typed "boot" and was finished.

It all seems so straight-forward and mnemonic now.  I don't know what my problem was this morning.  :-)