[Disclaimer: I do not work for Google or Twitter; I have no investments in Google or Twitter. ]

Google should buy Twitter. (link to rumors here)

Twitter isn't a good "MBA runs the numbers" acquisition. However could be used as a showcase for CGE. It would more than justify itself. In fact, the financial losses might be off-set by the marketing value it provides to CGE.

As part of integrating it into the internal Google stack, they should require their engineers to rebuild it on the Google Cloud Engine platform. GCE scales crazy-good. Twitter has a history of scaling problems. If Google could run it on the Google Cloud Engine, and show that it scales, it would be great advertising.

Google needs GCE to succeed (but that's for another blog post... or you can read Appendix B ofhttp://the-cloud-book.com .. especially the last few paragraphs.)

How difficult would it be to rebuild Twitter on GCE? I think it would be easier than you'd imagine. Every talk I've seen by Twitter engineers at conferences is about technology that (and I don't mean this with any disrespect) is reproducing something in Google's stack. Most of those technologies being re-invented are available in GCE now, and the rest really should be. In fact, if they aren't available in CGE they should be. The project of porting Twitter to GCE would generate a list of high-quality feature requests. Interestingly enough, the re-invented technologies don't seem to be as scalable as Google's original. Oh, and it seems like a lot of people re-implementing those technologies at Twitter are ex-Google employee so ... you have that.

Sadly the few Google executives that I know think that Twitter is a joke, nobody uses it, and isn't worth saving. I disagree. I think it is "the world's chatroom". If you think Twitter "doesn't matter" then why does every news program, TV show, and billboard list a Twitter handle? (Hint: they don't list G+ handles... does G+ even have handles?)

So, in summary:

  • Google should buy Twitter.
  • It would help save this important resource that the world finds very useful.
  • It would be the best showcase of GCE evah.... which is something Google needs more than the revenue of Twitter.
  • Sadly Google executives dis Twitter as a niche application that a very small number of people find compelling. (Spoiler alert: I think they're wrong)

I wonder what will happen.

Tom

NOTE: This article was written by Thomas Limoncelli and included no involvement by current or past co-authors or friends.

Posted by Tom Limoncelli in Rants

Pearson is doing their annual "Back to Business" sale until Monday, September 19. You can save 35-45%, which is a big deal IMHO.

The Practice of Cloud System Administration is 35% off, or 45% off if you buy 2 copies. Buy one for yourself and get a copy for a friend for their birthday. Just use this link to receive the discount.

You can also get a copy of the Cloud Administration book plus the new 3rd Edition of The Practice of System and Network Administration (when it ships in November) and save 45% if you use this link and enter offer code "B2B".

These offers end on Monday. Act now!

(NOTE: I'm not saying our books are great birthday gifts. However think about it this way. What's the worst that could happen? If your friend is offended and never talks to you again at least you'll have more time to read your book. Isn't that what you wanted in the first place?)

Posted by Tom Limoncelli in Book News

15 years ago today (or August 24, 2001 depending on who you talk to) the first edition of The Practice of System and Network Administration reached bookstores.

We had been working on the book for 2+ years, having first met during a Usenix conference in 1999. Writing it was quite an experience, especially since this was before voice-chat on the internet was common, and we were on different continents (Christine in London and Tom in New Jersey). We collaborated via email, used CVS for our source code repository, and we had monthly phone calls (which Tom dialed from work... Thanks, Lucent!). At the time collaborating this way was considered quite radical. Most authors emailed chapters back and forth, and had a hell of a time with merge conflicts. Our publisher was amazed at our ability to collaborate so seamlessly. This kind of collaboration is now commonplace.

The book did quite well. We've sold more than 38,000 copies, in 3 editions (2001, 2007, and 2016), and many printings. It is available on softcover, ebook, and as a web page. It has been translated into Chinese and Russian. In 2005 we received the Usenix SAGE/LISA Outstanding Achievement Award. The 2nd and 3rd editions added an additional co-author, Strata R. Chalup. Strata's experience and project management skills have been a real asset. We've worked with many editors and other production people at Pearson / Addison-Wesley, starting with Karen Gettman who originally recruited us. Thanks Karen, Catherine, Mark, Debra, Kim, Michael, Julie and many others!

We've had a number of surprises along the way. Our favorite was visiting Google (before Tom worked there) and shown a supply closet full of copies of the book. It turned out all new Sysops members were issued a copy. Wow!

Most importantly, we've received a lot of fan mail. Hearing how the book helped people is the biggest joy of all.

In November, the 3rd edition will reach bookstores. We're very excited about the new edition. It has over 300 pages of new material. Dozens of new chapters. It is more modern, better organized, and has a lot of great new stories. You can pre-order the book today. You can read drafts online at SBO. Visit the-sysadmin-book.com for more info. (This is not to be confused with the sequel book, The Practice of Cloud System Administration.)

Thank you to everyone that has purchased a copy of The Practice of System and Network Administration. We really appreciate it!

Posted by Tom Limoncelli in Book News

The last time you joined a new operations/devops/sysadmin team, what make it easy or difficult to get started?

For example... if procedures aren't documented, it can be very difficult to know how to perform them. The rest of the team does them by memory, but you are spending your time asking for help, or guessing your way through them. Well-documented procedures (or even tiny bullet lists) make it easy to be self-sufficient quickly.

What have you found made assimilating into a new team difficult? What have you seen teams do that made it easy? What did you wish a team had done to make it easy?

Please answer in the comments.

Posted by Tom Limoncelli in Survey

Every sysadmin knows that you can protect a server though cryptographic or other means, but if someone has physical access "all bets are off". Right? With physical access they can do physical damage (smash it with a hammer) or pull out a hard disk and read the bits directly. Even security systems that are highly respected (I'm looking at you, Kerberos!) are an "all bets are off"-situation if someone gets the private key through physical access.

Sadly we forget this when it comes to smartphones. We'll plug our phones into any darn USB charger we find... especially when we are desperate! Those Pokemon's ain't gonna catch themselves!

Have we forgotten that our phone is a computer and the USB port gives better access than sitting at a server's console?

This article by Alexey Komarov was a very painful reminder of just how much access a USB port gives to an attacker. USB is a vector for malware and spying. Not just that, but USB is how we upgrade firmware on most phones. The commands to upload and activate new firmware are almost entirely unprotected. Giving USB access is providing unrestricted ability to install new software and firmware. That's crazy!

I was recently reminded of this when I plugged into a USB charging port on an airplane. My iPhone popped up a window asking if I wanted to trust the device. Wait... what?? Why is that power charging port not "charge-only"? Why is it trying to make a data connection to my phone? Oh, it turns out that I could play my iPhone's music though the airplane's audio system. (This, of course, is a stupid feature... the airplane headphones aren't nearly as good as what comes with my phone.)

The airline makes this feature available only in the business/first class cabin. I don't believe in conspiracy theories, but if I was a state that wanted to hijack the phones of important business people, politicians, and government officials, these are the USB ports that I'd be subverting.

So... what can you do?

  1. Use charge-only USB cables. These simply pass on the power wires but not the data wires. They are 100% effective against bad actors. The downside: when you do need data, you need to carry a different set of cables. Available in USB 2.0, iPhone Lightning, and USB C.

  2. Use a USB-condom. This is a device that plugs in between your normal cable and the computer and blocks the data lines. The downside is that you now have a second device to carry around. The upside is that your phone will charge faster! The PortaPow brand has an extra little circuit that tells the device to go into fast-charge mode! I love this feature! (Available for USB 2.0, buy 2 at a time, or built-into a USB 3.0 cable, or buy 3 at a time. In the PortaPow product line, make sure it mentions "PortaPow SmartCharge" otherwise it may be an older model that is specific to Apple or Android but not both.

  3. Use a USB cable with a "data switch". This cable is normally power-only, which is what you want 90% of the time. However there is a button ("Data Transfer Protection On/Off Switch") you can press that will enable data. An LED indicates the mode. This kind of cable is much safer and secure, plus more convenient for the users. It follows the security principle that if you make the defaults what you want users to do, they're more likely to follow your security policy. Available in Micro USB.

I recommend that all IT departments give out USB cables with "Data Transfer Protection On/Off Switch" as their default. Include one with every new laptop or mobile device that you hand out. For a tiny additional cost you get a lot of benefit.

The USB condoms are useful when you need to support a variety of USB connector types or cable lengths, since it requires that you use your own cable. I keep one in my travel bag. I also put a few in the datacenter so that when someone is tempted to charge their phone by plugging it into one of our servers, we can instead hand them a condom. No matter what type of connector is on their phone, they can use the condom because it connects to the USB B port on the server side.

Lastly, these devices make great gifts for the holidays. For the geek that has everything, they probably haven't thought of this!

Notes:

  • PortaPow
  • Monoprice
  • Thanks to Scott Hazen Mueller for alerting me to the Komarov article!

Posted by Tom Limoncelli in Technical Tips

One of the things my team at StackOverflow does is maintain the CI/CD system which builds all the software we use and produce. This includes the Stack Exchange Android App.

Automating the CI/CD workflow for Android apps is a PITA. The process is full of trips and traps. Here are some notes I made recently.

First, [this is the paragraph where I explain why CI/CD is important. But I'm not going to write it because you should know how important it is already. Plus, Google definitely knows already. That is why the need to write this blog post is so frustrating.]

And therefore, there are two important things that vendors should provide that make CI/CD easy for developers:

  • Rule 1: Builds should work from the command line on a multi-user system.
    1. Builds must work from a script, with no UI popping up. A CI system only has stdin/stdout/sterr.
    2. Multiuser systems protect files that shouldn't be modified from being modified.
    3. The build process should not rely on the internet. If it must download a file during the build, then we can't do builds if that resource disappears.
  • Rule 2: The build environment should be reproducible in an automated fashion.
    1. We must be able to create the build environment on a VM, tear it down, and built it back up again. We might do this to create dozens (or hundreds) of build machines, or we might delete the build VM between builds.
    2. This process should not require user interaction.
    3. It should be possible to automate this process, in a language such as Puppet or Chef. The steps should be idempotent.
    4. This process should not rely on any external systems.

Android builds can be done from the command line. Hw, but the process itself updates files in the build area. Creating the build environment simply can not be automated, without repackaging all of the files (something I'm not willing to do).

Here are my notes from creating a CI/CD system using TeamCity (a commercial product comparable to Jenkins) for the StackOverflow mobile developers:

Step 1. Install Java 8

The manual way:

CentOS has no pre-packaged Oracle Java 8 package. Instead, you must download it and install it manually.

Method 1: Download it from the Oracle web site. Pick the latest release, 8uXXX where XXX is a release number. (Be sure to pick "Linux x64" and not "Linux x86").

Method 2: Use the above web site to figure out the URL, then use this code to automate the downloading: (H/T to this SO post)

# cd /root
# wget --no-cookies --no-check-certificate --header \
    "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
    "http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jdk-8u102-linux-x64.rpm"

Dear Oracle: I know you employ more lawyers than engineers, but FFS please just make it possible to download that package with a simple curl or wget. Oh, and the fact that the certificate is invalid means that if this did come to a lawsuit, people would just claim that a MITM attack forged their agreement to the licence.

Install the package:

# yum localinstall jdk-8u102-linux-x64.rpm

...and make a symlink so that our CI system can specify JAVA8_HOME=/usr/java and not have to update every individual configuration.

# ln -sf /usr/java/jdk1.8.0_102 /usr/java/jdk

We could add this package to our YUM repo, but the benfit would be negligible plus whether or not the license permits this is questionable.

EVALUATION: This step violates Rule 2 above because the download process is manual. It would be better if Oracle provided a YUM repo. In the future I'll probably put it in our local YUM repo. I'm sure Oracle won't mind.

Step 2. Party like it's 2010.

The Android tools are compiled for 32-bit Linux. I'm not sure why. I presume it is because they want to be friendly to the few developers out there that still do their development on 32-bit Linux systems.

However, I have a few other theories: (a) The Android team has developed a time machine that lets them travel back to 2010 because I happen to know for a fact that Google moved to 64-bit Linux internally around 2011; they created teams of people to find and eliminate any 32-bit Linux hosts. Therefore the only way the Android team could actually still be developing on 32-bit Linux is if they either hidden their machines from their employer, or they have a time machine. (b) There is no "b". I can't imagine any other reason, and I'm jealous of their time machine.

Therefore, we install some 32-bit libraries to gain backwards compatibility. We do this and pray that the other builds happening on this host won't get confused. Sigh. (This is one area where containers would be very useful.)

# yum install -y glibc.i686 zlib.i686 libstdc++.i686

EVALUATION: B-. Android should provide 64-bit binaries.

Step 3. Install the Android SDK

The SDK has a comand-line installer. The URL is obscured, making it difficult to automate this download. However you can find the current URL by reading this web page, then clicking on "Download Options", and then selecting Linux. The last time we did the the URL was: https://dl.google.com/android/android-sdk_r24.4.1-linux.tgz

You can install this in 1 line:

cd /usr/java && tar xzpvf /path/to/android-sdk_r24.4.1-linux.tgz

EVALUATION: Violates Rule 2 because it is not in a format that can easily be automated. It would be better to have this in a YUM repo. In the future I'll probably put this tarfile into an RPM with an install script that untars the file.

Step 4. Install/update the SDK modules.

Props to the Android SDK team for making an installer that works from the command line. Sadly it is difficult to figure out which modules should be installed. Once you know the modules you need, specifying them on the command line is "fun"... which is my polite way of saying "ugly."

First I asked the developers which modules they need installed. They gave me a list, which was wrong. It wasn't their fault. There's no history of what got installed. There's no command that shows what is installed. So there was a lot of guess-work and back-and-forth. However, we finally figured out which modules were needed.

The command to list all modules is:

/usr/java/android-sdk/tools/android list sdk -a

The modules we happened to need are:

  1- Android SDK Tools, revision 25.1.7
  3- Android SDK Platform-tools, revision 24.0.1
  4- Android SDK Build-tools, revision 24.0.1
  6- Android SDK Build-tools, revision 23.0.3
  7- Android SDK Build-tools, revision 23.0.2
  9- Android SDK Build-tools, revision 23 (Obsolete)
 19- Android SDK Build-tools, revision 19.1
 29- SDK Platform Android 7.0, API 24, revision 2
 30- SDK Platform Android 6.0, API 23, revision 3
 39- SDK Platform Android 4.0, API 14, revision 4
141- Android Support Repository, revision 36
142- Android Support Library, revision 23.2.1 (Obsolete)
149- Google Repository, revision 32

If that list looks like it includes a lot of redundant items, you are right. I don't know why we need 5 versions of the build tools (one which is marked "obsolete") and 3 version of the SDK. However I do know that if I remove any of those, our builds break.

You can install these with this command:

/usr/java/android-sdk/tools/android update sdk \
    --no-ui --all --filter 1,3,4,6,7,9,19,29,30,39,141,142,149

However there's a small problem with this. Those numbers might be different as new packages are added and removed from the repository.

Luckily there is a "name" for each module that (I hope) doesn't change. However the names aren't shown unless you specify the -e option:

# /usr/java/android-sdk/tools/android list sdk -a -e

The output looks like:

Packages available for installation or update: 154
----------
id: 1 or "tools"
     Type: Tool
     Desc: Android SDK Tools, revision 25.1.7
----------
id: 2 or "tools-preview"
     Type: Tool
     Desc: Android SDK Tools, revision 25.2.2 rc1
 ...
 ...

Therefore a command that will always install that set of modules would be:

/usr/java/android-sdk/tools/android update sdk --no-ui --all \
    --filter tools,platform-tools,build-tools-24.0.1,\
    build-tools-23.0.3,build-tools-23.0.2,build-tools-23.0.0,\
    build-tools-19.1.0,android-24,android-23,android-14,
    extra-android-m2repository,extra-android-support,\
    extra-google-m2repository

Feature request: The name assigned to each module should be listed in the regular listing (without the -e) or the normal listing should end with a note: "For details, add the -e flag."

EVALUATION: Great! (a) Thank you for the command-line tool. The docs could be a little bit better (I had to figure out the -e trick) but I got this to work. (b) Sadly, I can't automate this with Puppet/Chef because they have no way of knowing if a module is already installed, therefore I can't make an idempotent installer. Without that, the automation would blindly re-install the modules every time it runs, which is usually twice an hour. (c) I'd rather have these individual modules packaged as RPMs so I could just install the ones I need. (d) I'd appreciate a way to list which modules are installed. (e) update should not re-install modules that are already installed, unless a --force flag is given. What are we, barbarians?

Step 4: Install license agreements

The software won't run unless you've agreed to the license. According to Android's own website you do this by asking a developer to do it on their machine, then copy those files to the CI server. Yes. I laughed too.

EVALUATION: There's no way to automate this. In the future I will probably make a package out of these files so that we can install them on any CI machine. I'm taking suggestions on what I should call this package. I think android-sdk-lie-about-license-agreements.rpm might be a good name.

Step 5: Fix stupidity.

At this point we though we were done, but the app build process was still breaking. Sigh. I'll save you the long story, but basically we discovered that the build tools want to be able to write to /usr/java/android-sdk/extras

It isn't clear if they need to be able to create files in that directory or write within the subdirectories. Fuck it. I don't have time for this shit. I just did:

chmod 0775 /usr/java/android-sdk/extras
chown $BUILDUSER /usr/java/android-sdk
chown -R $BUILDUSER /usr/java/android-sdk/extras

("$BUILDUSER" is the username that does the compiles. In our case it is teamcity because we use TeamCity.)

Maybe I'll use my copious spare time some day to figure out if the -R is needed. I mean... what sysadmin doesn't have tons of spare time to do science experiments like that? We're all just sitting around with nothing to do, right? In the meanwhile, -R works so I'm not touching it.

EVALUATION: OMG. Please fix this, Android folks! Builds should not modify themselves! At least document what needs to be writable!

Step 6: All done!

At this point the CI system started working.

Some of the steps I automated via Puppet, the rest I documented in a wiki page. In the future when we build additional CI hosts Puppet will do the easy stuff and we'll manually do the rest.

I don't like having manual steps but at our scale that is sufficient. At least the process is repeatable now. If I had to build dozens of machines, I'd wrap all of this up into RPMs and deploy them. However then the next time Android produces a new release, I'd have to do a lot of work wrapping the new files in an RPM, testing them, and so on. That's enough effort that it should be in a CI system. If you find that you need a CI system to build the CI system, you know your tools weren't designed with automation in mind.

Hopefully this blog post will help others going through this process.

If I have missed steps, or if I've missed ways of simplifying this, please post in the comments!

P.S. Dear Android team: I love you folks. I think Android is awesome and I love that you name your releases after desserts (though I was disappointed that "L" wasn't Limoncello.... but that's just me being selfish.). I hope you take my snark in good humor. I am a sysadmin that wants to support his developers as best he can and fixing this problems with the Android SDK would really help. Then we can make the most awesome Android apps ever.... which is what we all want. Thanks!

Posted by Tom Limoncelli in DevOpsTechnical Tips

I'll be giving a new talk called "Teaching DevOps to Ops without Devs" at the NYC SRE Tech Talks meetup, Wednesday, August 17, 2016, from 6pm to 8:30pm, at Squarespace's offices in the West Village (8 Clarkson St, 12th Floor). There are 4 speakers during the event. This is a free event. Seating is limited. Info and registration here.

Posted by Tom Limoncelli in Speaking

I'm reposting this survey. The researcher is trying to identify what makes configuration languages difficult to use. If you use Puppet/Chef/CfEngine/Ansible (and more importantly... if you DON'T use any configuration languages) please take this survey. I took it in under 10 minutes.

Tom


I would like to invite you to take a survey on configuration languages. The survey is about 10 minutes long and does not require any specific skills or knowledge, anyone can participate in it.

https://edinburgh.onlinesurveys.ac.uk/configlang

The survey is a part of ongoing research at University of Edinburgh School of Informatics on configuration languages. As Paul Anderson has mentioned previously, we are interested in studying the usability of configuration languages and how it relates to the configuration errors.

The survey will be open through the 2nd of August, 2016. It is possible to take survey in parts by choosing to finish it later, but it cannot be completed past the deadline.

Please feel free to forward this survey to others.

If you have any questions or concerns, contact information is linked on the first page of the survey.

Your answers to the survey will be used in aggregate to improve our understanding of how configuration languages are understood and used.

Thank you
Adele Mikoliunaite
Masters student at
University of Edinburgh

Posted by Tom Limoncelli in Guest Post

Have you seen/read The Martian?

What's so sad about the movie/book is that it is a reminder of what could have been.

Part of the premise is that after the Apollo program, the U.S. continued their plans for landing on Mars. Such plans were dropped for the less ambitious Shuttle program.

Think about it. In most science fiction the science is unbelievable. In The Martian, the science was pretty darn accurate and the unbelievable part is that U.S. politicians had the audacity to continue NASA's funding level.

Posted by Tom Limoncelli in Rants

Episode 11 of LISA Conversations is Russell Pavlicek, who presented Unleashing the Power of the Unikernel at LISA '15.

You won't want to miss this!

Posted by Tom Limoncelli in LISA Conversations

 
  • Register Today
  • LISA16