Systems: Because You Can’t Count That Fast


Me trying to explain a global SAP ERP network to kids.

On Thursday, it was bring your daughters and sons to work day at Genentech. My little Marlowe is only 6 weeks old so I didn’t bring her, but my VP shot me a note Wednesday saying she was bringing her two boys, knew of some other kids who would be there, and wondered if I could give a 30 minute talk on systems. I responded, “sure.”

But why the heck do you tell kids ranging from 6-12 years old about your corporate ERP system? While fascinating, I doubt they’d care about the usual things that I work on and I wouldn’t dare show them PowerPoint slides. So what to do?
Continue reading

Cleanup Unused Linux Kernels in Ubuntu

I update Ubuntu with a very simple script I call apt-update that looks like this:

$ cat ./apt-update 
sudo apt-get update; sudo apt-get dist-upgrade; sudo apt-get autoremove

Nothing too crazy there. It updates the apt-get cache, performs the upgrade, and then removes all the residual junk that’s laying around. Well, almost all. If you do this enough, eventually you’ll see the following (assuming you’ve got the default motd Ubuntu script running and you’re logging in from a terminal):

=> /boot is using 86.3% of 227MB

This is because that script I mentioned doesn’t consider old kernel images to be junk. However, unless you’ve got an abnormal /boot partition, it doesn’t take too many old images to fill it up.

A quick Google search found Ubuntu Cleanup: How to Remove All Unused Linux Kernel Headers, Images and Modules. The solution on the page had exactly what I’m looking for, however, I couldn’t take it at face value. While the article offers an adequate solution, it doesn’t offer much explanation. The remainder of this article explains the details for this one-liner noted in the article above:

$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

Note: Only run this if you’ve rebooted after installing a new kernel.

Ick. Let’s dig into what’s going on here. The pipe characters are chaining a bunch of commands together. Each command’s output becomes the input for the next. Given that, let’s walk through what’s going on in 3 steps.
Continue reading

Controlling Movies in QuickTime Player X (10.1) With AppleScript

The following works for Quicktime Player X (10.1) in OS X 10.7 (Lion). It takes arguments from the command line and opens a movie, resizes it, sets the volume, and starts playing the movie (there are other properties you could add). Hopefully this is useful for someone like me who was trying to find examples of this, but only finding mounds of info for QuickTime Player 7 (and earlier), which doesn’t help much as many of the properties have changed.

-- AppleScript to open and control movie files in QuickTime Player X

-- The script is meant to run at the command line, like so:
-- $ osascript /path/to/movie.mp4 movie.mp4 800 60
-- If you don't want to use a shell, just remove this and 
-- the last line and manually populate the first 4 variables
on run argv
	
	set movieFile to item 1 of argv -- the full path with movie filename
	set movieName to item 2 of argv -- the file name only of the movie
	set desiredWidth to item 3 of argv -- the width of the movie on the screen
	set startOffset to item 4 of argv -- how many seconds into the movie to start
	
	-- this finds the right side of the screen, then offsets to the left
	-- the desired width of the movie
	tell application "Finder"
		set desktopBounds to bounds of window of desktop
	end tell
	set rightX to item 3 of desktopBounds
	set leftX to (rightX - desiredWidth)
	
	tell application "QuickTime Player"
		
		-- open the movie and bring it to the forefront (if other movies are open)
		open movieFile
		set lastOpenedWindow to (first window whose name contains movieName)
		set visible of lastOpenedWindow to true
		
		-- resize the movie using its original aspect ratio
		set movieBounds to bounds of lastOpenedWindow
		set widthOrigin to leftX
		set heightOrigin to 1
		set originalWidth to ((item 3 of movieBounds) - (item 1 of movieBounds))
		set originalHeight to ((item 4 of movieBounds) - (item 2 of movieBounds))
		set calculatedHeight to (round ((desiredWidth / originalWidth) * originalHeight) rounding down)
		set desiredWidth to (desiredWidth + widthOrigin)
		set calculatedHeight to (calculatedHeight + heightOrigin)
		set the bounds of lastOpenedWindow to {widthOrigin, heightOrigin, desiredWidth, calculatedHeight}
		
		-- if opening multiple movies, ensure they don't overlap
		set slideDown to 1
		repeat with aWindow in (get every window)
			set boundsOfAWindow to bounds of aWindow
			set heightOfAWindow to item 4 of boundsOfAWindow
			if (heightOfAWindow > slideDown) then
				if (name of aWindow does not contain movieName) then
					set slideDown to (heightOfAWindow + 1)
				end if
			end if
		end repeat
		set the bounds of lastOpenedWindow to {widthOrigin, (heightOrigin + slideDown), desiredWidth, (calculatedHeight + slideDown)}
		
		-- note, here we select a document, not a window
		set myMovie to document 1
		tell myMovie
			set audio volume to 0
			set current time to startOffset
			play
			activate -- makes QuickTime Player the frontmost application
		end tell
		
	end tell
end run

Ruby Script to Search Apache Logs for High Frequency Clients

I wrote a quick Ruby script to scour through my Apache access logs and look for IPs that are hitting my site too frequently, e.g., bad bots, etc. The command line arguments are simple:

$ ruby find-frequent-clients.rb \
--apache-access-log=/path/to/your/log \
--seconds=3600 \
--request-limit=7200 \
--log-time-zone=PST

That command is going to find any client IPs that are hitting my web server in the last 10 minutes more twice or more per second. The output will be a line separated list of IP addressess (optionally with a hit count if --show-count=1 is added). Here’s how it works:

File: find-frequent-clients.rb
require 'date'
require 'time'
# Process command line arguments.  Filter only args starting with --
args = {}
$*.each do |arg|
  spl=arg.split("=")
  if spl[0][0..1] == "--"
    args[spl[0][2..spl[0].length-1].gsub("-","_").intern]=spl[1]
  end
end

# Check that we have the bare essentials to proceed
raise "You must specify the full path to an Apache access log file with --apache-access-log" unless args[:apache_access_log]
raise "You must specify the maximum amount of recent seconds to consider with --seconds" unless args[:seconds]
raise "You must specify the maximum requests allowed per #{args[:seconds]} seconds with --request-limit" unless args[:request_limit]
raise "You must specify the time zone of the Apache logs with --log-time-zone e.g., EST" unless args[:log_time_zone]
raise "The Apache access log file specified does not exist or is not readable: #{args[:apache_access_log]}" unless FileTest.readable?(args[:apache_access_log])

# Open the file and read the lines in reverse; exit once time stamps are beyond our time threshold
file = File.open(args[:apache_access_log], "r")
log_array = []
log_snapshot = file.readlines
file.close
start_time = Time.now.to_i
log_snapshot.reverse_each do |line|
  line_array = line.split(" ")
  date_time = line_array[3][1..line_array[3].length-1]
  date_time[11] = " "
  date_time = Time.iso8601(DateTime.parse(date_time+" "+args[:log_time_zone]).to_s).to_i
  if date_time > (start_time - args[:seconds].to_i)
    log_array << [line_array[0], date_time]
  else
    break
  end
end

# Use a hash to collect the counts of the IPs
log_hash = Hash.new(0)
log_array.each do |log|
  log_hash[log[0]]+=1
end

# collect the offenders in an array
offenders = log_hash.to_a.collect{|h| h if h[1] > args[:request_limit].to_i}.compact

# output the offending IPs, 1 per line; optionally show the offending count
offenders.each{|o| puts o[0].to_s+"#{" => "+o[1].to_s if args[:show_count]}"}

Note: This makes the assumption that your logs are in the format: aa.bb.cc.dd – - [datetime]