hross.net

The best thing about Netflix is the instant queue and its constant ability to surprise me. I end up watching movies and documentaries about things I would normally never see, simply because I wouldn’t know they exist. One such example is the documentary Helvetica.

I am sure there are many consultants, programmers and others in the IT world who, like me, end up wearing the graphic designer hat from time to time. It may be to create icons or images for a web site, or it may be to edit CSS styles.

If you are one of these types of people, and you have even a passing interest in design, Helvetica is for you. It takes you down the rabbit hole of typeface design, various artistic movements, and the “why’s” and “how’s” of what make a good typeface. This is one of those movies that could actually make you better at your job and provide some interesting historical context around something we take for granted when we design web sites.

It will also make you look at normal everyday signage in a totally different light.

In one of my previous posts I mentioned Apache Directory Studio as a great way to view LDAP directories, but the entire Apache Directory project really deserves its own post (and here it is).

First of all, Apache has implemented a fully featured directory browser (Apache Directory Studio) and a fully featured LDAP directory (Apache Directory Server). Both of these projects are entirely Java based. Studio is an eclipse based directory browser with all the bells and whistles. JXplorer is nice, but not as powerful or as easy to use.

However, the real power of ApacheDS is the server. It’s entirely java based and is available for a variety of platforms, including Windows. I have yet to find an easier platform to install and use on a variety of operating systems (especially Windows). Rather than trying to build a confusing OpenLDAP implementation, you can simply download, install, and start ApacheDS in 5 to 10 minutes.

And, oh by the way, you can also embed and manipulate ApacheDS in your own applications, since its written entirely in Java and the source code is freely available.

So, if you are looking for an easy, free, directory implementation for your next proof of concept, demo, or unit test, look no further than ApacheDS.

I am not sure this is even enough for a blog post, but… well it is. Here are just a few the of cool things you can do with Google Translate:

1. Use the new audio capability to generate audio translations of text, automatically.
2. Use the translate API to translate your entire web page by simply including some javascript.
3. Programmatically translate stuff using the unofficial Java API. There is probably one for .NET, but I haven’t tried it.
4. Translate bits and pieces of your web site, on demand, using a very cool jQuery plugin called Sunday Morning.

Recently, I was in the midst of a Windows 7 installation when my company decided to migrate my email to a new mail server. As we in the IT world are aware, migrations rarely go as planned. It seemed like as good a time as any for me to start a project I have long considered: migrating all of my email to Gmail.

I guess this is technically something I’m not supposed to do. Then again, it is no less secure than downloading and sending email using my local laptop and a standard email client (provided the passwords/accounts are properly encrypted). Either way, I love Gmail for personal email and there is no way my entire work organization is going to switch to Gmail, so I decided to set up my own little synchronization process.

Here is what I did:

1.) Enable IMAP on my Gmail account. My work email is already IMAP, so this let me drag and drop folders from one mail account to another using Thunderbird. Once all my folders were migrated, I only had to worry about new email in my inbox.

2.) Set up a synchronization process from my work email to my Gmail account. Transferring mail itself is pretty simple. There is an RFC that defines what mail messages look like, so they are the same data from one mail account to the other. The trick is moving them automatically.

Since I already have a Linux host that runs full time (this site), it seemed like my most sane option would be to write something that I could schedule using cron. Since I am a member of the cargo cult, I thought I could pretty easily find something on Google written in Java.

After some searching, though, it seemed like the best and simplest examples were all written in Ruby. Unfortunately, none of them did exactly what I wanted so I figured I would have to write a bit of code. Before I began this endeavor, I knew nothing about Ruby (yes, I am way behind), but it seemed like a good time to learn.

I started off with some setup:

http://wiki.dreamhost.com/Ruby

http://wiki.dreamhost.com/index.php/Ruby_on_Rails

Next I went with a few blogs/docs and some source from the beginnings of larch:

http://ruby-doc.org/stdlib/libdoc/net/imap/rdoc/classes/Net/IMAP.html

http://wonko.com/post/ruby_script_to_sync_email_from_any_imap_server_to_gmail

http://codeclimber.blogspot.com/2008/06/using-ruby-for-imap-with-gmail.html

While I like larch, it doesn’t delete from my inbox, has way more features than I need, and it is more object oriented (and thus harder to understand) than I would like. Since I am a Ruby novice, I wanted something simple that I could make sure was working. Here is what I came up with:

#! ~/run/bin/ruby
require 'net/imap'

puts "Synchronizing mailboxes..."

# create destination imap
dest = Net::IMAP.new("imap.gmail.com",993,true)
dest.login("my.account@gmail.com", "password")

# create source imap
source = Net::IMAP.new("imap.work.com",993,true)
source.login("my.account@work.com", "password")

puts "Logins complete. Checking source mailbox for mail."

source.select('INBOX')
source.search(["NOT", "DELETED"]).each do |message_id|
        puts "Found message: #{message_id}"
        msg = source.fetch(message_id, ['RFC822', 'FLAGS',
                  'INTERNALDATE'])[0]

        puts "Transferring message with id: #{message_id}"
        dest.append('INBOX', msg.attr['RFC822'], msg.attr['FLAGS'],
            msg.attr['INTERNALDATE'])

        puts "Deleting message from source inbox."
        source.store(message_id, "+FLAGS", [:Deleted])
end
puts "Transfer complete. Logging out."

source.logout()
source.disconnect()

dest.logout()
dest.disconnect()

puts "done."

Simple, huh? I set this up to run as a cron job every ten minutes and that’s all it took.

It came out just in time for my birthday this year: Scott Hanselman’s 2009 Windows Power Tools list. It is indeed a great list. If you are a developer working in Windows and have a few minutes to spare, you might just find something on that list that makes your life a million times easier.

As some of you may know, I recently took a new job within Oracle, and as such I had the privilege of getting a brand new laptop for my birthday (okay, I happened to start the job the same week as my birthday, but it *felt* like I got a birthday present). There is nothing quite like a fancy new laptop and a list of awesome tools and utilities to install on it.

Anyway, since I’ve finally got it perfectly configured (never to be this fast or nice again), and since searching through the 2009 list is a bit of a grind, I thought I’d share my own, much smaller, list of tools I love (I’ll try to keep this not-so-development-oriented):

PS Hot Launch – I hate the start menu, and the quick start tray is never big enough for all my icons. Enter PS Hot Launch, where I can now keep all my frequently used programs and bind hotkeys for startup. There are a lot of heavy duty hotkey managers and the like, but for my money (or lack thereof) this is the easiest, best, and most free version.

Console – I never use a dos prompt anymore. This one looks way cooler and supports more features (like easy copy and paste).

GNU Win32 – One of my biggest peeves about Windows is not being able to do find . | xargs grep “blah” and get results. I sometimes use cygwin, but this is much less overhead and more native. Thank you GNU.

Microsoft TimeZone – This is a weird one, but I find myself constantly having to check what time it is on the west coast, central, etc.. I am never sure how many hours to subtract. This super simple free utility runs in your tray and lets you customize up to five locales to show current times for when you click on it. A lot easier than googling “current time pacific”.

Google Desktop – When it first came out, I thought “will I really use this?” Now I can’t go a day without it. How many times have you thought about an email you sent two months ago but could only remember various key words? Or an old coding project that has something specific you did that you now can’t remember? I can’t even count the number of hours this has saved me in “hunting for stuff” time. Seriously… if you don’t have this you don’t even know what you’re missing.

Textpad – My favorite of the “enhanced text editors” crowd. It has a vast array of pluggable syntax highlighting (no more Eclipse to edit one line of Java code), explorer shell integration, and an intuitive interface without a ton of annoying bells and whistles.

Postbox – A lightweight email client. If it had better calendar integration I’d give it a gold star. Still a great quick and dirty client that has some nice search capability.

PuTTY – Remember when people made applications that fit in a single executable and just did the job? Yeah… I do too. If you’re not using PuTTY I have no idea why not.

Xming – Perhaps a limited audience on this one, but if you need to use X in Windows, this one is for you.

SQLDeveloper – Wait… you mean to tell me Oracle made a lightweight, user friendly, super powerful database tool? And I don’t have to use sqlplus or Toad anymore? And it’s free?! Say no more…

Apache Directory Studio/Server – I absolutely love these two. I used to use Softerra LDAP Browser, which is a great tool in its own right, but I cannot tell you how happy I am to see an easy to install, easy to configure LDAP directory for Windows. I can now test LDAP integrations with impunity! Documentation is a bit sparse, but the first time I downloaded and installed this sucker I got all warm and fuzzy inside.

WinMerge – I blogged about this earlier and it is great for file and directory comparisons.

PDF Split and Merge – This is a great tool if you find yourself having to create expense reports or combine PDF’s of scanned documents. The name says it all.

Darkroom – I tend to take a lot of notes in text editors and this one just looks awesome. You may not use it all the time, but it definitely gets an A+ on style points.

Here are a few more that are pretty common so I won’t write blurbs about them: Pidgin, ImgBurn, PaintDotNet, LiveWriter, and Wireshark. And of course there are the myriad plugins you can get for Firefox, but they deserve a separate post.

Alright... happy downloading!

A while back I wrote this post detailing my struggle to find a quick and dirty way of displaying paginated table data in the portal (or anywhere, for that matter). I ended up settling on the method I found at Spartan Java.

While I enjoyed the series of articles, I still ended up having to review a jQuery primer, the DWR documentation and the iBATIS user guide. I would have liked to have been able to download the finished application. On top of that there were the usual struggles with portletizing the code, and scrambling to jam a bunch of features into the finished web application before launch.

Thus, when I got asked to provide some “development best practices” portal code, I was in a bit of a bind. I wanted to show off the WCI portal’s ability to consume and use a variety of frameworks, but my original source was… pretty crappy. In the end I went back and rewrote a very simple sample application from scratch (including documentation) using the methods described above.

Hopefully I can save you some time and effort in similar endeavors. Without further ado here is a link to a sample database search application using the Oracle HR sample data.

It’s nothing fancy (yes, there are even possible SQL injection vulnerabilities), but if you want a simple example of jQuery, iBATIS, Java and the portal, this is definitely something to check out. All you should have to do is import the war file into eclipse and read the index.html documentation.

Lately I have found myself setting up a lot of Amazon EC2 instances, new computers, virtual machines and the like. As such, I’ve come to know and love Oracle XE. While it “just works”, there are a few tweaks that vastly improve performance and behavior. All of these tweaks require you to log in as the system user and run the noted SQL. In no particular order, they are:

1. Modifying the default listening port of the web server (why is it listening on the default Tomcat port?!):
   

   begin
   	dbms_xdb.sethttpport('7080');
   	dbms_xdb.setftpport('2100');
   end;

2. Increasing the number of sessions and processes so you don’t get locked out of the database:
   

   alter system set sessions=250 scope=spfile;
   alter system set processes=200 scope=spfile;

3. Making the web server available for non-local access (in case you are running out of a console):

   begin 
   	dbms_xdb.setlistnerlocalaccess(false); 
   end;

Here are the original links for these tips:

• Mark C. Allen
• daust_de
• blog.gralike.com

Finally, I have a tip of my own. If you are running Oracle XE in an EC2 instance (see this article), you will undoubtedly notice that when you restart your brand new AMI with a new IP, Oracle will fail to start (doh!). In order to fix this, you need to do the following:

1. Make copies of your listener.ora and tnsnames.ora files. Modify them so that your current hostname in EC2 is replaced with “localhost”, and rename them to listener.ora.localhost and tnsnames.ora.localhost.
2. Add this script to the /etc/init.d/oracle-xe startup script (under start is preferable):

   NEWHOST=`hostname`
   
   sed s/localhost/$NEWHOST/ /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/admin/listener.ora.localhost > /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/admin/listener.ora
   sed s/localhost/$NEWHOST/ /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/admin/tnsnames.ora.localhost > /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/admin/tnsnames.ora

3. Profit (you may want to modify the above to run as the oracle user -- not as root).

(sorry Windows users, you will have to create your own variation of this)

As I mentioned in my previous post, I wanted to capture some real world info on hosting a game server in the cloud. The results were a rousing success. We had 5 or 6 people connected at various times, played some Deathmatch and Capture the Flag, and everyone had a ping of 40 or less the entire time. I didn’t notice any latency whatsoever and there were absolutely no packet loss or lag complaints throughout.

Cost

I haven’t broken down the numbers yet, but all told I started up an EC2 instance and hosted a game for 2 hours. I also attached an elastic IP for ease of use. That cost me less than $0.50. I’d say that’s a pretty good deal.

Usage

Below are the usage stats for network I/O and CPU usage. I gathered these using my simple Java application and created these no-frills charts in Microsoft Excel (all told, this took about 5 minutes to put together):

Figure 1 – Network I/O over a 2 hour F.E.A.R. game.

Figure 2 – CPU Usage over a 2 hour F.E.A.R. game.

Conclusion

This is a short and imperfect analysis, but overall I’d say the “small” EC2 instance could easily have handled a 16 person game, both from a load and network traffic standpoint, and it would have cost me a dollar or so to host for 2 hours. That seems like great bang for your buck if you’re looking to crank up a quick game and then move on to something else.

I have recently been working on a utility for porting ALUI databases from a production environment to a development environment. Fabien Sanglier started this effort, and I hope to have some code to contribute to his ALUI toolbox project very soon.

In the meantime, however, I have been banging my head against the pain that is migrating Publish and Preview target URL’s in Publisher. These URL’s are stored in a binary BLOB in the Publisher database, and are actually serialized Java classes, making them extremely difficult to update (especially when you don’t have access to the original Publisher source code).

My original plan was to wrap all of this stuff into one “uber-utility” and then blog about it. Recently, though, I saw this post on the Oracle Webcenter Interaction discussion forums: http://forums.oracle.com/forums/thread.jspa?threadID=900736&tstart=0 and it made me think I should probably post the code for migrating Publishing Targets, for the benefit of the sanity of the community at large.

Here is a link to a jar file which will update Publisher publish targets. If you crack the jar file with a zip editor, you will be able to update the configuration.properties file in the root directory to suit your needs.

I took the liberty of including the Publisher classes in my own jar, making it simpler to run from a command line. To run it, you will only need to download the correct jdbc driver for your database:

Oracle JDBC Driver

SQL Server JDBC Driver

Next, simply execute it from a java command line with the driver in your classpath, like so:

java -cp updatepublishtargets.jar;ojdbc14.jar net.hross.content.UpdatePublishTargets

Note that the utility is in debug mode by default, so nothing will happen to your Publisher database until you set debug to false in the configuration, although now is probably a good time to let you know that I provide no warranties of any kind with this code.

In order to build and run the source, you will need the content.jar and dom4j.jar found in the WEB-INF/lib directory of your ptcs.war. Here is the relevant source code, in case you are looking to build your own version of the utility (source is also in the jar):

package net.hross.content;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;

import net.hross.utility.Configuration;

import com.plumtree.content.data.AttributeKey;
import com.plumtree.content.data.impl.RdbiPublishingTarget;

public class UpdatePublishTargets {

    public static void main(String[] args) {
        Connection connection = Configuration.getConnection();

        if (null == connection) {
            System.out.println("Unable to connect to database. Exiting.");
        }

        int directoryId = Integer.parseInt(Configuration
                .getString(Configuration.CONFIG_DIRECTORY_ID));
        boolean debug = Boolean.parseBoolean(Configuration
                .getString(Configuration.CONFIG_DEBUG_MODE));
        String newPublishTarget = Configuration
                .getString(Configuration.CONFIG_PUBLISH_TARGET);

        System.out.println("Updating publish targets for directory ID: "
                + directoryId);

        System.out.println();
        if (debug) {
            System.out.println("** DEBUG MODE ON ** Nothing will be updated.");
        } else {
            System.out.println("** DEBUG MODE OFF ** This is happening for real.");
        }
        System.out.println();

        updatePublishTarget(connection, directoryId, newPublishTarget, debug);
    }

    /***
     * Update the publishing target for a specified directory ID (-1 for all
     * items).
     * 
     * @param connection
     *            Publisher database connection.
     * @param directoryId
     *            Directory ID to update. -1 for all items.
     * @param newPublishTarget
     *            - New publishing target.
     * @param debug
     *            - if true, no replace will be made, data will just be output.
     */
    public static void updatePublishTarget(Connection connection,
            int directoryId, String newPublishTarget, boolean debug) {

        // create a statement to query the directory id
        try {

            // create prepared statement for directory query
            PreparedStatement psDirectory = null;
            if (directoryId > 0) {
                psDirectory = connection
                        .prepareStatement("SELECT * FROM PCSDIRECTORY WHERE ITEMTYPE=0 AND DIRECTORYID=?");
                psDirectory.setInt(1, directoryId);
            } else {
                psDirectory = connection
                        .prepareStatement("SELECT * FROM PCSDIRECTORY WHERE ITEMTYPE=0");
            }
            ResultSet rs = psDirectory.executeQuery();

            // loop through any rows we need to check
            while (rs.next()) {

                // get basic info about the object
                String itemName = rs.getString("ITEMNAME");
                int size = rs.getInt("DATASIZE");
                
                // reset directory ID in case it was generic
                directoryId = rs.getInt("DIRECTORYID");

                // get binary input stream
                InputStream input = rs.getBinaryStream("DATABYTES");

                // if there's actually some settings, let's check them
                if ((null != input) && (0 != size)) {

                    // generic catch statement for problems with this item
                    try {
                        byte[] buffer = new byte[size];
                        input.read(buffer);

                        // load the hash map from the database
                        Map map = (HashMap) deserialize(buffer);

                        // loop through the keys in the hash map
                        Iterator keys = map.keySet().iterator();
                        while (keys.hasNext()) {
                            Object key = keys.next();

                            // this should probably always be true
                            if (key.getClass().equals(AttributeKey.class)) {
                                AttributeKey akey = (AttributeKey) key;

                                // if we found a publishing target...
                                if (akey.getKeyString().equals(
                                        "PUBLISHING_TARGET")) {
                                    System.out.println();
                                    System.out.println("--------------------");
                                    System.out
                                            .println("Updating publishing target for:");
                                    System.out.println(directoryId + " - "
                                            + itemName);

                                    // get the publishing target info
                                    RdbiPublishingTarget val = (RdbiPublishingTarget) map
                                            .get(key);
                                    String publishTarget = val
                                            .getPublishDetail()
                                            .getTargetLocation();
                                    String publishBrowser = val
                                            .getPublishDetail()
                                            .getBrowserLocation();
                                    String previewTarget = val
                                            .getPreviewDetail()
                                            .getTargetLocation();
                                    String previewBrowser = val
                                            .getPreviewDetail()
                                            .getBrowserLocation();
                                    String ftpUser = val.getPublishDetail()
                                            .getUsername();
                                    String ftpPassword = val.getPublishDetail()
                                            .getPassword();

                                    System.out
                                            .println("Publish  browser location: "
                                                    + publishBrowser);
                                    System.out.println("Preview target: "
                                            + previewTarget);
                                    System.out
                                            .println("Preview browser location: "
                                                    + previewBrowser);
                                    System.out.println("FTP user: " + ftpUser);
                                    System.out.println("FTP password: "
                                            + ftpPassword);
                                    System.out.println("Old publish target: "
                                            + publishTarget);
                                    System.out.println("New publish target: "
                                            + newPublishTarget);

                                    // if we are doing this for real, update
                                    // values
                                    if (!debug) {
                                        val.setTargetValues(newPublishTarget,
                                                publishBrowser, previewTarget,
                                                previewBrowser, ftpUser,
                                                ftpPassword);

                                        map.put(key, val);

                                        // update the directory
                                        serializeToDirectory(connection,
                                                directoryId, map);
                                        System.out.println("Update successful.");
                                    }
                                    System.out.println("--------------------");
                                    System.out.println();
                                }
                            }
                        }

                        // clean up
                        input.close();
                    } catch (IOException ex) {
                        System.out.println("Something bad happened.");
                        ex.printStackTrace();
                    }
                } // if null
            } // while next rs
        } catch (SQLException ex) {
            System.out.println("Something bad happened.");
            ex.printStackTrace();
        }

        System.out.println("Procedure successfully completed.");
    }

    private static Object deserialize(byte bytes[]) {
        try {
            ByteArrayInputStream byteStream = new ByteArrayInputStream(bytes);
            ObjectInputStream objectStream = new ObjectInputStream(byteStream);
            return objectStream.readObject();
        } catch (Exception ex) {
            return null;
        }
    }

    private static void serializeToDirectory(Connection conn, int directoryId,
            Object obj) throws IOException, SQLException {
        byte bytes[] = getBytes(obj);
        ByteArrayInputStream byteStream = new ByteArrayInputStream(bytes);

        PreparedStatement ps = conn
                .prepareStatement("UPDATE PCSDIRECTORY SET DATASIZE=?, DATABYTES=? WHERE DIRECTORYID=?");
        ps.setInt(1, bytes.length);
        ps.setBinaryStream(2, byteStream, bytes.length);
        ps.setInt(3, directoryId);
        ps.execute();
        conn.commit();
    }

    public static byte[] getBytes(Object obj) throws java.io.IOException {
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(bos);
        oos.writeObject(obj);
        oos.flush();
        oos.close();
        bos.close();
        byte[] data = bos.toByteArray();
        return data;
    }
}

It is rare that I am on the bleeding edge of technology. Normally, I don’t think its worth the time and effort necessary to learn something brand new unless it has been at least somewhat widely adopted and accepted by the community at large.

Oddly enough, my blog post about running a game server on EC2 turned out to be perfectly timed, as Amazon launched its new CloudWatch, Elastic Scaling and Load Balancing services on Sunday. And since, as I discussed earlier, I have been looking at ways to monitor the usage of my EC2 game server, I somehow find myself on the bleeding edge of the cloud.

Why CloudWatch?

As I discussed in my previous post, setting up monitoring on an EC2 instance wasn’t that hard to do. However, it did come with some drawbacks:

• Maintenance – Although it can be fun to install new software and learn its in’s and out’s, the actual task of upgrading that software, maintaining it, patching it, watching it for security risks, etc, etc is a major pain in the rear end. CloudWatch solves this problem by providing a simple service for retrieving performance data, no maintenance or special setup required.
• Granularity – As I discovered with munin, there are limitations to the frequency with which you can store performance data, not to mention the storage requirements for vast quantities of it. Again, this is hidden from us in the case of CloudWatch.
• Performance – Last but certainly not least, monitoring something usually incurs a performance hit. In my previous article I was sampling data on the same host I was tracking statistics from. The very act of collecting performance data could cause that data to be skewed. Since CloudWatch abstracts this away from individual instances, this is no longer a problem.

Getting Started With CloudWatch

There are quite a few resources available to get you started with CloudWatch. I recommend taking a look at the javascript scratch pad and the other various developer libraries already available (more on this later).

If you really want to get down to the nitty gritty, you should start with the CloudWatch command line interface (CLI). Here are some simple steps to get you started:

1. Download the EC2 API Tools first (you’ll need them to set up monitoring). Check out the Getting Started Guide for instructions on extracting the tools and setting up the proper environment variables.
2. Download the CloudWatch API Tools. Check out the included readme for details on environment variable setup.
3. Start up an EC2 instance like you normally would (see my previous post).
4. Enable monitoring on your running instance using the EC2 API Tools command: ec2-monitor-instances <instanceId>.
5. Take a look at the CloudWatch Getting Started Guide for details on the available monitoring parameters, etc.
6. Run the CloudWatch command mon-get-stats to get some statistics from your running instance (mon-get-stats –help should give you some examples).

Here are a few things to keep in mind when running the command line utility:

• I normally output data to a CSV file so I can create fancy graphs in Excel. Here is an example command (Windows) that delimits stats by comma and outputs to a CSV file:

  mon-get-stats CPUUtilization --start-time 2009-05-19T21:00:00
   --end-time 2009-05-19T22:00:00 --period 60 --statistics Average 
  --namespace AWS/EC2 --delimiter "," 
  --dimensions "InstanceId=i-2bb5cc42" > stats.csv

• Timestamps – As per the forums, input timestamps are in ISO-8601 format with the default timezone UTC (Eastern Standard Time + 4 hours). Output timestamps are in UTC and cannot be changed (so start thinking in Greenwich Mean Time).
• Virtually as soon as monitoring is enabled, statistics are retrieved from your instances. Data is available up to a per-minute frequency and is stored for two weeks.

Writing a Simple Java Monitoring Utility

As much fun as I was having trying to parse and decipher various command line inputs, I was somewhat disappointed in the output. For one thing, there was the time formatting problem. For another, only one set of statistics (CPU utilization, network I/O, etc) were available at one time.

I am not one to do more work than I need to, so instead of setting off to invent an uber-utility for aggregating data, I simply downloaded the Java library for CloudWatch and hacked up some of the sample code until I had a very basic utility for downloading and aggregating the data I wanted. I present it below in case someone finds it useful:

import java.io.File;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Properties;

import com.amazonaws.cloudwatch.AmazonCloudWatch;
import com.amazonaws.cloudwatch.AmazonCloudWatchClient;
import com.amazonaws.cloudwatch.AmazonCloudWatchException;
import com.amazonaws.cloudwatch.model.Datapoint;
import com.amazonaws.cloudwatch.model.GetMetricStatisticsRequest;
import com.amazonaws.cloudwatch.model.GetMetricStatisticsResponse;
import com.amazonaws.cloudwatch.model.GetMetricStatisticsResult;

public class GrabStats {

    public static void main(String[] args) {
        
        String fileName = "C:\\stats.csv";

        String startTime = "2009-05-19T20:00:00";
        String endTime = "2009-05-20T00:00:00";
        
        String[] statList = { "CPUUtilization","NetworkIn","NetworkOut" }; //(%, bytes, bytes)
        
        HashMap<String, HashMap<String, Double>> map = new HashMap<String, HashMap<String, Double>>();
        
        // grab stats for each stat value
        for (int i = 0; i < statList.length; i++) {
            HashMap<String, Double> stats = getStatistics(startTime, endTime, statList[i]);
            map.put(statList[i], stats);
        }
        
        // write to disk
        try {
            FileWriter fw = new FileWriter(fileName);
            
            // write the header
            fw.write("Date");
            for (int i = 0; i < statList.length; i++) {
                fw.write(",");
                fw.write(statList[i]);
            }
            fw.write("\n");
            
            // get a date iterator from our first statistic
            Iterator<String> dateIterator = map.get(statList[0]).keySet().iterator();

            while(dateIterator.hasNext()) {
                String date = dateIterator.next();
                fw.write(date);
                
                // get values for each stat at this date
                for (int i = 0; i < statList.length; i++) {
                    Double value = map.get(statList[i]).get(date);
                    fw.write(",");
                    fw.write(value.toString());
                }
                
                fw.write("\n");
            }
            
            fw.close();
        } catch (IOException ex) {
            // error storing data
            System.out.print("Error writing file: " + fileName);
        }

    }

    // define the cloudwatch service (should be a singleton)
    private static final String _accessKeyId = "<insert key here>";
    private static final String _secretAccessKey = "<insert access key here>";
    private static AmazonCloudWatch _service = new AmazonCloudWatchClient(
            _accessKeyId, _secretAccessKey);

    public static HashMap<String, Double> getStatistics(String startTime,
            String endTime, String statName) {
        HashMap<String, Double> map = new HashMap<String, Double>();

        // build the request with some defaults
        GetMetricStatisticsRequest request = new GetMetricStatisticsRequest();
        ArrayList<String> stats = new ArrayList<String>();
        stats.add("Average");
        request.setStartTime(startTime);
        request.setEndTime(endTime);
        request.setPeriod(60); // statistics every minute
        request.setMeasureName(statName);
        request.setNamespace("AWS/EC2");
        request.setStatistics(stats);

        try {

            GetMetricStatisticsResponse response = _service
                    .getMetricStatistics(request);

            if (response.isSetGetMetricStatisticsResult()) {
                GetMetricStatisticsResult getMetricStatisticsResult = response
                        .getGetMetricStatisticsResult();
                java.util.List<Datapoint> datapointsList = getMetricStatisticsResult
                        .getDatapoints();
                for (Datapoint datapoints : datapointsList) {
                    map.put(datapoints.getTimestamp(), datapoints.getAverage());
                }
            }

        } catch (AmazonCloudWatchException ex) {

            System.out.println("Caught Exception: " + ex.getMessage());
            System.out.println("Response Status Code: " + ex.getStatusCode());
            System.out.println("Error Code: " + ex.getErrorCode());
            System.out.println("Error Type: " + ex.getErrorType());
            System.out.println("Request ID: " + ex.getRequestId());
            System.out.print("XML: " + ex.getXML());
        }

        return map;
    }

}

Conclusion

The CloudWatch tools and utilities are nothing less than I’d expect from Amazon. Everything worked as expected, the documentation was well put together and there were no real surprises with the API. Overall, I am very satisfied with the finished product of my meager efforts.

There are, of course, a few shortcomings:

1. It would be nice to have more statistics available (memory usage being the main one I’m thinking of). Having the ability to define and collect your own statistics via an API would be even better. Since the API already has a flexible way of defining statistic and type, I have to assume this is coming.
2. Output visualization is certainly lacking. It would be great to see someone hack a Google Chart generator into the javascript scratch pad (given my lack of copious amounts of free time, this person won’t be me).
3. Adding some statistic collection and enablement to ElasticFox would certainly make things easier to set up and administer.

I have to assume these drawbacks will be addressed in future updates, as they have been in the past. I am willing to accept them as the price to pay for being on the bleeding edge of the cloud.