Today is a milestone. We are freezing the beta for HDXPRT 2012. It’s taken a lot of work–developing something with this many moving parts is a bit like herding cockroaches. However, the workloads are set and the benchmark seems stable.

So, what happens next?

First, the beta goes to a team who will spend a few days intensively testing it. The test team is isolated from the development team because, let’s face it, developers can’t test their own code. (I should know. I’ve developed a lot of products in my time.)

We’ll run it on a variety of systems, checking it for stability and repeatability. We will also look at the results and make sure that they are sensible. That is, they reflect that capabilities of the systems we are running the benchmark on.

The beta is approximately 10 GB. It includes installers for nine applications and a number of large content files. Because of its size, the beta will only be available on DVD. Once the tests look good, we’ll start reproducing and verifying the DVDs. Once the DVDs are verified, we’ll mail them out. The whole process should take about a week, so community members should have their beta DVDs within a couple of weeks.

While the testing is happening, we’ll be contacting each community member to confirm that we have their correct physical mail address.

Once you have the DVDs, please run the benchmark and send feedback. We need your help to make this beta into a truly great release candidate. We will work hard to address any feedback we get. If you find a problem, let us know. If you have a complaint, let us know. It you have a compliment, by all means, let us know!

Next week, Bill will be back from his travels.

- Eric Hale

Comment on this post in the forums

We’re in the home stretch of testing the HDXPRT 2012 before releasing the beta to the community members. As is often the case, we’ve run into some issues that I want to share with you.

We’ve encountered some technical challenges in creating the 4K video playback workload that the design specification called for. We could not use the existing HDXPRT playback automation and instrumentation because Windows Media Player currently does not by default play 4K H.264 content. We tried other video players but ran into stability issues on slower systems.

These technical problems led us to step back and think about the role of video playback in HDXPRT 2012. In HDXPRT 2011, there wasn’t much differentiation between most current systems on the video playback results. The video playback tests were also a source of confusion. As I traveled and talked to users, they thought that the stars of the video playback were the primary metric, not the performance number. More generally, however, having both quality and performance numbers can be confusing.

Our plan is to remove the video playback tests from the beta. We will include them as optional inspection tests in an updated version later this year. By removing video playback from the primary tests and metrics, we’ll be able to include stressful test that won’t work on all systems while avoiding unnecessary confusion. Over time, I think we can include a number of interesting tests this way. We’d love to hear your ideas for what might be good to include that way as well as whether you think we’re making the correct call for video playback tests in HDXPRT 2012.

I will be traveling next week. Eric Hale, our project lead for HDXPRT 2012, will be writing the next blog. He will hopefully have some good news about the beta!

-Bill Catchings

Comment on this post in the forums

For HDXPRT 2011, we did quite a bit of testing to characterize the benchmark. Those results appeared in an initial white paper and a follow up one. In the first, we ran tests on different processors, various amounts of RAM, internal vs. external graphics, hard disk vs. SSD, and the effects of Intel Turbo Boost Technology. In the follow up white paper, we looked in more depth at the effect of graphics cards with different processors.

I mentioned a couple weeks ago that we are starting to put together a testbed to help us characterize HDXPRT 2012. What is in that testbed will define what characteristics of the upcoming benchmark we measure. We would like to get your help defining that testbed.

Our current thinking is to do a similar set of tests this year with updated hardware. However, plenty of additional things would be interesting to look at. First, I would like to increase the range of processors we test, including AMD processors. I would also like to do some testing varying different processor characteristics such as threads, cores, and frequency. It might also be good to look at the effect of new technologies like hybrid drives (which combine a small SSD with a hard disk to try and have the best of both).

We face two challenges in doing these characterization tests. One is to try and change one only variable at a time. That is very difficult in some cases, such as comparing Intel and AMD processors—you can’t just swap them in the same motherboard. Fortunately, it is usually possible to find very similar motherboards and keep other components (like disks, graphics, and RAM) constant. The other challenge is getting all of the necessary hardware in house.

So, we have two requests for you. First, let us know what you would like to see us test. Second, help us by supplying some of that equipment. If you supply the equipment we will do our best to include results from it in the characterization study and in the new HDXPRT 2012 results database. As always, thanks for your help!

-Bill Catchings

Comment on this post in the forums

As we get close to the beta version of HDXPRT 2012, I wanted to let you know how it compares with the original design specification. As inevitably happens in any software project, there are differences between the original design and the final product. Generally, things have stayed pretty close with HDXPRT 2012, but there are two changes worth noting.

First, in the design specification, we specified Audacity 1.3.14 Beta in the Music Maker scenario as that was the only version that supported Windows 7 at the time. Audacity 2.0 with Windows 7 debuted in the interim so we are using that version.

The second and more significant issue was with Picasa, which was to be part of the Media Organizer scenario. Unfortunately, we couldn’t create a stable script because scripting tools like AutoIT could not properly recognize some of Picasa’s application UI elements. Somewhat reluctantly, we ended up replacing Picasa with Photoshop Elements. We still think the scenario is a good one and Photoshop Elements is an appropriate tool. I would have liked, however, to have Picasa in there.

There are probably some more minor differences between the beta and the design specification. We’ll let you know what they are when we have the beta ready in a couple weeks. (Hopefully!) We’re looking forward to getting that into your hands and getting your feedback. If you’re not already a member of the Development Community, I encourage you to join so that you can get your copy of the beta when it is available.

-Bill Catchings

Comment on this post in the forums

We’re currently starting testing on alpha versions of HDXPRT 2012. In order to do that, we’re putting together a testbed. We have two goals for the testbed that are somewhat contradictory. The first is to make the testbed as diverse as possible in terms of vendors and configurations. We want notebooks and desktops from as many vendors as possible. We want to make sure we have systems that will push the edges—both slower systems that may even be below the minimum recommended configuration and faster ones representing the current latest and greatest. These systems will help us shake out bugs and provide some raw data that we can publish when the benchmark debuts in the new results database.

The second goal for the testbed is to have systems where we can easily change one variable at a time to help us understand the characteristics of the benchmark. Typically, these are white box systems where we can swap processors, disks, RAM, and so on. We will use the results from these systems in the benchmark characterization white paper we will create for the debut of HDXPRT 2012.

We’d like your opinions on what we should be certain to test. We think we have a good handle on what to include, but we want your ideas as well.

We also are looking for additional systems to include in our testbed. If you can supply some, please let me know. That is one way to make sure HDXPRT 2012 works on your system and to get your results in the results database. Rest assured, we will not publish those results without your permission. Regardless, the more systems we can test, the better the final product will be.

There will, of course, be opportunities for you to help with the testing as we get to the beta stage in the near future.

-Bill Catchings

Comment on this post in the forums

One advantage of being in the industry for a long time is seeing the development pendulum go back and forth. One such pendulum is the way of interacting with touch interfaces. Touch interfaces existed long before the current phone and tablet devices. I remember the HP-150, an early touchscreen PC, from my days working on Kermit in the 1980s. It was not a big seller, so you probably never used one. However, you may have used early touchscreen technology in devices like kiosks. While those touch interfaces were fairly simple, you used your fingertip on the screen to indicate your selections.

When PDAs became a big deal in the 1990s, the stylus rather than the fingertip became the way to touch the screen. If you lost your stylus or did not feel like pulling it out of the case, you could use your fingernail. I became very good at writing in the odd script that the Palm OS used. (I still sometimes write the letter A as an upside-down V.) Though the stylus was easier, you could do most things using your fingernail. I also used a stylus (and my fingernail) with Windows smartphones.

Smartphones, especially the iPhone, swung the pendulum back to touching the screen with your fingertip. It took me a decent bit of time to adjust to touching the screen that way. I also had to get used to staring at screens through fingerprints. The ability to multi-touch, however, made that worthwhile. (And, caused me to make sure I always carry those screen-cleaning cloths.)

Recent tablets have generally utilized multi-touch, fingertip interfaces. I still find myself wishing for a stylus at times. I’ve purchased a few different styli for using my iPad, but the mushy, fingertip-like ends leave much to be desired. I just ordered an interesting compromise, the Adonit Jot Classic Stylus. I’m hopeful, but won’t be surprised if I’m disappointed.

The stylus on some Windows 7 tablets like the Dell Latitude ST shows what is possible with a stylus. The stylus can be really useful in some work environments. Hopefully, we’ll see more innovation in touch interfaces. In my ideal world, I could use a simple stylus or my fingernail some of the time and my fingertips when multi-touch is better—all on a single device, of course! For now, I just have to keep cleaning off my iPad’s screen while I try to find the ideal stylus.

Whatever way the touch interface pendulum swings, we’ll try to make sure that TouchXPRT will be the right tool to measure it.

-Bill Catchings

Comment on this post in the forums

Today I’m spending a good chunk of the day participating in a panel discussion on the Kermit file transfer protocol as part of an oral history project with the Computer History Museum. A little over 30 years ago, I worked at Columbia University on the original versions of Kermit. In preparing for the panel discussions, I’ve been thinking about projects with available source code, like Kermit and HDXPRT.

Kermit was a protocol and set of programs for moving files before the Internet. We designed Kermit to work between a wide variety of computers—from IBM mainframes to DEC minicomputers to CP/M microcomputers. As such, we wrote the code to accommodate the lowest common denominator and assume as little as possible. That meant we could not assume that the computers all used ASCII characters (IBM mainframes used EBCDIC), that 8-bit characters would transmit over a phone line, or that packets of more than 100 characters were possible (DEC-20 computers specifically had an issue with that). The pair of Kermit programs negotiated what was possible at the beginning of a session and were able to work, often in situations where nothing else would.

We developed Kermit before the open-source movement or Gnu. We just had the simple notion that the more people who had access to Kermit, the better. Because we did not want incompatible versions of Kermit or the code to be used for the wrong purposes, we retained control (via copyright) while allowing others to use the code to create their own versions. We also encouraged them to share their code back with us so that we could then share it with others. In this way, Kermit grew to support all sorts of computers, in just about every corner of the planet as well as outer space.

In many ways, what we are doing with HDXPRT and its source code is similar. We are working to create a community of interested people who will work together to improve the product. Our hope is that by having the HDXPRT source code available to the Development Community, it will encourage openness, foster collaboration, and spark innovation.

I believe that what made Kermit successful was not so much the design as it was the community. I’m hoping that through the Development Community here, we can make just as successful HDXPRT, TouchXPRT, and who knows what else in the future. If you have not already joined, please do—the more folks we have, the better the community and its resulting benchmarks will be. Thanks!

-Bill Catchings

Comment on this post in the forums

Since the beginning of the year and especially in the last couple of weeks, I’ve been discussing in the blog our thoughts on what should be in TouchXPRT. Based on those thoughts and on feedback we’ve gotten, we are working on scenarios, apps, and workloads for two of the seven possible roles I mentioned in an earlier blog—consuming and manipulating media and browsing the Web. These seemed like two of the more important and common roles and ones where performance might have a noticeable impact.

For the consuming and manipulating media portion, we are working on building a limited app (or apps) that can do some of the functions in the scenario I described in last week’s blog. We’re also working on the necessary content (photos, videos, and sound clips) for TouchXPRT to manipulate and show using the app(s). For the Web browsing role, we are putting together Web pages and HTML5 that emulate sites and applications on the Web.

The goal is to release both of these roles as the first Community Build (CB1) of TouchXPRT by the end of April. As the name implies, CB1 will be available only to members of the Development Community. If you have not joined the Development Community, hopefully TouchXPRT CB1 will give you some additional incentive!

Once we have CB1 ready to release to the community, we will need your help with debugging, results gathering, and general critiquing. As always, thanks in advance for whatever help you are able to offer.

-Bill Catchings

Comment on this post in the forums

Last week I looked at the roles in TouchXPRT that would make sense on a touch-based device like a tablet. I suggested seven possible ones. The next step is to create usage models and scenarios based on those roles. In turn, we would need to develop simple apps to do these things. To get the ball rolling, here are some activity and scenario ideas we came up with for one of the roles—consuming and manipulating media.

After doing email and reading books, this is one of the main things I do on my iPad. Originally, in this role I mostly showed pictures or videos (especially of my grandsons) to people. (Yes, people do hide when they see me coming with my iPad in hand saying, “You gotta see this!”) As the iPad and its apps have grown, I’ve found myself doing some cleaning up of photos, video, and even sound directly on the iPad. I think a person in this role is not necessarily an expert in media, but like most of us enjoys playing with media. So, the person might do something like scale/trim a video or add a nice cross-dissolve between two video clips. Maybe the person would even create a video montage by combining stock travel footage with personal video clips. Beyond simply rotating and cropping photos, the person might add some stock preset effects like making them sepia toned, adding titles, or creating a postcard. The person might create a slideshow based on a set of travel photos and use some visual or audio effects. They might also add sound by manipulating audio clips. Based upon these kinds of usages, the apps would include some of the features found in apps like iMovie, Instagram, SnapSeed, PhotoGene, iPhoto, and GarageBand.

What do you think? How do those activities match your usage of touch-based devices? What would you add, subtract, or change? Do you have suggestions for the other roles? Thanks for your help in defining what TouchXPRT will be.

-Bill Catchings

Comment on this post in the forums

We have been busy the last couple of months with TouchXPRT. We have been investigating and trying out things on Windows 8 Metro. While we are excited by the possibilities for a benchmark in that space, the task is a bit daunting.

The first key question is what are people likely to do with a device using touch-based environment like Metro? The best way to answer that is to look at what people are currently doing with IOS- and Android-based devices. We have been playing with those as well as some units running the Metro beta. To create an initial list of roles or usage categories, we spent some time looking at what is available on the iTunes App Store, the Android Play store, and the Windows Store. Here, in no particular order, is the list of uses we came up with:

• Consume and manipulate media – Touch devices are heavily used for consuming media (music, photos, and video), but now are being used for some simple manipulation tasks like adding simple visual effects to video, mixing and changing audio, and enhancing photos.
• Browse the Web – Touch devices are becoming one of the main ways people consume Web content, both normal Web pages and specially crafted “mobile” pages. Touch devices are what I use to find the phone number for the nearest takeout Chinese.
• Watch video for entertainment – Through movie apps like Netflix and TV network apps, touch devices (especially tablets) are becoming a major force in this area.
• Play games – This is obviously something folks do on their touch devices. As best we can tell, no consumer device can ship without Angry Birds!
• Interact with others – Through apps like Facebook and Foursquare, touch devices are becoming a big way that people interact with each other.
• Get news and information – Another big area is general news and information, including things like stock quotes and weather.
• Use utilities – This is a broad category—there are a ton of utilities for doing everything from moving files to backing up data.

That list covers a lot of ground and some of the areas, like games, would be particularly difficult to benchmark. We thought, however, that it would be best to get everything out and then figure out what to tackle first. The big challenges we face are the lack of apps available for Metro and having no good ability to script or drive applications. Our current thinking is to write some minimal sample apps that mimic common apps out there. These would not be complete apps, but would do some of the key functions. Then, we could build scenarios around these functions. That seems like the best approach to completing something in a timely fashion. Initially, we would aim for two or three of those areas and then add others over time.

As always, we need your feedback. Let us know what you think about the list of uses and the approach in general. And, let us know if you can help with any of the sample app development. Thanks!

-Bill Catchings

Comment on this post in the forums