HDXPRT Development Community Blog

February 17

Between talking about CES, the new touch benchmark, and sausage making, it seems like it’ has been a while since I’ve said anything about HDXPRT. Here’ is a quick status update. The short form is that folks are heads- down coding, debugging, and testing. We still have some significant hurdles to overcome, such as trying to script Picassa. We also are going to have to make some difficult decisions in the near future about possibly swapping out one or two of the applications due to either licensing or scripting issues. (Sausage making at its best!) We’ll keep you posted in the forums when we have to make those decisions.

There is still a lot to get done, but things still appear to be on schedule. That schedule means that we are still hoping to have a beta version available for the Development Community to test in late March. At that point, the beta version will be available to our members and we will really need your help to try and shake things out. (Join at http://hdxprt.com/forum/register.php if you are not yet a member of the Development Community and want to help in our effort.) The more different systems and configurations we can all test together, the better the benchmark will be. There will also be at least some time for feedback on whether HDXPRT 2012 matches the design specification and if there are any last- minute tweaks you think would help make for a better benchmark.

So, stay tuned! We look forward to continuing to work with you on making HDXPRT 2012 even better than the current version.

-Bill Catchings

Comment on this post in the forums

February 10

I discussed in my previous blog how weighing the tradeoffs between real science and real world in benchmark is a real art. One person felt it was more akin to sausage making than art! In truth, I have made that comparison myself.

That, of course, got me thinking. Is the process of creating a benchmark like that of creating sausage? With sausage, the feeling is that if you knew what went into sausage, you probably wouldn’t eat it. That may well be true, but I would still like to know that someone was inspecting the sausage factory. Sausage that contains strange animal parts is one thing, but sausage containing E. coli is another.

We are trying with the Development Community to use transparency to create better benchmarks. My feeling is that the more inspectors (members) there are, the better the benchmark will be. At least to me, unlike making sausage, creating benchmarks is actually cool. (There are probably sausage artisans who feel the same way about sausage.)

What do you think? Would you prefer to know what goes into making a benchmark? We hope so and hope that is why you are a part of this community. If you are not part of the Development Community, we encourage you to join at http://hdxprt.com/forum/register.php. Come join us in the sausage-making art house!

-Bill Catchings

Comment on this post in the forums

February 3

In my last blog entry, I noted the challenge of balancing real-world and real-science considerations when benchmarking Web page loads. That issue, however, is inherent in all benchmarking. Real world argues for benchmarks that emphasize what users and computers actually do. For servers, that might mean something like executing real database transactions against a real database from real client computers. For tablets, that might mean real fingers selecting and displaying real photos. There are obvious issues with both—setting up such a real database environment is difficult and who wants to be the owner of the real fingers driving the tablet? It is also difficult to understand what causes performance differences—is it the network, the processors, or the disks in the server? There are also more subtle challenges, such as how to make the tests work on servers or tablets other than the original ones. Worse, such real-world environments are subject to all sorts of repeatability and reproducibility issues.

Real science, on the other hand, argues for benchmarks that emphasize repeatable and reproducible results. Further, real science wants benchmarks that isolate the causes of performance differences. For servers, that might mean a suite of tests targeting processor speed, network bandwidth, and disk transfer rate. For tablets, that might mean tests targeting processor speed, touch responsiveness, and graphics-rendering rate. The problem is that it is not always obvious what combination of such factors actually delivers better database server performance or tablet experience. Worse, it is possible that testing different databases and transactions would result in very different characteristics that these tests don’t at all measure.

The good news is that real world and real science are not always in opposition. The bad news is that a third factor exacerbates the situation—benchmarks take real time (and of course real money) to develop. That means benchmark developers need to make compromises if they want to bring tests to market before the real world they are attempting to measure has changed. And, they need to avoid some of the most difficult technical hurdles. Like most things, that means trying to find the right balance between real world and real science.

Unfortunately, there is no formula for determining that balance. Instead, it really is somewhat of an art. I’d love to hear from you some examples of benchmarks (current or from the past) that you think do a good job implementing this balance and showing the real art of benchmarking.

-Bill Catchings

Comment on this post in the forums

January 27

I think that an important part of any touch benchmark will be a Web component. After all, the always (or almost always) connected nature ofthese devices is a critical part of their identities. I think such a Web benchmark needs to include a measurement of page load speed (how long it takes to download and render a page).

Creating such a test seems straightforward. Pick a set of sites, such as the five or ten most popular, and then time how long the home page of each takes to load. The problem, however, is that those pages are constantly changing. Every few months, most popular sites do a major redesign. That would obviously affect the results for a test and make it difficult to compare the results of a current test to one from a few months back. It is even more of a problem that the page will be different for one user than another as sites typically know things like where you are and what your computer is and adjust things to match those characteristics. And, the ads and the content of the site are constantly changing and updating. Even hitting Refresh on a page can give you different page.

Given all of those problems, how is it possible to test page loads? One way is to create pages that are similar those of leading Web sites in terms of things like size, amount of graphics, and dynamic elements. This allows the tests to be consistent over time and from different devices and locations. (Or, at least, as consistent as the variability of the Internet from moment to moment allows.) The problem with this approach, however, is that the pages will age out as Web sites update themselves and they will not be the real sites.

Such are the tradeoffs in benchmarking. The key is how to balance real science with real world considerations. What do you think? Which approach is the better balance of real science and real world?

-Bill Catchings

Comment on this post in the forums

January 20

While talking with people at CES about HDXPRT and the upcoming touch benchmark, I encountered the same question a few times—Who are the current members of the Development Community? My answer was something along the lines of “About 10 PC hardware vendors, about the same number of press people, and a few other folks from companies around the world.” I was, however, itching to name the companies because the list is really pretty impressive. We haven’t asked for permission from the Development Community members, though, so I left my answer vague.

Given our goal of expanding the Development Community, I find myself weighing two possible outcomes if we were to make public the names of the companies represented. On the one hand, it could encourage others to join us (“All the other cool kids are doing it, I guess I will too!”). On the other hand, it could discourage others from joining us (“Not sure how my company would feel about this. Should I ask Legal? I’m too busy, never mind.”)

My best plan for now is to email each member individually and ask where he or she stands on company anonymity. And to give all new members the option of keeping their affiliation off the record. Rest assured that we will definitely not reveal this information without your permission.

We’d like to know what you think. Would you have joined the Development Community if doing so required identifying your company and allowing us to share it? Would you now be willing to let us say that someone from your company is a member?

-Bill Catchings

Comment on this post in the forums

January 13

I never thought I would say this, but there are more electronic gadgets and toys than I want. While walking the many cavernous show floors of CES, I saw cool bicycle gadgets from iBike (www.ibikedash.com). One device is a case for your iPhone that transforms it into a cycling computer. Because it measures wind speed, it actually is more capable than any existing bike computer—it uses data you supply like your type of bike and your weight, GPS info and knowledge of the terrain, and readings on wind speed and your heart rate to calculate your power output. If it works reliably, it would provide data that normally requires a cycle power meter costing a couple thousand dollars. If you are not into cycling, you probably don’t care, but it does show how our phones are becoming the gathering point for a myriad of data sources around us. I definitely need to try one of these out when they become available in March.

I also saw solar panels from Sharp (SunSnap) that have the inverter built in so that they output AC power directly. This gets around the messy inverter and wiring problems of typical panels that output DC power. Now, if I can get my homeowners association to agree, I need some of these.

I also saw TVs that were enormous, like 84-inch LCD, and gorgeous, like the 55-inch OLED, both from LG. I saw Windows 8 tablets and cars and iPhone cases and e-cigarettes. Basically, I reached gadget overload. At least the future of technology does not appear to be boring!

Thanks so much to the folks that stopped by our suite to talk about HDXPRT, the upcoming touch benchmark, and what they see as the future of benchmarking. We will be doing our best over the coming months to incorporate your ideas and suggestions. If you were not able to visit with us, please feel free to drop me an email and let me know what you are thinking.

-Bill Catchings

Comment on this post in the forums

January 6

Hopefully, you caught our press release yesterday: http://www.hdxprt.com/touch_release.php. The short form of it is that we (PT and the HDXPRT Development Community) are going to create another benchmark—one aimed at touch-based devices. The obvious place to start will be Windows 8 Metro. Because it is a Windows-based touch platform, the development process should not be too different from work we have done in the past and may even allow us to use some code from HDXPRT. The longer-term goal of going cross-platform will be much more challenging. One step at a time!

Touch-based computing devices and Windows 8 Metro are emerging areas without any good tools for comparing performance. We are all excited at this opportunity to create a benchmark to meet that need and hope you are as well. After all, we will need your help to pull this off!

As I mentioned last week, I will be at CES. I would love to talk with as many of you as possible about the new touch benchmark and HDXPRT 2012. We’ll have a suite at the Hilton and I will be on the show floor checking out the latest gadgets, especially touch devices. Drop us an email at hdxprt_ces@hdxprt.com if you would like to stop by the suite or will be working a booth or suite and would like me to stop by. Either way, I’m hoping to see quite a few of you folks there.

-Bill Catchings

Comment on this post in the forums

December 29

This is just a quick entry to wish everyone a great and productive 2012 and to let you know that I will be at CES. My goal is to meet with as many of you in the Development Community as possible. Please let me know if you have some time and would like to escape the craziness of the show to relax and chat. We’ll have a suite at the Hilton and would love to offer you the opportunity to kick back and talk about HDXPRT, the future of benchmarks, or about the cool things you’ve seen at the show. If you plan to be at CES, but are stuck working a booth or suite, let me know and I’ll try to stop by and say hi. Drop us an email at hdxrpt_CES@principledtechnologies.com and we will set up an appointment.

I hope to see quite a few of you folks there. And, to play with lots of cool new toys! Have a Happy New Year!

-Bill Catchings

Comment on this post in the forums

December 22

I’ve spent some time over the last week looking into computing devices with touch interfaces. It has become clear to me that touch is not just the next way of interacting with a computing device. As has been the case with earlier input mechanisms such as the punch card, keyboard, and mouse, the interface defines the experience and the whole platform. I have been in the industry long enough to have started on punch cards. When you wrote programs in that environment, you made sure that everything was right before you even attempted to run the program. (You also tried really hard to not drop your deck of punch cards!)

Keyboards and monitors drove command-line interfaces. In those interfaces, I learned how to enter really complex commands to do things like repeat or modify earliercommands. (If !! or !$ mean anything to you, that probably means you understood how to get the most out of the C shell command-line interface.)

The mouse, as an input device, worked naturally with windowed environments. Click, drag, and drop all took on new meaning in the mouse-centric world. The mouse lends itself well to precision but can be cumbersome for something like turning the page in a document—I still often use the page-down key instead.

Touch-based computing devices are beginning to define their environments. Not only does touch determine what is easy (turning a page is trivial, but picking an exact point on the screen is hard), but a way of working. Applications tend to be less complex, and cheaper and easier to get. Consequently, applications (and their usages) are much more disposable. I used the same family of keyboard-based text editors for at least fifteen years (EMACS) before moving to a mouse-based, windowed word-processing environment and have not changed that (Word) for even longer. However, I have used multiple note-taking programs on my iPad in just the last couple of months. In the world of the mouse and windows, deciding on a particular program required a large investment of time and money. In the world of touch and application markets, I try a few programs, pick one I like, and do not hesitate to change when a new one comes out.

All of this, of course, will need to come into play in any attempt to benchmark a touch environment such as Windows 8 Metro. For example, there may be less emphasis on particular applications than on categories of applications. It also means that it will likely be important to have more small, targeted usage scenarios than a few large scenarios. What do you think will be different (or the same) in a touch benchmark?

I, and the HDXPRT team, wish everyone a great holiday! Enjoy some time at home with family. And, enjoy playing with whatever new devices you get this year!

-Bill Catchings

Comment on this post in the forums

December 16

As is often the case for me, Christmas shopping has given me the chance to look at all sorts of gadgets. (No, I’m not sure who to buy them for, but that isn’t the point.) The wealth of touch-based devices like the iPad, the Kindle Fire, the Galaxy Tab, and phones of all sorts is either incredibly exciting or amazingly confusing. Touch-based interfaces have moved well beyond the devices they started on and are showing up pretty much everywhere. Even my car (a Nissan Leaf) uses a touch interface. When I use a device with a screen, like my camera, and can’t touch the screen, it just feels wrong.

The power of computing devices like the iPad and other tablets is bringing touch into what we traditionally think of as the PC marketplace. The debut next year of Windows 8 with its touch-based Metro user interface will add another serious player to the mix. Touch will be in your desktop and notebook future. (Which for me means a steady supply of cloths for wiping screens will be a necessity, but that’s another story.) I think that touch will be the dominant interface—surpassing the mouse—in the near future.

When I see that kind of shift in the marketplace, and the resulting product diversity, my background makes me think that such an area is ripe for some good tools to compare the products. What do you think? Do we need a new generation of touch-based benchmarks for Metro? For other touch-based platforms?

Back here in HDXPRT Central, I do want to mention that the HDXPRT 2012 design specification is now available. Check it out at http://www.hdxprt.com/forum/2012_design_specification.php!

-Bill Catchings

Comment on this post in the forums