Futuremark is a well-respected provider of computer benchmark applications. Their PCMark and 3DMark benchmarks have been around for almost 20 years, and provide a good indication of the system performance for various workloads. Today, Futuremark is launching PCMark 10, their seventh major update to the PCMark series of benchmarks first launched in 2002. PCMark 10 builds upon the PCMark 8 platform, adds a few workloads and streamlines the rest in order to present a vendor-neutral, complete, and easy-to-use benchmark for home and office environments. Futuremark provided us with a preview copy of the benchmark, and we took it for a test drive.

Introduction and Evaluation Setup

Futuremark's first unified benchmark for evaluation of system performance was PCMark 2002, released in March 2002. It was followed by PCMark 04 in late 2003, and PCMark 05 (with support for Windows Vista). PCMark Vantage was introduced in 2007 for evaluation of systems running Windows 7, and PCMark 7 in 2011. PCMark 8 was launched in 2013, with full support for Windows 8 / 8.1. Even though Windows 10 is not an officially supported OS, we have had no issues in processing the PCMark 8 workloads on machines running that OS. PCMark 10 brings official support for Windows 10.

PCMark 10, like BAPCo's SYSmark 2014 SE, caters mainly to system vendors who want to bid for government contracts / tenders for high-volume PC purchases from businesses and enterprises. The PCMark scores help the vendors in advertising a performance level for their systems. From the other perspective, it can also help businesses to mandate a minimum performance level for systems that they hope to purchase. PCMark 10 measures system performance using workloads based on real-world applications and activities reflective of the usage of modern-day PCs in an office environment.

PCMark 10 comes in three different flavours - the Basic edition is free to download, while the Advanced edition costs $30 (suitable for home consumers). The Professional edition is the one suitable for business use, and it comes with a host of features that make the benchmark easy to use in an automated manner. Pricing for the professional edition is not public.

The PCMark 10 benchmark can be processed in one of three built-in modes - default, express or extended. These built-in modes correspond to benchmark definitions (in XML) that are supplied along with the installation. User-generated XML files conforming to the correct format can be created for custom benchmarks. The 'Express' benchmark is a subset of the default one, which, in itself, is a subset of the 'Extended' benchmark. The express benchmark and focuses on typical office workloads such as web browsing and selected productivity tasks. The 'Extended' tests add demanding graphics and physics test to provide an idea of the CPU as well as the GPU performance.

Each of these workloads is composed of sub-tests, as shown below.

The application start-up workload involves measuring the time taken to launch a number of programs of varying complexity such as the Chromium web browser, the Firefox web browser, LibreOffice Writer word processor, and the GIMP image manipulation program. Web browsing tests involve navigation of a social media site, online shopping, map navigation, video playback (H.264 and VP9 at 1080p and 4K resolutions), and a static web page, using both Firefox and Chromium. The video conferencing workload deals with both low-quality one-on-one conferencing (720p30 H.264 encode and decode, as well as face detection running on an available OpenCL device or CPU), as well as high-quality group conferencing (1080p30 H.264 encode, and 3x 720p30 H.264 decode, as well as face detection). Productivity tests involve word processing (using LibreOffice Writer) and spreadsheet editing (using LibreOffice Calc). Digital content creation tests involve photo editing (using ImageMagick, capable of OpenCL acceleration) with various filters, video editing (using ffmpeg, capable of OpenCL acceleration) for downscaling, sharpening and deshaking, and rendering and visualization. The last workload uses the OpenGL 4.3 code path from 3DMark's Sling Shot workload, and a custom POV-Ray workload. The gaming tests are directly from the 3DMark Fire Strike workload.

Setting up the Evaluation Flow

We processed PCMark 10 on a number of systems that we had readily configured. The details of the systems are presented in the next section. The first step was to create an installation script. Though PCMark 10 offers an uninstall option, I have found it better to create a system image prior to installation of such benchmarks. This provides us with an option to perform 'System Image Recovery' to go back to the prior version. Towards this, I created the following Powershell script, assuming that a USB drive with the volume label 'PCImageBackups' is connected to the system.


​$BackupVolume = Get-Volume -FileSystemLabel PCImageBackups

​$BackupDrive = $BackupVolume.DriveLetter + ":"
Write-Host "`nUsing ${BackupDrive} for backing up..."
wbAdmin start backup -backupTarget:$BackupDrive "-include:C:,D:" -allCritical -quiet

Write-Host "`nInstalling PCMark 10..."
cmd /c start /wait .\pcmark10-setup.exe /quiet /silent
& 'C:\Program Files\Futuremark\PCMark 10\PCMark10Cmd.exe' --register=<PCMark10 Pro Key>


After executing the above script (resident in the same folder as pcmark10-setup.exe), the following run script was then copied over to a local folder on the machine, and the system was restarted after dismounting all attached USB drives.


​$CWD = (Get-Item -Path ".\" -Verbose).FullName

​& 'C:\Program Files\Futuremark\PCMark 10\PCMark10Cmd.exe' --definition=pcm10_extended.pcmdef --out=$CWD\bmark.pcmark10-result --export-pdf=$CWD\bmark.extended.pdf --systeminfo on --systeminfomonitor on


Processing the above script generated the benchmark results in the same folder as that of the script. The results file could be opened in the PCMark 10 application to observe the monitoring data and low-level workload scores, while the PDF held a portable copy of the high-level results and system information.

BAPCo SYSmark 2014 SE - A Comparison

BAPCo's SYSmark 2014 SE was released recently, and it caters to the same market segment as the Futuremark PCMark 10. Having set up an evaluation flow using SYSmark 2014 SE recently, it was interesting to try out PCMark 10 and evaluate its approach to system benchmarking. The SYSmark 2014 SE flow is considerably more complicated, as it uses commercial versions of programs such as Adobe Acrobat. This creates issues with clean uninstallation of the benchmark (the system image creation scripting was done as a solution for that initially). Fortunately, PCMark 10 doesn't use any commercial software as part of its workloads. SYSmark 2014 SE tries to ensure that all OS settings are consistent across different benchmark runs as long as a particular profile is loaded. Unfortunately, PCMark 10 has a set of default internal settings, but really doesn't care about the OS settings. In the course of our evaluation of PCMark 10, we found that scores could vary widely with different OS settings, making it the prerogative of the benchmark processor to ensure consistency while evaluating different systems. We document that, as well as additional selected sensitivity analysis, in the next section.

 

Benchmark Processing and Sensitivity Analysis
POST A COMMENT

17 Comments

View All Comments

  • surfnaround - Monday, June 05, 2017 - link

    pricing for the commercial version is not mentioned, because it is 1000 dollars. I would assume it is the same price as the previous versions... how could it get anymore expensive? Reply
  • surfnaround - Monday, June 05, 2017 - link

    Ignore the post above... i though i was talking about 3Dmark... D'oh! Reply
  • waltsmith - Monday, June 05, 2017 - link

    Well, sounds like they are headed in the right direction!! Reply
  • TelstarTOS - Monday, June 05, 2017 - link

    Only open souce software is a step back IMO. Is LibreOffice the most used office suite? Doubt that. Same for the lack of acrobat. HEVC is another disappointment. Reply
  • BrokenCrayons - Monday, June 05, 2017 - link

    There's no doubt that LibreOffice is far behind MS Office, however there are signs of growth. In 2015, there were an estimated 100 million users.

    Source: http://news.softpedia.com/news/libreoffice-now-has...
    Reply
  • JoeyJoJo123 - Monday, June 05, 2017 - link

    They'd need some collaboration effort with Microsoft to get their blessing on being able to hook into Office applications at such a native level when doing these benchmarking tasks.

    LibreOffice is a lot easier to do this native testing in due to its open source nature; they can look up the code to see how they can invoke opening a document, generating 1000 pages of "Lorem Ipsum", doing a Ctrl + H find and replace all, etc. from the runtime of their own executable benchmark, and evaluate to a fine level the amount of time it took to complete these tasks to the clock cycle.

    Can you do Word and Excel based productivity benchmarking? Yeah, but it's manual and I hope you're quick and accurate with that stop watch. Why is it manual? Well go ask Microsoft why they continue to build walled garden closed source software where making extensions is difficult or impossible to accomplish.
    Reply
  • ganeshts - Monday, June 05, 2017 - link

    I doubt Word & Excel-based productivity benchmarking is that difficult. SYSmark 2014 SE does it without any issues. Also, you can always use AutoIt scripting to achieve the required 'workload' and 'automate' it similar to the way a real user would interact (PCMark 10 also uses AutoIt). Reply
  • JoeyJoJo123 - Friday, June 09, 2017 - link

    Wasn't actually aware of AutoIt existing. In that case, yes, I do agree that dropping Word/Excel automation, which could otherwise be done via AutoIt is short-sighted. Reply
  • FMJarnis - Thursday, June 22, 2017 - link

    PCMark 8 includes this exact thing as a separate application test. Implementing one is not that difficult. It is likely that a separate test for MS Office is added to PCMark 10 at a later date.

    But requiring everyone who wants to run PCMark 10 to first buy and install Microsoft Office is not really feasible as it is commercial software. For the baseline tests that should be usable without any other prerequisites, what you propose unfortunately doesn't work.
    Reply
  • FMJarnis - Thursday, June 22, 2017 - link

    HVEC is not used because it is not supported on Windows 7 by the OS.

    Also when PCMark 10 development started, HVEC licensing was... complicated. Software implementations on computers still had per-application licensing fees for the codec. Open source solutions were in a legal limbo. Since then HVEC licensing terms have changed, but it was too late for this benchmark.
    Reply

Log in

Don't have an account? Sign up now