Sunday, August 14, 2011

Leaving Blogger

All, after a few years of publishing my blog on the Blogger service provided by Google, I've decided to move to a paid blog with my own domain name hosted by Wordpress. Whilst Blogger is a great free blogging service, lately I have been reading about Google deleting people's accounts, for non-fee paying users Google has little accountability and it is difficult - if at all possible - to restore accounts later.

Please point your browsers here for my regular RSS digest and new and original tehnical articles.

Friday, June 10, 2011

RSS Digest: Week Ending 10-Jun-2011

Java

Java SE 6 Update 26 is released!
http://blogs.oracle.com/javase/entry/java_se_6_update_26

Moving to OpenJDK as the official Java SE 7 Reference Implementation
http://blogs.oracle.com/henrik/entry/moving_to_openjdk_as_the


Technology

Google Publishes C++, Go, Java and Scala Performance Benchmarks
http://www.readwriteweb.com/hack/2011/06/cpp-go-java-scala-performance-benchmark.php

Windows 8: could it be more than lipstick on a pig?
http://www.pcpro.co.uk/features/367813/windows-8-could-it-be-more-than-lipstick-on-a-pig

Apple Introduces iCloud
http://www.apple.com/uk/pr/library/2011/06/06icloud.html

New Version of iOS Includes Notification Center, iMessage, Newsstand, Twitter Integration Among 200 New Features
http://www.apple.com/uk/pr/library/2011/06/06ios.html

Iomega Home Media Network Hard Drive Cloud Edition 2TB
http://www.pcpro.co.uk/reviews/storage-appliances/367861/iomega-home-media-network-hard-drive-cloud-edition-2tb

Will 10 GigE Finally Go Big in HPC?
http://insidehpc.com/2011/06/08/will-10-gige-finally-go-big-in-hpc/

Friday, April 8, 2011

RSS Digest: Week Ending 08-Apr-2011

Java

DZone Meets The Well-Grounded Java Developers
http://java.dzone.com/articles/dzone-meets-well-grounded-java

Next step on the road...
http://nighthacks.com/roller/jag/entry/next_step_on_the_road

Java founder Gosling joins Google
http://www.javaworld.com/javaworld/jw-03-2011/110328-gosling-joins-google.htm

Cliff Click in "A JVM Does What?"
http://jeremymanson.blogspot.com/2011/04/cliff-click-in-jvm-does-what.html


UNIX/Linux

My Bash Profile - Part V: Prompt
http://fahdshariff.blogspot.com/2011/03/my-bash-profile-part-v-prompt.html

My vimrc
http://fahdshariff.blogspot.com/2011/04/my-vimrc.html


Technology

Informatica, HP, and Mellanox/Voltaire Benchmark Report
http://www.a-teamgroup.com/article/informatica-hp-and-mellanoxvoltaire-benchmark-report/

AMD: quad-core Llano outshines Sandy Bridge
http://www.pcpro.co.uk/news/366544/amd-quad-core-llano-outshines-sandy-bridge

Evaluating Multi-core for HPC
http://sicorp.com/blogs/75-evaluating-multi-core-for-hpc

Dell XPS 15 (2011) review
http://cdn.pcpro.co.uk/reviews/laptops/366409/dell-xps-15-2011

Samsung SyncMaster C27A750 review
http://cdn.pcpro.co.uk/reviews/monitors/366613/samsung-syncmaster-c27a750

Intel unveils next-generation Xeons with ten cores
http://cdn.pcpro.co.uk/news/enterprise/366562/intel-unveils-next-generation-xeons-with-ten-cores

Friday, March 25, 2011

RSS Digest: Week Ending 25-Mar-2011

Java

The Performance Paradox of the JVM: Why More Hardware Means More Failures
http://docs.media.bitpipe.com/io_25x/io_25445/item_394514/Azul_TheServerSide_SO%23033729_Expert-Tip_031611.pdf

Java Tip: Find process id of running Java process
http://boxysystems.com/index.php/java-tip-find-process-id-of-running-java-process/

The impact of Garbage Collection on Java performance
http://blog.dynatrace.com/2011/03/24/the-impact-of-garbage-collection-on-java-performance/


UNIX/Linux

My Bash Profile - Part IV: Functions
http://fahdshariff.blogspot.com/2011/03/my-bash-profile-part-iv-functions.html


Technology

May’s Law and Parallel Software
http://www.linux-mag.com/id/8422/

Apple iPad 2 review
http://www.pcpro.co.uk/reviews/tablets/366115/apple-ipad-2

Satellite snaps extent of Japan's post-quake blackout
http://www.newscientist.com/blogs/shortsharpscience/2011/03/extent-of-post-quake-blackout.html

New Intel Turbo Boost Technology
http://drdobbs.com/blogs/high-performance-computing/229301335

Nintendo 3DS: the essential guide
http://www.guardian.co.uk/technology/gamesblog/2011/mar/22/nintendo-3ds-review

Firefox 4 finally ready to roll
http://www.pcpro.co.uk/news/366235/firefox-4-finally-ready-to-roll

iPad 2 Arrives in 25 More Countries This Friday
http://www.apple.com/uk/pr/library/2011/03/22ipad.html

CUDAfy - General Purpose GPU Development on .NET
http://insidehpc.com/2011/03/23/cudafy-general-purpose-gpu-development-on-net/

Friday, March 4, 2011

RSS Digest: Week Ending 04-Mar-2011

Java

Java Bytecode Fundamentals: Using Objects and Calling Methods
http://www.zeroturnaround.com/blog/java-bytecode-fundamentals-using-objects-and-calling-methods/

Performance tuning @ JFokus
http://kirk.blog-city.com/performance_tuning__jfokus.htm

The Azul Garbage Collector
http://www.infoq.com/articles/azul_gc_in_detail

Nondeterministic Floating-Point Conversions in Java
http://www.exploringbinary.com/nondeterministic-floating-point-conversions-in-java/

Java Synchronization (Mutual Exclusion) Benchmark
http://www.baptiste-wicht.com/2010/09/java-synchronization-mutual-exclusion-benchmark/

JavaOne and Oracle Develop 2010
http://www.oracle.com/us/javaonedevelop/index.html

JVM Language Summit 2010
http://wiki.jvmlangsummit.com/Main_Page


Technology

Thunderbolt: The Fastest Data Connection to Your PC Just Arrived
http://newsroom.intel.com/community/intel_newsroom/blog/2011/02/24/thunderbolt-technology-the-fastest-data-connection-to-your-pc-just-arrived

The essential tools for IT pros in 2011
http://www.pcpro.co.uk/realworld/365539/the-essential-tools-for-it-pros-in-2011

How long does it take to make a context switch?
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html

Apple iPad 2 review: first look
http://www.pcpro.co.uk/blogs/2011/03/02/apple-ipad-2-review-first-look/

Apple staffer: wait for iPad 3
http://www.pcpro.co.uk/news/365659/apple-staffer-wait-for-ipad-3


Football

I'm a romantic, says Xavi, heartbeat of Barcelona and Spain
http://www.guardian.co.uk/football/2011/feb/11/xavi-barcelona-spain-interview

Friday, January 28, 2011

RSS Digest: Week Ending 28-Jan-2011

Java

The Future Of Java
http://blogs.forrester.com/john_r_rymer/11-01-23-the_future_of_java

Looking Ahead to Java SE 7 and 8: A Discussion with Oracle’s Java Language Architect, Brian Goetz
http://www.oracle.com/technetwork/articles/java/briangoetzchief-188795.html


Programming

TIOBE Programming Community Index for January 2011
http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html


Finance

A Day in the Life of an Emerging Market Trader
http://www.quantnet.com/day-life-emerging-market-trader/

Financial crisis was 'avoidable', says official US report
http://www.guardian.co.uk/business/2011/jan/27/financial-crisis-avoidable-fcic-report

Friday, January 21, 2011

Friday, January 7, 2011

Inspecting HotSpot JVM Options

Introduction

A method for inspecting the comprehensive set of HotSpot JVM options and some examples comparing different outputs.


Background

The Oracle JVM provides an enormous spectrum of options to control the runtime JVM. These options are given a criminally short and non-comprehensive discussion at the OTN here. OTN divides these options into three categories: Behavioural, Performance and Debugging. This is a useful abstraction but some examples Oracle gives in each category can be misleading due to the overlap between Behavioural and Performance.

The JVM options themselves can be controlled in a number of ways:
  • via the command line on JVM startup.
  • via JMX for certain options where this is allowed.
  • indirectly via the command line by a super-option which then sets other options.
  • automatically by the JVM. The JVM has ergonomic capability to detect features of the host and set options accordingly.
Using the first two methods we are explicitly setting options ourselves and can easily track what value each option has (true, false, 20, 100 etc). These last two methods for setting options can cause developers the most confusion, as we have less insight and therefore less understanding over what options are being set. This puts us in an unfavourable position when explaining the runtime behaviour of our applications. To dump out every JVM option and its value there is an option to do this hidden away in the JVM source code: 

-XX:+PrintFlagsFinal


Usage

Let's see this in action! First let's run the following:


$ jdk1.6.0_23/bin/java -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

 (I have added -XX:+UnlockDiagnosticVMOptions as this unlocks more options for viewing and control. Also I have added -version at the end so Java doesn't complain that it has nothing to run and to supplement the results).

The sysout is quite large so I've saved it here. Firstly you may be surprised at the sheer quantity of JVM options available: 717! That's a lot of configuration! Before we dive into the options lets look at our host specification and the output of the -version option. The host has 8 dual core 64bit AMD Opteron processors running at 3.2GHz so 16 cores in total with 64GB of RAM. The Java -version output shows:

java version "1.6.0_23" 
Java(TM) SE Runtime Environment (build 1.6.0_23-b05) 
Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)

this tells us we are running Java 6 update 23 using the Server version of the HotSpot VM in 32bit mode. Now let's look closer at the output. For each option we can see the following columns:
  1. Datatype. bool for booleans, uintx for unsigned integers etc
  2. Name
  3. Assignment. = means this is the default. := means this option was modified from the default value by command line or ergonomics.
  4. Value
  5. Category. There are two large categories diagnostic or product. (I find this a better grouping than Behavioural, Performance and Debugging.) and also manageable if this option can be set via JMX. C1 relates to the JIT compiler used in the Client JVM. C2 means the same but for the Server JVM and I suspect pd means Platform Dependent as most pd options are Solaris only.

Let's look closer at the modified options which have the := assignment:

uintx InitialHeapSize                     := 67108864         {product} 
uintx MaxHeapSize                         := 1073741824       {product} 
uintx ParallelGCThreads                   := 13               {product} 
bool PrintFlagsFinal                     := true             {product} 
bool UnlockDiagnosticVMOptions           := true             {diagnostic} 
bool UseParallelGC                       := true             {product}

apart from the two flags which I set myself on the command line we can see Java ergonomics has given the heap a sensible range of 64MB to 1GB. The fairly old Parallel Garbage Collector has been chosen with an enormous 13 threads spawned to do the collection. Straightaway we've got a lot more insight into how our JVM is configured and a few ideas for improvements.


Client vs Server JVM Analysis 

We know from looking at the -version output Java automatically chose the server JVM instead of the client JVM (Java automatically chooses server if it detects 64bit processors). But how do we know what options are set because Java chose the Server JVM? What options were set because of our host specification? Lets run the following commands and review the output:


$ jdk1.6.0_23/bin/java -client -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version
$ jdk1.6.0_23/bin/java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

I have saved the client output here and the diff client server here. Firstly lets look at the output from the Client JVM; it has 661 options, 56 fewer than the Server version. Now let's look solely at the Client options which have been modified, excluding the two options I set on the command line there are only two values which were changed:

uintx InitialHeapSize                     := 16777216         {product} 
uintx MaxHeapSize                         := 268435456        {product}

so Java has chosen some fairly sensible values for the Client JVM: 16MB initial heap size with a max size of 256MB. There are many entries in the diff file mostly by virtue of the extra options available to the Server JVM or options which relate to memory simply having higher values. Here are the more interesting differences:

<  intx CICompilerCount                      = 1                {product} 
>  intx CICompilerCount                      = 2                {product} 
<  intx CompileThreshold                     = 1500             {pd product} 
>  intx CompileThreshold                     = 10000            {pd product}

 The Client VM waits only 1,500 invocations before compiling a method to native code and uses only one thread for compilation vs the Server which waits for 10,000 invocations and uses two threads.

>  bool DoEscapeAnalysis                     = true             {C2 product}

 In this build the Server VM has escape analysis enabled by default whereas the Client VM cannot enable escape analysis at all. I don't think this is sensible on Oracle's part as whilst escape analysis can improve performance for scientific computing (where many primitives are created within a method and then discarded once a final value has been computed) for other applications escape analysis can lower performance when arguments do escape their methods and are better left on the heap. Slipping in such a change transparently can also be problematic as developers upgrading to a new JVM may see performance degradation in their applications when no visible command line option has changed. Developers should benchmark accordingly when upgrading JVM builds.

<  bool RewriteBytecodes                     = false            {pd product} 
<  bool RewriteFrequentPairs                 = false            {pd product} 
>  bool RewriteBytecodes                     = true             {pd product} 
>  bool RewriteFrequentPairs                 = true             {pd product}

 An optimisation to rewrite two bytecode operations as one operation.


Super Option Activation

As mentioned earlier some options are super-options and will enable other options. Let's take a look at a common super option: -XX:+AggressiveOpts

The two commands are:

jdk1.6.0_23/bin/java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version
jdk1.6.0_23/bin/java -server -XX:+AggressiveOpts -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

here are all the differences:


<  bool AggressiveOpts                       = false            {product} 
>  bool AggressiveOpts                      := true             {product} 
<  intx AutoBoxCacheMax                      = 128              {C2 product} 
>  intx AutoBoxCacheMax                      = 20000            {C2 product} 
<  intx BiasedLockingStartupDelay            = 4000             {product} 
>  intx BiasedLockingStartupDelay            = 500              {product} 
<  bool EliminateAutoBox                     = false            {C2 diagnostic} 
>  bool EliminateAutoBox                     = true             {C2 diagnostic} 
<  bool OptimizeFill                         = false            {C2 product} 
>  bool OptimizeFill                         = true             {C2 product} 
<  bool OptimizeStringConcat                 = false            {C2 product} 
>  bool OptimizeStringConcat                 = true             {C2 product}

So a number of options have been enable and some others have been given more aggressive values. We know -XX:+AggressiveOpts changes with each JVM build. Let's look at an older build:

(NB. Java6u20 is the oldest build to support this form of option dumping)

jdk1.6.0_20/bin/java -server -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version
jdk1.6.0_20/bin/java -server -XX:+AggressiveOpts -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal


<  bool AggressiveOpts                       = false            {product} 
>  bool AggressiveOpts                      := true             {product} 
<  intx AutoBoxCacheMax                      = 128              {C2 product} 
>  intx AutoBoxCacheMax                      = 20000            {C2 product} 
<  intx BiasedLockingStartupDelay            = 4000             {product} 
>  intx BiasedLockingStartupDelay            = 500              {product} 
<  bool DoEscapeAnalysis                     = false            {C2 product} 
>  bool DoEscapeAnalysis                     = true             {C2 product} 
<  bool EliminateAutoBox                     = false            {C2 diagnostic} 
>  bool EliminateAutoBox                     = true             {C2 diagnostic} 
<  bool UseLoopPredicate                     = false            {C2 product} 
>  bool UseLoopPredicate                     = true             {C2 product}

We can see in 6u20 DoEscapeAnalysis was an AggressiveOpt but in the new build 6u23 DoEscapeAnalysis became the default (as did UseLoopPredicate). Along the same lines OptimizeFill and OptimizeStringConcat may become default values in future builds.


Conclusion

There is enormous scope for coarse and fine grained control of JVM behaviour. The option -XX:+PrintFlagsFinal allows comprehensive reporting of the options and their values. The available options vary by build and JVM type (server or client). Recording and auditing this output is an important step in any Java benchmarking or continuous monitoring exercise. Oracle can surreptitiously enable options in new Java builds which may cause inexplicable changes in performance and behaviour of existing applications.

RSS Digest: Week Ending 07-Jan-2011

Java

Java Bytecode Fundamentals
http://arhipov.blogspot.com/2011/01/java-bytecode-fundamentals.html


Linux

What's new in Linux 2.6.37
http://www.h-online.com/open/features/What-s-new-in-Linux-2-6-37-1163376.html


Technology

2nd UK GPU Computing Conference, December 13-14 2010, Cambridge http://www.many-core.group.cam.ac.uk/ukgpucc2/programme.shtml

CES: Sony's 3D visor - a hit, or garish headgear?
http://www.newscientist.com/blogs/shortsharpscience/2011/01/ces-sonys-3d-visor---a-hit-or.html



Finance

Quantnet's Best-selling Books of 2010
http://www.quantnet.com/forum/showthread.php?t=8309

Bloomberg: 100 Richest Hedge Funds
http://www.quantnet.com/forum/attachment.php?attachmentid=3834&d=1294353871

Tuesday, January 4, 2011

RSS Digest: Week Ending 31-Dec-2010

Java

Java concurrency bug patterns for multicore systems
http://www.ibm.com/developerworks/java/library/j-concurrencybugpatterns/

Josh Bloch on Java and Programming
http://www.infoq.com/interviews/josh-bloch-java-prog

Java SE & Java EE Performance: A Lot Of Garbage
http://ua.sun.com/java/Performance_SRitter.pdf

A Comparative Study of JVM Implementations with SPECjvm2008
http://web-ext.u-aizu.ac.jp/~hitoshi/RESEARCH/iccea2010.pdf

Azul's Pauseless Garbage Collector
http://www.artima.com/lejava/articles/azul_pauseless_gc.html

NUMA-Aware Java Heaps for Server Applications
http://www.cs.umd.edu/~hollings/papers/ipdps05.pdf

Help for the NUMA Weary
http://blogs.sun.com/jonthecollector/entry/help_for_the_numa_weary

Java Knobs: How to Change Performance of the Sun JVM
http://www4.java.no/presentations/javazone/2007/slides/5334.pdf

Do Java 6 threading optimizations actually work?
http://www.infoq.com/articles/java-threading-optimizations-p1


Other

Scientific Computing on Heterogeneous Architectures
http://babrodtk.at.ifi.uio.no/files/publications/brodtkorb_phd_thesis.pdf

Google Body Browser
http://bodybrowser.googlelabs.com/
(requires a WebGL browser)

The disposable academic - Why doing a PhD is often a waste of time
http://www.economist.com/node/17723223?story_id=17723223

Sick of reading? Five cool things you can do with your Amazon Kindle
http://www.pcpro.co.uk/blogs/2010/12/27/sick-of-reading-five-cool-things-you-can-do-with-your-amazon-kindle/

Intel's Sandy Bridge Architecture Exposed
http://www.anandtech.com/show/3922/intels-sandy-bridge-architecture-exposed