Sorting out sensible JVM memory settings in a production environment can be a tricky business. There are some great articles, blogs and lengthy white-papers out there explaining the details of how the JVM manages memory, but I’ve yet to come across a simple guide that actually helps configure the key JVM startup parameters.

For the impatient: Right down at the bottom of this page I have included a simple tool for helping to set your JVM parameters.

For those who are new to this, I suggest doing some web searches around “java heap”. Other people have done a much better job of explaining the theory behind this.

Whilst the theory is well worth reading, I’m hoping this provides some more practically useful information for people running production Java applications. In this context, I’m dealing with a Tomcat 7 server running on the Oracle JVM with an IO intensive application that performs a lot of in-memory caching.

First up, fire up VisualVM (included with your JDK in the /bin folder) if you haven’t already. This will allow you to interrogate the memory of your running application. If you’re trying to interrogate a remote system, you’ll want to run jstatd in order to get a connection. There’s information elsewhere on the web about how to do this; I’ll probably summarise it in another post later.

That will give you a beautiful screen like this:

The VisualVM Visual GC View

The server in this example has a 6GB heap allocation and 150MB PermGen allocation. The applicable JVM startup parameters are:

-server
-Xms6g
-Xmx6g
-XX:MaxPermSize=150m
-XX:PermSize=150m
-XX:NewRatio=2
-XX:SurvivorRatio=10

To run through these quickly; ‘server’ tells the JVM to run in server mode. This mainly changes the behaviour of the HotSpot compiler. In general things will take a little longer to ‘warm up’, but long-running performance will be improved. There are other implications too, which are well documented elsewhere.

Setting the Xmx and Xms values to be the same, in this instance 6GB, we create a single (hopefully un-fragmented) heap allocation for the virtual machine. It’s this 6GB that then get’s carved up into the areas shown in the screenshot above.

The PermGen space is assigned with MaxPermSize and PermSize. In general you don’t need to allocate much here; this mostly relates to the number of classes that you have loaded; relatively simple applications without loads of library inclusions get away with very small values here. We’re using 150MB in this example, which is more than enough. Again, setting the values the same saves the JVM scaling up and down with usage.

Now for the interesting stuff. The heap is split into Eden Space, Survivor Space and Old Generation. Without going into excruciating detail, objects are created in the Eden space and then moved to the Survivor Space and finally Old Generation. The idea is that the majority of the objects that you create only need to last for a very short period of time (consider an Iterator for example; it only needs to exist for the duration of a single loop). By creating objects in the Eden space, the initial garbage collections on this space are very efficient, since they potentially scrub very large percentages of all objects. Anything which is still in use is promoted to one of the survivor spaces, and then eventually to the Old Generation.

Performing garbage collection on the Old Generation is usually more time consuming, so that’s what you ideally want to avoid when scaling the other spaces.

It’s worth noting that the Survivor Space is actually split into two parts, known as S0 and S1. Only one of these is active at a time, and it receives the objects being tenured from the Eden space.

The JVM settings that you use to define these spaces are the ‘NewRatio’ and the ‘SurvivorRatio’.

The NewRatio is the ratio between the Old Generation and the sum of Survivor Space (both S0 and S1) and Eden Space (in this context, ‘New’ means Survivor and Eden). For example, a NewRatio of 2 will create an Old Generation that is twice the size of Eden plus Survivor (or 2/3 of the heap).

The SurvivorRatio is the ratio of one of the Survivor Spaces and the Eden Space. For example, a Survivor Ratio of 10 will mean that the Eden space is 10 times the size of each of S0 and S1.

So clearly it’s not a simple task to visualise how the heap will be allocated when you’re setting these values.

To that end, I have created this little bit of javascript which you can use to plug in your Xmx (heap size), NewRatio and SurvivorRatio. The resultant heap split is then shown in the table below. All byte values are in MB. Don’t forget to multiple GB values by 1024 to get MB.

I have already put in values from my example above; that is a heap of 6GB with a NewRatio of 2 and a SurvivorRatio of 10. This is actually an unusually high survivor ratio, however having watched our application in production for a few weeks we’ve been able to tune it to our [rather unusual] workload.

Input Values

Resultant Allocations

Please leave a comment if you have any questions – I’d be really interested to hear if this helps to solve your problems.

For anyone interested, I’m http://twitter.com/TimCreswick (@TimCreswick)

With our recent development work with the Yubikey, a one-time password (OTP) token from Yubico we decided that it would be useful to have a Yubikey authentication realm for the Glassfish application server.

A lot of our existing J2EE applications use the JDBC Realm that is included in the Glassfish distribution. The JDBC Realm allows you to configure the authentication of your application to any JDBC back-end accessible by your application server (i.e. enabled by an appropriate JDBC driver library). Typically this will take the form of a MySQL database table with a list of usernames and password hashes.

Here’s an example of the Glassfish v2.1 JDBC Realm configuration template:

Glassfish v2.1 JDBC Realm Configuration

Our custom Yubikey Realm is essentially an extension of the JDBC Realm (although not in the sense of a Java “extends”). The Yubikey Realm supports all the same functionality as the JDBC Realm, so you can drop it in-place with all the same settings and it’s totally backwards compatible.

We’ve added the functionality to specify an additional column in the users database table containing the Base 64 Yubikey Public Identifier (ID) of the Yubikey assigned to that user. This is configured via the yubikey-column property. By default, the Yubikey Realm will authenticate users as normal, unless the password length is >= 32 characters. In this instance, the password is assumed to be a Yubikey OTP, and the Realm authenticates accordingly.

We have the additional parameters yubikey-force and yubikey-auth-url, which respectively control whether all logins are treated as Yubikey OTPs, and which validation server you would like to use.

Here’s an example of the realm configuration screen:

Yubikey Authentication Realm Configuration

…and there we have it. Instant in-place implementation of Yubikey authenticated logins for all of our existing J2EE applications without changing any application code. Obviously it’s worth noting that in this context we’re using the Yubikey OTP as a convenient password replacement, not as a 2^nd authentication factor.

Please feel free to contact me if you’re interested in using our authentication realm; we may consider publishing it online in due course, along with the installation instructions and a few enhancements that are pending.

The initial requirement for our client project is 2F protection of all remote-access to their systems, which presently only amounts to VPN connections and webmail access via Outlook Web Access (OWA) provided by Exchange 2007.

We’ve now finished implementing this, and the diagram below shows how we protected the OWA service (I might cover VPN in a later post). Naturally there are a wide variety of ways to provide 2F authentication for OWA, but I think this presents the best end-user experience:

Authentication Process Overview

First, a quick overview of the highlights:

• We implemented a domain-joined ISA Server 2006 Standard.
• We customised the default 2F ISA Server OWA login wrapper. By default it requests Username, OTP (one-time password) and finally the “internal” password (i.e. the domain password). We thought this was a little counter-intuitive for our users, since the OTP is the new credential being requested (in addition to the usal username/password combination) we thought it should be the last field on the form. Also, our Yubikeys are configured to emulate an ‘enter’ keypress after populating the field, which means that the Yubikey OTP needs to be the last field filled out by the user since it causes the form to be submitted. Finally, the default ISA OWA login form contains an option to add a second username for internal use. We removed this since we don’t have different usernames for remote access.
• We modified the Active Directory Schema to include a new field on the User object, contianing the user’s Yubikey Public Indentifier. In this way we are able to verify that the Yubikey used to generate the OTP matches the Yubikey assigned to the AD user. This appears to be a unique solution; all the other implementations that I’ve seen require a separate mapping table of user accounts to Yubikeys, usually on the RADIUS server. This is annoying, since it means that you’ve got another system to modify every time a new key is deployed.
• All the FreeRADIUS modules that we used are entirely custom in-house code. Their function is described in detail below.

Examining our final implementation step by step:

1. We present our custom ISA / OWA login form to the user over a secure HTTPS connection, where we collect the AD username, AD password and their Yubikey OTP.
2. The ISA Server is configured to use RADIUS-backed OTP authentication for this service entry, which means that it sends the username and OTP to our FreeRADIUS server (note that the AD password isn’t sent, since we don’t need that).
3. On the FreeRADIUS Server we have a custom validation module that communicates with the Domain Controller via LDAPS (secure LDAP) in order to confirm that the user exists in the Active Directory (AD). The module allows us to specify particular CNs and OUs in the LDAP tree to search within.
4. Once the user has been located in the AD, the LDAP function retrieves the Yubikey Public ID of the Yubikey assigned to the user being authenticated. As mentioned earlier, we store this mapping in a custom AD field that we added to the schema. This Yubikey ID is compared with the ID of the key used to generate the OTP to ensure that the key is authorised for that user (for those unfamiliar with the Yubikey, the public ID can be trivially extracted from the OTP).
5. Next, the FreeRADIUS module communicates with our own Yubikey Validation Server running on Glassfish (we could easily switch this to another server) using the standard Yubikey validation API. This ensures that the submitted OTP is valid.
6. Assuming all of the above was successful the RADIUS server sends an ‘Accept’ response back to the ISA Server. To clarify, the following must be true:
   1. The user must exist in one of the configured AD OUs or CNs.
   2. The user must have a Yubikey ID assigned in their AD record.
   3. The submitted Yubikey ID must match the ID stored in the AD.
   4. The OTP must validate successfully against the validation service.
7. Finally, if the RADIUS response was positive, the ISA server provides a delegated login to the OWA service. This means that the user doesn’t need to login a second time, so it’s essentially a form of SSO (single sign on). In order for this to work, the OWA service on your Exchange CAS server must be configured for NTLM (or basic) authentication as opposed to forms authentication.

Hopefully I have provided a coherent overview of our solution. I will try and follow this up with more details on the custom ISA forms, the LDAP schema modifications and FreeRADIUS integration in future posts.

As always, feedback is welcome!

Here’s my response as posted to the list:

Hi Christoph.

Several reasons, I’ll try and outline those that I can recall:

1. We found the performance of NFS to be far better than iSCSI, although we probably hadn’t spent sufficient time tweaking the iSCSI configuration. NB: we were exporting ZFS block devices as LUNs to virtual machines as RDMs (raw device mappings) as opposed to formatting them as VMFS, so the comparison isn’t perhaps completely fair.

2. Overall the administration of iSCSI was overcomplicated, and prone to human error. Training our people to administer and troubleshoot it was likely to be too costly, and the additional ongoing costs associated with the increased administration have to be considered.

3. Similar to point (2) above, we found that you had to be careful to keep track of IQNs and LUN Ids when mapping in VMware, it’s only possible to put names on targets, not LUNs, so when mapping multiple disks into a VM the process was potentially error-prone.

4. The provisioning of iSCSI storage between ESXi and COMSTAR *felt* a little bit unstable; we had numerous instances of lengthy HBA rescans, targets appearing and disappearing which although they were all explained within ‘expected behaviour’ were either a little counter-intuitive or too time-consuming.

5. My understanding of the nature of ESX NFS connections is that less IO blocking takes place, which explains why so many people get better throughput when running multiple VMs against the same SAN connection.

6. The limit of 48 NFS mappings in ESX wasn’t going to be a constraint for us for the foreseeable future; we rarely load more than 15 VMs per node. The flexibility of creating a separate ZFS filesystem for each VM and exporting it separately over NFS was therefore possible.

7. Everyone understands files; when you login to the Solaris system you can navigate to the ZFS filesystem and see .vmx and .vmdk files – these are nice and simple to manage, clone, backup, export etc etc.

8. Related to point (7), when you use NFS in this way the virtual machine configuration is stored alongside the rest of the virtual machine data. This means that a snapshot of the ZFS filesystem for that VM gives us a true complete backup at that point in time.

I think that just about covers it. Essentially, although iSCSI feels like a more clever way of solving our problem, and certainly it poses as a more ‘enterprise’ solution, it really was a case of overcomplicating the solution.

Several times in the last few years we’ve found that simple is better, even if it doesn’t satisfy one’s desire to implement the ‘technically perfect solution’. It’s more a question of balancing the economics (I’m running a business, after all) with the actual requirements. Oftentimes just because you *can* do something doesn’t mean you should, and I’ve often found that PERCEIVED requirements can out-grow ACTUAL requirements just because some technology exists to solve problems that you don’t have [yet].

In short, I like to keep it fit for purpose, even if it does feel like a more agricultural solution.

Finally, I should add that the one major shortcoming with our NFS solution is the lack of any equivalent to the iSCSI multipathing. If we had any machines that required true high availability or automated failover this would probably have negated all of the points above – iSCSI multipathing is a beautiful thing, and it creates some awesome possibilities for fault tolerance.

As it stands, we take care of link failure at the network level (as opposed to the iSCSI MPIO protocol level) and deal with ESX node or storage node failure by manually remapping NFS filesystems from elsewhere. This is actually preferable to automated recovery since sometimes we don’t want to take the ‘default’ action during a failure scenario. By having a carefully documented failure plan I believe we have more flexibility, and can deal with recovery on a per-client basis, rather than a system-wide basis.

Ultimately we are a business dealing with multiple clients hosted on shared hardware, so it’s important to keep our implementations client-centric, rather than system-centric.

Finally, I should add that although we don’t have automated failover of these systems, our solution does still permit us to stay well within our contracted SLAs, which serves the business need.

Regards,

Timothy Creswick

I found it very hard to find how well these pre-formatted PDFs would display on the device, with a lot of people saying that they were barely legible. Obviously with most of these documents you don’t want to spend time reformatting them for the Reader’s screen, and zooming is cumbersome and often breaks the formatting.

Other people’s posts notwithstanding I went ahead and bought one.

The good news is that I find it perfectly usable, and although the rendered font size ends up being rather smaller than you might choose, text and diagrams are rendered very clearly. In case it helps anyone else deciding to buy one of these devices for this kind of work, I’ve included some photos below.

First, a regular eBook page rendered in “Small” (click for hi-res):

An eBook

Then a page from a Sun technical manual (click for hi-res):

A regular PDF

Finally, here’s a close up including some very small text, all of which I find perfectly fine for my purposes. Again, click for a larger version, and apologies for the lens distortion which I haven’t bothered to correct:

Regular PDF Close-up

A better PowerShell prompt

In order to achieve this, I have the following script:

function prompt { Write-Host("%") -nonewline -foregroundcolor Yellow; return " "; }
c:
cd Scripts
" ";

You’ll also note that I change the default working directory to C:\Scripts, which is where I keep all my PowerShell scripts and saves me a little extra time.

In order to implement this, you’ll need to create the folder C:\Users\{USERNAME}\Documents\WindowsPowerShell and save the file as profile.ps1.

To make provisioning new LUNs as simple as possible I’ve written an interactive script that first creates a ZFS backing store, then creates the LU for STMF, gives you the option of creating a new iSCSI target or using an existing one, and finally attaches the LU to the selected target.

Here’s a sample output from the script:

-----------------------------------------------------------------
ZFS / iSCSI / COMSTAR Simple Management Toolset
(c)Copyright 2009 Thoughtspace Ltd. Developed by Timothy Creswick
-----------------------------------------------------------------

Please specify the name for the new ZFS backing store.
Your new store will be created inside mpool/clients/iscsi

Existing backing stores are:
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
mpool/clients/iscsi                  271G   973G    19K  /mpool/clients/iscsi
mpool/clients/iscsi/testing           24K   973G    24K  -
mpool/clients/iscsi/testing-three     24K   973G    24K  -
mpool/clients/iscsi/testing-two       24K   973G    24K  -

New backing store: mpool/clients/iscsi/test

Should the backing store be created as a sparse volume? [Yes/No] no

Please enter the size of the backing store: 10GB

Will create a ZFS store with the following details:
  Name:    mpool/clients/iscsi/test
  Size:    10GB
  Sparse:  No

Proceed? [Yes/No] y

Creating ZFS backing store... OK

Creating the STMF LU... OK (GUID is 600144f094ab000000004aa29327000b)

You can either attach this LU / device to an existing iSCSI target as a new LUN,
or you can create a new iSCSI target and STMF target group for this ZFS backing
store.

In general, you want to have a single iSCSI target per Virtual Machine, so if the
virtual machine requires multiple disks, then you should attach each additional
disk as a new LUN on an existing target.

If this is the first disk that you are attaching via iSCSI to the Virtual Machine
you should create a new iSCSI / STMF target.

Create a new iSCSI target? [Yes/No] y

Creating iSCSI target... OK (iqn.1986-03.com.sun:02:2fff96a8-63b2-48bd-b447-bda2a1fc9ead)

Creating STMF target group... OK

Assigning new iSCSI target to target group:
  Waiting for STMF service to terminate.... OK
  Adding TG member... OK
  Starting STMF service.... OK

Adding LU view to target group... OK

All done. Your ZFS device should be available on target iqn.1986-03.com.sun:02:2fff96a8-63b2-48bd-b447-bda2a1fc9ead.

You can get a copy of the script here. Please feel free to use and modify – I’d be interested in any feedback.

Here’s one of my frequently used nmap commands – very simple, but gives you a quick list of IP addresses for all of the ProCurve switches on your network.

nmap -n -sP 10.0.0.0/24 | grep --before-context=1 "ProCurve Networking by HP" | grep "Host" | cut -f2 -d' '

Just change the subnet range to match your network. Note: this relies on doing vendor OUI lookups on device MAC addresses, which means that the scanning node needs to be on the same Layer 2 network segment for this to work.

One of the things that surprised me was the presence of a GPS transciever built into the laptop as part of the HSPA 3G wireless card. Viewing in Windows Device Manager it appears as a component connected to the USB sub-interface on the card:

Device Manager Screenshot

The interface to the GPS unit is simple NMEA over the virtual COM port 7. By default there’s not much you can do with this – Google Earth will interface pretty much out of the box, but not much else.

I decided it would be fun to write a little Java application to map the current position of the laptop, as given by the GPS unit.

I looked around a bit and found JXMapKit, which is part of the swingx project. Using some of this tutorial to get me started, I had a basic mapping application up and running in about 20 minutes.

The next step was to interface with the GPS unit. This is where Java falls down a little, espcially on Windows machines. In the end I settled for the RXTX library which uses native implementation (JNI) to provide access to serial devices from within a Java app.

The Latitude’s GPS unit spews out location information on the COM port as it becomes available, so all you need to do is ‘listen’, and then handle the stream in an event-driven manner.

A sample GGA ‘fix’ sentence looks like this:

$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47

I couldn’t find any NMEA parsing libraries for Java online, so another 2 hours and I’d written my own. There’s a core class which validates the NMEA sentence and then invokes a specific handler class depending on the NMEA sentence type. For example, GGA (fix) sentences are handled by NmeaLocationMessage, GSA (overall satellite data) sentences are handled by NmeaSatelliteGeneralDataMessage, GSV (detailed satellite data) sentences are handled by NmeaSatelliteDetailMessage and so on.

Each of these ‘handler’ classes contains the logic to further parse the specific sentence type. Just to get up and running I only implemented handlers for GGA, GSA, RMC and GSV messages, which were more than enough to get a fix on the map.

Finally I wrote the code to invoke the NMEA parser on serial port events. The core event handling code looks like this:

NmeaMessage message = new NmeaParser().parseSentence(line);

if (NmeaLocationMessage.class.isInstance(message)) {
	NmeaLocationMessage location = (NmeaLocationMessage) message;

	if (location.getLatitude() != lastLat || location.getLongitude() != lastLon) {

		lastLat = location.getLatitude();
		lastLon = location.getLongitude();

		map.setAddressLocation(
				new GeoPosition(
				location.getLatitude(),
				location.getLongitude()));

	}
}

Nice and simple!

The net result is that everytime the GPS reports a new Latitude or Longitude, the map is re-centered accordingly.

So finally, here’s a location plot as I’m sitting on a train at London Paddington Station waiting to depart:

GPS Tracker Screenshot

I wrote this really for a bit of fun, since clearly it’s not massively practical. If you have an E6400 with GPS, or a similar notebook and would like to play around with it please let me know and I’ll gladly send you the app and/or sourcecode.