Manually debugging the SugarCRM REST API with cURL

The SugarCRM REST API is particularly difficult to debug; it’s full of bugs and inconsistencies in behaviour, and in general a malformed request simply returns an HTTP 200 code with the data string “null”. Nothing useful is posted in the application log.

To that end, I have found it essential to have cURL at my disposal to quickly push different query types into Sugar, in order to figure out which fields or parts of my API submission are causing problems.

I start by creating a couple of files containing the parameter and JSON data that I want to submit. First, we need to authenticate with the API, so we create a file like this one:

   "application_name":"Custom API"

This can then be submitted to the server like so:

curl -d @login.txt http://<hostname>/serivce/v4_1/rest.php

Note that in this instance, the @ in the curl parameter means that we’re specifying a file to sumbit as data. If we wanted to submit a field directly, that would be done like this:

curl -d method=login -d input_type=json -d response_type=json -d @rest-data.txt http://<hostname>/serivce/v4_1/rest.php

In order to get this format the JSON response nicely, I like to use the python JSON tool:

curl -d @login.txt http://<hostname>/dev/service/v4_1/rest.php | python -mjson.tool

This provides a nicely formatted response like this:

    "id": "88s7e18fbbrt5muo1mig1eov12",
    "module_name": "Users",
    "name_value_list": {
        "mobile_max_list_entries": {
            "name": "mobile_max_list_entries",
            "value": 10
        "mobile_max_subpanel_entries": {
            "name": "mobile_max_subpanel_entries",
            "value": 3
        "user_currency_id": {
            "name": "user_currency_id",
            "value": "-99"
        "user_currency_name": {
            "name": "user_currency_name",
            "value": "British Sterling"
        "user_decimal_seperator": {
            "name": "user_decimal_seperator",
            "value": "."
        "user_default_dateformat": {
            "name": "user_default_dateformat",
            "value": "d.m.Y"
        "user_default_team_id": {
            "name": "user_default_team_id",
            "value": "1"
        "user_default_timeformat": {
            "name": "user_default_timeformat",
            "value": "h:ia"
        "user_id": {
            "name": "user_id",
            "value": "4a4f2d1f-cbd4-b6b8-e3f1-102a322c6574"
        "user_is_admin": {
            "name": "user_is_admin",
            "value": true
        "user_language": {
            "name": "user_language",
            "value": "en_UK"
        "user_name": {
            "name": "user_name",
            "value": "api-test-user"
        "user_number_seperator": {
            "name": "user_number_seperator",
            "value": ","

The first part of this (the “id”) is the session ID which is required for subsequent queries.

Now you’re ready to submit general queries to the API, for example:


This will fetch the account with ID 76d9f1ab-312e-c673-d542-102a322c6574.

If you want to create an entry, you can test it like this:

         "value":"2012-11-26 19:36:02"
         "value":"Test call via API"
         "value":"This is the description"

I hope this is helpful!

Posted in APIs, Linux, PHP, SugarCRM | Leave a comment

Quickly see how many of your users are on Gravatar

A couple of the companies that I work with have wanted to evaluate the business case for integrating with Gravatar. In order to do this, we wanted to know how many of our users had Gravatar profiles, so I built this very simple PHP script which reads an input file containing email addresses (one per line), and outputs whether or not a Gravatar profile was found.

If you have PHP-CLI installed, you can simple execute this from a bash console.


# This is the list of email addresses
# that we're going to parse. Should be
# one per line.
$handle = fopen("uniques.txt", "r");

# Print a clear line
echo "\n";

# Iterate over each line in the file
while (!feof($handle)) {

  # Per Gravatar spec, trim the whitespace and
  # convert all to lowercase.
  $address = strtolower(trim(fgets($handle)));

  # Take the MD5 hash of the address
  $hash = md5($address);
  echo "$address,$hash,";

  # Perform the web lookup
  $web = curl_init("${hash}.xml");
  curl_setopt($web, CURLOPT_RETURNTRANSFER, TRUE);
  $response = curl_exec($web);

  # Get the response code from the server.
  # For purposes of this process we don't care
  # about the contents of the result.
  $http_code = curl_getinfo($web, CURLINFO_HTTP_CODE);
  switch ($http_code) {
  case 200:
    # Response was 200, so print out the URL for
    # the profile.
    echo "${hash}\n";
  case 404:
    # API response was 404, so no gravatar exists
    echo "NOT_FOUND\n";
    # Not one of the expected return codes from
    # the API server. Print an error.
    echo "ERROR($http_code)\n";



I hope you find this helpful!

Posted in APIs, General, Linux, PHP | 2 Comments

Email files directly from the Linux console

It’s not a great habit to get into, but sometimes it can be very quick and convenient to email a file to yourself directly from a bash SSH console.

Note: Because this uses email, this method isn’t secure. Think carefully about the contents of what you’re sending before you email files directly from your system. Even /etc/ configuration files can give away too much about your system.

Personally, I like to use the mailx utility in Debian to email from the console; it’s an alternative to sendmail, and it’s less fussy about some of the parameter format.

If you haven’t got mailx, you can install it with:

apt-get install mailx

If you haven’t already got an MTA (mail transfer agent) installed, this will prompt you to install one. The default in Debian is exim, which is fine. I prefer to use postfix, so, I run an apt-get install postfix first. That’s a matter of preference.

You’re also going to need the uuencode tool. This is part of the sharutils package. Install this with:

apt-get install sharutils

Now you’re ready. Let’s assume you’ve got a gzipped log file that you want to email to yourself. The command is simply:

uuencode /var/log/apache.log.gz apache.log.gz | mailx -s "Apache Log File (compressed)"

…and that’s it. Note that the uuencode utility takes two parameters:

Usage: uuencode [INFILE] REMOTEFILE

The first argument (/var/log/apache.log.gz above) is the local filename. This could include a path to the file. The second argument is the attachment ‘name’. This is the filename that will show up in the email when the recipient opens it.

Posted in Debian, General, Linux, Systems Administration | Leave a comment

Debian APT repos not found? Switch to the archive!

If you’re running an older release of Debian and you’re seeing messages like this when you try and apt-get installs, then it’s probably because your release has been taken off the main mirrors and can no longer be updated that way.

W: Failed to fetch  404 Not Found [IP: 80]
W: Failed to fetch  404 Not Found [IP: 80]
W: Failed to fetch  404 Not Found [IP: 80]
W: Failed to fetch  404 Not Found [IP: 80]
E: Some index files failed to download, they have been ignored, or old ones used instead.

You essentially have two choices; either upgrade your distribution to the latest release, or switch to the archive mirrors.

In general when I’m in a hurry I like to just switch to the archive mirrors because it’s quick and I usually don’t want to get into distribution upgrades in the middle of the day.

With that in mind, it’s pretty simple.

First, edit your /etc/apt/sources.list file. Comment out (or delete) everything except your primary mirror reference. It’ll look something like this:

deb lenny main
deb-src lenny main

To update this to the archive mirror, change these lines to:

deb lenny main
deb-src lenny main

Save the file and return to the console.

Next, you need to purge the APT cache, this is a simple:

apt-get clean all

Finally, you probably need to update the GPG keyring. If not, you’ll probably see errors that look like this:

W: GPG error: lenny Release: The following signatures were invalid: KEYEXPIRED 1337087218
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AED4B06F473041FA

To update the keyring, simply run this command:

apt-get install debian-archive-keyring

…and you’re all done. You should be able to use apt-get install normally as before.

Posted in Debian, Linux, Systems Administration | 4 Comments

Tuning JVM Memory Settings

Update: Andrew Wang’s blog post ( is a pretty nice quick guide, well worth reading in conjunction with this page.

Sorting out sensible JVM memory settings in a production environment can be a tricky business. There are some great articles, blogs and lengthy white-papers out there explaining the details of how the JVM manages memory, but I’ve yet to come across a simple guide that actually helps configure the key JVM startup parameters.

For the impatient: Right down at the bottom of this page I have included a simple tool for helping to set your JVM parameters.

For those who are new to this, I suggest doing some web searches around “java heap”. Other people have done a much better job of explaining the theory behind this.

Whilst the theory is well worth reading, I’m hoping this provides some more practically useful information for people running production Java applications. In this context, I’m dealing with a Tomcat 7 server running on the Oracle JVM with an IO intensive application that performs a lot of in-memory caching.

First up, fire up VisualVM (included with your JDK in the /bin folder) if you haven’t already. This will allow you to interrogate the memory of your running application. If you’re trying to interrogate a remote system, you’ll want to run jstatd in order to get a connection. There’s information elsewhere on the web about how to do this; I’ll probably summarise it in another post later.

That will give you a beautiful screen like this:

VisualVM Screenshot

The VisualVM Visual GC View

The server in this example has a 6GB heap allocation and 150MB PermGen allocation. The applicable JVM startup parameters are:


To run through these quickly; ‘server’ tells the JVM to run in server mode. This mainly changes the behaviour of the HotSpot compiler. In general things will take a little longer to ‘warm up’, but long-running performance will be improved. There are other implications too, which are well documented elsewhere.

Setting the Xmx and Xms values to be the same, in this instance 6GB, we create a single (hopefully un-fragmented) heap allocation for the virtual machine. It’s this 6GB that then get’s carved up into the areas shown in the screenshot above.

The PermGen space is assigned with MaxPermSize and PermSize. In general you don’t need to allocate much here; this mostly relates to the number of classes that you have loaded; relatively simple applications without loads of library inclusions get away with very small values here. We’re using 150MB in this example, which is more than enough. Again, setting the values the same saves the JVM scaling up and down with usage.

Now for the interesting stuff. The heap is split into Eden Space, Survivor Space and Old Generation. Without going into excruciating detail, objects are created in the Eden space and then moved to the Survivor Space and finally Old Generation. The idea is that the majority of the objects that you create only need to last for a very short period of time (consider an Iterator for example; it only needs to exist for the duration of a single loop). By creating objects in the Eden space, the initial garbage collections on this space are very efficient, since they potentially scrub very large percentages of all objects. Anything which is still in use is promoted to one of the survivor spaces, and then eventually to the Old Generation.

Performing garbage collection on the Old Generation is usually more time consuming, so that’s what you ideally want to avoid when scaling the other spaces.

It’s worth noting that the Survivor Space is actually split into two parts, known as S0 and S1. Only one of these is active at a time, and it receives the objects being tenured from the Eden space.

The JVM settings that you use to define these spaces are the ‘NewRatio’ and the ‘SurvivorRatio’.

The NewRatio is the ratio between the Old Generation and the sum of Survivor Space (both S0 and S1) and Eden Space (in this context, ‘New’ means Survivor and Eden). For example, a NewRatio of 2 will create an Old Generation that is twice the size of Eden plus Survivor (or 2/3 of the heap).

The SurvivorRatio is the ratio of one of the Survivor Spaces and the Eden Space. For example, a Survivor Ratio of 10 will mean that the Eden space is 10 times the size of each of S0 and S1.

So clearly it’s not a simple task to visualise how the heap will be allocated when you’re setting these values.

To that end, I have created this little bit of javascript which you can use to plug in your Xmx (heap size), NewRatio and SurvivorRatio. The resultant heap split is then shown in the table below. All byte values are in MB. Don’t forget to multiple GB values by 1024 to get MB.

I have already put in values from my example above; that is a heap of 6GB with a NewRatio of 2 and a SurvivorRatio of 10. This is actually an unusually high survivor ratio, however having watched our application in production for a few weeks we’ve been able to tune it to our [rather unusual] workload.

Input Values

Xmx / Xms MB

Resultant Allocations

Old Generation MB
Eden Space MB
Survivor Spaces MB

Please leave a comment if you have any questions – I’d be really interested to hear if this helps to solve your problems.

Posted in Java, Systems Administration | Tagged , , , | Leave a comment

Twitter …finally

So I’ve finally got my act together and opened a twitter account.

For anyone interested, I’m (@TimCreswick)

Posted in General | Leave a comment

Yubikey Glassfish Java Authentication Realm

Container-managed authentication is very useful in Java web applications; it saves writing a lot of back-end authentication and authorisation code into our applications, makes SSO (single sign-on) trivial between J2EE apps and also allows developers to hook into existing enterprise authorisation frameworks with relative ease.

With our recent development work with the Yubikey, a one-time password (OTP) token from Yubico we decided that it would be useful to have a Yubikey authentication realm for the Glassfish application server.

A lot of our existing J2EE applications use the JDBC Realm that is included in the Glassfish distribution. The JDBC Realm allows you to configure the authentication of your application to any JDBC back-end accessible by your application server (i.e. enabled by an appropriate JDBC driver library). Typically this will take the form of a MySQL database table with a list of usernames and password hashes.

Here’s an example of the Glassfish v2.1 JDBC Realm configuration template:

Glassfish v2.1 JDBC Realm Configuration

Glassfish v2.1 JDBC Realm Configuration

Our custom Yubikey Realm is essentially an extension of the JDBC Realm (although not in the sense of a Java “extends”). The Yubikey Realm supports all the same functionality as the JDBC Realm, so you can drop it in-place with all the same settings and it’s totally backwards compatible.

We’ve added the functionality to specify an additional column in the users database table containing the Base 64 Yubikey Public Identifier (ID) of the Yubikey assigned to that user. This is configured via the yubikey-column property. By default, the Yubikey Realm will authenticate users as normal, unless the password length is >= 32 characters. In this instance, the password is assumed to be a Yubikey OTP, and the Realm authenticates accordingly.

We have the additional parameters yubikey-force and yubikey-auth-url, which respectively control whether all logins are treated as Yubikey OTPs, and which validation server you would like to use.

Here’s an example of the realm configuration screen:

Yubikey Authentication Realm Configuration

Yubikey Authentication Realm Configuration

…and there we have it. Instant in-place implementation of Yubikey authenticated logins for all of our existing J2EE applications without changing any application code. Obviously it’s worth noting that in this context we’re using the Yubikey OTP as a convenient password replacement, not as a 2nd authentication factor.

Please feel free to contact me if you’re interested in using our authentication realm; we may consider publishing it online in due course, along with the installation instructions and a few enhancements that are pending.

Posted in Java, Security | Tagged , , | Leave a comment

Yubikey Authentication with Outlook Web Access

I’ve recently been evaulating Two-factor Authentication (2F) implementations for one of our clients, including the excellent Yuibikey product from Yubico.

The initial requirement for our client project is 2F protection of all remote-access to their systems, which presently only amounts to VPN connections and webmail access via Outlook Web Access (OWA) provided by Exchange 2007.

We’ve now finished implementing this, and the diagram below shows how we protected the OWA service (I might cover VPN in a later post). Naturally there are a wide variety of ways to provide 2F authentication for OWA, but I think this presents the best end-user experience:

Authentication Process Overview

Authentication Process Overview

First, a quick overview of the highlights:

  • We implemented a domain-joined ISA Server 2006 Standard.
  • We customised the default 2F ISA Server OWA login wrapper. By default it requests Username, OTP (one-time password) and finally the “internal” password (i.e. the domain password). We thought this was a little counter-intuitive for our users, since the OTP is the new credential being requested (in addition to the usal username/password combination) we thought it should be the last field on the form. Also, our Yubikeys are configured to emulate an ‘enter’ keypress after populating the field, which means that the Yubikey OTP needs to be the last field filled out by the user since it causes the form to be submitted. Finally, the default ISA OWA login form contains an option to add a second username for internal use. We removed this since we don’t have different usernames for remote access.
  • We modified the Active Directory Schema to include a new field on the User object, contianing the user’s Yubikey Public Indentifier. In this way we are able to verify that the Yubikey used to generate the OTP matches the Yubikey assigned to the AD user. This appears to be a unique solution; all the other implementations that I’ve seen require a separate mapping table of user accounts to Yubikeys, usually on the RADIUS server. This is annoying, since it means that you’ve got another system to modify every time a new key is deployed.
  • All the FreeRADIUS modules that we used are entirely custom in-house code. Their function is described in detail below.

Examining our final implementation step by step:

  1. We present our custom ISA / OWA login form to the user over a secure HTTPS connection, where we collect the AD username, AD password and their Yubikey OTP.
  2. The ISA Server is configured to use RADIUS-backed OTP authentication for this service entry, which means that it sends the username and OTP to our FreeRADIUS server (note that the AD password isn’t sent, since we don’t need that).
  3. On the FreeRADIUS Server we have a custom validation module that communicates with the Domain Controller via LDAPS (secure LDAP) in order to confirm that the user exists in the Active Directory (AD). The module allows us to specify particular CNs and OUs in the LDAP tree to search within.
  4. Once the user has been located in the AD, the LDAP function retrieves the Yubikey Public ID of the Yubikey assigned to the user being authenticated. As mentioned earlier, we store this mapping in a custom AD field that we added to the schema. This Yubikey ID is compared with the ID of the key used to generate the OTP to ensure that the key is authorised for that user (for those unfamiliar with the Yubikey, the public ID can be trivially extracted from the OTP).
  5. Next, the FreeRADIUS module communicates with our own Yubikey Validation Server running on Glassfish (we could easily switch this to another server) using the standard Yubikey validation API. This ensures that the submitted OTP is valid.
  6. Assuming all of the above was successful the RADIUS server sends an ‘Accept’ response back to the ISA Server. To clarify, the following must be true:
    1. The user must exist in one of the configured AD OUs or CNs.
    2. The user must have a Yubikey ID assigned in their AD record.
    3. The submitted Yubikey ID must match the ID stored in the AD.
    4. The OTP must validate successfully against the validation service.
  7. Finally, if the RADIUS response was positive, the ISA server provides a delegated login to the OWA service. This means that the user doesn’t need to login a second time, so it’s essentially a form of SSO (single sign on). In order for this to work, the OWA service on your Exchange CAS server must be configured for NTLM (or basic) authentication as opposed to forms authentication.

Hopefully I have provided a coherent overview of our solution. I will try and follow this up with more details on the custom ISA forms, the LDAP schema modifications and FreeRADIUS integration in future posts.

As always, feedback is welcome!

Posted in Security | Tagged , , , | 5 Comments

The business case for NFS backing stores with VMware ESXi

I mentioned in this post (relating to my Solaris ZFS / iSCSI management script) to the storage-discuss OpenSolaris mailing list that we were mostly using NFS as opposed to iSCSI for our ESXi backing stores, and was asked by Christoph Jahn to provide some background on this.

Here’s my response as posted to the list:

Hi Christoph.

Several reasons, I’ll try and outline those that I can recall:

1. We found the performance of NFS to be far better than iSCSI, although we probably hadn’t spent sufficient time tweaking the iSCSI configuration. NB: we were exporting ZFS block devices as LUNs to virtual machines as RDMs (raw device mappings) as opposed to formatting them as VMFS, so the comparison isn’t perhaps completely fair.

2. Overall the administration of iSCSI was overcomplicated, and prone to human error. Training our people to administer and troubleshoot it was likely to be too costly, and the additional ongoing costs associated with the increased administration have to be considered.

3. Similar to point (2) above, we found that you had to be careful to keep track of IQNs and LUN Ids when mapping in VMware, it’s only possible to put names on targets, not LUNs, so when mapping multiple disks into a VM the process was potentially error-prone.

4. The provisioning of iSCSI storage between ESXi and COMSTAR *felt* a little bit unstable; we had numerous instances of lengthy HBA rescans, targets appearing and disappearing which although they were all explained within ‘expected behaviour’ were either a little counter-intuitive or too time-consuming.

5. My understanding of the nature of ESX NFS connections is that less IO blocking takes place, which explains why so many people get better throughput when running multiple VMs against the same SAN connection.

6. The limit of 48 NFS mappings in ESX wasn’t going to be a constraint for us for the foreseeable future; we rarely load more than 15 VMs per node. The flexibility of creating a separate ZFS filesystem for each VM and exporting it separately over NFS was therefore possible.

7. Everyone understands files; when you login to the Solaris system you can navigate to the ZFS filesystem and see .vmx and .vmdk files – these are nice and simple to manage, clone, backup, export etc etc.

8. Related to point (7), when you use NFS in this way the virtual machine configuration is stored alongside the rest of the virtual machine data. This means that a snapshot of the ZFS filesystem for that VM gives us a true complete backup at that point in time.

I think that just about covers it. Essentially, although iSCSI feels like a more clever way of solving our problem, and certainly it poses as a more ‘enterprise’ solution, it really was a case of overcomplicating the solution.

Several times in the last few years we’ve found that simple is better, even if it doesn’t satisfy one’s desire to implement the ‘technically perfect solution’. It’s more a question of balancing the economics (I’m running a business, after all) with the actual requirements. Oftentimes just because you *can* do something doesn’t mean you should, and I’ve often found that PERCEIVED requirements can out-grow ACTUAL requirements just because some technology exists to solve problems that you don’t have [yet].

In short, I like to keep it fit for purpose, even if it does feel like a more agricultural solution.

Finally, I should add that the one major shortcoming with our NFS solution is the lack of any equivalent to the iSCSI multipathing. If we had any machines that required true high availability or automated failover this would probably have negated all of the points above – iSCSI multipathing is a beautiful thing, and it creates some awesome possibilities for fault tolerance.

As it stands, we take care of link failure at the network level (as opposed to the iSCSI MPIO protocol level) and deal with ESX node or storage node failure by manually remapping NFS filesystems from elsewhere. This is actually preferable to automated recovery since sometimes we don’t want to take the ‘default’ action during a failure scenario. By having a carefully documented failure plan I believe we have more flexibility, and can deal with recovery on a per-client basis, rather than a system-wide basis.

Ultimately we are a business dealing with multiple clients hosted on shared hardware, so it’s important to keep our implementations client-centric, rather than system-centric.

Finally, I should add that although we don’t have automated failover of these systems, our solution does still permit us to stay well within our contracted SLAs, which serves the business need.


Timothy Creswick

Posted in Virtualisation | Tagged , , , , | Leave a comment

Normal PDF viewing with Sony Reader PRS-505

My primary reason for buying an eBook Reader was for the large number of technical PDFs, white papers and carefully formatted user guides that I read for work.

I found it very hard to find how well these pre-formatted PDFs would display on the device, with a lot of people saying that they were barely legible. Obviously with most of these documents you don’t want to spend time reformatting them for the Reader’s screen, and zooming is cumbersome and often breaks the formatting.

Other people’s posts notwithstanding I went ahead and bought one.

The good news is that I find it perfectly usable, and although the rendered font size ends up being rather smaller than you might choose, text and diagrams are rendered very clearly. In case it helps anyone else deciding to buy one of these devices for this kind of work, I’ve included some photos below.

First, a regular eBook page rendered in “Small” (click for hi-res):

Then a page from a Sun technical manual (click for hi-res):

Finally, here’s a close up including some very small text, all of which I find perfectly fine for my purposes. Again, click for a larger version, and apologies for the lens distortion which I haven’t bothered to correct:

Posted in General | Leave a comment