OpsProject

Java 1.5

OPS webservices use a number of libraries, a number of those libraries are the Axis libraries, included was version 1.1 of Axis, which is not compatible with Java 1.5. So we now added version 1.3 of Axis which does work with Java 1.5.

Resulting in using Axis 1.3 libraries, the SOAP response it sends after a webcall has been made is a little different from version 1.1, we had to make some changes in handling it so it can read the values again.

LDAP binding

Binding to LDAP by the OPS webservices used to bind anonymously and then tried to bind using username and password. We changed this to binding with username and password right away, so LDAP servers can refuse anonymous binds.

OpsEmail.java

This class connects to the IMAP server to check whether an e-mail message with a certain message id is in one of the project folders the caller has access to. The Java mail implementation for IMAP creates a new connection for every IMAP folder it uses, unless the a connection is available. So we need to make sure we close each connection to a folder after we are done with it and before proceeding to a next one so this an all be done on 1 connection.

Speed issues

Retrieving the project list and searching the IMAP folders on a message id takes time, too much time, waiting 30 seconds or something before a user can see to what project, if any, an e-mail message is assigned to.

getProjectList()

Fetching the project list can be solved by caching it at the client, adding a new webservice which retrieves a timestamp of the last changes for the projects. The client will check with this webservice every time it needs a project list, if the timestamp returned does not equal the one the client had stored it asks for the entire project list again and updates the timestamp.

Update: Speeding up LDAP seems to make this method responsive enough, leaving it for now.

getProjectForEmail()

A sollution to the slow retrieval of projectGuid could be caching.

Mozilla Thunderbird uses client side caching of e-mail headers for IMAP, which it stores in a local DB, we could search this DB for the e-mail with the specified message id in the shared folders, problem would be the shared folder name, which is now only set in webservices configuration. Requires big changes on all plugins. Maybe Thunderbird doesn't cache shared folders? Should be checked out.

Serverside caching, for example keeping message id, projectGuid pairs in a hashmap and checking this before searching the IMAP folders, this would increase speed on e-mail messages assigned to a project. This won't work for e-mail messages not assigned to a project, we could add those to the hashmap too, but how to be sure they have not been assigned to a project after our check? We could do periodic checks on the project folders, checking per folder if the amount of messages in it changed since last check and if so, update the cache. (Maybe some IMAP folder features we can use instead?)

ServerSideCachingOpsEmail

See ServerSideCachingOpsEmail.

Caching using the message header of e-mails

Another possible way of caching projectGuid's for e-mail messages is by adding a tag to the e-mail header containing the projectGuid. This could work if we checked everytime on selecting an e-mail message if its header contains a projectGuid, if not we call the webservice to retrieve it's projectGuid and write it in the message header so next time we will find it.

This way we will update messages on selection with header tags, but now they will remain at the selected project even after they are deleted from project folder. We need to do a checkup when a tag exists in the header, to determine if the information is still correct.

It would probably be good to add folder and message uid to the header aswell so we can check if the e-mail is there very fast, if we don't we need to get the mailfolder from LDAP then do an IMAP search(which takes a lot of time) just to see if a tag in the header is valid.

How to change the e-mail header? I think this can only be done by fetching the raw message, changing it and then appending it again.

• fetch e-mail header
• if ops tag exists
  • if the message is still in it's project folder
    • display selected project
  • else
    • this tag is invalid, continue to search
• else
  • do search
  • add/replace tag
• display selected project

LDAP performance

Some tips on LDAP server performance can be found here.

We tested getting the entire project list(about 400 projects), here are the results:

time ldapsearch -D "uid=bastiaan,ou=Users,dc=linops01,dc=func,dc=nl" -w "extremely secretly hidden" -b "ou=Projects,dc=linops01,dc=func,dc=nl" -Z -x -v -d 0 -L "(objectClass=*)" name projectGuid > ldapsearch.result
ldap_initialize( <DEFAULT> )
filter: (objectClass=*)
requesting: name projectGuid

real    0m3.288s
user    0m0.031s
sys     0m0.002s

This takes far too long, how to get these results faster?

LDAP log level

After turning down the loglevel, form 768 to 0 I got the following results:

time ldapsearch -D "uid=bastiaan,ou=Users,dc=linops01,dc=func,dc=nl" -w "extremely secretly hidden" -b "ou=Projects,dc=linops01,dc=func,dc=nl" -Z -x -v -d 0 -L "(objectClass=*)" name projectGuid > ldapsearch.result
filter: (objectClass=*)
requesting: name projectGuid

real    0m0.233s
user    0m0.059s
sys     0m0.017s

So clearly we should keep logging at a minium for the LDAP server.

OpsTimesheet

The performance problems with OpsTimesheet all seem related to the ical library we use. Not that its a bad library, but seeing that a object is made for every property and every event and a lot of method calls surround the parsing of an ical file, generating a report is becoming very cpu and memory intensive. We are testing with about 30 ical files 0.5 MB each and this already proves to be problematic.
A possible solution:

• Split the ical files by year(or even shorter periods) and extend the webservices so that by default only the most recent files are taken in account. In ldap we would need additional attributes to label the files with start end end dates so that we can select only the needed files based on the request.