On Saturday evening I set a reindex of our JIRA 4.4.4 service running at 17:52. By 23:31 it had completed just 8%. Based on this timing I didn't expect it to finish for around 66 hours.
I raised an incident with Atlassian (JSP-150046) and they asked me to run 6 thread dumps (attached and numbered 1 to 6) about 2 minutes apart. Their analysis was:
I believe I've found the cause of the problem. There's one thread in the indexing that is blocking others to run and should be causing the slowness.
The first thread dump and the last thread dump were checked. The messages relate to a class com.innovalog.jmcf.fields.CalculatedNumberField.getValueFromIssue that belongs to the JIRA Misc Custom Fields.
I am not sure what version of the plugin you currently have, but looking in our Marketplace, looks like Version 1.5.2 fixes one bug related to Oracle that may be affecting you.
Furthermore, if the plugin is up to date, I recommend disabling it and then reaching the plugin developers to let them aware of this issue.
We were using JMCF v1.2.2 so I downloaded and installed JMCF v1.5.2 and restarted the reindex. The result was a reduced reindex time of 13.75 hours. Whilst this is significantly better, it is still way too long.
I took another set of thread dumps (numbered 7 to 12) which again pointed to an issue with JMCF.
I will upload the jira log along with the thread dumps to this issue.
+61 439 049 668
Windows 2008 Server, Tomcat 6.0, JIRA 4.4.4