Uploaded image for project: 'Reporting Compatability Module'
  1. Reporting Compatability Module
  2. RCM-20

Data Export Slow and Out of Memory Error - java.lang.OutOfMemoryError: Java heap space

    XMLWordPrintable

Details

    Description

      Data exports are very slow, consuming more than 100% of CPU resources for 7+ hours. The system is very slow or inaccessible for other users. When I try to run some data exports, it fails. I get the following error in the log file...

      Jul 29, 2009 7:18:43 AM org.apache.catalina.startup.Catalina start
      INFO: Server startup in 16337 ms
      ERROR - CommonsLoggingOutput.error(75) |2009-07-29 07:18:43,993| Line=65 The content of element type "dwr" must match "(init?,allow?,signatures?)".
      ERROR - CommonsLoggingOutput.error(75) |2009-07-29 07:18:44,031| Parameter mismatch parsing signatures section in dwr.xml on line: DWRMatchingConfigUtilities
      .getAllBlockingRuns()                 DWRMatchingConfigUtilities.getAllReports()                 DWRMatchingConfigUtilities.deleteBlockingRun(String)
               DWRMatchingConfigUtilities.doAnalysis()                 DWRMatchingConfigUtilities.deleteReportFile(String)                 DWRMatchingConfigUtiliti
      es.getStatus()                 DWRMatchingConfigUtilities.setStatus(String)                 DWRMatchingConfigUtilities.getNextPage()                 DWRMatch
      ingConfigUtilities.getPrevPage()                 DWRMatchingConfigUtilities.resetStep()                 DWRMatchingConfigUtilities.getStep()
      ERROR - CommonsLoggingOutput.error(75) |2009-07-29 07:18:44,043| Parameter mismatch parsing signatures section in dwr.xml on line: DWRHtmlFormEntryService.ch
      eckIfLoggedIn()
      DEBUG - Extension.initialize(69) |2009-07-29 07:19:36,379| Initializing extension for point: org.openmrs.admin.list
      DEBUG - Extension.initialize(69) |2009-07-29 07:23:34,982| Initializing extension for point: org.openmrs.admin.list
      Exception in thread "Timer-2" java.lang.OutOfMemoryError: Java heap space
              at java.util.Arrays.copyOfRange(Arrays.java:3209)
              at java.lang.String.<init>(String.java:215)
              at java.lang.StringBuilder.toString(StringBuilder.java:430)
              at java.lang.Thread.<init>(Thread.java:377)
              at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.<init>(ThreadPoolAsynchronousRunner.java:490)
              at com.mchange.v2.async.ThreadPoolAsynchronousRunner.recreateThreadsAndTasks(ThreadPoolAsynchronousRunner.java:393)
              at com.mchange.v2.async.ThreadPoolAsynchronousRunner.access$300(ThreadPoolAsynchronousRunner.java:36)
              at com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector.run(ThreadPoolAsynchronousRunner.java:627)
              at java.util.TimerThread.mainLoop(Timer.java:512)
              at java.util.TimerThread.run(Timer.java:462)
      Jul 29, 2009 7:42:14 AM org.apache.catalina.connector.CoyoteAdapter service
      SEVERE: An exception or error occurred in the container during the request processing
      java.lang.OutOfMemoryError: Java heap space
              at java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:314)
              at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267)
              at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186)
              at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209)
              at sun.nio.cs.US_ASCII$Encoder.<init>(US_ASCII.java:121)
              at sun.nio.cs.US_ASCII$Encoder.<init>(US_ASCII.java:118)
              at sun.nio.cs.US_ASCII.newEncoder(US_ASCII.java:43)
              at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215)
              at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207)
              at java.lang.StringCoding.encode(StringCoding.java:266)
              at java.lang.String.getBytes(String.java:946)
              at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
              at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
              at java.io.File.exists(File.java:733)
              at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:785)
              at org.apache.naming.resources.FileDirContext.getAttributes(FileDirContext.java:429)
              at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:747)
              at org.apache.naming.resources.ProxyDirContext.cacheLoad(ProxyDirContext.java:1531)
              at org.apache.naming.resources.ProxyDirContext.cacheLookup(ProxyDirContext.java:1454)
              at org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:288)
              at org.apache.tomcat.util.http.mapper.Mapper.internalMapWrapper(Mapper.java:777)
              at org.apache.tomcat.util.http.mapper.Mapper.internalMap(Mapper.java:667)
              at org.apache.tomcat.util.http.mapper.Mapper.map(Mapper.java:557)
              at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:457)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:291)
              at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859)
              at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:574)
              at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527)
              at java.lang.Thread.run(Thread.java:619)
      Jul 29, 2009 7:44:54 AM unknown unknown
      SEVERE: Exception invoking periodic operation:
      java.lang.OutOfMemoryError: Java heap space
              at java.util.Arrays.copyOf(Arrays.java:2882)
              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
              at java.lang.StringBuilder.append(StringBuilder.java:203)
              at java.io.UnixFileSystem.resolve(UnixFileSystem.java:93)
              at java.io.File.<init>(File.java:312)
              at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:541)
              at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:491)
              at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1274)
              at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296)
              at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
              at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
              at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
              at java.lang.Thread.run(Thread.java:619)
      Jul 29, 2009 7:44:59 AM org.apache.catalina.core.StandardWrapperValve invoke
      SEVERE: Servlet.service() for servlet jsp threw exception
      java.lang.OutOfMemoryError: Java heap space
              at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:636)
              at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:412)
              at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:1959)
              at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1384)
              at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1133)
              at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:670)
              at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1024)
              at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
              at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)
              at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
              at org.hibernate.loader.Loader.doQuery(Loader.java:674)
              at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
              at org.hibernate.loader.Loader.loadEntity(Loader.java:1860)
              at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
              at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42)
              at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3044)
              at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:395)
              at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375)
              at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
              at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
              at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
              at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
              at org.hibernate.impl.SessionImpl.get(SessionImpl.java:815)
              at org.hibernate.impl.SessionImpl.get(SessionImpl.java:808)
              at org.openmrs.api.db.hibernate.HibernateAdministrationDAO.getGlobalProperty(HibernateAdministrationDAO.java:234)
              at org.openmrs.api.impl.AdministrationServiceImpl.getGlobalProperty(AdministrationServiceImpl.java:591)
              at org.openmrs.api.impl.AdministrationServiceImpl.getGlobalProperty(AdministrationServiceImpl.java:599)
              at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:301)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
      

      This happens on OpenMRS Version: 1.4.4.00 Build 9364

      Test performance by measuring the time between clicking the "Generate" export button until the response page is fully loaded.

      We will need to select a specific export with which to do performance testing.

      Gliffy Diagrams

        Attachments

          1. deadpatients_cohort_profiler_snapshot.zip
            5.11 MB
          2. diabetes.csv
            5 kB
          3. employee_cohort_profiler_snapshot.zip
            3.36 MB
          4. short_report.csv
            0.2 kB

          Issue Links

            Activity

              People

                bwolfe Ben Wolfe
                arbaughj James Arbaugh
                Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved: