Troubleshooting

Normal Start Up

When eXist-db starts up it writes log output to the console.

• If you started eXist-db via the system tray launcher, the console output is captured. It can be viewed by selecting the Show Tool Window menu entry in the system tray pop-up. Then select Show console messages.

• If you launched eXist-db via one of the shell scripts, the output directly appears.

Output for a properly launched eXIst-db should be similar to the following (Linux) example:

05 Dec 2017 13:53:19,795 [main] INFO  (JettyStart.java [run]:137) - Running with Java 1.8.0_131 [Oracle Corporation (OpenJDK 64-Bit Server VM) in /usr/lib/jvm/java-8-openjdk-amd64/jre] 
05 Dec 2017 13:53:19,795 [main] INFO  (JettyStart.java [run]:144) - Running as user 'root'
05 Dec 2017 13:53:19,795 [main] INFO  (JettyStart.java [run]:145) - [eXist Home : /exist]
05 Dec 2017 13:53:19,797 [main] INFO  (JettyStart.java [run]:146) - [eXist Version : 3.6.0]
05 Dec 2017 13:53:19,797 [main] INFO  (JettyStart.java [run]:147) - [eXist Build : 201712051111]
05 Dec 2017 13:53:19,797 [main] INFO  (JettyStart.java [run]:148) - [Git commit : ceaab01]
05 Dec 2017 13:53:19,798 [main] INFO  (JettyStart.java [run]:150) - [Operating System : Linux 4.9.49-moby amd64]
05 Dec 2017 13:53:19,798 [main] INFO  (JettyStart.java [run]:151) - [log4j.configurationFile : file:///exist/log4j2.xml]
05 Dec 2017 13:53:19,803 [main] INFO  (JettyStart.java [run]:152) - [jetty Version: 9.4.7.v20170914]
05 Dec 2017 13:53:19,803 [main] INFO  (JettyStart.java [run]:153) - [jetty.home : /exist/tools/jetty]
05 Dec 2017 13:53:19,804 [main] INFO  (JettyStart.java [run]:154) - [jetty.base : /exist/tools/jetty]
05 Dec 2017 13:53:19,804 [main] INFO  (JettyStart.java [run]:155) - [jetty configuration : /exist/tools/jetty/etc/standard.enabled-jetty-configs]
05 Dec 2017 13:53:20,143 [main] INFO  (JettyStart.java [run]:164) - Configuring eXist from /exist/conf.xml
05 Dec 2017 13:53:21,433 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty.xml]
05 Dec 2017 13:53:21,518 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-gzip.xml]
05 Dec 2017 13:53:21,547 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-http.xml]
05 Dec 2017 13:53:21,573 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-jaas.xml]
05 Dec 2017 13:53:21,578 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-jmx.xml]
05 Dec 2017 13:53:21,606 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-requestlog.xml]
05 Dec 2017 13:53:21,616 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-ssl.xml]
05 Dec 2017 13:53:21,636 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-ssl-context.xml]
05 Dec 2017 13:53:21,652 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-https.xml]
05 Dec 2017 13:53:21,657 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-deploy.xml]
05 Dec 2017 13:53:21,677 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-plus.xml]
05 Dec 2017 13:53:21,697 [main] INFO  (JettyStart.java [run]:188) - [loading jetty configuration : /exist/tools/jetty/etc/jetty-annotations.xml]
05 Dec 2017 13:53:21,701 [main] INFO  (JettyStart.java [startJetty]:501) - [Starting jetty component : org.eclipse.jetty.server.Server]
05 Dec 2017 13:53:21,701 [main] INFO  (JettyStart.java [lifeCycleStarting]:649) - Jetty server starting...
05 Dec 2017 13:53:23,197 [main] INFO  (JettyStart.java [lifeCycleStarted]:658) - Jetty server started.
05 Dec 2017 13:53:23,198 [main] WARN  (JettyStart.java [run]:239) - Could not find OpenID extension. OpenID will be disabled!
05 Dec 2017 13:53:23,199 [main] WARN  (JettyStart.java [run]:246) - Could not find OAuthServlet extension. OAuth will be disabled!
05 Dec 2017 13:53:23,199 [main] WARN  (JettyStart.java [run]:253) - Could not find IPRangeServlet extension. IPRange will be disabled!
05 Dec 2017 13:53:23,205 [main] WARN  (JettyStart.java [getURI]:468) - java.net.UnknownHostException: 5773b75f8e8d: 5773b75f8e8d: Name or service not known
05 Dec 2017 13:53:23,208 [main] WARN  (JettyStart.java [getURI]:468) - java.net.UnknownHostException: 5773b75f8e8d: 5773b75f8e8d: Name or service not known
05 Dec 2017 13:53:23,209 [main] INFO  (JettyStart.java [run]:262) - -----------------------------------------------------
05 Dec 2017 13:53:23,209 [main] INFO  (JettyStart.java [run]:263) - Server has started, listening on:
05 Dec 2017 13:53:23,209 [main] INFO  (JettyStart.java [run]:268) - Configured contexts:
05 Dec 2017 13:53:23,211 [main] INFO  (JettyStart.java [run]:274) - 	/exist
05 Dec 2017 13:53:23,211 [main] INFO  (JettyStart.java [run]:274) - 	/
05 Dec 2017 13:53:23,211 [main] INFO  (JettyStart.java [run]:328) - -----------------------------------------------------

When you see the "Server has started" message and no further errors appear, your eXist-db installation is working normally.

When you do not (even) see this message,try the following troubleshooting steps.

Database Refuses to Start

If eXist-db was not shutdown properly, it will start a recovery process to redo committed transactions and roll back uncommitted ones. If an inconsistency is found during this process, eXist-db will automatically abort the start-up and print out a warning. This emergency stop is done to avoid potential damage and give an administrator a chance to check the database and create a backup. It does not necessarily indicate a real problem. In most cases, the database should be ok and restarting it will be save.

We recommend to run a consistency check in when this happens. If inconsistencies are found, make sure you have a backup before continuing.

Going Back to an Empty Database

During development and testing you may sometimes wish to go back to a completely empty, fresh, database. Here's how to really remove everything and reset the database to its initial state:

1. Make sure eXist-db is not running

2. If you installed the source code (and therefore the development tools), call ant as follows:

   build.sh clean-default-data-dir

   If you do not have build.sh (or build.bat), you may can manually remove the contents of your data directory.

   The default data directory is $EXIST_HOME/webapp/WEB-INF/data

JAVA_HOME and EXIST_HOME Environmental Variables

When using one of the shell or batch scripts, eXist-db can fail to start up properly if either of the two key environmental variables JAVA_HOME and/or EXIST_HOME are not set properly. Both variables are used in the startup.bat and startup.sh scripts and have to be set correctly before the scripts are run (you can of course also insert the lines required in the beginning of the scripts themselves).

JAVA_HOME

This should point to the directory where Java (JRE or JDK) is installed.

Windows

Setting JAVA_HOME on Windows

Linux

Setting JAVA_HOME on Linux

macOS

Setting JAVA_HOME on macOS

EXIST_HOME

This must point to the directory that contains eXist-db's configuration file conf.xml. For example, if the EXIST_HOME path is C:\Program Files\eXist, the server will look for C:\Program Files\eXist\conf.xml.

You can set EXIST_HOME in the same way that you set JAVA_HOME.

On macOS, depending on how you installed eXist, you would open a terminal and enter either:

for JAR based installs

export EXIST_HOME=/Applications/eXist

for DMG and homebrew installs

export EXIST_HOME=/Applications/eXist-db.app/Contents/Resources/eXist-db

.

Ensure that you have "write" permissions for the data directory (located by default in webapp/WEB-INF/).

Port Conflicts

eXist-db can fail to start up if another service on your system is using the ports that eXist's embedded web server is using (port 8080 or 8443 by default). If another service occupies these ports, you cannot start up eXist-db unless you shut down the service in question or make eXist-db use another port.

To see whether this is the case, enter http://localhost:8080/ in your browser. To make eXist-db use another port, open /tools/jetty/etc/jetty-http.xml and change the value 8080 to a port number that is not used (for instance 8899):

<Set name="port">
                    <Property name="jetty.http.port" deprecated="jetty.port">
                        <Default>
                            <SystemProperty name="jetty.port" default="8899"/>
                        </Default>
                    </Property>
                </Set>

eXist-db uses port 8443 for confidential communication. Another service may be using this port. To make eXist-db use another port, open the file /tools/jetty/etc/jetty-ssl.xml and change the value 8443 in to a port number that is not used (for instance 8444):

<Set name="port">
                    <Property name="jetty.ssl.port" deprecated="ssl.port">
                        <Default>
                            <SystemProperty name="jetty.ssl.port" deprecated="ssl.port" default="8444"/>
                        </Default>
                    </Property>
                </Set>

If these changes do not launch eXist-db try the following: Change to the directory where you installed eXist-db and enter the following command:

java -Xmx1024M -Djava.endorsed.dirs=lib/endorsed -jar start.jar jetty

If you have problems running the shell/batch scripts, read the section about alternative scripts.

Using the Logs

If you experience any problems while using eXist-db, your first step should be to check the log files to get additional information about the source of the problem. eXist-db uses the log4j-package to write output logs to files.

• By default, this output is written to the directory webapp/WEB-INF/logs

• If you are running eXist as a service, check the directory tools/yajsw/logs

The main log files for eXist are exist.log and xmldb.log.

Out of Memory

Running out of memory typically throws Java into an inconsistent state. Some threads may still be alive and continue to run while others have died. It is therefore important to avoid memory errors up-front by checking the memory consumption of your queries before they go into production. Should you encounter an out of memory error, please make sure to restart eXist and follow the emergency procedure.

Killing the Database

If you ever feel you have to kill the database (because it does not respond for whatever reason), the recommended procedure is as follows:

1. Check if a query is running wild and try to kill it. This can be done either through the "Scheduler" plug-in in the dashboard, or the "Running Jobs" section in the Java Admin Client. Try to kill the query there and wait for a minute if the system returns to normal operations:

   

2. Attempt to trigger a proper shutdown, either via the system tray icon or the dashboard. Wait for at least 3 minutes. Even if eXist-db does not stop completely, it may still be able to complete the shutdown procedure for the core database.

3. It may now be safe to kill the eXist-db process. Check the logs to see if the database has properly shut down. The last message in the logs will indicate this.

4. If the logs indicate a proper shutdown: before restarting, remove any .log and .lck files from the data directory. This will prevent a recovery run, which would certainly take a lot of time.

5. Otherwise:

   • You are sure you have no valuable changes in this db instance, usually because it's a development system: follow the step above and remove the .log files before restart to reduce startup time.

   • Before restart, archive the contents of the data directory: you may need them if anything goes wrong. Restart the database but be prepared for a recovery run, which may take considerable time (depending on the size of your db).

   • If inconsistencies are detected during the recovery, eXist will switch to read-only mode. In this case, stop it again and run a consistency check, which can also create a low-level backup.

     If the consistency check reports errors, eXist may still be able to run, but there might be inconsistencies in its data structures. Prepare for a complete restore into a clean data directory as soon as you can take the database offline for maintenance.