General troubleshooting and problem solving

LDAP: Testing authentication

You can troubleshoot LDAP authentication by running the ldapsearch tool on your Dash Enterprise server. If you installed Dash Enterprise following the Installation on a Cloud Provider instructions, this tool has already been installed, otherwise you can install it as below:

• Ubuntu: sudo apt install ldap-utils

• RHEL or CentOS: yum install openldap-clients

ldapsearch syntax can be complicated, so here is an example to get you started:

ldapsearch -H ldap://ad.plot.ly -x -D 'plotly\mrplot' -w 'ldappass' -b  'CN=Users,DC=ad,DC=plot,DC=ly' SAMAccountName -v

Breaking it down:

• ldap://ad.plot.ly is the server URI

  • This is the the auth_ldap_server_uri parameter

• plotly\mrplot is the bind DN used for authentication

  • This is the auth_ldap_bind_dn parameter

  • mrplot is the username

  • Your LDAP server may or may not need you to include the LDAP domain and backlash (plotly in this example); check with your server administrator

• ldappass is the password for the bind DN

  • This is the auth_ldap_bind_password parameter

    CN=Users,DC=ad,DC=plot,DC=ly is the *search DN

  • This is the auth_ldap_search_dn parameter

  • This is usually based on the domain name of your LDAP server

The above command returns a list of all accounts in the search DN; in this case, the list of all usernames allowed to log in to Dash Enterprise.

If you receive any other output, contact your LDAP server administrator or Plotly support for help.

Testing without adding entries to your DNS server (using /etc/hosts)

You can test your Dash Enterprise instance locally by adding the DNS entries directly to the hosts file on each machine intended to access the instance. Here are instructions on how to do that for various operating systems. The line to add will look something like this:

12.34.56.78 plotly.your-company.com

• Replace 12.34.56.78 with the IP of your Dash Enterprise server

• plotly.your-company.com is the Base Domain value that you specified during configuration

If you set your server’s current hostname as the Base Domain in your Dash Enterprise configuration, you do not need to modify your hosts file.

SSL Error: certificate verify failed when using the Plotly Python library

These kinds of errors look like this in your console:

SSLError: [Errno 1] _ssl.c:510: error:14090086:SSLroutines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

You can solve this in one of two ways:

• Set the path to a valid certificate file in your Plotly Python configuration (recommended):

import plotly.tools as tls
tls.set_config_file(plotly_ssl_verification='/absolute/path/to/file.crt')

• Turn off SSL verification in your Plotly Python configuration (not recommended):

tls.set_config_file(plotly_ssl_verification=False)

Only use this option if you are sure the endpoint of the HTTPS request points to your server. This option leaves your data vulnerable to man-in-the-middle attacks.

When using IPython, if you modify plotly_ssl_verification as above, you may receive formatter warnings similar to:

Exception image/{format} formatter: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

These warnings can safely be suppressed as follows:

import warnings
import IPython.core.formatters
warnings.filterwarnings('ignore', category=IPython.core.formatters.FormatterWarning)

All other warnings will continue to be displayed.

Private key/certificate debugging

If you suspect your SSL certificate and key do not match, you can verify by finding the sha256sum of each as follows:

openssl x509 -in /path/to/your/plotly_certificate.crt -pubkey -noout -outform pem | sha256sum

openssl pkey -in /path/to/your/plotly_key.key -pubout -outform pem | sha256sum

If the results of the above commands match, then the key and certificate match as well.

Running devicemapper in loopback mode

If your system defaults to using devicemapper in loopback mode (typical for RHEL 7 and CentOS 7), you will see an error during installation that looks like this:

The running Docker daemon is configured to use the 'devicemapper' storage driver in loopback mode.
This is not recommended for production use. Please see the following URL for more information.

https://help.replicated.com/docs/kb/developer-resources/devicemapper-warning/.

Do you want to proceed anyway?

To solve this, cancel installation and configure your system to use direct-lvm according to this Docker guide.

Dash app manager boot failure -- Error response from daemon: max depth exceeded

During startup, you may encounter an error that corresponds with the following message in your Dash container logs:

Error response from daemon: max depth exceeded

You can resolve this as follows:

1 - Find the Dash container ID: sudo docker ps

2 - Stop the Dash container: sudo docker stop DASH_CONTAINER_ID

3 - Delete all Dash-related containers/images:

sudo docker rm -f $(sudo docker ps -a | grep -E "web.1 | Exited" | awk "{print $ 1}")

sudo docker rmi -f $(sudo docker images -a | grep -E "<none> | dokku" | awk "{print $ 3}")

4 - In the Dash Enterprise Server Manager, click Start Now

5 - Redeploy your Dash application

Plotly data directory verification failure

If attempting to set the Plotly data directory and the verification test is failing:

1 - Confirm the directory exists on the system and you have correctly entered the path to it

2 - Ensure SELinux is enabled and running with sudo sestatus, and enable it if not

3 - If SELinux is enabled, add SELinux permissions to your data directory with:

sudo chcon -Rt svirt_sandbox_file_t /path/to/your_data_directory

Missing container-selinux dependency (Error: nothing to do)

In RHEL, the container-selinux dependency may be missing from the system, which commonly results in the following error when running the installation script:

Error: nothing to do.

To solve this, we recommend the following options:

• Enable the appropriate repository:

  sudo subscription-manager repos --enable=rhel-7-server-extras-rpms

• If the above doesn’t work, install the package directly:

  yum install http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.36-1.gitff95335.el7.noarch.rpm

OCI runtime error

On RHEL < v7.4, OCI runtime issues have been known to occur, preventing Server Manager containers (names beginning with replicated) from starting when using Docker >= 17.12.1. You can resolve this by upgrading to >= RHEL v7.4.

Dash Enterprise Kubernetes setup problems

If you followed the steps in Set up your cloud infrastructure and Dash Enterprise Kubernetes additional required configuration and On-Premise does not start up correctly, here are some troubleshooting steps to try:

1 - In your cloud provider’s console, confirm the following:

• All of your cluster resources were created on the same VPC network

• The VPC has the appropriate firewall rules/security groups to allow traffic between your resources

• Your service accounts/IAM roles were created with the appropriate permission sets

• Your EC2/compute instances and Kubernetes clusters were created with the appropriate role account assigned to them

2 - Check for any typos in the values configured in your Server Manager Settings

3 - ssh into the EC2 or Google Compute Engine instance you created in Step 3 and check the docker logs output for the dashauth and dash containers

4 - Inspect your pod logs with kubectl logs for clues

5 - To manually apply a recent configuration change, manually scale your deployments to 0 and back to their original number again

Support

Our support team is available to help if you run into any issues with your installation. When contacting us, please also include a support bundle, which is a collection of log files and app state that will help us diagnose your problem. There are two methods available for creating and sending one:

Directly upload support bundle to our portal (recommended)

1 - SSH into the server running Dash Enterprise

2 - Copy and paste the following command, which generates a bundle and prompts you to upload it to our support portal:

docker pull replicated/support-bundle && \
docker run --rm -it --name support-bundle \
  -v $PWD:/out -w /out \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --net host --pid host  \
  -e HTTP_PROXY -e HTTPS_PROXY -e NO_PROXY \
  replicated/support-bundle generate \
    --channel-id 2981105c67e98b19d315128c52286160

3 - E-mail Plotly support with a description of the issue and let us know you have made a support bundle available for analysis

Download and e-mail support bundle (for airgapped instances)

1 - Browse to your Dash Enterprise Server Manager

2 - Click the Support tab

3 - Click Download Support Bundle

4 - Upload the resulting file to a cloud storage service of your choice

5 - E-mail us with a description of the issue and a link to the support bundle