TLS encryption between Splunk hosts

In today's data-driven world, security is paramount. Protecting sensitive information as it traverses across networks is crucial, especially within environments like Splunk where valuable data is stored and analyzed. One effective method to enhance security is by implementing TLS encryption between Splunk hosts. In this guide, we'll walk you through the process of setting up TLS encryption for Splunk hosts, ensuring data integrity and confidentiality.

 

Step 1: Creating Certificates

First, let's generate the necessary certificates. Open a terminal and follow these commands:

mkdir /opt/splunk/etc/auth/mycerts
cd /opt/splunk/etc/auth/mycerts
# 1.Generate CA Private Key
/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out CAPrivateKey.key 2048
# Enter password
 
# 2.Create CSR
/opt/splunk/bin/splunk cmd openssl req -new -key CAPrivateKey.key -out CACertificate.csr
# Enter password from .key
# Fill in the required information
 
# 3.Create PEM file
/opt/splunk/bin/splunk cmd openssl x509 -req -in CACertificate.csr -sha512 -signkey CAPrivateKey.key -CAcreateserial -out CACertificate.pem -days 1095
# Enter .key password
 

Step 2: Generate Private Certificate for Specific Server

Now, let's generate a private certificate for the specific server:

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out ServerPrivateKey.key 2048
/opt/splunk/bin/splunk cmd openssl req -new -key ServerPrivateKey.key -out ServerCertificate.csr
/opt/splunk/bin/splunk cmd openssl x509 -req -in ServerCertificate.csr -SHA256 -CA CACertificate.pem -CAkey CAPrivateKey.key -CAcreateserial -out ServerCertificate.pem -days 1095
 

Step 3: Create Certchain

Create a certificate chain by concatenating the server certificate, private key, and CA certificate:

cat ServerCertificate.pem ServerPrivateKey.key CACertificate.pem > SplkServerCertificate.pem
 

Step 4: Configure Splunk Server

Configure port 9997 on the Splunk server to use the certificate chain:

/opt/splunk/bin/splunk cmd openssl s_server -accept 9997 -cert SplkServerCertificate.pem

Note: Use s_server, not your server name.

Step 5: Configure Inputs and Server

Update the Splunk configuration files:

nano /opt/splunk/etc/system/local/inputs.conf

Add the following:


[splunktcp-ssl:9997]
disabled = 0
[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/SplkServerCertificate.pem
sslpassword = password_used_for_creating_pemfile
requireClientCert = false
 

Next, edit the server.conf file:

nano /opt/splunk/etc/system/local/server.conf

Add the following under the [sslconfig] stanza:

[sslconfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/CACertificate.pem
 

Step 6: Configure Second Splunk Host

Copy the certificate files to the second Splunk host:

cp SplkServerCertificate.pem CACertificate.pem /home/usr/
chown -R usr /home/usr/
cd /home/usr

On the remote machine (second indexer/forwarder):

mkdir /opt/splunk/etc/auth/mycerts
mv *.pem /opt/splunk/etc/auth/mycerts
 

Adjust the [sslConfig] stanza in /opt/splunk/etc/system/local/server.conf:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/CACertificate.pem
 

Create /opt/splunk/etc/system/local/inputs.conf:

[splunktcp-ssl:9997]
disabled = 0
[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/SplkServerCertificate.pem
sslpassword = password_used_for_creating_pemfile
requireClientCert = false
 

Enable listen on port 9997 if needed and restart the Splunk service.

 

Step 7: Configure Universal Forwarder

For Universal Forwarder, copy the certificates and adjust the outputs.conf file:

cp *.pem /opt/splunk/etc/auth/mycerts

Edit outputs.conf:

[tcpout]
defaultGroup = default-autolb-group
sslPassword = password_used_for_creating_pemfile
 
[tcpout:default-autolb-group]
server = 192.168.1.20:9997
clientCert = /opt/splunk/etc/auth/mycerts/SplkServerCertificate.pem
sslVerifyServerCert = true
useClientSSLCompression = true
 
[tcpout-server://192.168.1.20:9997]
 

Additionally, in server.conf, add the following under the [sslConfig] stanza:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/CACertificate.pem
 

With these steps, you've successfully configured TLS encryption between Splunk hosts. However, if you encounter any issues during the setup process, here are some common troubleshooting steps:

  • Check Permissions: Ensure that the certificate files have the correct permissions set.
  • Verify Paths: Double-check file paths in configuration files to ensure they are correct.
  • Restart Services: After making changes, restart Splunk services to apply configurations.
  • Review Logs: Check Splunk logs for any error messages that might provide clues to the issue.

By following this guide and troubleshooting steps, you can enhance the security of your Splunk environment with TLS encryption, safeguarding your data from potential threats.