Replace InfluxDB Enterprise cluster meta nodes and data nodes
Introduction
Nodes in an InfluxDB Enterprise cluster may need to be replaced at some point due to hardware needs, hardware issues, or something else entirely. This guide outlines processes for replacing both meta nodes and data nodes in an InfluxDB Enterprise cluster.
Concepts
Meta nodes manage and monitor both the uptime of nodes in the cluster as well as distribution of shards among nodes in the cluster. They hold information about which data nodes own which shards; information on which the anti-entropy (AE) process depends.
Data nodes hold raw time-series data and metadata. Data shards are both distributed and replicated across data nodes in the cluster. The AE process runs on data nodes and references the shard information stored in the meta nodes to ensure each data node has the shards they need.
influxd-ctl is a CLI included in each meta node and is used to manage your InfluxDB Enterprise cluster.
Scenarios
Replace nodes in clusters with security enabled
Many InfluxDB Enterprise clusters are configured with security enabled, forcing secure TLS encryption between all nodes in the cluster.
Both influxd-ctl and curl, the command line tools used when replacing nodes, have options that facilitate the use of TLS.
influxd-ctl -bind-tls
In order to manage your cluster over TLS, pass the -bind-tls flag with any influxd-ctl commmand.
If using a self-signed certificate, pass the
-kflag to skip certificate verification.
# Pattern
influxd-ctl -bind-tls [-k] <command>
# Example
influxd-ctl -bind-tls remove-meta enterprise-meta-02:8091
curl -k
curl natively supports TLS/SSL connections, but if using a self-signed certificate, pass the -k/--insecure flag to allow for “insecure” SSL connections.
Self-signed certificates are considered “insecure” due to their lack of a valid chain of authority. However, data is still encrypted when using self-signed certificates.
# Pattern
curl [-k, --insecure] <url>
# Example
curl -k https://localhost:8091/status
Replace meta nodes in a functional cluster
If all meta nodes in the cluster are fully functional, simply follow the steps for replacing meta nodes.
Replace an unresponsive meta node
If replacing a meta node that is either unreachable or unrecoverable, you need to forcefully remove it from the meta cluster. Instructions for forcefully removing meta nodes are provided in the step 2.2 of the replacing meta nodes process.
Replace responsive and unresponsive data nodes in a cluster
The process of replacing both responsive and unresponsive data nodes is the same. Simply follow the instructions for replacing data nodes.
Reconnect a data node with a failed disk
A disk drive failing is never a good thing, but it does happen, and when it does, all shards on that node are lost.
Often in this scenario, rather than replacing the entire host, you just need to replace the disk.
Host information remains the same, but once started again, the influxd process doesn’t know
to communicate with the meta nodes so the AE process can’t start the shard-sync process.
To resolve this, log in to a meta node and use the influxd-ctl update-data command
to update the failed data node to itself.
# Pattern
influxd-ctl update-data <data-node-tcp-bind-address> <data-node-tcp-bind-address>
# Example
influxd-ctl update-data enterprise-data-01:8088 enterprise-data-01:8088
This will connect the influxd process running on the newly replaced disk to the cluster.
The AE process will detect the missing shards and begin to sync data from other
shards in the same shard group.
Replace meta nodes in an InfluxDB Enterprise cluster
Meta nodes together form a Raft cluster in which nodes elect a leader through consensus vote. The leader oversees the management of the meta cluster, so it is important to replace non-leader nodes before the leader node. The process for replacing meta nodes is as follows:
- Identify the leader node
- Replace all non-leader nodes
2.1. Provision a new meta node
2.2. Remove the non-leader meta node
2.3. Add the new meta node
2.4. Confirm the meta node was added
2.5. Remove and replace all other non-leader meta nodes - Replace the leader node
3.1. Kill the meta process on the leader node
3.2. Remove and replace the old leader node
1. Identify the leader node
Log into any of your meta nodes and run the following:
curl -s localhost:8091/status | jq
Piping the command into
jqis optional, but does make the JSON output easier to read.
The output will include information about the current meta node, the leader of the meta cluster, and a list of “peers” in the meta cluster.
{
"nodeType": "meta",
"leader": "enterprise-meta-01:8089",
"httpAddr": "enterprise-meta-01:8091",
"raftAddr": "enterprise-meta-01:8089",
"peers": [
"enterprise-meta-01:8089",
"enterprise-meta-02:8089",
"enterprise-meta-03:8089"
]
}
Identify the leader of the cluster. When replacing nodes in a cluster, non-leader nodes should be replaced before the leader node.
2. Replace all non-leader nodes
2.1. Provision a new meta node
Provision and start a new meta node, but do not add it to the cluster yet.
For this guide, the new meta node’s hostname will be enterprise-meta-04.
2.2. Remove the non-leader meta node
Now remove the non-leader node you are replacing by using the influxd-ctl remove-meta command and the TCP address of the meta node (ex. enterprise-meta-02:8091):
# Pattern
influxd-ctl remove-meta <meta-node-tcp-bind-address>
# Example
influxd-ctl remove-meta enterprise-meta-02:8091
Only use
remove-metaif you want to permanently remove a meta node from a cluster.
For unresponsive or unrecoverable meta nodes:
If the meta process is not running on the node you are trying to remove or the node is neither reachable nor recoverable, use the
-forceflag. When forcefully removing a meta node, you must also pass the-tcpAddrflag with the TCP and HTTP bind addresses of the node you are removing.
# Pattern
influxd-ctl remove-meta -force -tcpAddr <meta-node-tcp-bind-address> <meta-node-http-bind-address>
# Example
influxd-ctl remove-meta -force -tcpAddr enterprise-meta-02:8089 enterprise-meta-02:8091
2.3. Add the new meta node
Once the non-leader meta node has been removed, use influxd-ctl add-meta to replace it with the new meta node:
# Pattern
influxd-ctl add-meta <meta-node-tcp-bind-address>
# Example
influxd-ctl add-meta enterprise-meta-04:8091
You can also add a meta node remotely through another meta node:
# Pattern
influxd-ctl -bind <remote-meta-node-bind-address> add-meta <meta-node-tcp-bind-address>
# Example
influxd-ctl -bind enterprise-meta-node-01:8091 add-meta enterprise-meta-node-04:8091
This command contacts the meta node running at
cluster-meta-node-01:8091and adds a meta node to that meta node’s cluster. The added meta node has the hostnamecluster-meta-node-04and runs on port8091.
2.4. Confirm the meta node was added
Confirm the new meta-node has been added by running:
influxd-ctl show
The new meta node should appear in the output:
Data Nodes
==========
ID TCP Address Version
4 enterprise-data-01:8088 1.5.x-c1.5.x
5 enterprise-data-02:8088 1.5.x-c1.5.x
Meta Nodes
==========
TCP Address Version
enterprise-meta-01:8091 1.5.x-c1.5.x
enterprise-meta-03:8091 1.5.x-c1.5.x
enterprise-meta-04:8091 1.5.x-c1.5.x # <-- The newly added meta node
2.5. Remove and replace all other non-leader meta nodes
If replacing only one meta node, no further action is required. If replacing others, repeat steps 2.1-2.4 for all non-leader meta nodes one at a time.
3. Replace the leader node
As non-leader meta nodes are removed and replaced, the leader node oversees the replication of data to each of the new meta nodes. Leave the leader up and running until at least two of the new meta nodes are up, running and healthy.
3.1 - Kill the meta process on the leader node
Log into the leader meta node and kill the meta process.
# List the running processes and get the
# PID of the 'influx-meta' process
ps aux
# Kill the 'influx-meta' process
kill <PID>
Once killed, the meta cluster will elect a new leader using the raft consensus algorithm. Confirm the new leader by running:
curl localhost:8091/status | jq
3.2 - Remove and replace the old leader node
Remove the old leader node and replace it by following steps 2.1-2.4. The minimum number of meta nodes you should have in your cluster is 3.
Replace data nodes in an InfluxDB Enterprise cluster
Data nodes house all raw time series data and metadata. The process of replacing data nodes is as follows:
- Provision a new data node
- Replace the old data node with the new data node
- Confirm the data node was added
- Check the copy-shard-status
1. Provision a new data node
Provision and start a new data node, but do not add it to your cluster yet.
2. Replace the old data node with the new data node
Log into any of your cluster’s meta nodes and use influxd-ctl update-data to replace the old data node with the new data node:
# Pattern
influxd-ctl update-data <old-node-tcp-bind-address> <new-node-tcp-bind-address>
# Example
influxd-ctl update-data enterprise-data-01:8088 enterprise-data-03:8088
3. Confirm the data node was added
Confirm the new data node has been added by running:
influxd-ctl show
The new data node should appear in the output:
Data Nodes
==========
ID TCP Address Version
4 enterprise-data-03:8088 1.5.x-c1.5.x # <-- The newly added data node
5 enterprise-data-02:8088 1.5.x-c1.5.x
Meta Nodes
==========
TCP Address Version
enterprise-meta-01:8091 1.5.x-c1.5.x
enterprise-meta-02:8091 1.5.x-c1.5.x
enterprise-meta-03:8091 1.5.x-c1.5.x
Inspect your cluster’s shard distribution with influxd-ctl show-shards.
Shards will immediately reflect the new address of the node.
influxd-ctl show-shards
Shards
==========
ID Database Retention Policy Desired Replicas Shard Group Start End Expires Owners
3 telegraf autogen 2 2 2018-03-19T00:00:00Z 2018-03-26T00:00:00Z [{5 enterprise-data-02:8088} {4 enterprise-data-03:8088}]
1 _internal monitor 2 1 2018-03-22T00:00:00Z 2018-03-23T00:00:00Z 2018-03-30T00:00:00Z [{5 enterprise-data-02:8088}]
2 _internal monitor 2 1 2018-03-22T00:00:00Z 2018-03-23T00:00:00Z 2018-03-30T00:00:00Z [{4 enterprise-data-03:8088}]
4 _internal monitor 2 3 2018-03-23T00:00:00Z 2018-03-24T00:00:00Z 2018-03-01T00:00:00Z [{5 enterprise-data-02:8088}]
5 _internal monitor 2 3 2018-03-23T00:00:00Z 2018-03-24T00:00:00Z 2018-03-01T00:00:00Z [{4 enterprise-data-03:8088}]
6 foo autogen 2 4 2018-03-19T00:00:00Z 2018-03-26T00:00:00Z [{5 enterprise-data-02:8088} {4 enterprise-data-03:8088}]
Within the duration defined by anti-entropy.check-interval,
the AE service will begin copying shards from other shard owners to the new node.
The time it takes for copying to complete is determined by the number of shards copied and how much data is stored in each.
4. Check the copy-shard-status
Check on the status of the copy-shard process with:
influxd-ctl copy-shard-status
The output will show all currently running copy-shard processes.
Source Dest Database Policy ShardID TotalSize CurrentSize StartedAt
enterprise-data-02:8088 enterprise-data-03:8088 telegraf autogen 3 119624324 119624324 2018-04-17 23:45:09.470696179 +0000 UTC
Important: If replacing other data nodes in the cluster, make sure shards are completely copied from nodes in the same shard group before replacing the other nodes. View the Anti-entropy documentation for important information regarding anti-entropy and your database’s replication factor.
Troubleshoot
Cluster commands result in timeout without error
In some cases, commands used to add or remove nodes from your cluster timeout, but don’t return an error.
add-data: operation timed out with error:
Check your InfluxDB user permissions
In order to add or remove nodes to or from a cluster, your user must have AddRemoveNode permissions.
Attempting to manage cluster nodes without the appropriate permissions results
in a timeout with no accompanying error.
To check user permissions, log in to one of your meta nodes and curl the /user API endpoint:
curl localhost:8091/user
You can also check the permissions of a specific user by passing the username with the name parameter:
# Pattern
curl localhost:8091/user?name=<username>
# Example
curl localhost:8091/user?name=bob
The JSON output will include user information and permissions:
"users": [
{
"name": "bob",
"hash": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"permissions": {
"": [
"ViewAdmin",
"ViewChronograf",
"CreateDatabase",
"CreateUserAndRole",
"DropDatabase",
"DropData",
"ReadData",
"WriteData",
"ManageShard",
"ManageContinuousQuery",
"ManageQuery",
"ManageSubscription",
"Monitor"
]
}
}
]
In the output above, bob does not have the required AddRemoveNode permissions
and would not be able to add or remove nodes from the cluster.
Check the network connection between nodes
Something may be interrupting the network connection between nodes.
To check, ping the server or node you’re trying to add or remove.
If the ping is unsuccessful, something in the network is preventing communication.
ping enterprise-data-03:8088
If pings are unsuccessful, be sure to ping from other meta nodes as well to determine if the communication issues are unique to specific nodes.