Today I faced a weird problem with elasticsearch and I don't know how to fix the issue. The scenrario is somewhat like this
Elasticsearch cluster with 2 nodes : Total docs 1000
Now one of my server goes down and all write and reads are handled by second server. Now say there are 10000 docs in the ES2.
Due to system problem now whole elasticsearch network is down and both ES1 and ES2 are down. Now somehow I manually go and make ES1 up and now its taking all the write requests.
ES2 comes up and now discovers ES1 as its master.
ES2 syncs up with master and all the data written to ES2 node is lost.
Is there any way to recover the lost data. Is this an expected behaviour in distributed system?
Please let me know if something is not clear.