Etcdserver: mvcc: database space exceeded

teleport
#1

Ported from support portal

Hello,

We encountered the following error when trying to log into our Teleport 3.1 cluster.

etcdserver: mvcc: database space exceeded

I looked up the error and found the error in here: https://coreos.com/etcd/docs/latest/faq.html

I did as the documentation said by compacting, defragging, and clearing the alarm, but my question is why did we get this error? The different in usage between our Teleport v2 and Teleport V3 cluster is huge, and Teleport V2 is still getting tons more usage.

#2

I believe I figured it out. We didn’t have automatic revision compaction enabled, and for whatever reason, the Teleport 3 cluster caused a lot of revisions to build up. Once I cleared the old revisions and ran a defrag, the database size went from 2.1GB down to around 175kb. I’ve updated the etcd startup options to enable automatic discarding of revisions prior to 6 hours old and will be deploying the change to the rest of our etcd hosts next week.

#3

More detailed description of the solution is here:

#4

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.