n3s0 || journal

    Sysadmin Pain: When Percona and MySQL Community Repos aren't Synced

    Posted on 4 mins

    MySQL Sysadmin Pain

    Just thought I’d talk a little bit about some Sysadmin pain that I’ve come across. Might prove beneficial in the future. Not complaining. This is more of a “don’t do what I did” article. I’ll probably provide more of these as I run into issues.

    Percona Xtrabackup is a wonderful tool for conducting backups on MySQL servers. Especially when users come up with one off issues where they weren’t paying attention and found themselves deleting a mound of data by accident. It’s fine. That’s what we have backups for right?

    Though, what happens when your backups start failing because the database is utilizing a mix of different repositories? Backups start failing because Percona MySQL Xtrabackup is built for a specific version of MySQL. I get it.

    Backup status of something like this should probably be monitored. Which thankfully it was in this case. This is how it was caught.

    I’ve run into an issue a couple of times where the Percona Xtrabackup repositories and the MySQL Community backup repositories are out of sync on RHEL systems. This can prove problematic when I’ve conducted patching and it takes a little while for Percona to catch up with MySQL Community and visa versa.

    Generally, in this case. When the packages are out of sync. The only option is to wait because it’s unsafe to rollback to the previous version of MySQL. You just hop on the server here and there and check to see if the packages are updated. OR, check the repositories to see which version is being used. Within this time you could have multiple failed backups. Sometimes for weeks to a month.

    Vendor lock-in is a contributing factor of this problem. Percona has their own MySQL Server packages and it seems like the expectation is to utilize both Percona packages for your database needs.

    One thing I would do if you have a mixed Percona Xtrabackup and MySQL Community environment is to check to see if the updates are on the same version before proceeding. If they are, exclude the package from the update. If they are. Fire away! I haven’t checked Debian or Ubuntu systems to see if they have the same issue. But, that goes into another thing I would like to touch up on.

    But, to pull back from the next point. Here are some commands for checking to verify that packages will stay in sync.

    On CentOS 7 system. Check to see if the packages will be updated. If the update shows the version as not the same. Then it should be excluded from the update.

    yum check-update

    On CentOS 8/Stream and Fedora Server the following command can be issued to look into if the repositories are in sync before updating. If they aren’t in sync. Exclude them. If they are. Go right ahead.

    dnf check-update

    If already updated and curious if backups are probably failing. One way to indicate this is to check the mysql-community packages and the percona-xtrabackup package versions. Just to determine if they’re the same. Backups will fail even if the minor version number of the application isn’t in sync.

    CentOS 7 check current version for MySQL. This will provide the version number in the second column.

    yum list installed | grep -i mysql

    CentOS 7 check current version for Xtrabackup. This will also provide the version number in the second column.

    yum list installed | grep -i xtrabackup

    Same for MySQL on most recent Fedora distros and Centos 8/Stream.

    dnf list installed | grep -i msyql

    Same for Xtrabackup on most recent Fedora distros and Centos 8/Stream.

    dnf list installed | grep -i xtrabackup

    Now to the next point. If I would have read the documentation or just installed Percona MySQL Server instead of mixing two separate repositories. I probably wouldn’t be in this mess that I find myself in from time to time.

    Again. This is more of a shame on me for not paying attention to the packages I’m updating MySQL to. This is also a shame on me for not moving the database over to the locked in version of Percona MySQL when the packages were finally in sync. We live and we learn. Hopefully this was comprehensible so others can learn too.

    Another option that probably needs to be explored is not using Xtrabackup at all. Although, it’s a wonderful product. Locking in with a specific vendor creates a dependency that I don’t want/need in my environment. Percona is a wonderful company and they provide a lot of great options for database software. But, they aren’t the only database solution out there. They also don’t the only MySQL backup solution out there.

    I’ll still experiment with their products and provide posts. But, I think it’s time to expand to different things and find a backup solution that provides more flexibility regarding backups.