0

Damaged community forum, lost data, again !!!!

Sometime I wish you an happy new year with music mixes, this time, this new year start by an issue, with redis database server, again. Looks like 2 years back with this post.

And we just lost 4.5 months of community data on the NodeBB community forum. That’s insane and a great lost for everybody.

How all started

On August 2017 I moved personal community forum from ovh VPS (debian 8.6) to Azure CH2i VM (debian 8.10). I spend one day for full migration (nodebb, redis and some other things) and all worked fine. Then I spent another 1/2 day to make redis-database backup working. I was unable to make this working even if redis-server was same version. Finally I got it working (not really sure what I done, typed so much command in this day) but backup file was created, so I though all was OK

Then Microsoft Azure maintenance email came

On December I received a maintenance update for my azure VM, no problem, got it

Reboot does not mean all is okay

2 days ago while I was working on the VM, I remember that this one needed a reboot, so as usual I issued a   sudo reboot then checked CH2i web site was back and left all alone, it was working (at least at first view)

Happy new year, start fine

Today I needed to read back a article I wrote on this blog to put a customer Raspberry PI Zero into read-only mode and I noticed that the comments on bottom of the article was missing (and some was very useful).

So I went to community forum web site and it was down (makes sense why I had no comments). No problem ssh on the server and look at nodeBB logs. It’s indicate nodeBB can’t start because it need a database schema update for current running nodeBB version. Humm, that’s strange but issue the famous ./nodebb upgrade , and then ./nodebb start and nodeBB came back to life.

But, … WTF?

Last forum post is dated from August 10th 2017? whoo, ok maybe a hacker got a security hole in nodeBB or redis-database and played with post (it already happened), so no problem I’ve got backup every day. restore the redis-database backup and yes it is, start again !!!

But why again nodeBB ask for upgrade database, it was working 2 days ago, I didn’t upgraded nodeBB version so what ?

You know what, WTF again

Hey, with database restore of last day, same things, data are 4.5 month old !!!

Ok go to server again, and take a look at those backup

As you can see there is a backup every day at 11H13, but all the backup have the same size (except the last one after reboot)

What’s wrong, does this mean that all backup are with data from August, sounds like so !!!

What happened on reboot

Even if backup are wrong, why a clean shutdown/reboot break the current redis database ?

Let’s look at the log when rebooted the VM, what DB saved to disk ? that’s what redis says

So what, looks like something went wrong and I don’t know what. But what I know, for sure, is that we lost more than 4 months of data, and I apologize and feeling bad for this, how can this happen ? and how to prevent to happen again ?

Conclusion

I’m just stuck, For now I opened back a ticket on github because of course I’m not the only one who encounter this issue.

Any comments or help to understand this problem and avoid it again are welcome!

By the way I apologize for this, I’ve lost 2 month of living data and interesting posts and topics. Users account registered in the interval are also lost, it’s a big loss for my community and all fans.

To avoid this happening again, I’ll investigate on backup and check them.

While writing this post, a user posted a new message on forum, so after that, I rebooted the server, and the message is still there, unbelievable it seems to work and I didn’t changed anything.

Charles

Charles