0

Damaged community forum, lost data

Last night was not really a good one for our new NodeBB community forum. I was working on other project, with, as usual indecent number of tab opened from Chrome. When I went back on my community tab, it asked me to login back ?

Hey what’s ? strange, does not ask me to authenticate when I’m already authenticated. Never mind, I clicked login then selected google auth and once logged, strange screen (sorry did not took a picture). No more categories, no more post, well just a starting screen with 2 poor unaligned icons!

1st check

My first tough was, “ok nodbb got a problem”, so I sshed to my hosted VM, go to nodeBB install folder and restart it with  ./nodebb restart

Refreshed my browser, same thing, launched another browser (Firefox) same thing, oups!

2nd check, humm, I don’t like this

Ok, let restart redis database, /etc/init.d/redis-server restart and restart nodebb   ./nodebb restart

same thing, grrr!

3st check, It smells bad !

Ok, learned my base IT, may be should I started with this, let restart everything rebooting the computer with reboot

Waited for all back to life, refresh browser, and same thing, oh my god, this will bring me into trouble, it’s almost 03:00AM not sure I’ll start my night soon !!

Going into deep investigation

Connect to database with with redis desktop manager, I already have some experience with nodeBB and redis database structure, so I know how it should look like, something like that (db0 is for emoncms database, db1 is nodeBB)

NodeBB redis database structure

NodeBB redis database structure

But when I connected there was only 2 folders (once again I was in a hurry and did not took time to capture screen and remember the folder names)

Lucky, or not, when I was writing this article redis clear happened again, so here is the picture

NodeBB redis database structure cleared

NodeBB redis database structure cleared

So at this point I found the problem, my database is corrupted !!

Check this assumption

Ssh again to my server and take a look at the database files going into  /var/lib/redis

Ohhhh, WTF? dump.rdb  file is 20263 bytes and last backup 2 months ago size if 3MB ? Ok my database went flushed for a reason I’m still ignoring. It was working fine 3 hours ago. May be I’ve been hacked even with protection I’ve put in place to avoid this.

Ok, let’s put old backup back in place, but it’s 2 months old, my fault, I should have put backup in place!

Ok everything is now back but I’ve lost 2 months data, that’s weird. I lost users, posts, topics, all my community value!!!

Check log files

Ok now it’s time to check log files. As it was working on Nov 07  at 23H45, the problem happened between this time and the time I detected it, Nov 08 at 02:51 AM, that’s about 3 hours windows.

redis

As you can see everything is fine until I restarted redis at 02:52, so redis says no problem!

nodebb

As you can see everything is fine (except the github auth error see this unsolved ticket) until I restarted nodebb at 02:51, so nodeBB says no problem!

Configuration files

Here the needed configuration file for those interested, if anyone as a answer, let me know

nodebb config.json

 

redis.conf

nginx sites-enabled.conf

nginx.conf

Any comments or help to understand this problem and avoid it again are welcome!

By the way I apologize for this, I’ve lost 2 month of living data and interesting posts and topics. Users account registered in the interval are also lost, it’s a big loss for my community and all fans.

To avoid this happening again, I’ve implemented a backup every hour now !! but it seems the problem is happening to other users, so I don’t think I’ve been hacked (and to logs confirms this point)

A ticket has been openened by another user on nodeBB forum

Charles

Charles

Comments on this topic in community Forums.