Rev. Voodoo

I have received some requests from WP users about how exactly to use access logs to track down rogue php files on their server which can be used to reinfect their blogs after they thought they were nice and clean.  This write-up is from my experience using Godaddy as a host, but the process would be nearly identical on any host where you have access to your access logs.

I unfortunately did not save a copy of my access log from when I was hacked.  However, I can try to describe a bit better what I did/found.  I had never seen an access log before being hacked either, and so it was a bit daunting.  But here is what I found. 

First off, here is a sample line from an access log (This is a legitimate log entry, not from a hack….I’m just using it to try to explain what you are looking at a bit):

208.106.281.6 – - [01/Mar/2010:02:26:53 -0700] “POST www.rvoodoo.com/empire/wp-cron.php?doing_wp_cron HTTP/1.0″ 200 0 “-” “WordPress/2.9.2; http://www.rvoodoo.com”

So that’s a possible line from an access log.  First in that line is an IP address (208.106.281.6) you can kind of ignore that for now, it’s not important to this investigation.

Next up, we have the timestamp ([01/Mar/2010:02:26:53 -0700]) this is pretty important, it’ll help you match things up, I’ll explain that in a minute

Next up, the method (POST), it’ll be HEAD, POST, or GET……frequently for a hack, you are looking for the POST method

Then we have the url of the file being affected (www.rvoodoo.com/empire/wp-cron.php?doing_wp_cron ) – this is important, helping us find things which I’ll explain momentarily

Then there is this stuff between the urls, which isn’t so important for us, its user agents and whatnot (HTTP/1.0″ 200 0 )

And finally, the second url, (WordPress/2.9.2; http://www.rvoodoo.com) this tells us which file is doing the action, it’s also important…..this could point to your rogue file if you have one

So now you have a pretty basic understanding of what the lines in your access log look like, and how to read them…..how to use this info to investigate for rogue files?

It’s pretty simple really.  When your files get hacked, some code is added to them…..well, when  that happens…the timestamp of the file changes in your godaddy file manager.  So, find a file that has been hacked.  Check the timestamp.  Now, open your access log for that date.  Cross reference in your access log for the time and date that your hacked file was changed.  Especially look, at that date and time, for any requests using the POST method.  Then, in the first url find the path to your file that was changed.  When you find the url to the file that had been changed, take a look at the second url in that line of your log.  The second url will be the path to the file which was used to change the hacked file.  For instance for me it was (http://www.rvoodoo.com/shop/includes/products/2008/06/images/settings.php).  This file is totally outside of my Wordpress install, but it was accessing my WP install for some reason.  When I went to that folder and took a look at the code from that file from within Godaddy’s file editor, it was just a bunch of gibberesh.  So I deleted the file, and along with the other cleanup steps in the reading links I provided, that took care of my problem!

Hopefully this can help you track down whether or not you have any rogue files sitting around, or at least give you a bit of a better understanding on how to use your access logs!  Good luck!

Join the forum discussion on this post

Seems like more and more, I see people having a terrible time with their Wordpress installs being hacked.  While I’m no expert on the internets, or security, or anything like that….I did suffer through 2 rounds of attacks here on The Voodoo Empire, and recovered fully….and learned a lot.  So here is the process I went through.  The first time I wasn’t thorough enough, and that is how it happened again.  I’ve been clean for months now, after doing things properly the second time.  Before I describe my process, here are some great reads that pretty much set me straight and got me going on the cleanup process.

• http://codex.wordpress.org/FAQ_My_site_was_hacked
• http://ocaoimh.ie/did-your-wordpress-site-get-hacked/
• http://smackdown.blogsblogsblogs.com/2008/06/24/how-to-completely-clean-your-hacked-wordpress-installation/
• http://www.snipe.net/2010/01/when-wordpress-gets-hacked/

These got me going through the cleanup process, and one other good link, after you get everything cleaned up:

• http://codex.wordpress.org/Hardening_WordPress

So that’s the required reading out of the way….now onto my process.

I first noticed some gibberesh (the base64 stuff) showing up in my theme’s files when I went in to do some editing.  It had been months since I had worked on my theme, so who knows how long the spam had been in there.  I deleted all the rogue code from all my theme’s files and thought I was golden.  Until it showed up again, I layed the find and delete game for some time…..but it kept coming back.  Then I started to dig around some more…..The Voodoo Empire has 5 Wordpress installs, and at the time we had Drupal, Joomla, ZenCart, MediaWiki, and many other software packages.  This totals up to a few thousand php files on the server.  Each and every php file had the base64 stuff in it.  So next up, every php file was either cleaned, or replaced.  I thought I’d had it beat.  A month later it was back.  I’d developed the habit of viewing my source code from within my browser each time I visited my websites now to look for bad stuff.  But this time, even though there was spam in my source code, there wasn’t any files altered.  So I dug and dug, and finally found a php file that didn’t belong buried about 4 level deep.  It was inserting spam links.  I deleted it and thought I was good.  But alas, it was too good to be true…..soon, all my php files on my server got altered again.  This time though, I did some research and finally cleaned up properly.

First thing I did was change all my passwords.  FTP, WP (and all other programs), all DB passwords, etc.  After I changed all my passwords, I took a look at one of the files that had been altered, and checked out it’s timestamp.  My host (GoDaddy) offers server access logs to peruse at my leisure (if you’ve never seen an access log, they are scary).  I had no idea what was in a log, I’d never seen one, but I decided to dive in.  I checked my access logs agains the times that the altered files had last been changed and found that a rogue php file buried way deep in the 2008 uploads folder of a different WP install was affecting my main WP install.  Upon further investigation, I found another rogue php file that was altering my other files.  So I deleted the rogue php files.  I then reinstalled every single software package on my server.  Then I went in and manually cleaned the rest of the php files that didn’t get replaced (WP theme files, wp-config file, etc).  After that it was on to plugins.  I deleted all plugins and installed from clean downloads.  So now every php file on my server was accounted for, and clean.  No rogue php files existed anywhere.  No rogue code was added on to any of my existing php files. 

It was time to finish thing up….on to the database.  I manually scanned through exports of my database looking for any suspicious code that I found, and ran some basic sql queries on my existing database looking for suspicious code, and got my database all squeeky clean.  And now, with everything nice and clean, I changed up all my passwords once again.

There you have it!  My long and painful travel to a clean WP.  Keep everything up to date, WP installs, themes, plugins, etc.  Check your source code often. Maybe install some of the various security plugins that are out there (none of them stopped my attacks, but they do have a use).  And if you do get hacked…..be thorough.  If you are not, you are just wasting your time!  Remember, server access logs can be your friend!  Hopefully you can avoid being hacked, if you are reading this because it happened to you, I hope I helped you just a little, be patient, be thorough, and good luck!

Check out my post here if you are looking for more information on rogue files, and using your access logs to find them.

Join the forum discussion on this post

Rev. Voodoo just stumbled across a few discussions on the Wordpress Forums about setting up post thumbnails.  This is obviously a very cool feature, and we set about integrating it immediately into our theme.  You can enjoy the results on this very post.  While that is cool and all, it got us thinking.  We already include nice updates from the blog over on our main Voodoo Empire homepage…. but this new feature allows us to incorporate photos on that page real easily…..a feature that was not available to us previously.  It definately adds a touch more interest to the blog updates over there!

Join the forum discussion on this post

Rev. Voodoo continues to work hard at recovering from the latest hacks.  The front end of all the websites look and feel fine once again.  This website, along with Ankhjii’s Fitness Site, AngelaandJamie.com, and The Voodoo Empire main page are fully recovered.  That still leaves the wedding services site, DJ Odawa’s page, the forums, and the wiki.

With these problems, and especially with the restoration we continually have to go through, we are considering paring down our sites once again.  Mainly, the forums.  It looks like sometime over the next few months, there will be a major new release of Wordpress 3.0.  Perhaps after that happens, we may try to merge in a WP based forum, instead of the SMF forum we use now.  The more consolidated things are, the easier it will be to recover. 

Until then though, you can continue to count on us fighting the good fight to keep things running here, and all around the empire!

It seems yesterday, just after Rev. Voodoo spent about 2 hours upgrading Wordpress to it’s newest release, v2.9, we were hit pretty hard by another hack.  So, if you see anything strange going on around the Empire, or if something doesn’t work quite right.  We are probably working on it.  Feel free to drop us a line and let us know exactly what is wrong.  This site, and the main Voodoo Empire site are currently totally clean.  It’ll take a little bit to get everything all cleaned up though.  Luckily this time, the offending rogue php file was not well hidden, so the source has been deleted….now its just a matter of cleaning up the damage.  It seems that this may be something going around with people hosted at Godaddy.  We’ve talked to several people who were hit pretty hard.  It may be an issue on their end, or it may be a coincidence.  Regardless, they won’t be helping any.

Well, Rev. Voodoo has been a big fan of the Wordpress Support forums for some time.  The kind folks over there have helped this site become what it is today.  Many times during development, Rev. Voodoo has had to head over there to search for answers to problems, and even post many questions!  Yes, Rev. Voodoo is not allknowing!  Over time Rev. Voodoo has been able to gather a few tidbits of knowledge, and has tried to return the favour to the forums.  So this little shoutout over there was pretty nice!  Thanks folks!

Seems Rev. Voodoo had a setting messed up on here.  I just noticed that when I replied to most of the comments people have made on the site, about half didn’t show up.  Strange!  Anyway, all fixed, you won’t be ignored any more!  So talk to me people!

Rev. Voodoo is trying to get better integration between websites, this is a test of that.

Luckily Rev. Voodoo frequently checks on the status of all Voodoo Empire websites.  This attack was caught just after it happened, and before any damage was done, such as being delisted from Google.  All is currently well….but I am confused.  According to several posts around the Wordpress community, including this one, as long as I keep The Voodoo Empire up to date with the latest Wordpress releases, all should be well in the security world.  But that just ain’t the case.  I always upgrade this site on the day a new version is released.  But we still got hit….  Oh well…Rev. Voodoo will maintain vigilence in protecting the site, andhopefully you all will continue to enjoy our offerings!

After launching the new Voodoo Empire homepage, and adding a filesharing service skinned to match, Rev. Voodoo thought it might be cool to build up a new theme for his blog.  That is what you are looking at now!  It still needs some tweaks here and there, but it’s near complete!  It fits in real well with the overall Empire feel.  Now that this site is upgraded, the previous theme will be released to the public very shortly!