Rev. Voodoo

I have received some requests from WP users about how exactly to use access logs to track down rogue php files on their server which can be used to reinfect their blogs after they thought they were nice and clean.  This write-up is from my experience using Godaddy as a host, but the process would be nearly identical on any host where you have access to your access logs.

I unfortunately did not save a copy of my access log from when I was hacked.  However, I can try to describe a bit better what I did/found.  I had never seen an access log before being hacked either, and so it was a bit daunting.  But here is what I found. 

First off, here is a sample line from an access log (This is a legitimate log entry, not from a hack….I’m just using it to try to explain what you are looking at a bit):

208.106.281.6 – - [01/Mar/2010:02:26:53 -0700] “POST www.rvoodoo.com/empire/wp-cron.php?doing_wp_cron HTTP/1.0″ 200 0 “-” “WordPress/2.9.2; http://www.rvoodoo.com”

So that’s a possible line from an access log.  First in that line is an IP address (208.106.281.6) you can kind of ignore that for now, it’s not important to this investigation.

Next up, we have the timestamp ([01/Mar/2010:02:26:53 -0700]) this is pretty important, it’ll help you match things up, I’ll explain that in a minute

Next up, the method (POST), it’ll be HEAD, POST, or GET……frequently for a hack, you are looking for the POST method

Then we have the url of the file being affected (www.rvoodoo.com/empire/wp-cron.php?doing_wp_cron ) – this is important, helping us find things which I’ll explain momentarily

Then there is this stuff between the urls, which isn’t so important for us, its user agents and whatnot (HTTP/1.0″ 200 0 )

And finally, the second url, (WordPress/2.9.2; http://www.rvoodoo.com) this tells us which file is doing the action, it’s also important…..this could point to your rogue file if you have one

So now you have a pretty basic understanding of what the lines in your access log look like, and how to read them…..how to use this info to investigate for rogue files?

It’s pretty simple really.  When your files get hacked, some code is added to them…..well, when  that happens…the timestamp of the file changes in your godaddy file manager.  So, find a file that has been hacked.  Check the timestamp.  Now, open your access log for that date.  Cross reference in your access log for the time and date that your hacked file was changed.  Especially look, at that date and time, for any requests using the POST method.  Then, in the first url find the path to your file that was changed.  When you find the url to the file that had been changed, take a look at the second url in that line of your log.  The second url will be the path to the file which was used to change the hacked file.  For instance for me it was (http://www.rvoodoo.com/shop/includes/products/2008/06/images/settings.php).  This file is totally outside of my Wordpress install, but it was accessing my WP install for some reason.  When I went to that folder and took a look at the code from that file from within Godaddy’s file editor, it was just a bunch of gibberesh.  So I deleted the file, and along with the other cleanup steps in the reading links I provided, that took care of my problem!

Hopefully this can help you track down whether or not you have any rogue files sitting around, or at least give you a bit of a better understanding on how to use your access logs!  Good luck!

Join the forum discussion on this post

Seems like more and more, I see people having a terrible time with their Wordpress installs being hacked.  While I’m no expert on the internets, or security, or anything like that….I did suffer through 2 rounds of attacks here on The Voodoo Empire, and recovered fully….and learned a lot.  So here is the process I went through.  The first time I wasn’t thorough enough, and that is how it happened again.  I’ve been clean for months now, after doing things properly the second time.  Before I describe my process, here are some great reads that pretty much set me straight and got me going on the cleanup process.

• http://codex.wordpress.org/FAQ_My_site_was_hacked
• http://ocaoimh.ie/did-your-wordpress-site-get-hacked/
• http://smackdown.blogsblogsblogs.com/2008/06/24/how-to-completely-clean-your-hacked-wordpress-installation/
• http://www.snipe.net/2010/01/when-wordpress-gets-hacked/

These got me going through the cleanup process, and one other good link, after you get everything cleaned up:

• http://codex.wordpress.org/Hardening_WordPress

So that’s the required reading out of the way….now onto my process.

I first noticed some gibberesh (the base64 stuff) showing up in my theme’s files when I went in to do some editing.  It had been months since I had worked on my theme, so who knows how long the spam had been in there.  I deleted all the rogue code from all my theme’s files and thought I was golden.  Until it showed up again, I layed the find and delete game for some time…..but it kept coming back.  Then I started to dig around some more…..The Voodoo Empire has 5 Wordpress installs, and at the time we had Drupal, Joomla, ZenCart, MediaWiki, and many other software packages.  This totals up to a few thousand php files on the server.  Each and every php file had the base64 stuff in it.  So next up, every php file was either cleaned, or replaced.  I thought I’d had it beat.  A month later it was back.  I’d developed the habit of viewing my source code from within my browser each time I visited my websites now to look for bad stuff.  But this time, even though there was spam in my source code, there wasn’t any files altered.  So I dug and dug, and finally found a php file that didn’t belong buried about 4 level deep.  It was inserting spam links.  I deleted it and thought I was good.  But alas, it was too good to be true…..soon, all my php files on my server got altered again.  This time though, I did some research and finally cleaned up properly.

First thing I did was change all my passwords.  FTP, WP (and all other programs), all DB passwords, etc.  After I changed all my passwords, I took a look at one of the files that had been altered, and checked out it’s timestamp.  My host (GoDaddy) offers server access logs to peruse at my leisure (if you’ve never seen an access log, they are scary).  I had no idea what was in a log, I’d never seen one, but I decided to dive in.  I checked my access logs agains the times that the altered files had last been changed and found that a rogue php file buried way deep in the 2008 uploads folder of a different WP install was affecting my main WP install.  Upon further investigation, I found another rogue php file that was altering my other files.  So I deleted the rogue php files.  I then reinstalled every single software package on my server.  Then I went in and manually cleaned the rest of the php files that didn’t get replaced (WP theme files, wp-config file, etc).  After that it was on to plugins.  I deleted all plugins and installed from clean downloads.  So now every php file on my server was accounted for, and clean.  No rogue php files existed anywhere.  No rogue code was added on to any of my existing php files. 

It was time to finish thing up….on to the database.  I manually scanned through exports of my database looking for any suspicious code that I found, and ran some basic sql queries on my existing database looking for suspicious code, and got my database all squeeky clean.  And now, with everything nice and clean, I changed up all my passwords once again.

There you have it!  My long and painful travel to a clean WP.  Keep everything up to date, WP installs, themes, plugins, etc.  Check your source code often. Maybe install some of the various security plugins that are out there (none of them stopped my attacks, but they do have a use).  And if you do get hacked…..be thorough.  If you are not, you are just wasting your time!  Remember, server access logs can be your friend!  Hopefully you can avoid being hacked, if you are reading this because it happened to you, I hope I helped you just a little, be patient, be thorough, and good luck!

Check out my post here if you are looking for more information on rogue files, and using your access logs to find them.

Join the forum discussion on this post

Rev. Voodoo just stumbled across a few discussions on the Wordpress Forums about setting up post thumbnails.  This is obviously a very cool feature, and we set about integrating it immediately into our theme.  You can enjoy the results on this very post.  While that is cool and all, it got us thinking.  We already include nice updates from the blog over on our main Voodoo Empire homepage…. but this new feature allows us to incorporate photos on that page real easily…..a feature that was not available to us previously.  It definately adds a touch more interest to the blog updates over there!

Join the forum discussion on this post

If this works, it’ll add thumbnail support automatically to the theme in a couple of places.  Nice, tidy resizing.  Lets see!

Well, Rev. Voodoo has a brand new new Voodoo Forums setup, and is considering a full launch of it.  It can be found here.  Or just by clicking the word forums up in the text navbar on every page.  It’s tightly integrated with this site which is kind cool.  Thing is, if I do the switch, the old users and posts won’t be here.  This forum will be a brand new start.  I think my database over at the old forums might be corrupt…if that is the case, it’ll pretty much force my hand and I’ll close the old forum and open the new one.  Check them both out…see what ya think.  (Note, the old forum doesn’t have our custom Voodoo Empire theme on it at the moment, I’m having a hard time with admin functions).  If you feel like posting in our forums, go ahead and use the new forum…..

Join the forum discussion on this post

Well, as Rev. Voodoo finds time, more sites are being fixed from the latest hack a couple weeks ago.  This time the Voodoo Empire Filesharing service is completely restored.  Everything is testing and ready to go.  It’s the location we keep our branding stuff at, Voodoo Empire art, logos, etc….. there’s also some music there.  But you can also use it if you need a place to dump some files, or if you want to get some music to me to post.  It’s got a pretty large upload limit, and we are pretty generous with the storage space.  Plus, if you need more you can ask us….we can raise your limit or even give you ftp access to upload big stuff, just in case your internet connection resets you on big browser based uploads!

Due to the recent hacks that hit the Voodoo Empire, we are going to take the opportunity to integrate the shop directly into the Rev. Voodoo website.  The products are still available online at our Zazzle shop and cafepress shops.  It’s too hard to keep trying to dig out multiple software installations every time something happens, so we will try to pull of some tighter integration, and see where we end up!

It seems yesterday, just after Rev. Voodoo spent about 2 hours upgrading Wordpress to it’s newest release, v2.9, we were hit pretty hard by another hack.  So, if you see anything strange going on around the Empire, or if something doesn’t work quite right.  We are probably working on it.  Feel free to drop us a line and let us know exactly what is wrong.  This site, and the main Voodoo Empire site are currently totally clean.  It’ll take a little bit to get everything all cleaned up though.  Luckily this time, the offending rogue php file was not well hidden, so the source has been deleted….now its just a matter of cleaning up the damage.  It seems that this may be something going around with people hosted at Godaddy.  We’ve talked to several people who were hit pretty hard.  It may be an issue on their end, or it may be a coincidence.  Regardless, they won’t be helping any.

Well, it was way easier than I had suspected to set up this site to automatically resize and photos uploaded by email.  It was literally 2 lines in my css.  So that’s it, no more giant pictures.  It took a bit of thinking to get my layout looking right, but it seems to now.  However, I can only check my website on one browser here from work.  So take a look around, if something looks bad, why not leave me a comment so I can get it fixed?

It’s been just over a week since Rev. Voodoo started working dayshift.  That’s right….dayshift…..after 12 years!  So far, I’m surviving…..but it’s pretty strange keeping these kinds of hours!  And now I’ve actually got things to do at work, which sure cuts into my design time!  But I’ll still find time for my projects!  I may have to begin doing my design work at home!  Horrors!  On that note, keep your eyes peeled for a full relaunch of the Digital Dissent website.  We are planning a full relaunch of the main webpage, to cater to a successful multimedia business design, along with a launch of the DigitalDissent blog!  It’s pretty exciting.  Watch for the launch, check in at the site from time to time to see what changes, and definitely consider them for your photo/video needs!