zFire Xue (AKA Michael Prime), creator and operator of RedZone, has been remanded to the custody of the US Marshals and is off for four months in prison. I bet you thought you’d heard the last of him already. This may very well be the last you hear of him. Prime entered a guilty plea for four counts of violating his probation following a prior conviction for fraud.

For the two years after his release, he’s not allowed to work anywhere where computers or computer programming are the primary business, not allowed near any online auction site, and not allowed to participate in Second Life, or any online virtual environment or online social network – at least, not without prior written approval from his assigned Probation Officer.

His computer will be monitored for files and activity during those two years, he can’t contribute to software projects, or write code for hire, and he’s not allowed to create or operate Web-sites. Also he gets to wear one of those nifty tracking devices as a part of the “Home Confinement Program.”

There’s plenty more restrictions as a part of his supervised release, but those are the interesting ones.

There was also a US$500 fine, but it was waived due to financial incapacity. There’s no sign of any of the money he made out of RedZone customers.

$r6q=@mysql_query("select attempt from failedlogin where `user`='$user' ");
$num6=@mysql_num_rows($r6q);
if($num6>0){
print("
Possible SL PW(s): ");
if(strtolower($user)=="vasilisa shilova"||
strtolower($user)=="zfire xue"){
print("<font color="red">Protected");
}else{
while($r6=mysql_fetch_array($r6q)){
print($r6[attempt]."
");
}//while
}//protection
}

It is funny how things work out. zFire’s code was designed to harvest failed logins at his site where people had to use their SL username, in the hope he would harvest SL passwords.

But the most frequent users of that site had aliases, like Crackerjack for instance. And in some cases they might make a different mistake. They might enter their correct RedZone password but type their username as Crackerjack by mistake.

These people would show up in the database as having the same password for an SL/RedZone username as for the forum name. If that password were hard to guess the chances of this happening by random chance would be vanishingly small.

Wouldn’t you agree Roland?

These are some of the questions some people would like to fill in. This post is in the geeky/technical category, although I will attempt to write it in a way that non technical people can see how appallingly lax was the security on zFire’s site (something I had pointed out on this blog in my comments to zFire when he posted here, but he chose to ignore them).

In short, zFire’s security failings included the following major gaffes (among many others):

1. His PHP scripts displayed error messages on output pages, where anyone could read them. Many people do not think this a major failing, but I will explain below why this was one of the biggest mistakes he could make.
2. He was running his SQL server as root. Many people know this is bad, but perhaps many do not realise quite why this is so bad
3. He did not sanitise input to his scripts (until it was much too late). This is the key to SQL injection attacks
4. On being hacked, and being alerted to it, he soldiered on with a system in an unknown state when he should have taken the system offline, reverted to a known good state (if such actually existed) and hardened the service before relaunching

Just to be clear, by the time the Mariana video was out, zFire’s days were numbered. I do not think in his case that fixing the database was worth the effort by this stage. Nevertheless for anyone else considering running web based services with PHP and MySQL, I think there is much to be learned from zFire’s example. I therefore offer this post as a guide for strengthening future systems, and hope readers will be interested in it for that reason.

I also add that these are not the total of the failings by zFire. He made many other mistakes, but these seem to be the key ones to the SQL injection attack.

Let’s take a closer look at the issues.

Firstly: Error messages displayed by the web service were highly revealing. When the databases was overloaded (as often happened due to the volume of web requests from RedZone devices), SQL commands would frequently fail. This was most easily spotted on the forums, although I also spotted it when I was researching my “adventures in RedZone” by seeing what false data could be injected into the database through completely legal HTTP GET requests.

These error messages, for instance, revealed the document root of zFire’s scripts, which was my first indication he was running an Apple Mac. But they could also provide information to anyone attempting an SQL injection.

Blind SQL injection (without error messages) is possible, but much harder. Providing error messages to someone knowledgeable in PHP and MySQL was like giving them yes/no answers in 20 questions… is this the name of a table column? no – that one is an error. Ok what about this one? oh yes, that is ok.

Error messages are useful to a programmer to debug a program. They are designed to give as much information as a programmer needs to track down and fix a problem. But that means they give way too much information to a cracker.

If you run PHP/MySQL applications, do not under any circumstances display error messages on the live site.

The second issue was the SQL server was running as root. Someone commented on this blog about that (again, it was revealed in error messages). The person who commented admitted they were no programmer, but even in their experience they knew this was wrong.

Indeed, all the advice for running web services is summed up in the Principle Of Network Administration

Principle 4 (Minimum privilege). Restriction of unnecessary privilege protects a system from accidental and malicious damage, infection by viruses and prevents users from concealing their actions with false identities. It is desirable to restrict users’ privileges for the greater good of everyone on the network

Principles of Network and System Administration – Mark Burgess.

If people applied that principle both for services and for user accounts in use, we would see most security issues vanish away overnight.

But what is specifically wrong with running an SQL service as root? Specifically, pretty much everything. Inexperienced SQL users may think that SQL is all about manipulating tables. They may think that the worst an SQL server can do is drop the tables and kill the database. Inexperienced SQL users like zFire are not aware of the SQL commands that allow the SQL user to read from and write to the filesystem with the privelege of the SQL service user.

If your SQL service runs as root, and someone can run arbitrary SQL on your service, then that cracker can read *every* file on your filesystem.

Yes, even the system password hashes file (if they know where to look). Is your system root password uncrackable? No? Do you allow remote access through SSH or VNC or some other service? Yes?

Bang! You now no longer have a cracker in your database, you have a cracker all over your system.

And even if you do not leave things that wide open, be sure that the cracker can look at all your log files and every other document you thought was private on your system.

Never, ever, ever even consider running a live service under the root user. Run the service with the lowest privelege you can. SQL usres should specifically not have read permissions pretty much anywhere on the filesystem, other than the database directories themselves.

Thirdly Always, Always, Always sanitise your database inputs. To be honest, even this is not really good enough. You should really write your databases differently. Instead of inserting code into SQL statements, send the sanitised input as variables to stored procedures. But zFire would have done so much better if he had taken the minimal step of sanitising input.

Let’s look at some code examples.

Suppose that the web form requests username and password and then passes it to a PHP code snippet similar to this:

$items=@mysql_query("SELECT username, password FROM users WHERE username='$username' AND `password`='$password' );

The problem is that if you allow someone to enter a username something like this, and leave the password blank:

zFire Xue' #

then look what the select statement becomes:

SELECT username, password FROM users WHERE username='zFire Xue' #' AND `password`='$password'

Everything after the # is ignored as a comment (other versions of SQL use different comments, but whatever comment character you use, the implication is the same). This SQL code always returns a row from the database, as long as the name you choose exists in the database. Consequently you are allowed in. With the privelege of the name you chose!

And that is it. The key to breaking into zFires database was to add ‘# after whichever user you wished to impersonate! No password was required and you could control all that user’s redzones. This hole, which falls under the category of “stupidly obvious” would allow you full administrative access to zFire’s redzone account (or anyone elses).

Now suppose that zFire tried to log access to his site (we know he did), his log statement would look like:


INSERT into log (username, password, accesstime)
VALUES ($username, $password, CURDATE())


This, with our input above, becomes:


INSERT into log (username, password, accesstime)
VALUES ('zfire xue' #' , '', '2011-6-4 04:13:54')


The italicised bit after the # is ignored, and the insert fails as it is not properly formed. In other words, no line gets inserted into the log. All such accesses to the database were invisible to the log files.

But, in fact, you can do so much more than gain access to the web forms. For instance, if an SQL statement returns data, you can merge the data returned to the user with, say, the contents of a file from the filesystem.

You could use this to, say, look at the code of the pages themselves – thus finding more security holes – or hidden links to secret pages with videos meant for your girlfriend.

You could look at the SQL connection string file, which includes the SQL password. You could investigate system files and generally wlts all over the system. Which is why we turn to lesson 4.

Fourthly, when hacked, do not paper over the cracks.

Rémy Evard produced a software lifecycle that noted how systems move from a configured state towards an unknown state through an entropic process of constant update, change and debug.

When your system is hacked it takes the fast track route from the configured state to the unknown state.

Sure if some dumb script kiddy downloads software that justs repeatedly crashes your server then its likely you know what is going on. But if you have had a more sophisticated cracker in there – the kind who has been poking your system for weeks or months, and you haven’t spotted them – then really you have no idea what state your system is in.

zFire’s system was vulnerable to attack from day one, and with such obvious security holes in it, it is implausible that no one was inside that system before the script kiddies had a go at it. As we have seen, it is quite likely that those crackers had far more than access to the web site interface. There is no doubt they held a copy of his code, his SQL passwords and pretty much anything else they wanted.

So zFire’s response to being hacked was a lesson in stupidity. (As was the challenge that got the script kiddies running). On realising he has a fundamental security hole in his system, the correct response, in my view would have been to:

1. Take the system offline to evaluate the extent of intrusion. This is the point you should be contacting the police if it happens to you. Pull out the network cable and call the police. Don’t touch the evidence. However, if you do not wish to involve the police, still take it off line. Look at the logs and the code, and see if you can determine when the crack firts occurred.
2. Restore the system to a known state – which means restore from a back up before the earliest time a hack could have occurred. This is bad news for zFire – he was using time machine backups and frankly he did not have a backup from that far back. Nevertheless this is good advice. restore to a known good state – which may be your original build. Consider if your system has a kernel root kit installed – it will not be found. You need to wipe everything and start over.
3. Change ALL your passwords EVERYWHERE. Not just your web site password. Not just the connection string. ALL passwords. And this time don’t make them so crackable. After the crack, zFire changed his RedZone password, although not the passowrds of his alts. After being hacked again, he changed the passwords of users AND alts. When he was promptly hacked again he really should have noticed that passwords were not helping.
4. Fix the code. this seems obvious. But what is not obvious is how to fix it. zFire’s solution was to try to capture certain SQL strings. What he should have done was rewrite the whole application from scratch. Take your time. You are now a target, so get the rewrite right. Buy in some consultancy if necessary
5. Don’t trust your data. People have been modifying it. It is no longer correct. restore to the last known good data or just start over

So there you have it – four things zFire really should have done (well the fourth is a list in itself I know, but the point is – never soldier on with a crippled system. Start over).

On April 20, 2011, a violation report (for anyone truly that interested, the case number is U.S. District Court, Western District of Washington, case number 01CR00310RSL- and this information is public- call them yourself if you want) was submitted alleging that zFire (Mike Prime) had violated his conditions of supervision by:

1. Committing the criminal offense of Possession of Stolen Property 1st degree;
2. Committing the criminal offense of Trafficking in Stolen Property 2nd degree;
3. Associating with a convicted felon;
4. Associating with a convicted felon; (note these are two separate charges, which would indicate two separate people/felons)
5. Failing to allow the U.S. Probation Officer to inspect any personal computer owned or operated by the defendant;
6. Failing to notify the USPO of all computer software owned or operated by the defendant;
7. Beginning employment without prior approval of USPO and working for cash.

A warrant was issued for his arrest and on May 2, he surrendered to U.S. Marshals. He appeared before a U.S. Magistrate Judge and denied the allegations. An evidentiary hearing is currently scheduled for May 18, 2011. At this time, pending that evidentiary hearing, he remains in custody.

I think both counts 6 and 7 have some bearing on RedZone, but it is not clear to the extent that thse actually figured in the action by the authroities. I note Avril suggests only count 7 relates to his activities in Second Life, and she is in a better position to know.

In any case, it is an interesting epilogue to the RedZone affair. In the end the bad guy goes to jail.

zFire Xue is banned, as are his alts. RedZone is gone. As others point out, the privacy issue in Second Life has not gone away – but we now have much better tools to discover and fight future privacy invasions with Sione’s media patch. The point that this kind of data collection is neither legal nor moral has been made. Most importantly, this is a single issue blog. I called it “no2redzone”, and that issue has vanished down the plug hole of history. And good riddance.

I will continue to watch the privacy issue, and no doubt will contribute again in the future – but this blog is not really needed anymore.

Having said that, there are some loose ends that could be tied up – I may blog about some of these. In particular, I may write an article about how that database could have been so much better protected. I am still considering whether that would be worthwhile to my readers and how I should do that.

But other than a couple of follow up articles, I do not intend to keep up this site. I may archives some news posts but I will keep the general interest stuff available for a historical perspective. However in a couple of weeks I intend to turn comments off on the articles and leave things be.

Now having said that, I could not resist a couple of things. One correction: zFire claimed in his Alphaville Herald article that no one had proven he had a page tracking locations of IP addresses.

In fact I can reveal that the page that did this was hidden in his error404.php document. If logged in as zFire or his girlfriend, the 404 page showed a search box that allowed you to enter an Avatar name and it showed you the location associated with their IP address. Even though I was given access to this information, I did not bother report this as, amongst all the other revelations, that page seemed pretty lame. Just thought I would mention it now though.

The second thing, I just felt that as we are all missing Crackerjack, we should have a quote from him. This written by Crackerjack on 19 February:

my particular expertise is in network security and although i am not an expert in database programming i do work with programmers and know what an sql injection is and how it can be performed and they havent the expertise nor the methodology to do it

Once again many thanks to readers of this blog for your interest.

Soft Linden Nukes the RedZone Store As Crowds Party

Thanks to everyone who has written ARs, blogged, commented, posted, voted and in any way helped make this happen!

There will be some more news on this blog soon.

Michael Stefan Prime has multiple criminal convictions, including convictions for ebay fraud. This is an article about Mike Prime – zFire Xue. A longer PDF court record is here, and a New York Times article is here. Note from the court record that he had a string of previous convictions for first and second degree theft, two counts of possession of stolen property in the second degree, and forgery – at the age of 19!!

Bronxelf released this link to the court records for this case in a sensational post on SLUniverse a short while ago. Theia Magic confirmed this was the information she passed on to Linden Lab on Friday, and that the Lab is therefore now aware of zFire’s criminal record.

As zFire’s usual response is to fib and deny, I will pre-empt questions as to whether this is the same Michael Prime by saying that the age and details in this report fit him perfectly (19 in 1999 – he is 30 now, which we know is correct), and I had already scoured the web some months ago and I am 99.9% confident that he is the only Mike Prime in the Seattle area in the region of 30 years old. Theia Magic has also spent the weekend confirming identity and turning up some interesting names to others we have seen before – including John Hamlin. Theia’s blog is linked on the right, so check there to see if she posts updates.

So there you have it. It is as we always knew it: The fight between us and RedZone was always about thousands of honest sim owners and content creators on the one side, and a small band of thieves and criminals on the other. And the criminals built a system called RedZone.

Now, in our view, Linden Lab must act.

Consider what we have here:

1. Mike Prime set up a system that unreliably but with some success links alts by IP address. He does not care about false positives, as long as he finds a few real positives because:
2. Most people who have lats use the same password or some variation on the same.
3. Mike Prime has been harvesting password information from the 5,000+ users of his site by logging real passwords and failed attempts.

there is a very real risk that Mike Prime intends to use stolen accounts and stolen alt accounts for more petty theft and fraud.

URGENT: Please put aside all grudges and tell all users of RedZone, and anyone who ever registered at Mike Prime’s site to CHANGE THEIR PASSWORD IMMEDIATELY IN ALL PLACES THEY USE THAT PASSWORD!

zFire is a victim of his own nasty forum

I have always said the neighbourhood watch is one of the nastiest things on zFire’s site. Right now, however, I am enjoying it.

I expect zFire will delete these when he sees them. We are not meant to criticise him after all.

But not before it is all too late. Password outing functionality, and indeed the veracity of the video we carried this week has been confirmed by the hackers from last night who released their findings to the Alphaville Herald. It may be they attempted to contact us with the information first, for which I thank them but I think the Alphaville Herald is a good place for that report.

Yesterday’s hack was still annoyingly obvious – and today’s moreso. I can allay some fears however in that I understand that significant quantities of false data have been injected into that database by yet another person or persons who have demonstrated they understood the security vulnerability well enough to do this. This same source suggests that zFire was about to manually add the names of all members of the inworld GreenZone users group to the list of “known copybotters”[sic]. Attached is the evidence provided – snipped away are well over 1000 names take from the group membership.

Letter to zFire Xue from Merlin Swordthain

Since today’s hack the forums appear to have had it although it looks like there was a recent database backup. If anyone else is thinking of cracking this database I should point out that its no great challenge but at this time the working database is zFire’s biggest albatross It shows he has been a very very bad boy so please do not be tempted to take it offline. False IP address reports will do no harm though.

To end on a lighter note, Theia was confused by this remark from new RedZone poster arooga:

by arooga » Fri Mar 11, 2011 1:48 am

I would like to have crackerjack’s babies for the way he got Theia Magic
Done Up Like A Kipper she was, hung by her own petard

Her comment to that was amusing bit this is even more amusing in the light of this:

Arooga is Crackerjack

[Edit: Someone challenged the image showing that Arooga is Crackerjack, saying anyone could have written that on the forum. I edited down the screenshot I was given and now include a bit more to show this was a message sent directly to zFire. The message and the screenshot predate Friday's crack on the database.]

It seems Crackerjack, in an attempt to beef up his security by changing his email address, locked himself out of that account. He decided Arooga would be fun for alt games. Strange from someone who finds alt outing so important.

So Arooga wants to have Crackerjack’s babies? Nice to see him getting in touch with his feminine side.

It may be that someone took zFire up on his challenge to beat his security. I hope not really. But if they did, we could see quite an extended outage!

Move along now…there is nothing more to see.

Oh, but another theory: Merlin may have been watching for the posting of that URL. This may have been an attempt to bow out graciously – take the server down when people will think it is hacking… blame the griefers and walk away.

Either way – RedZone could be gone.

EDIT 3:12 SLT – This is a confirmed crack on the database.

ZFire had posted this earlier:

Originally Posted by zFire Xue
Let me be very clear when I say:
zFire did not “underestimate the tech savvy community of Secondlife if he thinks they will not [insert illegal hack attack here]…”

My server remains online, DDOS, URL probing, port scans, and seriously did you just try to “NUKE” me on port 139 Mr Germany?
They offer technical resumes, and warnings of everything they feel I did wrong.
My server is still online, even with low tech abuse reports to my ISP, DDOS of 860 million a second (Impressive but pointless), and whatever else.
This therefore means that my server is the most secure server and database in all of Secondlife.
That is a challenge.
Many people have already made battle cries, suggested methods, or claimed not to support methods of hacking.
Bring it on.

I am the guy that logs your shoe size right? Do you think any server software exists that does NOT log the IP, date and time of an attempted cybercrime? Wow this will be fun.

“My computer is bigger than your computer”
Cyber criminals need banning, so please feed attempts to isellsl.ath.cx

His site was actually an exercise in how not to do security, but I am annoyed that this crack was so unsubtle. That’s what happens when you challenge the whole Internet to come hack your server.