Traceroute, CrackerJack’s Ignorance, and Network Information

26 02 2011

Crackerjack continues to amuse us with his ignorance. Last Wednesday he said something that is not strictly relevant to the RedZone debate, but it is amusing because he does not realise that. I repeat it partly for the amusement but partly because I know a few of my readers appreciate some geeky stuff, and the rest can ignore this post as irrelevant and move onto something more interesting.

On Wednesday Crackerjack said:

well the greenzoners are a bit stumped when it comes to tracking the link to the database when all their traceroute enquiries end at dyndns

Now I did not immediately correct him, because frankly who cares? We all know where zFire’s computer is, right down to his flat number! But even if we did not have this, we don’t need to use traceroute to find out the information (and in any case its just nonsense again. The packets do not touch the DynDNS network at all, as we will prove later on). Given any IP address, you can quickly look up the records on its registration. In many cases this will give you the area where the address is in use (although not always. Sometimes it is not even the right country).

I did a lookup on the IP address for online. Note a few things that this tells me:

IP Location: United States Los Angeles Comcast Cable Communications Inc

So zFire uses a Comcast cable connection. This gives him a fair bit of bandwidth, although it is essentially a home grade solution. It looks like it is the Washington State part of the Comcast service.

OrgAbuseHandle: NAPO-ARIN
OrgAbuseName: Network Abuse and Policy Observance
OrgAbusePhone: +1-856-317-7272

Now you know who to call to complain.

Also we have:

Reverse IP: 2 websites use this address. (examples:

The reverse ip misses the dyndns domains, but still we have zFire’s other hobbies (porn and pretending to be a pro gambler).

So I am really at a loss as to why Crackerjack says greenzoners (and I presume he really means all non spyware operators) are stumped. What precisely does he think we don’t know?

He was replying to this query:

Can you comment zFire, is it feasible to cloak the domain, effectively neutralising this viewer change?

Just for the record – the answer to that is no. If you want us to send data to the redzone database you can’t “cloak” it. That is like saying “can we make everyone take a train at Euston station, but not tell them where the stayion is?” It is just not going to happen. The only thing I am stumped about is what on earth Crackerjack thinks it is that he can hide here!

But…. never one to pass up a chance to get geeky, let’s actually try traceroute.

As expected my traceroute does not get stuck at all. The only difference between me and crackerjack is I don’t use windows! How is it that crackerjack’s traceroute fails and mine succeeds?

To understand this you need to understand a little about how traceroute works – and what it tells you.

Traceroute is a program that is designed to tell you the interface IPs of nearside[0] interfaces of the routers between the person running a traceroute and another node on the Internet. It does this by using a particular value in the IP datagram header called the TTL (or hop count in IPv6). The TTL – Time to Live – is a number in the datagram header that is decremented by 1 each time the datagram passes through a router.[1] If the TTL is decremented to 0 then the packet is discarded as undeliverable, and an ICMP message (think of it as an Internet error or control message) is sent back to the originating node to say the delivery failed for the reason “time exceeded”.

The purpose of the TTL is to protect the Internet from routing loops. Imagine if Router A was told to pass all traffic to router B, and router B is told to send all traffic to C, and C is told to pass all traffic to A…then once router A receives a packet it will spin around the routing loop A-B-C-A-B-C add infinitum, until everything collapses because all the other packets are doing the same thing.

Most real routing loops are more subtle than that, but they do happen.

So inserting a TTL ensures that if a packet cannot be delivered in a reasonable number of hops, it is thrown away to save bandwidth and we send back an error message (as long as this error does not also get stuck in the loop. If that happens the failed error message does not generate another error message). The initial TTL is usually set at 64, 128 or 255 hops before the packet is discarded. No routes on the Internet are that long, nor ever likely to be.

That is great, but then Steve Deering postulated in the 1980s that we could use the TTL value to recover information about the routing nodes a packet passes through on its way to the destination node. To do this is actually quite simple: we send a packet to the destination node but we set the initial TTL value to 1. This means that the very first router that receives our packet will decrement the TTL to 0 and thus discard it, sending back an ICMP time exceeded message.

That ICMP time exceeded message is sent by the router and thus has the IP address of the router on it. We have thus discovered the nearside IP address of the first router on the network path. Just in case we have packet loss, we repeat a couple more times to be sure.

Now how to find the next hop? I know – let’s make the TTL 2 now. So now we deliver our packet to the end node with a TTL of 2. The first router decrements it to 1, and the second hop now decrements to 0. Thus the second router sends back our error message and we have the address of hop 2.

We repeat the process until either our packet is delivered or until we have tried all the TTL values (255 is the largest TTL).

There is a problem. How do we know the packet was successfully delivered?

The traditional solution was to send the packet to the destination node to a port that is not listening. Choose an unlikely UDP port and send the node a packet…when the datagram is eventually delivered, the node we sent it to will send back a different ICMP error message. This time it is still a destination unreachable message, but the cause is that there is nothing listening on our chosen port (port unreachable). This is the default operation for Unix systems.

Microsoft Windows has a program called tracert that does much the same thing but does not try to send a UDP packet al all. Instead it sends and ICMP echo message (a ping packet). If it gets an echo reply (a reply to the ping) then the packet reached the destination.

Sometimes one method works, sometimes the other does. The problem here is firewalls. Many firewalls will filter out ICMP echo replies because of, for instance, the problems with the Nachi virus that used these packets to locate other machines to attack. Other firewalls block all traffic to all but a few UDP ports. If a firewall silently blocks packets (with no error message), then traceroute just receives no reply and cannot tell this from packet loss. The program inserts a * in the data it receives for lost packets.

Sometimes a router will not respond with ICMP destination unreachable packets, but it will pass them from following nodes. That is fine – that node just appears as a row of “* * *” in our output, but traceroute still discovers all other nodes. But if all traffic is filtered beyond a certain node, and we receive no more ICMP messages, we cannot tell when traceroute completes. All the program can do is work through all TTLs, and stop when the TTL reaches 255. This looks like a lot of “* * *” lines in the output, although usually the final destination is only a hop or two after this starts to happen.

If we get this problem, we can try to work around it. For instance, maybe trying traceroute with TCP packets or packets to port 80 will work. There are all kinds of options to traceroute, and you can retry the trace at a certain node to save time. With a bit of work, many apparently dark routes can be lightened.

Not necessary for ZFire’s route though… here is the result of my traceroute with ICMP (as windows does it by default). I only include the last few hops:

11 ( 213.193 ms 212.087 ms 229.980 ms
10 ( 215.151 ms 295.391 ms 215.994 ms
11 ( 295.998 ms 306.183 ms 218.683 ms
12 ( 217.853 ms 279.288 ms 217.092 ms
13 * * *
14 * * *
15 * * *
16 * * *

See it failing there? not at dyndns of course. DynDNS is used for the name look up but no packets are routed through their service. It fails in the seattle area of the comcast network.

Now here is my default UNIX traceroute (using UDP packets). I get exactly the same with TCP incidentally:

12 ( 264.004 ms 280.467 ms 216.272 ms
10 ( 295.643 ms 271.381 ms
11 ( 307.256 ms
12 ( 282.131 ms 217.690 ms 320.098 ms
13 * * *
14 ( 257.890 ms 307.147 ms 307.233 ms

So hop 13 did some ICMP filtering but zFire’s router seems to reply fine. In fact, I think maybe his router is a NAT box set up to forward all UDP/TCP traffic to the Mac, in which case the Mac replied. I could investigate further if I cared. But I don’t.


[0] All routers by definition must have at least two non loopback interfaces. The router takes packets from one interface, and makes a decision as to which of its other interfaces to pass the packet onto. Basically a router takes a packet and forwards it somewhere else. Think of it like the postal service. A router is a sorting office. Letters arrive at the office, and someone looks at the address on the envelope and throws the letter into one of a number of sacks. Some sacks go to, say, London. Some go to Birimingham and so on. On arrival at the destination there is more sorting and the process continues until a postman takes his sack to your door and posts the letter through your letterbox (straight into the waiting mouth of your dog, who swallows it).

Now traceroute tells you the address on the door of the sorting office as you go in, not the address on the sack as you go out. That is what I mean by nearside interface.

[1] The RFCs actually say the TTL also gets decremented for each second a router holds the datagram without forwarding it, but as this never actually happens, we can ignore that technicality.

On Geekiness and the Perils of Bemoaning Stupidity

17 02 2011

I would like to highlight a discussion in the RedZone forums because it leads to some interesting speculations and reveals the lack of technical knowledge in the RedZone community. The subject is “Static IP .. need to know more, please”. Before I start, I should say this post is geeky and anyone choosing to ignore it will not have missed much!

Cole Cybertar said in that discussion: “Now if something started recording MAC addresses then there would be an issue.” What he meant is that it is his belief that a MAC address is private, and its use in a detection system would be a privacy issue, whereas the use of IP addresses, in his opinion, is not. zFire agreed with a telling comment on 10th February:

I have not researched MAC Addresses.

Now if zFire would like to know more about MAC addresses, he need but ask. I won’t bore my readers with all the details now, but MAC addresses are used to communicate between devices on a single network link. They are usually tied to the hardware of your interface and often you can use the top 3 bytes of the address (less two bits) to identify a hardware manufacturer. So yes, they give away a little more than IP addresses in terms of hardware, but as they have no network structure, they give away less in terms of location. All that is moot of course, because a spyware operation based on hijacked GET requests has no way to access your MAC address. Your address is not on any packet beyond your router, and the MAC address on the frame that zFire receieves will be the MAC address of his router. So he cannot harvest these.

We have seen that Linden Labs do harvest this information in the authentication packet your SL client uses to connect to their service, and they do use this information to identify alts (it is much better than IP addresses for this purpose, although not perfect). They can do this because the client specifically packages up and sends them the information, and this is ok because Linden Labs are up front about the data collection and we agreed to it. It is also okay because Linden Labs use it for administration of their service, and do not share it with third parties.

But zFire, who has not researched MAC addresses enough to know any of this, cannot collect the data.

That is… until we move to version 6 of the Internet Protocol (IPv6).

And here, we cue crackerjack for this absolutely wonderful explanation of IPv6 in the same forum thread:

with the phasing out of ipv4 and the bringing in of ipv6, devices such as redzone should be more accurate since every interface will have a unique routable ip address, now NAT routing will no longer present a problem, for example – to save ip addresses the router was configured so that machines behind it would have a local ip address and the router would interface the internet on behalf of all the machines behind it. Somework would be required to show that two avatars were not a single rl person but two people. With ipv6 that would not be a problem since ipv6 would come with its own security and there will be no need for a router, just a network switch so the problem would boil down to a single computer and whether for example two avatars from the very same computer represented one or two rl people

Whilst this is nearly all nonsense, I was particularly tickled by “there will be no need for a router”. [0]

Again I have to resist the temptation to bore my readers with lots of irrelevant detail. But in short – IPv6 will continue to need just as many routers as IPv4.

His attempted clarification only further muddied the waters:

i believe under ipv6 both the router information and the originating computers information get passed along and that the originating computer will have an address that can be determined, so under ipv4 you would route information to the router that hides the local adress but under ipv6 i think the originating computers address is given too unless i am badly mistaken, this implies also that you dont route under ipv4 or have a ipv6 translator to deal with ipv4 requests and will be probably something that becomes more relevent after 2012

What I think crackerjack was trying to say was that we will not need to share IP addresses using network address translation devices (NAT) in IPv6. I think he has some idea that all routers are NATs, or maybe that there are no other routers other than his home NAT. Either way it is nonsense.

His argument that we don’t need NATs is, however, correct because the 128 bit address space is so astoundingly huge that, by my calculation, we could number every single network device on a global Internet 100 times larger than ours on every likely inhabitable world in the entire universe![1] So he is indeed right that we no longer need to recycle and aggregate addresses.

Thus indeed Network Address Translators will be a thing of the past (unless the world is filled with clueless media people who argue that a NAT is a security device. Hopefully reason will prevail though[2]). Every interface on every device connecting to the Internet will be able to have its own IP address. And on that crackerjack is correct, if somewhat confused.

Here is the problem: The global unicast addresses in IPv6 are designed with the enormous address space, 64 bits of interface id, to allow interface autoconfiguration. This highly attractive feature of IPv6 will allow devices to essentially choose their own IPv6 addresses, within a set structure. How do they do it?

Structure of a Globbal Unicast IPv6 Address

The default mechanism is to create a 64 bit EUI-64 global identifier based on – you guessed it – the 48 bit MAC address!

Creating an EUI-64 from an IEEE MAC-48

Yes, that is right! IPv6 addresses will, by default, include your MAC address.

Just to remind you: Cole Cybertar said in that discussion:

Now if something started recording MAC addresses then there would be an issue.

I guess even the redzoneophiles realise now that we have an issue.

Of course, IPv6 will also make RedZone spyware trivially easy to avoid. I can reconfigure my hostid how I like. I do not need to settle for the EUI-64 interface id. With 2^64 IP addresses to choose from, I could have a new IP address every minute for the next 35,000,000,000,000 years without recycling any. I would like to see zFire match my alt from that![3]

All this is moot though. I doubt that the RedZone database even has the data structures to handle IPv6 addresses, and at present people using IPv6 are tunnelling to his server through IPv4, and are thus another source of IP aggregation. That is, they keep matching each other in that brain dead database.


[0] I only quote crackerjack’s nonsense in full because he also said: “These are people who have no idea how their computer or the internet or second life actually works”. I just thought it an opportune moment to remind everyone that, in fact, crackerjack’s own knowledge of the Internet Protocols appears to be at the level of an interested user. There is nothing wrong with that. We cannot all be experts in these things, and it is no reflection on an individual if they have not studied the RFCs in depth! But two things I have learned are (1) Never underestimate your chosen out group. It is human nature to think your in group is more clever than your out group, but that human nature is not rational. (2) Never spout off on something you really do not understand – especially at the same time you are attempting to mock people for spouting off on things they do not understand. Oh and (3) Never trust me to stick to just two things!

Actually crackerjack went on: “… start telling the others they know about […] data protection law etc. then they go on to spout the most utter boulder dash you ever heard”. That would be “balderdash”[4], but as I think that was a jibe at me I would just make the point that it is a part of my RL work to know about and advise people in Data Protection Law.

Still crackerjack is venting though: “OK The anti redzone people are infact so stupid they have made me very nervous but only because i hadnt realised just how many truly uneducated morons “

I just want to take this point to make it quite plain that I am, in fact, an educated moron.

[1] I really have made this calculation. I have, of course, made some finger in the air guesses on number of inhabitable planets, loosely based on the current thoughts on this following the discovery of large numbers of exoplanets – but I think if anything I have overestimated the number of these. I will dig out the calculation if anyone is interested.

[2] One popular tech commentator argued in his IT Security podcast some years ago that NAT is a security device because it blocks out the home network. He is wrong though. NATs are typically implemented using stateful firewalls that do IP header rewriting. It is the stateful firewall that offers the protection to a network, not the header rewriting. Keep the firewall and dump the NAT.

[3] Put another way – given 1 /64 IP address range – the key to IPv6 autoconfiguration – I could set up a network of computers. If I kept the spacing between computers to about one meter, which is about as close together as people could comfortably sit at them to work, and I connected all these to a single router (because we cannot aggregate at less than a /64), then my network link would stretch roughly from my house to here:

The M25 – not the London orbital motorway, the other one – is about 2,000 light years away. Thus the network latency on my network would be about 4,000 years – plus a small processing latency at the router. I think this could make Second Life a little laggy, but I am willing to give it a try if anyone will donate me the computers.

[4] Balderdash: A word that was used in the tudor period to refer to a mixture of liquors that would not normally go together.