The GSM security model is based on a shared secret that is located on the mobile phone inside the SIM chip and the subscriber’s HLR. This shared secret is a 128-bit key that is used to generate a 32-bit response. It is identified as Ki.
When a mobile station first comes online, it connects to the local gateway based on its position. This gateway then tracks down the phone home gateway and receives a random challenge and signed response from its home HLR. This also comes with two other pairs of challenge and signed responses that are used later. After receiving both pieces of information, the local gateway will only send the challenge to the mobile station. When the mobile station sees this challenge, it will use its shared secret to create a signed response and send it to the local gateway. To create this response, it will use the A3 cryptographic algorithm with the shared secret and the challenge. Once the gateway sees this, it will compare it to the already sent signed response from the subscriber carrier. Now we have taken care of authentication; this has only gotten the phone onto the network and able to place calls.
Now let us return to the second pair of challenge and signed response messages. These messages were created with an algorithm other than A3. They use an algorithm called A8 to create a key that is used with another algorithm called A5 to encrypt the data. This works much the same way as authentication works. The mobile station uses the challenge and their shared secret programmed into the mobile devices to create a key. This process uses another algorithm, called A8, to create a session key called Kc. The Kc key is used with a frame number to create a unique key stream for every frame.
Another algorithm used is called COMP 128; it is used for both A3 and A8 algorithms in most GSM networks. The COMP 128 algorithm generates both the signed response and the session key in one run. The key length of COMP 128 is 54 bits instead of 64 bits, which is the length of the A5 algorithm key. Ten zeros are appended to the key as padding when generated by the COMP 128 algorithm. This means that the keyspace used to protect the key is not 64 bits, but rather 54 bits.
Over the years since GSM came out, some attacks have been released that defeat this security method. A key note that needs addressing here is that the A3, A5, and A8 algorithms are not publicly available. It has always been correct procedure for any algorithm to have open source available to prove a good level of security by subjecting it to academic and professional scrutiny. This ensures that an algorithm is actually safe. The process of having an algorithm open has been default practice for all commercial, financial, and governmental usage for more than 20 years. The GSM algorithm for A3 and A8 uses the COMP 128 algorithm as well. This algorithm was leaked to the public; and once it was securitized, it was quickly noted that a problem exists.
In 1998, members of the Smart Card Developers Association demonstrated that they could crack the A5 authentication method of GSM in a matter of hours on a single PC. This experimentation led to more in-depth analysis and later two men (Alex Biryukov and Adi Shamir) said they were able to perform an attack on A5 with a single computer in less than two minutes. Because of these threats, it is widely known that GSM systems have a poor security method.