Investigating iPad keyboard lag
Whenever I go to type a URL on my iPad Pro, there’s a good chance it’ll start lurching. I type a key, two keys, ten, and it’ll just sit there dumbly until it suddenly unblocks and all the keypresses buzz through. Sometimes it can lag ten to twenty seconds, and as soon as another key is hit, it’ll lag again.
I’m using iOS 11 beta, but it’s been doing this for ages… in fact, my iPads of times past did this too.
It happens with Amazon Search strings too, and other apps.
I’m using Google GBoard, but it does the same with the stock Apple keyboard too.
I’ve wiped the keyboard dictionary, hoping it’s something weird going on with autocomplete.
It happens when I use a hardware keyboard too.
I’ve searched; and I’ve followed forum post suggestions.
It’s incredibly annoying.
Hmm.
So, what’s going on? Damned if I know. It feels like network lag, somehow. I know that’s probably just my imagination, but it does feel like the kind of pause we used to get with Netscape Navigator when the DNS server was running hinky and they hadn’t figured out asynchronous lookups.
“Where” is the problem.
The one thing that seems to work is turning off Location Services: Go to Settings → Privacy → Location Services. It usually starts working properly then. Keeping Location Services switched off isn’t my ideal state of affairs, though.
My iPad is a WiFi-only iPad Pro 12.9” 128GB; as I have free tethering on my phone and I rarely go anywhere with it that I’d need cell data, I don’t feel the need to pay for another SIM card. However, this does mean (apparently) that the iPad doesn’t include GPS either. Not a big loss, but it does mean that Location is less than precise. It doesn’t tend to bother me.
Or does it?
I’m wondering if the lack of a GPS means Location Services has to rely on WiFi to triangulate the old-fashioned way using local WiFi signals and a big fat database of known WiFi MAC addresses and their latitude/longitudes. So maybe it’s waiting on Apple to tell it where it is?
My iPhone (with its tiny GPS chip) doesn’t seem to have the same problem; I’ve replaced my iPhone and iPad regularly and this bug only seems to affect the iPad. I’ve done installs from backup, as well as fresh installs — thanks iCloud for refusing to restore from a cloud backup and losing my kid’s digital paintings…
EDIT: Hang on… yes, my iPhone does have the same problem. Damn. I’ve never noticed that before. Not that I tend to type much on my iPhone, but I just tried to type a long passphrase and it took about five minutes. Turning off Location Services did help, as well.
Let’s investigate further.
My home network is currently in a bit of a bodge; after I noticed the crappy Sky fibre modem letting through a few ports when it shouldn’t’ve, I clammed up and rigged up an OPNsense firewall as quickly as I could. However, as it’s still not quite working right, it’s bypassed. It does listen promiscuously, though, so I can eavesdrop on my iPad to some extent.
I fire up tcpdump with the command:
tcpdump -n -i bridge0 host 192.168.7.51 and port 443 | tee dump | perl tcpdump_filter_ips.pl
where tcpdump_filter_ips.pl is a quick Perl script I just wrote that filters out my iPad’s IP, prints the other IP, looking it up from a list of known hosts.
Well, I had to have at least one image in this story.
This gives a slew of IP addresses to check. Now, since the traffic is (quite rightly) encrypted with TLS, I don’t know the contents of the messages. However, I can see what server certificate they’re set up with by default, and that should be enough to get an idea what’s going on:
curl -Iv https://54.239.35.198/
* Trying 54.239.35.198...
* TCP_NODELAY set
* Connected to 54.239.35.198 (54.239.35.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /usr/local/share/certs/ca-root-nss.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
**** subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=completion.amazon.co.uk
**** start date: Sep 23 00:00:00 2016 GMT
* expire date: Oct 22 23:59:59 2017 GMT
* subjectAltName does not match 54.239.35.198
* SSL: no alternative certificate subject name matches target host name '54.239.35.198'
* stopped the pause stream!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name '54.239.35.198'`
In the middle of this, there’s the certificate subject: subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=completion.amazon.co.uk telling us the server is Amazon and unless they’re going for multiple virtual hosts on a server (unlikely here), it’s the site https://completion.amazon.co.uk/ which we can assume is an autocomplete API server.
So, while I’m sure a more competent security research than me could wring more information out of this, I just want to try to see what’s causing the massive pause. This method of detecting the server name should be enough when playing with big names like Amazon and Apple that will divide functions between servers cleanly.
Results
I took tcpdumps of a few “phases” of a typing delay: waking from sleep to resume the Amazon app; tapping the search box (summoning the keyboard after a pause of a few seconds); and typing a character into the search box.
Upon waking the iPad, a slew of traffic occurs. Most of this is (unsurprisingly) with Amazon, under the IP ranges 54.224.0.0/12, 52.94.216.0/21 and 52.0.0.0/11, which are all registered to Amazon, presumably for both their store and for AWS.
There does appear to be some other traffic, including a lot of Apple IPs.
Upon summoning the keyboard for search, again a lot of Amazon traffic. The keyboard does take a couple of seconds to activate, and the dump shows that the instant the keyboard actually appears, there’s a send to *.ls.apple.com followed by a flurry of calls to and from *.ls.apple.com hosted by Apple, and configuration.apple.com hosted by Akamai.
root@OPNsense:/home/gid # perl -n tcpdump_filter_ips.pl < tap_search
08:29:01.7 > 52.94.217.61 completion.amazon.co.uk
[...repeated...]
08:29:01.7 > 52.94.217.61 completion.amazon.co.uk
08:29:01.7 > 95.101.84.226 www.amazon.co.uk (Akamai)
[...repeated...]
08:29:01.9 > 95.101.84.226 www.amazon.co.uk (Akamai)
[...pause...]
08:29:03.8 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.8 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.8 < 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.8 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.8 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.8 < 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.8 < 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.8 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 < 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 < 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 < 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 < 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 < 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 < 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 < 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 > 72.246.227.237 configuration.apple.com (Akamai)
08:29:03.9 < 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 < 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
08:29:03.9 > 17.130.137.79 *.ls.apple.com (OU=Map Operations)
root@OPNsense:/home/gid #`
*.ls.apple.com has the full subject: subject: C=US; ST=California; L=Cupertino; O=Apple Inc.; OU=Map Operations; CN=*.ls.apple.com
I reckon ls probably stands for “Location Services”, part of the overall “Map Operations” department at Apple.
Now, this is key. I originally assumed this was a case of Apple’s location servers being slow. However, the dump shows that it’s not waiting on a response from Location Services, but it’s pausing before sending Location Services a request.
So, what’s the hold up? Could it be my iPad doing an impromptu WiFi environment survey to pick up all networks and strengths, and then passing that to Apple to triangulate?
I think it might be that.
Unfortunately, as I was repeatedly running the tests to fine-tune the process, the delay on my iPad became less and less. I imagine this is due to the unknown networked being looked-up and cached. Maybe I should move around a bit and have another go, or start a test mobile network with a random MAC and SSID for the iPad to try to look up… actually, that does sound like a good test of my hypothesis.
Using hostapd to intentionally confuse
Okay, so I’ve set up hostapd quickly on a spare hardwired Linux box with an unused WiFi port, and configured it to start with a random BSSID (a.k.a. MAC address) and a dummy SSID. This should appear as a very strong signal of a new wireless network that the iPad hasn’t seen before.
Indeed, it suddenly slows down the keyboard launch, causing an avalanche of packets to/from Location Services, and other Apple servers.
I’m fairly happy with this hypothesis. I won’t call it “confirmed”, but it does seem to fit at the moment.
Solutions
What can I do about it? Not much, really. I could rig the walls with a Faraday cage to block all WiFi signals and thus only have known networks appear.
Or I could just leave Location Services off. That’ll annoy me. I do actually use it.
Or I could buy a new iPad Pro with a GPS just to speed up typing… £989? I think not.
Or Apple could fix the damn bug.
(And yes, I have filed a Radar on it.)
EDIT: After realising my iPhone does the same thing, I’ve a few other possibilities to try, including switching off WiFi on the iPhone without switching off Location Services. That particular test might give some insight into whether it’s a site survey that’s going on.
