De Nederlandse
Mozilla-gemeenschap

Update (2023-01-16): This article is now available in Korean.

I wrote about South Korea’s mandatory so-called security applications a week ago. My journey here started with TouchEn nxKey by RaonSecure which got my attention because the corresponding browser extension has more than 10 million users – the highest number Chrome Web Store will display. The real number of users is likely considerably higher, the software being installed on pretty much any computer in South Korea.

That’s not because people like it so much: they outright hate it, resulting in an average rating of 1,3 out of 5 stars and lots of calls to abolish it. Yet using it is required if you want to do things like online banking in South Korea.

The banks pushing for the software to be installed claim that it improves security. People call it “malware” and a “keylogger.” I spent some time analyzing the inner workings of the product and determined the latter to be far closer to the truth. The application indeed contains key logging functionality by design, and it fails to sufficiently restrict access to it. In addition, various bugs range from simple denial of service to facilitating remote code execution. Altogether I reported seven security vulnerabilities in the product.

Contents

• The backdrop
• What does TouchEn nxKey actually do?
• How do websites communicate with TouchEn nxKey?
• Abusing TouchEn extension to attack banking websites
  • Side-note: Browser extensions similar to TouchEn
• Using keylogging functionality from a website
• Attacking the application itself
• Abusing the helper application
• Accessing the driver’s keylogging functionality directly
  • Side-note: Driver crashes
• Will it be fixed?
  • Side-note: The information leak
• Can the nxKey concept even work?

The backdrop

After I gave an overview of South Korea’s situation, people started discussing my article on various Korean websites. One comment in particular provided crucial information that I was missing: two news stories from 2005 on the Korea Exchange Bank hacking incident [1] [2]. These are light on technical details but let me try to explain how I understand this.

This was apparently a big deal in Korea in 2005. A cybercrime gang managed to steal 50 million Won (around $50,000 at the time) from people’s banking accounts by means of a Remote Access Trojan. This way they not only got the user’s login credentials but also information from their security card. From what I can tell, this security card was similar to indexed TANs, a second factor authentication method banished in the European Union in 2012 for the exact reason of being easily compromised by banking trojans.

How did the users’ computers get infected with this malicious application? From the description this sounds like a drive-by download when visiting a malicious website with the browser, a browser vulnerability was likely exploited. It’s also possible however that the user was tricked into installing the application. The browser in question isn’t named, but it is certain to be Internet Explorer as South Korea didn’t use anything else at this point.

Now the news stress the point that the user didn’t lose or give away their online banking credentials, they’ve done nothing wrong. The integrity of online banking in general is being questioned, and the bank is criticized for not implementing sufficient security precautions.

In 2005 there have been plenty of stories like this one in other countries as well. While I cannot claim that the issue has been completely eliminated, today it is far less common. On the one hand, web browsers got way more secure. On the other hand, banks have improved their second factor. At least in Europe you usually need a second device to confirm a transaction. And you see the transaction details when confirming, so you won’t accidentally confirm a transfer to a malicious actor.

South Korea chose a different route, the public outrage demanded quick results. The second news story identifies the culprit: a security application could have stopped the attack, but its use wasn’t mandatory. And the bank complies. It promises to deliver an “anti-hacking” application and to make its use mandatory for all users.

So it’s likely not a coincidence that I can find the first mentions of TouchEn Key around 2006/2007. The application claims to protect your sensitive data when you enter data into a web page. Eventually, TouchEn nxKey was developed to support non-Microsoft browsers, and that’s the one I looked into.

What does TouchEn nxKey actually do?

All the public sources on TouchEn nxKey tell that it is somehow meant to combat keyloggers by encrypting keyboard input. That’s all the technical information I could find. So I had to figure it out on my own.

Websites relying TouchEn nxKey run the nxKey SDK which consists of two parts: a bunch of JavaScript code running on the website and some server-side code. Here is how it works:

1. You enter a password field on a website that uses the nxKey SDK.
2. JavaScript code of the nxKey SDK detects it and notifies your local nxKey application.
3. nxKey application activates its device driver in the Windows kernel.
4. Device driver now intercepts all keyboard input. Instead of having it processed by the system, keyboard input is sent to the nxKey application.
5. The nxKey application encrypts the keyboard input and sends it to the JavaScript code of the nxKey SDK.
6. The JavaScript code puts the encrypted data into a hidden form field. The actual password field receives only dummy text.
7. You finish entering your login credentials and click “Login.”
8. The encrypted keyboard input is sent to the server along with other data.
9. The server-side part of the nxKey SDK decrypts it and retrieves the plain text password from it. Regular login procedure takes over.

So the theory is: a keylogger attempting to record data entered into this website will only see encrypted data. It can see the public key used by the website, but it won’t have the corresponding private key. So no way to decrypt, the password is safe.

Yes, it’s a really nice theory.

How do websites communicate with TouchEn nxKey?

How does a website even know that a particular application is installed on the computer? And how does it communicate with it?

It appears that there is an ongoing paradigm shift here. Originally, TouchEn nxKey required its browser extension to be installed. That browser extension forwarded requests from the website to the application using native messaging. And it delivered responses back to the webpage.

Yet using browser extensions as intermediate is no longer state of the art. The current approach is for the websites to use WebSockets API to communicate with the application directly. Browser extensions are no longer required.

I’m not sure when exactly this paradigm shift started, but it is far from complete yet. While some websites like Citibank Korea use the new WebSocket approach exclusively, other websites like that of the Busan Bank still run older code which relies exclusively on the browser extensions.

This does not merely mean that users still need to have the browser extension installed. It also explains the frequent complains about the software not being recognized despite being installed. These users got the older version of the software installed, one that does not support WebSocket communication. There is no autoupdate. With some banks still offering these older versions for download, it’s a mistake I made myself originally.

Abusing TouchEn extension to attack banking websites

The TouchEn browser extension is really tiny, its functionality being minimal. It should be hard to do much wrong here. Yet looking through its code, we see comments like this one:

result = JSON.parse(result); var cbfunction = result.callback; var reply = JSON.stringify(result.reply); var script_str = cbfunction + "(" + reply + ");"; //eval(script_str); if(typeof window[cbfunction] == 'function') { window[cbfunction](reply); }

So somebody designed a horribly bad (meaning: actually dangerous) way of doing something. Then they either realized that it could be done without eval(), or somebody pointed it out to them. Yet rather than removing the bad code, they kept it around just in case. Quite frankly, to me this demonstrates a very bad grasp of JavaScript, security and version control. And maybe it’s just me, but I wouldn’t let this person write code for a security product unsupervised.

Either way, the dangerous eval() calls have already been purged from the browser extension. Not so much in the JavaScript part of the nxKey SDK used by banking websites, but these are no concern so far. Still, with the code quality so bad, there are bound to be more issues.

And I found such an issue in the callback mechanism. A website can send a setcallback request to the application in order to register for some events. When such events occurs, the application will instruct the extension to call the registered callback function on the page. Essentially, any global function on the page can be called, by name.

Could a malicious webpage register a callback for some other web page then? There are two hurdles:

1. The target webpage needs to have an element with id="setcallback".
2. Callbacks are delivered to a specific tab.

The first hurdle means that primarily only websites using nxKey SDK can be attacked. When communicating via the browser extensions these will create the necessary element. Communication via WebSockets doesn’t create this element, meaning that websites using newer nxKey SDK aren’t affected.

The second hurdle seems to mean that only pages loaded in the current tab can be attacked, e.g. those loaded in a frame. Unless the nxKey application can be tricked into setting a wrong tabid value in its response.

And this turned out surprisingly easy. While the application uses a proper JSON parser to process incoming data, the responses are generated by means of calling sprintf_s(). No escaping is performed. So manipulating some response properties and adding quotation marks to it allows injecting arbitrary JSON properties.

touchenex_nativecall({ … id: 'something","x":"y' … });

The id property will be copied into the application’s response, meaning that the response suddenly gets a new JSON property called x. This vulnerability allows injecting any value for tabid into the response.

How does a malicious page know the ID of a banking tab? It could use its own tab ID (which TouchEn extension helpfully exposes) and try guessing other tab IDs. Or it could simply leave this value empty. The extension is being helpful in this case:

tabid = response.response.tabid; if (tabid == "") { chrome.tabs.query({active: true, currentWindow: true}, function(tabs) { chrome.tabs.sendMessage(tabs[0].id, response, function(res) {}); }); }

So if the tabid value is empty it will deliver the message to the currently active tab.

Meaning that one possible attack looks like this:

1. Open a banking website in a new tab, it becoming the active tab.
2. Wait for the page to load, so that the element with id="setcallback" is present.
3. Send a setcallback message via the TouchEn extension to set a callback to some function, while also overwriting JSON response properties with "tabid":"" and "reply":"malicious payload".

The first call to the callback occurs immediately. So the callback function will be called in the banking website, with the malicious payload from the reply property as parameter.

We are almost there. A possible callback function could be eval but there is a final hurdle: TouchEn passes the reply property through JSON.stringify() before giving it to the callback. So we actually get eval("\"malicious payload\"") and this doesn’t do anything.

On the other hand, maybe the target page has jQuery? And calling $('"<img src=x onerror=alert(\'Hi,_this_is_JavaScript_code_running_on_\'+document.domain)>"') will produce the expected result:

Is expecting jQuery for an attack to succeed cheating? Not quite, the websites using TouchEn nxKey will most likely also use TouchEn Transkey (an on-screen keyboard) as well, and this one relies on jQuery. Altogether, all South Korean banking sites seem heavily dependent on jQuery which is a bad idea.

But update_callback, the designated callback of the nxKey SDK, can also be abused to run arbitrary JavaScript code when passed JSON-stringified data. Calling update_callback('{"FaqMove":"javascript:alert(\'Hi, this is JavaScript code running on \'+document.domain)"}') will attempt to redirect to a javascript: link and run arbitrary code as a side-effect:

So this attack allows a malicious website to compromise any website relying on the TouchEn extension. And none of the “security” applications South Korean banks force users to install detect or prevent this attack.

Side-note: Browser extensions similar to TouchEn

Back when I started my testing there were two TouchEn extensions in the Chrome Web Store. The less popular but largely identical extension has since been removed.

This isn’t the end of the story however. I found three more almost identical extensions: CrossWeb EX and Smart Manager EX by INISAFE as well as CrossWarpEX by iniLINE. CrossWeb EX is the most popular of those and currently listed with more than 4 million users. These extensions similarly expose websites to attacks.

My first thought was that RaonSecure and INISAFE belong to the same company group. That doesn’t appear to be the case.

But then I saw this page by the iniLINE software development company:

This lists Initech and RaonSecure as partners, so it would appear that iniLINE are the developers of these problematic browser extensions. Another interesting detail: the first entry in the “Major customers” line at the top is the Ministry of National Defense. I just hope that their defense work results in better code than what their other partners get…

Using keylogging functionality from a website

Now let’s say that there is a malicious website. And let’s say that this website tells TouchEn nxKey: “Hi there, the user is on a password field right now, and I want the data they enter.” Will that website get all the keyboard input then?

Yes, it will! It will get whatever the user types, regardless of which browser tab is active right now or whether the browser itself is active at all. The nxKey application simply complies with the request, it won’t check whether it makes any sense at this point. In fact, it will even give websites the administrator password entered into a User Access Control prompt.

But there certainly are hurdles? Yes, there are. First of all, such a website needs a valid license. It needs to communicate that license in the get_versions call prior to using any application functionality:

socket.send(JSON.stringify({ "tabid": "whatever", "init": "get_versions", "m": "nxkey", "origin": "https://www.example.com", "lic": "eyJ2ZXJzaW9uIjoiMS4wIiwiaXNzdWVfZGF0ZSI6IjIwMzAwMTAxMTIwMDAwIiwicHJvdG9jb2xfbmFtZSI6InRvdWNoZW5leCIsInV1aWQiOiIwMTIzNDU2Nzg5YWJjZGVmIiwibGljZW5zZSI6IldlMkVtUDZjajhOUVIvTk81L3VNQXRVd0EwQzB1RXFzRnRsTVQ1Y29FVkJpSTlYdXZCL1VCVVlHWlY2MVBGdnYvVUJlb1N6ZitSY285Q1d6UUZWSFlCcXhOcGxiZDI3Z2d0bFJNOUhETzdzPSJ9" }));

This particular license is only valid for www.example.com. So it can only be used by the www.example.com website. Or by any other website claiming to be www.example.com.

See that origin property in the code above? Yes, TouchEn nxKey actually believes that rather than looking at the Origin HTTP header. So it is trivial to lift a license from some website using nxKey legitimately and claim to be that website. It’s not even necessary to create a fake license.

Another hurdle: won’t the data received by the malicious website be encrypted? How does one decrypt it? It should be possible to use a different public key, one where the private key is known. Then one would only need to know the algorithm, and then decrypting the data would work.

Except: none of that is necessary. If TouchEn nxKey doesn’t receive any public key at all, it will simply drop the encryption! The website will receive the keyboard input in clear text then.

Behold, my proof of concept page (less than 3 kB with all the HTML boilerplate):

There is still a third hurdle, one that considerably reduces the severity of this vulnerability: keyboard input intercepted by a malicious web page no longer reaches its destination. A user is bound to get suspicious when they start typing in a password, yet nothing appears in the text field. My analysis of the nxKey application suggests that it only works this way: the keyboard input reaches either the web page or its actual target, but never both.

Attacking the application itself

We’ve already established that whoever wrote the JavaScript code of this product wasn’t very proficient at it. But maybe that’s because all their experts have a C++ background? We’ve already seen this before, developers trying to leave JavaScript and delegate all tasks to C++ code as soon as possible.

Sadly, this isn’t a suspicion I can confirm. I’m way more used to analyzing JavaScript than binary code, but it seems that the application itself is similarly riddled with issues. In fact, it mostly uses approaches typical to C rather than C++. There is lots of manual memory management here.

I already mentioned their use of sprintf_s(). An interesting fact about functions like sprintf_s() or strcpy_s(): while these are the “memory safe” versions of sprintf() or strcpy() functions which won’t overflow the buffer, these are still tricky to use. If you fail giving them a sufficiently large buffer, these will invoke the invalid parameter handler. And by default this makes the application crash.

Guess what: nxKey application almost never makes sure the buffer is sufficiently large. And it doesn’t change the default behavior either. So sending it an overly large value will in many cases crash the application. A crash is better than a buffer overflow, but a crashed application can no longer do its job. Typical result: your online banking login form appears to work correctly, but it receives your password as clear text now. You only notice something being wrong when submitting the form results in an error message. This vulnerability allows Denial-of-Service attacks.

Another example: out of all JSON parsers, the developers of the nxKey application picked out the one written in C. Not only that, they also took a random repository state from January 2014 and never bothered updating it. That null pointer dereference fixed in June 2014? Yeah, still present. So sending ] (a single closing square bracket) to the application instead of JSON data is sufficient to crash it. Another vulnerability allowing Denial-of-Service attacks.

And that WebSockets server websites connect to? It uses OpenSSL. Which OpenSSL? Actually, OpenSSL 1.0.2c. Yes, I can almost hear the collective sigh of all the security professionals here. OpenSSL 1.0.2c is seven years old. In fact, end of support for the 1.0.2 branch was three years ago: on January 1st, 2020. The last release here was OpenSSL 1.0.2u, meaning 18 more releases fixing bugs and security issues. None of the fixes made it into the nxKey application.

Let’s look at something more interesting than crashes. The application license mentioned above is base64-encoded data. The application needs to decode it. The decoder function looks like this:

size_t base64_decode(char *input, size_t input_len, char **result) { size_t result_len = input_len / 4 * 3; if (str[input_len - 1] == '=') result_len--; if (str[input_len - 2] == '=') result_len--; *result = malloc(result_len + 1); // Decoding input in series of 4 characters here }

I’m not sure where this function comes from. It has clear similarities with the base64 decoder of the CycloneCRYPTO library. But CycloneCRYPTO writes the result into a pre-allocated buffer. So it might be that the buffer allocation logic was added by nxKey developers themselves.

And that logic is flawed. It clearly assumes that input_len is a multiple of four. But for input like abcd== its calculation will result in a 2 bytes buffer being allocated, despite the actual output being 3 bytes large.

Is a one byte heap overflow exploitable? Yes, it clearly is as this Project Zero blog post or this article by Javier Jimenez explain. Writing such an exploit is beyond my skill level however.

Instead my proof of concept page merely sent the nxKey application randomly generated license strings. This was sufficient to crash the application in a matter of seconds. Connecting the debugger showed clear evidence of memory corruption: the application crashed because it attempted to read or write data using bogus memory locations. In some cases these memory locations came from the data supplied by my website. So clearly someone with sufficient skill and dedication could have abused that vulnerability for remote code execution.

Modern operating systems have mechanisms to make turning buffer overflows like this one into code execution vulnerabilities harder. But these mechanisms only help if they are actually being used. Yet nxKey developers turned Address space layout randomization off on two DLLs loaded by the application, Data Execution Prevention was turned off on four DLLs.

Abusing the helper application

So far this was all about web-based attacks. But what about the scenario where a malware application managed it into the system already and is looking for ways to expand its privileges? For an application meant to help combat such malware, TouchEn nxKey does surprisingly badly at keeping its functionality to itself.

There is for example the CKAgentNXE.exe helper application starting up whenever nxKey is intercepting keyboard input. Its purpose: when nxKey doesn’t want to handle a key, make sure it is delivered to the right target application. The logic in TKAppm.dll library used by the main application looks roughly like this:

if (IsAdmin()) keybd_event(virtualKey, scanCode, flags, extraInfo); else { AgentConnector connector; // An attempt to open the helper’s IPC objects connector.connect(); if (!connector.connected) { // Application isn’t running, start it now RunApplication("CKAgentNXE.exe"); while (!connector.connected) { Sleep(10); connector.connect(); } } // Some IPC dance involving a mutex, shared memory and events connector.sendData(2, virtualKey, scanCode, flags, extraInfo); }

Since the nxKey application is running with user’s privileges, it will fall back to running CKAgentNXE.exe in every sensible setup. And that helper application, upon receiving command code 2, will call SendInput().

It took me a while to get an idea of what the reason for this approach might be. After all, both nxKey application and CKAgentNXE.exe are running on the same privilege level. Why not just call SendInput()? Why is this indirection necessary?

I noticed however that CKAgentNXE.exe sets a security descriptor for its IPC objects to allow access from processes with integrity level Low. And I also noticed that the setup program creates registry entries under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Low Rights\ElevationPolicy to allow automatic elevation of CKAgentNXE.exe. And that’s where it clicked: this is all because of the Internet Explorer sandbox.

So when TouchEn Key runs as ActiveX in Internet Explorer, its integrity level is Low. Being sandboxed in this way effectively makes it impossible to use SendInput(). This restriction is circumvented by allowing to run and automatically elevate CKAgentNXE.exe from the Internet Explorer sandbox. Once the helper application is running, the sandboxed ActiveX control can connect to it and ask it to do something. Like calling SendInput().

Outside of Internet Explorer this approach makes no sense, yet TouchEn nxKey also delegates work to CKAgentNXE.exe. And this has consequences for security.

Let’s say we have a malware that is running on the integrity level Low. It likely got there by exploiting a browser vulnerability, but now it is stuck in that sandbox. What can it do now? Why, just wait for CKAgentNXE.exe to start up (bound to happen sooner or later) and use it to break out!

My proof of concept application asked CKAgentNXE.exe to generate fake keyboard input for it: Win key, then C, M, D and the Enter key. This resulted in a command line prompt being opened, this one running with the Middle integrity level (the default one). A truly malicious application could then type in an arbitrary command to run code outside the sandbox.

Not that a truly malicious application would do things in such a visible way. CKAgentNXE.exe also accepts command code 5 for example which will load an arbitrary DLL into any process. That’s a much nicer way to infect a system, don’t you think?

At least this time one of the mandatory security applications decided to make itself useful and flag the threat:

A malware author could probably figure out what triggers this warning and get around it. Or they could initiate a web socket connection to make sure CKAgentNXE.exe starts up without also activating AhnLab application like a real banking website would. But why bother? It’s only a prompt, the attack isn’t being stopped proactively. By the time the user clicks to remove the malicious application, it will be too late – the attack already succeeded.

Accessing the driver’s keylogging functionality directly

As mentioned above, TouchEn nxKey application (the one encrypting keyboard input it receives from the driver) is running with user’s privileges. It isn’t an elevated application, it has no special privileges. How is access to the driver’s functionality being restricted then?

The correct answer of course is: it isn’t. Any application on the system has access to this functionality. It only needs to know how nxKey communicates with its driver. And in case you are wondering: that communication protocol isn’t terribly complicated.

I am not sure what the idea here was. TKAppm.dll, the library doing the driver communication, is obfuscated using Themida. The vendor behind Themida promises:

Themida® uses the SecureEngine® protection technology that, when running in the highest priority level, implements never seen before protection techniques to protect applications against advanced software cracking.

Maybe nxKey developers thought that this would offer sufficient protection against reverse engineering. Yet connecting a debugger at runtime allows saving already decrypted TKAppm.dll memory and load the result into Ghidra for analysis.

Sorry, too late. I’ve already got what I needed. And it was no use that your application refuses to work when booting in Safe Mode.

Either way, I could write a tiny (70 lines of code) application that would connect to the driver and use it to intercept all keyboard input on the system. It didn’t require elevation, running with user’s privileges was sufficient. And unlike with a web page this application could also make sure this keyboard input is delivered to its destination, so the user doesn’t notice anything. Creating a keylogger was never so easy!

The best part: this keylogger integrated with the nxKey application nicely. So nxKey would receive keyboard input, encrypt it and send encrypted data to the website. And my tiny keylogger would also receive the same keyboard input, as clear text.

Side-note: Driver crashes

There is something you should know when developing kernel drivers: crashing the driver will crash the entire system. This is why you should make extra certain that your driver code never fails.

Can the driver used by nxKey fail? While I didn’t look at it too closely, I accidentally discovered that it can. See, the application will use DeviceIoControl() to ask the driver for a pointer to the input buffer. And the driver creates this pointer by calling MmMapLockedPagesSpecifyCache().

Yes, this means that this input buffer is visible to every single application on the system. But that’s not the main issue. It’s rather: what happens if the application requests the pointer again? Well, the driver will simply do another MmMapLockedPagesSpecifyCache() call.

After around 20 seconds of doing this in a loop the entire virtual address space is exhausted and MmMapLockedPagesSpecifyCache() returns NULL. The driver doesn’t check the return value and crashes. Boom, the operating system reboots automatically.

This issue isn’t exploitable from what I can tell (note: I am no expert when it comes to binary exploitation), but it is still rather nasty.

Will it be fixed?

Usually, when I disclose vulnerabilities they are already fixed. This time that’s unfortunately not the case. As far as I can tell, none of the issues have been addressed so far. I do not know when the vendors plan to fix these issues. I also do not know how they plan to push out the update to the users, particularly given that banks are already distributing builds that are at least three versions behind the current release. You remember: there is no autoupdate functionality.

Even reporting these issues was complicated. Despite specializing in security, RaonSecure doesn’t list any kind of security contact. In fact, RaonSecure doesn’t list any contact whatsoever, except for a phone number in Seoul. No, I’m not going to phone to Korea asking whether anyone speaks English there.

Luckily, KrCERT provides a vulnerability report form specifically for foreign citizens to use. This form will frequently error out and require you to re-enter everything, and some reports get caught up in a web firewall for no apparent reason, but at least the burden of locating the security contact is on someone else.

I reported all the vulnerabilities to KrCERT on October 4th, 2022. I still tried to contact some RaonSecure executives directly but received no response. At least KrCERT confirmed forwarding my reports to RaonSecure roughly two weeks later. They also noted that RaonSecure asked for my email address and wanted to contact me. They never did.

And that’s it. The 90 days disclosure deadline was a week ago. TouchEn nxKey 1.0.0.78 was apparently released on October 4th, 2022, the same day I reported these vulnerabilities. At the time of writing it remains the latest release, and all the vulnerabilities described here are still present in it. The latest version of the TouchEn browser extension used by millions of people is still five years old, released in January 2018.

Update (2023-01-10): In comments to Korean media, RaonSecure claims to have fixed the vulnerabilities and to distribute the update to customers. I cannot currently confirm this claim. The company’s own download server is still distributing TouchEn nxKey 1.0.0.78.

Side-note: The information leak

How do I even know that they are working on a fix? Well, thanks to something that never happened to me before: they leaked my proofs of concept (meaning: almost complete exploits for the vulnerabilities) prior to the deadline.

See, I used to attach files to my reports directly. However, these attachments would frequently end up being removed or otherwise destroyed by overzealous security software. So instead I now upload whatever files are needed to demonstrate the issue to my server. A link to my server always works. Additional benefit: even with companies that don’t communicate I can see in the logs whether the vendor accessed the proof of concept at all, meaning whether my report reached anyone.

A few days ago I checked the logs for accesses to the TouchEn nxKey files. And immediately saw Googlebot. Sure enough: these files ended up being listed in the Google index.

Now I use a random folder name, it cannot be guessed. And I only shared the links with the vendor. So the vendor must have posted a publicly visible link to the exploits somewhere.

And that’s in fact what they did. I found a development server, publicly visible and indexed by Google. It seems that this server was originally linking to my proof of concept pages. By the time I found it, it was instead hosting the vendor’s modified copies of them.

The first request by Googlebot was on October 17th, 2022. So I have to assume that these vulnerabilities could be found via a Google search more than two months prior to the disclosure deadline. They have been accessed many times, hard to tell whether it’s only been the product’s developers.

After reporting this issue the development server immediately disappeared from the public internet. Still, such careless handling of security-sensitive information isn’t something I’ve ever seen before.

Can the nxKey concept even work?

We’ve seen a number of vulnerabilities in the TouchEn nxKey application. By attempting to combat keyloggers, nxKey developers built a perfect keylogging toolset and failed to restrict access to it. But the idea is nice, isn’t it? Maybe it would actually be a useful security tool if built properly?

Question is: the keylogger that is being protected against, what level does it run on? The way I see it, there are four options:

1. In the browser. So some malicious JavaScript code is running in the online banking page, attempting to capture passwords. That code can trivially stop the page from activating nxKey.
2. In the system, with user’s privileges. This privilege level is e.g. sufficient to kill the CrossEXService.exe process which is also running with user’s privileges. This achieves the same results as my denial-of-service attacks, protection is effectively disabled.
3. In the system, with administrator privileges. That’s actually sufficient privileges to unload the nxKey driver and replace it by a trojanized copy.
4. In the hardware. Game over, good luck trying any software-based solutions against that.

So whatever protection nxKey might provide, it relies on attackers who are unaware of nxKey and its functionality. Generic attacks may be thwarted, but it is unlikely to be effective against any attacks targeting specifically South Korean banks or government organizations.

Out of these four levels, number 2 might be possible to fix. The application CrossEXService.exe could be made to run with administrator’s privileges. This would prevent malware from messing with this process. Effectiveness of this protection would still rely on the malware being unable to get into the user’s browser however.

I cannot see how this concept could be made to work reliably against malware operating on other levels.

We are pleased to announce that Android platform support in Rust will be modernized in Rust 1.68 as we update the target NDK from r17 to r25. As a consequence the minimum supported API level will increase from 15 (Ice Cream Sandwich) to 19 (KitKat).

In NDK r23 Android switched to using LLVM's libunwind for all architectures. This meant that

1. If a project were to target NDK r23 or newer with previous versions of Rust a workaround would be required to redirect attempts to link against libgcc to instead link against libunwind. Following this update this workaround will no longer be necessary.
2. If a project uses NDK r22 or older it will need to be updated to use r23 or newer. Information about the layout of the NDK's toolchain can be found here.

Going forward the Android platform will target the most recent LTS NDK, allowing Rust developers to access platform features sooner. These updates should occur yearly and will be announced in release notes.

I recently wrapped up a project on improving notifications in threads for Matrix. This is adapted from my research notes to understand the status quo before adapting the Matrix protocol for threads (in MSC3771 and MSC3773). Hopefully others find the information useful!

Note

These notes are true as of the v1.3 of the Matrix spec and also cover some Matrix spec changes which may or may not have been merged since. It is known to be out of date with the changes from MSC2285, MSC3771, and MSC3773.

Receipts

Matrix uses “receipts” to note which part of a room has been read by a user. It considers the history for a room to be split into three sections1:

1. Messages the user has read (or indicated they aren’t interested in them).
2. Messages the user might have read some but not others.
3. Messages the user hasn’t seen yet.

The fully read marker is between 1 & 2 while the read receipt is between 2 & 3. Note that fully read markers are not shared with other users while read receipts are.

Another way to consider this is2:

1. Fully read marker: a private bookmark to indicate the point which has been processed in the discussion. This allows a user to go back to it later.
2. Read receipts: public indicators of what a user has seen to inform other participants that the user has seen it.
3. Hidden read receipts: a private mechanism to synchronize “unread messages” indicators between a user’s devices (while still retaining the ability from 1 as a separate concept). (See MSC2285.)

Fully read markers

They are stored in the room account data for the user (but modified via a separate API).

The API is:

POST /_matrix/client/v3/rooms/{roomId}/read_markers

The read receipt can optionally be updated at the same time.

In Element Web your fully read marker is displayed as the green line across the conversation.

Read receipts

Only m.read is defined at the moment, but it is meant to be generic infrastructure.

Updating a read receipt updates a “marker” which causes any notifications prior to and including the event to be marked as read.3 A user has a single read receipt “marker” per room.

Passed to clients as an m.receipt event under the ephemeral array for each room in the /sync response. The event includes a map of event ID -> receipt type -> user ID -> data (currently just a timestamp). Note that the event is a delta from previous events. An example:

{ "content": { "$1435641916114394fHBLK:matrix.org": { "m.read": { "@rikj:jki.re": { "ts": 1436451550453 } } } }, "room_id": "!jEsUZKDJdhlrceRyVU:example.org", "type": "m.receipt" }

The API is:

POST /_matrix/client/v3/rooms/{roomId}/receipt/{receiptType}/{eventId}

In Element Web read receipts are the small avatars on the right hand side of the conversation. Note that your own read receipt is not shown.

Hidden read receipts (MSC2285)

A new receipt type (m.read.hidden) to set a read receipt without sharing it with other users. It also modifies the /read_markers API to accept the new receipt type and modifies the /receipts API to accept the fully read marker.

This is useful to synchronize notifications across devices while keeping read status private.

Push rules

A user’s push rules determine one or more user-specific actions on each event. They are customizable, but the homeserver provides default rules. They can result in an action to:

1. Do nothing
2. Notify the user (notify action), which can have additional actions (“tweaks”):
   1. Highlight the message (highlight action)
   2. Play a sound (sound action)

By default, all new m.room.message and m.room.encrypted events generate a notification, events with a user’s display name or username in them or @room generate highlights, etc.

Push rules for relations (MSC3664)

Augments push rules to allow applying them to the target of an event relationship and defines a default push rule for replies (not using the reply fallback).

Event notification attributes and actions (MSC2785)

A proposed replacement for push rules, the results are essentially the same actions (and presumedly would not change the data returned in /sync, see below).

Notification counts in /sync

The number of notification events and highlight events since the user’s last read receipt are both returned on a per room basis as part of a /sync response (for joined room).

Notification and highlight events are any messages where the push rules resulted in an action of notify or highlight, respectively. (Note that a highlight action must be a notify action, thus highlight_count <= notification_count.)

An example:

{ "account_data": [...], "ephemeral": [...], "state": [...], "summary": {...}, "timeline": {...}, "unread_notifications": { "highlight_count": 0, "notification_count": 0 } } Unread messages count (MSC2654)

A new field is added under the unread_notifications field (unread_count) which is the total number of events matching particular criteria since the user’s last read receipt.

This replaces MSC2625, which adds a new push rule action (mark_unread) to perform the same task. In this rendition, notify implies mark_unread and thus highlight_count <= notification_count <= unread_count.

Push notifications

Push notifications receive either the number of unread messages (across all rooms) or the number of rooms with unread messages (depending on the value of push.group_unread_count_by_room in the Synapse configuration). Unread messages are any messages where the push rules resulted in an action of notify.

This information is sent from the homeserver to the push gateway as part of every notification:

{ "notifications": { "counts": { "unread": 1, ... }, ... } }

1. From the fully read marker specification. ↩

2. See a discussion on MSC2285. ↩

3. Spec on receiving notifications ↩

Edit (2023-01-04): A Korean translation of this article is now available here, thanks to Woojin Kim. Edit (2023-01-07): Scheduled one more disclosure for February.

Last September I started investigating a South Korean application with unusually high user numbers. It took me a while to even figure out what it really did, there being close to zero documentation. I eventually realized that the application is riddled with security issues and, despite being advertised as a security application, makes the issue it is supposed to address far, far worse.

That’s how my journey to the South Korea’s very special security application landscape started. Since then I investigated several other applications and realized that the first one wasn’t an outlier. All of them caused severe security and privacy issues. Yet they were also installed on almost every computer in South Korea, being a prerequisite for using online banking or government websites in the country.

Before I start publishing articles on the individual applications’ shortcomings I wanted to post a summary of how (in my limited understanding) this situation came about and what exactly went wrong. From what I can tell, South Korea is in a really bad spot security-wise right now, and it needs to find a way out ASAP.

Contents

• Historical overview
• Current situation
• Software quality
• Security through obscurity
• Explanation attempts
• Getting out of the dead end
• Schedule of future disclosures

Historical overview

I’ve heard about South Korea being very “special” every now and then. I cannot claim to fully understand the topic, but there is a whole Wikipedia article on it. Apparently, the root issue were the US export restrictions on strong cryptography in the 90ies. This prompted South Korea to develop their own cryptographic solutions.

It seems that this started a fundamental distrust in security technologies coming out of the United States. So even when the export restrictions were lifted, South Korea continued adding their own security layers on top of SSL. All users had to install special applications just to use online banking.

Originally, these applications used Microsoft’s proprietary ActiveX technology. This only worked in Internet Explorer and severely hindered adoption of other browsers in South Korea.

Wikipedia lists several public movements aimed at improving this situation. Despite the pressure from these, it took until well after 2010 that things actually started changing.

Technologically, the solutions appear to have gone through several iterations. The first one were apparently NPAPI plugins, the closest thing to ActiveX in non-Microsoft browsers. I’ve also seen solutions based on browser extensions which are considerably simpler than NPAPI plugins.

Currently, the vendors appear to have realized that privileged access to the browser isn’t required. Instead, they merely need a communication channel from the websites to their application. So now all these applications run a local web server that websites communicate with.

Current situation

Nowadays, a typical Korean banking website will require five security applications to be installed before you are allowed to log in. One more application is suggested to manage this application zoo. And since different websites require different sets of applications, a typical computer in South Korea probably runs a dozen different applications from half a dozen different vendors. Just to be able to use the web.

Each of these applications comes with a website SDK that the website needs to install, consisting of half a dozen JavaScript files. So your typical Korean banking website takes quite a while to load and initialize.

Interestingly, most of these applications don’t even provide centralized download servers. The distribution and updates have been completely offloaded to websites using these security applications.

And that is working exactly as well as you’d expect. Even looking at mere usability, I’ve noticed an application that a few years ago went through a technology change: from using a browser extension to using a local web server for communication. Some banks still distribute and expect the outdated application version, others work with the new one. For users it is impossible to tell why they have the application installed, yet their bank claims that they don’t. And they complain en masse.

Obviously, websites distributing applications also makes them a target. And properly securing so many download servers is unrealistic. So a few years ago the North Korean Lazarus group made the news by compromising some of these servers in order to distribute malware.

Software quality

I took a thorough look at the implementation of several security applications widely used in South Korea. While I’ll go into the specific issues in future blog posts, some tendencies appear universal across the entire landscape.

One would think, being responsible for the security of an entire nation would make vendors of such software be extra vigilant. That’s not what I saw however. In fact, security-wise these applications are often decades behind state of the art.

This starts with a simple fact: some of these applications are written in the C programming language, not even C++. It being a low-level programming language, these days it is typically used in code that has to work close to hardware such as device drivers. Here however it is used in large applications interacting with websites in complicated ways.

The manual approach to memory management in C is a typical source of exploitable memory safety issues like buffer overflows. Avoiding them in C requires utmost care. While such bugs weren’t the focus of my investigation, I couldn’t fail noticing that the developers of these applications didn’t demonstrate much experience avoiding memory safety issues.

Modern compilers provide a number of security mechanisms to help alleviate such issues. But these applications don’t use modern compilers, relying on Visual Studio versions released around 15 years ago instead.

And even the basic security mechanisms supported by these ancient compilers, such as Address Space Layout Randomization (ALSR) and Data Execution Prevention (DEP), tend to be disabled. There is really no good reason for that, these are pure security benefit “for free.”

To make matters even worse, the open source libraries bundled in these applications tend to see no updates whatsoever. So far the record holder was a library which turned out to be more than a decade old. There have been more than 50 releases of this library since then, with many improvements and security fixes. None of them made it into the application.

Security through obscurity

Given how South Korea’s security applications are all about cryptography, they are surprisingly bad at it. In most cases, cryptography is merely being used as obfuscation, only protecting against attackers who cannot reverse engineer the algorithm. Other issues I’ve seen were dropping encryption altogether if requested or algorithm parameters that have been deprecated decades ago.

In fact, vendors of these applications appear to view reverse engineering as the main issue. There is very little transparency and much security through obscurity here. It’s hard to tell whether this approach actually works to deter hackers or we merely don’t learn about the successful attacks.

Either way, I’ve seen multiple applications use software “protection” that decrypts the code at runtime to prevent reverse engineering. While I don’t have much experience with such mechanisms, I found that attaching to the process with x64dbg at runtime and using the Scylla plugin does just fine to get a decrypted exe/dll file that can be fed into a disassembler.

There are services that will immediately shut down the application if a debugger is attached. And one application even attempts to prevent browser’s Developer Tools from being used. Neither mechanism mitigates security risks, the goal here is rather maintaining obscurity.

Explanation attempts

I think the main issue here is that the users are not the customers. While this is supposedly all about their safety, the actual customers are the banks. The users don’t get to choose whether to install an application, it is required. And banks can delegate liability away.

If somebody loses money due to a hack, the bank cannot possibly be at fault. The bank did everything right after all. It made the user install all the important security applications. That seems to be the logic here.

This creates a market for bogus security applications. Most of them fail at properly addressing an issue. Way too often they even make matters considerably worse. And in the few cases where meaningful functionality is present, a modern web browser is perfectly capable of it without any third-party software.

But none of this matters as long as banks continue to buy these applications. And whether they do is only related to whether they see a value for themselves, not whether the application does anything meaningful.

The vendors know that of course. That’s why they haven’t been investing into the security of their applications for decades, it simply doesn’t matter. What matters are the features that banks will see. Ways for them to customize the application. Ways for them to collect even more of users’ data. Ways for them to spend money and to get (perceived) security back without any noteworthy effort.

Getting out of the dead end

Unfortunately, I know too little about the Korean society to tell how they would get out of this less than perfect situation. One thing I’m pretty certain about however: improving the existing security applications won’t do it.

Yes, I reported the security and privacy issues I found. I gave the vendors some time to protect the users before my disclosure. And I hope they will.

It isn’t really going to change the situation however. Because many of these issues are by design. And if they fix all of them, they will no longer have a product to sell.

In fact, the ideal outcome is dismantling South Korea’s special security landscape altogether. Not relying on any of these security applications will be a huge win for security. This likely won’t be possible without some definitive legislative action however. Ideally one that will give users a choice again and outlaw forcing installation of third-party applications just to use basic online services.

Schedule of future disclosures

When I report security issues vendors generally get 90 days to fix them. Once that deadline is over I disclose my research. If you are interested in reading the disclosures, you can subscribe to the RSS feed of this blog. Alternatively, you could also check my blog on the scheduled disclosure dates:

• 2023-01-09: TouchEn nxKey: The keylogging anti-keylogger solution
• 2023-01-25: IPinside: Korea’s mandatory spyware
• 2023-02-06: Weakening TLS protection, South Korean style
• 2023-02-20: South Korea’s banking security: Intermediate conclusions
• 2023-03-06 (March 6th)

This article first appeared on the Firefox Attack & Defense blog.

Despite all the efforts of fixing Cross-Site Scripting (XSS) on the web, it continuously ranks as one of the most dangerous security issues in software.

In particular, DOM-based XSS is gaining increasing relevance: DOM-based XSS is a form of XSS …

If you happen to be using Firefox Beta 109 on an overpriced MacBook Pro that has a sticky letter s today (the 29th of December, 2022), this is what the User-Agent string looks like:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/109.0

And as of last week, the UA string in Firefox for versions 110 and higher looks like so:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0

If you managed to visually discover the difference (I guess in the world’s lamest “Spot the Difference” game), congrats. If you didn’t, take note that rv:109.0 did not change in the second one—but Firefox/110.0 did.

So why did Mozilla just freeze rv:109.0 in the User-Agent string? Perhaps forever, or just perhaps until Firefox 120 is released?

Presumably in an attempt to unburden itself from a legacy of UA-sniffing-driven workarounds for a browser that hadn’t historically supported a lot of useful things (like WebGL, or some ES5 or ES6 stuff - I don’t really remember and can’t be bothered to look it up), the IE team decided to change up their User-Agent string back in 2013.

Here’s the IE10 UA, which followed the same-ish predictable pattern they had since IE 2.0:

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0)

And here’s an IE11 UA:

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko

Basically they were trying to solve the problem of “now that we’ve invested seriously in web standards, how do we get access to content that makes use of those features, and still have a detectable version number for analytics (or whatever). And it doesn’t not makes sense to borrow Firefox’s rv: convention to accomplish that (slapping an extra “like Gecko” in there for good luck can’t hurt, I suppose (but more realistically, there was probably some bank or government site that sniffed for Safari’s like Gecko token)).

And then cut to today, about 9 years later where a handful of sites (including popular ones like bestbuy and cvs) tell Firefox users to upgrade to a modern browser, because there’s probably something really clever like var isIE = /rv:11/i.test(navigator.userAgent);.

That’s obviously lame, and hence, Mozilla has frozen another part of its UA string for compatibility. Anyways, happy new years, especially to the folks working to make sure the web is still usable in Firefox.

Hey SUMO nation,

Time surely flies, and here we are, already at the end of the year. 2022 has been an amazing year for the Customer Experience team. We welcomed 5 new people to our team this year, including 2 engineers, and 1 technical writer.

As an extension of our team, the SUMO community definitely plays an important role in our achievements. Let’s take a moment to reflect on what the community has accomplished this year. 

• Forum

From January 1 to November 11, we posted 49K answers (from non-OP* users) to 28K questions posted on the forum. On average, our answer rate within 72 hours is 75% while our solved rate is around 14%. In a month, around 300 users have contributed to the forum (including the OP).

*See Support glossary

• KB

From January 1 to November 11, the KB contributors have made 1746 revisions (all contributor revisions) with a 73% review rate and 95% approval rate. On average, we have in total of 30 contributors to our Knowledge Base on a monthly basis. 

• Localization

The localization community had been doing great things this year by submitting to 13K revisions from January 1 to November 11. The review rate for the localization is looking pretty good at 90%, while the approval rate is 99%. On average, there are around 73 contributors involved on a monthly basis, from around 30 locales. We saw the PT-PT community has been recently re-activated as well after the pandemic, which is amazing.

• Social Support

From January 1 to December 28, the Social Support Contributors have contributed to 908 responses in total (39.6% of our total responses). We also have been able to improve our resolved rate from 58% in 2021 to 70% this year. 

• Mobile Store

Last but not least, from January 1 to Dec 28, the Mobile Store Support contributors have contributed to 1.6K replies and onboarded 4 new contributors this year. The response conversion (comparison between total responses against total moderation) rate is also looking good, with 47% on average throughout the year. Meaning, 47% of reviews that contributors moderated are replied to.

Apart from that, we have also managed to work on a few projects throughout the year:

• Mobile hybrid support

In Q2, we hired a Community Support Advocate whose primary role is to support the mobile store ecosystem by moderating questions in Google Play Store and Apple App Store. This Community Support Advocate is working alongside contributors on Google Play Store and takes primary care of the App Store reviews as well as moderating forum questions (mainly by adding tags) for the mobile products to this day.

With the spirit of continuing the community program, we also rename the Mobile Support program to Mobile Store Support in Q4 with the introduction of the new contributor onboarding page.

• Locale audit

We also did a locale audit in Q2 to check on the stage of our localization community. I presented the result of the audit on the community call in June.

• Internal community dashboard

After the platform team fixed the data pipeline issue that was going on since the beginning of the year, Q3 follows with a project to create an internal community dashboard. I gave a brief overview of the project back then on the community call in July.

• MR2022

Major Release 2022 went smoothly in Q3 because of the support of you all. Similar to what we did for the Major Release last year, we also prepared a list of changes for contributors and monitor the inbounds closely across the channels that we oversee. This time, the product team also worked with the CMO team to collect rapid feedback about some of the major features that we released in Firefox 106.

• Contributor onboarding launch

In early November, we finally got to see the new face of our contributor onboarding page, which was formerly called the Get Involved page. You can learn more about this update in this blog post or by directly checking out the page.

It was not all rainbows and butterflies, though. On September 2022, we learned the news about the passing of one of our top contributors in the forum, FredMcD. It was definitely a great loss for the community.

Despite all the bumps, we do survive the year 2022, with grace and triumph. All the numbers that I presented at the beginning are not merely metrics. They are reflections of a collective effort from all of you, Mozillians around the world, who work tirelessly to keep the internet healthy and supported each other in the spirit of keeping the internet as a global public resource, open and accessible to all. I’m proud of working alongside you all and to reflect on what we have accomplished this year.

And yes, let’s keep on rocking the helpful web through 2023 and beyond!

PS!

If you’re a looker and interested in contributing to Mozilla Support, please head over to our Contribute page to learn more about our programs!

 

Kiki

 

LastPass has been breached, data has been stolen. I already pointed out that their official statement is misleading. I also explained that decrypting passwords in the stolen data is possible which doesn’t mean however that everybody is at risk now. For assessing whether you are at risk, a fairly hidden setting turned out critical: password iterations.

LastPass provides an instruction to check this setting. One would expect it to be 100,100 (the LastPass default) for almost everyone. But plenty of people report having 5,000 configured there, some 500 and occasionally it’s even 1 (in words: one) iteration.

Let’s say this up front: this isn’t the account holders’ fault. It rather is a massive failure by LastPass. They have been warned, yet they failed to act. And even now they are failing to warn the users who they know are at risk.

Contents

• What is this setting about?
• How did the low iteration numbers come about?
• What could LastPass do about it now?

What is this setting about?

This setting is actually central to protecting your passwords if LastPass loses control of your data (like they did now). Your passwords are encrypted. In order to decrypt them, the perpetrators need to guess your master password. The more iterations you have configured, the slower this guessing will be. The current OWASP recommendation is 310,000 iterations. So the LastPass default is already factor three below the recommendation.

What’s the impact if you have an even lower iterations number configured? Let’s say you have a fairly strong master password, 50 bits of entropy. For example, it could be an eight character random password, with uppercase and lowercase letters, digits and even some special characters. Yes, such password is already rather hard to remember but you want your passwords to be secure.

Or maybe you went for a diceware password. You took a word list for four dices (1296 words) and you randomly selected five words for your master password.

Choosing a password with 50 bits entropy without it being randomized? No idea how one would do it. Humans are inherently bad at choosing strong passwords. You’d need a rather long password to get 50 bits, and you’d need to avoid obvious patterns like dictionary words.

Either way, if this is your password and someone got your LastPass vault, guessing your master password on a single graphics card would take on average 200 years. Not unrealistic (someone could get more graphics cards) but usually not worth the effort. But that’s the calculation for 100,100 iterations.

Let’s look at how time estimates and cost change depending on the number of iterations. I’ll be using the cost estimate by Jeffrey Goldberg who works at 1Password.

Iterations Guessing time on a single GPU Cost 100,100  200 years $1,500,000 5,000  10 years $75,000 500  1 year $7,500 1  17 hours $15

And that’s a rather strong password. According to this older study, the average password has merely 40 bits of entropy. So divide all numbers by 1,000 for that.

How did the low iteration numbers come about?

The default for LastPass accounts wasn’t always 100,100 iterations. Originally it was merely 1 iteration. At some point this was changed to 500 iterations, later to 5,000. And the final change adjusted this value to 100,100 iterations.

I don’t know exactly when and how these changes happened. Except for the last one: it happened in February 2018 as a result of my research.

Edit (2022-12-30): I now know more, thanks to @Sc00bz@infosec.exchange. The switch to 500 iterations happened in June 2012, the one to 5,000 iterations in February 2013. To quote Sc00bz: “I shamed the CEO into increasing this. «I think it is irresponsible to tell your users the recommended iteration count is 500. When 12 years ago, PBKDF2 had a recommended minimum iteration count of 1000.»”

LastPass was notified through their bug bounty program on Bugcrowd. When they reported fixing the issue I asked them about existing accounts. That was on February 24th, 2018.

They didn’t reply. So I prompted them again in an email on March 15th and got the reply that the migration should take until end of May.

I asked again about the state of the migration on May 23rd. This time the reply was that the migration is starting right now and is expected to complete by mid-June.

On June 25th I was once again contacted by LastPass, asking me to delay disclosure until they finish migrating existing accounts. I replied asking whether the migration actually started now and got the response: yes, it did last week.

My disclosure of the LastPass issues was finally published on July 9th, 2018. After all the delays requested by LastPass, their simultaneously published statement said:

we are in the process of automatically migrating all existing LastPass users to the new default.

We can now safely assume that the migration wasn’t actually underway even at this point. One user reported receiving an email about their account being upgraded to a higher password iterations count, and that was mid-2019.

Worse yet, for reasons that are beyond me, LastPass didn’t complete this migration. My test account is still at 5,000 iterations, as are the accounts of many other users who checked their LastPass settings. LastPass would know how many users are affected, but they aren’t telling that.

In fact, it’s painfully obvious that LastPass never bothered updating users’ security settings. Not when they changed the default from 1 to 500 iterations. Not when they changed it from 500 to 5,000. Only my persistence made them consider it for their latest change. And they still failed implementing it consistently.

So we now have people report finding their accounts to be configured with 500 iterations. And for some it’s even merely one iteration. For example here. And here. And here.

This is a massive failure on LastPass’ side, they failed to keep these users secure. They cannot claim ignorance. They had years to fix this. Yet they failed.

What could LastPass do about it now?

There is one thing that LastPass could do easily: query their database for users who have less than 100,100 iterations configured and notify all of them. Obviously, these users are at heightened risk due to the LastPass breach. Some found out about it, most of them likely didn’t. So far, LastPass chose not to notify them.

Of course, LastPass could also deliver on their promise and fix the iterations count for the affected accounts. It won’t help with the current breach but at least it will better protect these accounts in future. So far this didn’t happen either.

Finally, LastPass could change the “Password Iterations” setting and make sure that nobody accidentally configures a value that is too low. It’s Security 101 that users shouldn’t be able to set settings to values that aren’t safe. But right now I changed the iterations count for my test account to 1 and I didn’t even get a warning about it.

Slightly confusion conditions around Jest module mocks.

Software-update: SOGo 5.8.0 - Computer - Downloads  Tweakers

'Ubuntu is Linux voor gewone mensen' - Interview  Tweakers

Mozilla zet in op Metaverse en neemt ‘Active Replica’ over  Newsbit.nl

Mozilla en Microsoft vertrouwen certificaatautoriteit TrustCor niet meer  Security.nl

Software-update: Mozilla Firefox 107.0.1 - Computer - Downloads  Tweakers

Thunderbird: minder dan één procent gebruikers doneert geld  Security.nl

"Criminelen stelen 675.000 wachtwoorden van Nederlandse computers"  Security.nl

Software-update: Pale Moon 31.4.0 - Computer - Downloads  Tweakers

During July I’ll be visiting family in Mongolia but I’ve also a few things that are very geeky that I want to do.

The first thing I want to do is plug the Ripe Atlas probes I have. It’s litle devices that look like that :

They enable anybody with a ripe atlas or ripe account to make measurements for dns queries and others. This helps making a global better internet. I have three of these probes I’d like to install. It’s good because last time I checked Mongolia didn’t have any active probe. These probes will also help Internet become better in Mongolia. I’ll need to buy some network cables before leaving because finding these in mongolia is going to be challenging. More on atlas at https://atlas.ripe.net/.

The second thing I intend to do is map Mongolia a bit better on two projects the first is related to Mozilla and maps gps coordinateswith wifi access point. Only a little part of The capital Ulaanbaatar is covered as per https://location.services.mozilla.com/map#11/47.8740/106.9485 I want this to be way more because having an open data source for this is important in the future. As mapping is my new thing I’ll probably edit Openstreetmap in order to make the urban parts of mongolia that I’ll visit way more usable on all the services that use OSM as a source of truth. There is already a project to map the capital city at http://hotosm.org/projects/mongolia_mapping_ulaanbaatar but I believe osm can server more than just 50% of mongolia’s population.

I got inspired to write this post by mu son this morning, look what he is doing at 17 months :

Last week I gave a talk at the Philly Tech Week 2015 Dev Day organized by the delightful people at technical.ly on some of the tricks/strategies we use in the Firefox OS Gaia Email app.  Note that the credit for implementing most of these techniques goes to the owner of the Email app’s front-end, James Burke.  Also, a special shout-out to Vivien for the initial DOM Worker patches for the email app.

I tried to avoid having slides that both I would be reading aloud as the audience read silently, so instead of slides to share, I have the talk script.  Well, I also have the slides here, but there’s not much to them.  The headings below are the content of the slides, except for the one time I inline some code.  Note that the live presentation must have differed slightly, because I’m sure I’m much more witty and clever in person than this script would make it seem…

Cover Slide: Who!

Hi, my name is Andrew Sutherland.  I work at Mozilla on the Firefox OS Email Application.  I’m here to share some strategies we used to make our HTML5 app Seem faster and sometimes actually Be faster.

What’s A Firefox OS (Screenshot Slide)

But first: What is a Firefox OS?  It’s a multiprocess Firefox gecko engine on an android linux kernel where all the apps including the system UI are implemented using HTML5, CSS, and JavaScript.  All the apps use some combination of standard web APIs and APIs that we hope to standardize in some form.

Here are some screenshots.  We’ve got the default home screen app, the clock app, and of course, the email app.

It’s an entirely client-side offline email application, supporting IMAP4, POP3, and ActiveSync.  The goal, like all Firefox OS apps shipped with the phone, is to give native apps on other platforms a run for their money.

And that begins with starting up fast.

Fast Startup: The Problems

But that’s frequently easier said than done.  Slow-loading websites are still very much a thing.

The good news for the email application is that a slow network isn’t one of its problems.  It’s pre-loaded on the phone.  And even if it wasn’t, because of the security implications of the TCP Web API and the difficulty of explaining this risk to users in a way they won’t just click through, any TCP-using app needs to be a cryptographically signed zip file approved by a marketplace.  So we do load directly from flash.

However, it’s not like flash on cellphones is equivalent to an infinitely fast, zero-latency network connection.  And even if it was, in a naive app you’d still try and load all of your HTML, CSS, and JavaScript at the same time because the HTML file would reference them all.  And that adds up.

It adds up in the form of event loop activity and competition with other threads and processes.  With the exception of Promises which get their own micro-task queue fast-lane, the web execution model is the same as all other UI event loops; events get scheduled and then executed in the same order they are scheduled.  Loading data from an asynchronous API like IndexedDB means that your read result gets in line behind everything else that’s scheduled.  And in the case of the bulk of shipped Firefox OS devices, we only have a single processor core so the thread and process contention do come into play.

So we try not to be a naive.

Seeming Fast at Startup: The HTML Cache

If we’re going to optimize startup, it’s good to start with what the user sees.  Once an account exists for the email app, at startup we display the default account’s inbox folder.

What is the least amount of work that we can do to show that?  Cache a screenshot of the Inbox.  The problem with that, of course, is that a static screenshot is indistinguishable from an unresponsive application.

So we did the next best thing, (which is) we cache the actual HTML we display.  At startup we load a minimal HTML file, our concatenated CSS, and just enough Javascript to figure out if we should use the HTML cache and then actually use it if appropriate.  It’s not always appropriate, like if our application is being triggered to display a compose UI or from a new mail notification that wants to show a specific message or a different folder.  But this is a decision we can make synchronously so it doesn’t slow us down.

Local Storage: Okay in small doses

We implement this by storing the HTML in localStorage.

Important Disclaimer!  LocalStorage is a bad API.  It’s a bad API because it’s synchronous.  You can read any value stored in it at any time, without waiting for a callback.  Which means if the data is not in memory the browser needs to block its event loop or spin a nested event loop until the data has been read from disk.  Browsers avoid this now by trying to preload the Entire contents of local storage for your origin into memory as soon as they know your page is being loaded.  And then they keep that information, ALL of it, in memory until your page is gone.

So if you store a megabyte of data in local storage, that’s a megabyte of data that needs to be loaded in its entirety before you can use any of it, and that hangs around in scarce phone memory.

To really make the point: do not use local storage, at least not directly.  Use a library like localForage that will use IndexedDB when available, and then fails over to WebSQLDatabase and local storage in that order.

Now, having sufficiently warned you of the terrible evils of local storage, I can say with a sorta-clear conscience… there are upsides in this very specific case.

The synchronous nature of the API means that once we get our turn in the event loop we can act immediately.  There’s no waiting around for an IndexedDB read result to gets its turn on the event loop.

This matters because although the concept of loading is simple from a User Experience perspective, there’s no standard to back it up right now.  Firefox OS’s UX desires are very straightforward.  When you tap on an app, we zoom it in.  Until the app is loaded we display the app’s icon in the center of the screen.  Unfortunately the standards are still assuming that the content is right there in the HTML.  This works well for document-based web pages or server-powered web apps where the contents of the page are baked in.  They work less well for client-only web apps where the content lives in a database and has to be dynamically retrieved.

The two events that exist are:

“DOMContentLoaded” fires when the document has been fully parsed and all scripts not tagged as “async” have run.  If there were stylesheets referenced prior to the script tags, the script tags will wait for the stylesheet loads.

“load” fires when the document has been fully loaded; stylesheets, images, everything.

But none of these have anything to do with the content in the page saying it’s actually done.  This matters because these standards also say nothing about IndexedDB reads or the like.  We tried to create a standards consensus around this, but it’s not there yet.  So Firefox OS just uses the “load” event to decide an app or page has finished loading and it can stop showing your app icon.  This largely avoids the dreaded “flash of unstyled content” problem, but it also means that your webpage or app needs to deal with this period of time by displaying a loading UI or just accepting a potentially awkward transient UI state.

(Trivial HTML slide)

<link rel=”stylesheet” ...> <script ...></script> DOMContentLoaded!

This is the important summary of our index.html.

We reference our stylesheet first.  It includes all of our styles.  We never dynamically load stylesheets because that compels a style recalculation for all nodes and potentially a reflow.  We would have to have an awful lot of style declarations before considering that.

Then we have our single script file.  Because the stylesheet precedes the script, our script will not execute until the stylesheet has been loaded.  Then our script runs and we synchronously insert our HTML from local storage.  Then DOMContentLoaded can fire.  At this point the layout engine has enough information to perform a style recalculation and determine what CSS-referenced image resources need to be loaded for buttons and icons, then those load, and then we’re good to be displayed as the “load” event can fire.

After that, we’re displaying an interactive-ish HTML document.  You can scroll, you can press on buttons and the :active state will apply.  So things seem real.

Being Fast: Lazy Loading and Optimized Layers

But now we need to try and get some logic in place as quickly as possible that will actually cash the checks that real-looking HTML UI is writing.  And the key to that is only loading what you need when you need it, and trying to get it to load as quickly as possible.

There are many module loading and build optimizing tools out there, and most frameworks have a preferred or required way of handling this.  We used the RequireJS family of Asynchronous Module Definition loaders, specifically the alameda loader and the r-dot-js optimizer.

One of the niceties of the loader plugin model is that we are able to express resource dependencies as well as code dependencies.

RequireJS Loader Plugins

var fooModule = require('./foo'); var htmlString = require('text!./foo.html'); var localizedDomNode = require('tmpl!./foo.html');

The standard Common JS loader semantics used by node.js and io.js are the first one you see here.  Load the module, return its exports.

But RequireJS loader plugins also allow us to do things like the second line where the exclamation point indicates that the load should occur using a loader plugin, which is itself a module that conforms to the loader plugin contract.  In this case it’s saying load the file foo.html as raw text and return it as a string.

But, wait, there’s more!  loader plugins can do more than that.  The third example uses a loader that loads the HTML file using the ‘text’ plugin under the hood, creates an HTML document fragment, and pre-localizes it using our localization library.  And this works un-optimized in a browser, no compilation step needed, but it can also be optimized.

So when our optimizer runs, it bundles up the core modules we use, plus, the modules for our “message list” card that displays the inbox.  And the message list card loads its HTML snippets using the template loader plugin.  The r-dot-js optimizer then locates these dependencies and the loader plugins also have optimizer logic that results in the HTML strings being inlined in the resulting optimized file.  So there’s just one single javascript file to load with no extra HTML file dependencies or other loads.

We then also run the optimizer against our other important cards like the “compose” card and the “message reader” card.  We don’t do this for all cards because it can be hard to carve up the module dependency graph for optimization without starting to run into cases of overlap where many optimized files redundantly include files loaded by other optimized files.

Plus, we have another trick up our sleeve:

Seeming Fast: Preloading

Preloading.  Our cards optionally know the other cards they can load.  So once we display a card, we can kick off a preload of the cards that might potentially be displayed.  For example, the message list card can trigger the compose card and the message reader card, so we can trigger a preload of both of those.

But we don’t go overboard with preloading in the frontend because we still haven’t actually loaded the back-end that actually does all the emaily email stuff.  The back-end is also chopped up into optimized layers along account type lines and online/offline needs, but the main optimized JS file still weighs in at something like 17 thousand lines of code with newlines retained.

So once our UI logic is loaded, it’s time to kick-off loading the back-end.  And in order to avoid impacting the responsiveness of the UI both while it loads and when we’re doing steady-state processing, we run it in a DOM Worker.

Being Responsive: Workers and SharedWorkers

DOM Workers are background JS threads that lack access to the page’s DOM, communicating with their owning page via message passing with postMessage.  Normal workers are owned by a single page.  SharedWorkers can be accessed via multiple pages from the same document origin.

By doing this, we stay out of the way of the main thread.  This is getting less important as browser engines support Asynchronous Panning & Zooming or “APZ” with hardware-accelerated composition, tile-based rendering, and all that good stuff.  (Some might even call it magic.)

When Firefox OS started, we didn’t have APZ, so any main-thread logic had the serious potential to result in janky scrolling and the impossibility of rendering at 60 frames per second.  It’s a lot easier to get 60 frames-per-second now, but even asynchronous pan and zoom potentially has to wait on dispatching an event to the main thread to figure out if the user’s tap is going to be consumed by app logic and preventDefault called on it.  APZ does this because it needs to know whether it should start scrolling or not.

And speaking of 60 frames-per-second…

Being Fast: Virtual List Widgets

…the heart of a mail application is the message list.  The expected UX is to be able to fling your way through the entire list of what the email app knows about and see the messages there, just like you would on a native app.

This is admittedly one of the areas where native apps have it easier.  There are usually list widgets that explicitly have a contract that says they request data on an as-needed basis.  They potentially even include data bindings so you can just point them at a data-store.

But HTML doesn’t yet have a concept of instantiate-on-demand for the DOM, although it’s being discussed by Firefox layout engine developers.  For app purposes, the DOM is a scene graph.  An extremely capable scene graph that can handle huge documents, but there are footguns and it’s arguably better to err on the side of fewer DOM nodes.

So what the email app does is we create a scroll-region div and explicitly size it based on the number of messages in the mail folder we’re displaying.  We create and render enough message summary nodes to cover the current screen, 3 screens worth of messages in the direction we’re scrolling, and then we also retain up to 3 screens worth in the direction we scrolled from.  We also pre-fetch 2 more screens worth of messages from the database.  These constants were arrived at experimentally on prototype devices.

We listen to “scroll” events and issue database requests and move DOM nodes around and update them as the user scrolls.  For any potentially jarring or expensive transitions such as coordinate space changes from new messages being added above the current scroll position, we wait for scrolling to stop.

Nodes are absolutely positioned within the scroll area using their ‘top’ style but translation transforms also work.  We remove nodes from the DOM, then update their position and their state before re-appending them.  We do this because the browser APZ logic tries to be clever and figure out how to create an efficient series of layers so that it can pre-paint as much of the DOM as possible in graphic buffers, AKA layers, that can be efficiently composited by the GPU.  Its goal is that when the user is scrolling, or something is being animated, that it can just move the layers around the screen or adjust their opacity or other transforms without having to ask the layout engine to re-render portions of the DOM.

When our message elements are added to the DOM with an already-initialized absolute position, the APZ logic lumps them together as something it can paint in a single layer along with the other elements in the scrolling region.  But if we start moving them around while they’re still in the DOM, the layerization logic decides that they might want to independently move around more in the future and so each message item ends up in its own layer.  This slows things down.  But by removing them and re-adding them it sees them as new with static positions and decides that it can lump them all together in a single layer.  Really, we could just create new DOM nodes, but we produce slightly less garbage this way and in the event there’s a bug, it’s nicer to mess up with 30 DOM nodes displayed incorrectly rather than 3 million.

But as neat as the layerization stuff is to know about on its own, I really mention it to underscore 2 suggestions:

1, Use a library when possible.  Getting on and staying on APZ fast-paths is not trivial, especially across browser engines.  So it’s a very good idea to use a library rather than rolling your own.

2, Use developer tools.  APZ is tricky to reason about and even the developers who write the Async pan & zoom logic can be surprised by what happens in complex real-world situations.  And there ARE developer tools available that help you avoid needing to reason about this.  Firefox OS has easy on-device developer tools that can help diagnose what’s going on or at least help tell you whether you’re making things faster or slower:

– it’s got a frames-per-second overlay; you do need to scroll like mad to get the system to want to render 60 frames-per-second, but it makes it clear what the net result is

– it has paint flashing that overlays random colors every time it paints the DOM into a layer.  If the screen is flashing like a discotheque or has a lot of smeared rainbows, you know something’s wrong because the APZ logic is not able to to just reuse its layers.

– devtools can enable drawing cool colored borders around the layers APZ has created so you can see if layerization is doing something crazy

There’s also fancier and more complicated tools in Firefox and other browsers like Google Chrome to let you see what got painted, what the layer tree looks like, et cetera.

And that’s my spiel.

Links

The source code to Gaia can be found at https://github.com/mozilla-b2g/gaia

The email app in particular can be found at https://github.com/mozilla-b2g/gaia/tree/master/apps/email

(I also asked for questions here.)

It was brought to my attention recently by reputable sources that the recent announcement of increased usage in recent years produced an internal firestorm within Mozilla. Key figures raised alarm that some of the tech press had interpreted the blog post as a sign that Thunderbird was not, in fact, dead. As a result, they asked Thunderbird community members to make corrections to emphasize that Mozilla was trying to kill Thunderbird.

The primary fear, it seems, is that knowledge that the largest open-source email client was still receiving regular updates would impel its userbase to agitate for increased funding and maintenance of the client to help forestall potential threats to the open nature of email as well as to innovate in the space of providing usable and private communication channels. Such funding, however, would be an unaffordable luxury and would only distract Mozilla from its central goal of building developer productivity tooling. Persistent rumors that Mozilla would be willing to fund Thunderbird were it renamed Firefox Email were finally addressed with the comment, "such a renaming would violate our current policy that all projects be named Persona."