A blog post on how I was able to identify unknown master passwords stored in the memory of the Bitwarden web extension and desktop client, after a vault has been locked. I also cover the decisions made for developing a proof of concept to automate the process of extracting potential passwords.

TL;DR

It is possible to identify unknown Bitwarden master passwords in memory, even after a vault is locked. We developed a proof of concept tool, called BW-dump, that works on Windows platforms. It was tested with Bitwarden desktop app version (2023.2.0).

The video belows shows a quick demo (getting the master password from the Bitwarden web browser extension)

Update (17/08/2023): This particular issue has been assigned CVE-2023-38840

Update (20/07/2023): A patch was released on GitHub (5813) which fixes the vulnerability. Bitwarden Desktop version 2023.7.0 and below are vulnerable.

Background

A few years ago, a GitHub issue relating to the Bitwarden client application (Erase Master Password in memory after login) was reported. This issue references an article by a German IT magazine who had tested different password managers.

Some of the tests involved checking whether master passwords are leaked before and after locking, and as a user myself I was curious, so I tried the tests myself.

I started off by focusing on the Bitwarden web browser extension (both for Chrome and MSEdge), where a initial proof of concept was developed. However, after an update in either Chrome and/or Bitwarden fixed the issue, so I was no longer able to identify the password in memory. I then moved onto the Bitwarden Desktop App, where the issue is still present.

How to find a known password?

To locate a known master password in memory, one can use a tool like Process Hacker. Process Hacker allows users to inspect every process that is currently running on their system. It also provides access to process memory space, including a neat string search feature.

Chromium web extensions

When looking at web browser processes (Chrome or MSEdge) with Process Hacker, you’ll notice there are several child processes. Each process has its own functionality and purpose. This includes web browser extensions, each of which will have a unique process:

To identify processes associated to web extensions you can inspect the Command Line options of each child process, and search for the string --extension-process. On Chrome browsers you will have to keep in mind that there are a few default extensions already installed.

Here is an example of an MSEdge.exe extension process:

For completeness, the full command line options were:

"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe" --type=renderer --extension-process 
 --disable-gpu-compositing --lang=en-GB --js-flags=--ms-user-locale= --device-scale-factor=1 
 --num-raster-threads=1 --renderer-client-id=19 --time-ticks-at-unix-epoch=-1677594276672537 
 --launch-time-ticks=198758688 --mojo-platform-channel-handle=3960 
 --field-trial-handle=2072,i,6587025175849962187,10098860286187380106,131072 /prefetch:1

I couldn’t find a way to identity a browser extension associated with a specific process by only using the Command Line option, so I had to checked them all manually. There is probably a better way of doing this.

Anyway, now that an extension process has been identified, we can go ahead and start looking for strings in memory. To do this, first go to process properties, and select the memory tab. Next, click on strings and select the default settings (unless your test password is 8 characters) type out your plaintext master password as the search query.

Where are web extensions stored on a Windows system?

The Bitwarden extension on MSEdge is installed in folder:

%LocalAppData%\Microsoft\Edge\User Data\Default\Local Extension Settings\jbkfoedolllekgbhcbcoahefnbanhhlh

Depending on how many profiles you have in your web browser, instead of Default, it could be Profile X, where X is the profile number. Also keep in mind that each web browser has a unique ID for each extension. For example the Bitwarden web extension has the following IDs:

• For MSEdge - jbkfoedolllekgbhcbcoahefnbanhhlh
• For Chrome - nngceckbapebfimnlniiiahkandclblb

Desktop app

The Bitwarden desktop application is built using Electron, which is similar to Chromium browsers. It too creates child processes for individual program features. Instead of --extension-process in the Command Line options, search for --no-zygote. This is the child process where most sensitive memory data is cached, including the master password.

Here is an example of the Bitwarden.exe Command Line options:

"C:\Users\Tester\AppData\Local\Programs\Bitwarden\Bitwarden.exe" --type=renderer 
 --user-data-dir="C:\Users\Tester\AppData\Roaming\Bitwarden" 
 --app-path="C:\Users\Tester\AppData\Local\Programs\Bitwarden\resources\app.asar" 
 --no-sandbox --no-zygote --first-renderer-process --lang=en-GB --device-scale-factor=1 
 --num-raster-threads=1 --renderer-client-id=4 --time-ticks-at-unix-epoch=-1677594276676209 
 --launch-time-ticks=5708174183 --mojo-platform-channel-handle=2428 
 --field-trial-handle=1892,i,2828961307611671252,14266672126368596536,131072 
 --disable-features=SpareRendererForSitePerProcess,WinRetrieveSuggestionsOnlyOnDemand /prefetch:1

And here is an example of the master password found in plaintext in the memory of that child process:

How to find an unknown password?

Searching for a known password is quite trivial, but searching for a password that you don’t know is much harder. Especially if you don’t know the length, or where to look. It’s like looking for a needle in a field of haystacks (memory regions).

Looking for patterns

Note: The following section is based on the Bitwarden MSEdge Chromium extension, which has since been patched. However, the same process was applied to the Bitwarden desktop client, which is still vulnerable.

The best place to start is looking at the bytes around a known master password. For this, I used Process Hacker to generate several process memory dumps, which I then searched through using 010 Editor.

Here is an example of a master password found in a memory dump:

Each time I took a snapshot, I restarted the process, unlocked the vault, and then locked it again. I observed that a few hex bytes remained consistent, even after a system reboot. You will notice in the above screenshot that at offset 0x0140 (different for new processes), the following bytes are shown, which I refer to as the password prefix pattern:

       1  2  3  4  5  6  7  8  9 10 11 12
Byte: 04 00 00 00 13 00 00 00 01 B1 6C AB XX XX XX ... 00 00

Here, XX represents an actual master password character byte. Apart from the null byte terminators, the other bytes were unknown. To try figure out what the other bytes represent I did a few tests.

What happens to these bytes when you change your master password?

Changing a master password by adding an extra character changes the pattern in two ways:

         1  2  3  4  5  6  7  8  9 10 11 12
Before: 04 00 00 00 13 00 00 00 01 B1 6C AB XX XX XX ... 00 00
After:  04 00 00 00 14 00 00 00 01 C3 5E 54 XX XX XX ... 00 00

• The 5th byte looks like it represents the master password length, as it increments by one, to 0x14 . The password only the be$t bacons contains 20 characters (including the spaces), which is 14 in hex.
• The 10th, 11th, and 12th bytes 0xC35E54 are also changed, although what they are for isn’t obvious.

What happens when you rotate the encryption keys?

Bitwarden allows users to rotate account encryption keys, a useful feature when you think your account has been compromised. This option is only available while changing the master password, and is is not enabled by default:

Rotating your account’s encryption key generates a new encryption key that is used to re-encrypt all Vault data. You should consider rotating your encryption key if your account has been compromised such in a way that someone has obtained your encryption key.

Bitwarden Docs

I opted to change the password back to my previous password, only the be$t bacon, so it’s easier to follow. And then I used the option to rotate encryption keys. This produced the following password memory entry:

         1  2  3  4  5  6  7  8  9 10 11 12 
Before: 04 00 00 00 14 00 00 00 01 C3 5E 54 XX XX XX XX XX XX ...
After:  04 00 00 00 13 00 00 00 01 B1 6C AB XX XX XX XX XX XX ...

You’ll notice that the 5th, 10th, 11th, and 12th bytes have been changed, and are back to the same as the original screenshot. This means these last three bytes (just before the password) are somehow related to the password plaintext itself, and not the encryption key, which is what I initially thought.

What happens to these bytes when you use a different Windows version?

In some cases, when using a different Windows version like 10 or 11, the first byte changes. But I found it not to be consistent enough, as it might be 05, 02, or 04 regardless of the version.

         1  2  3  4  5  6  7  8  9 10 11 12 
Before: 04 00 00 00 14 00 00 00 01 C3 5E 54 XX XX XX XX XX XX ...
After:  05 00 00 00 13 00 00 00 01 B1 6C AB XX XX XX XX XX XX ...

I tried out other tests but didn’t get much further.

With the information learned so far we known certain bytes remain static. We also know that some are dynamic and depend on the master password. Nevertheless, we can use this data to create a search pattern using a simple regular expression.

To quickly summarise the above I will use the following search template (in 010Editor syntax) as the password prefix pattern:

        1  2  3  4  5  6  7  8  9 10 11 12
 Byte: 04 00 00 00 ?? 00 00 00 01 ?? ?? ?? ... 00 00

• 1st byte 04 - possibly Windows version specific.
• 2nd, 3rd, 4th bytes 00 00 00 - some kind of separator.
• 5th byte ?? is master password length - must be 0x08 or above since Bitwarden registration requires passwords to have a minimum of 8 characters.
• 6th, 7th, and 8th bytes 00 00 00 - another separator.
• 9th 01 - possibly a platform indicator i.e. Windows = 01 and Linux = 03 (not confirmed).
• 10th, 11th, 12th bytes ?? ?? ?? - relate to the master password somehow.
• 13th byte is the start of the actual master password.

This is equivalent to a basic regular expression such as:

(\x04|\x05|\x02)\x00\x00\x00[^\x00]\x00\x00\x00\x01[^\x00][^\x00][^\x00]

Now we have a search pattern, we need to find the right memory region.

Note: The regex search pattern for Bitwarden Desktop app proof of concept is completely different \x01(?:[^\x00]{3})(?:[(\x20-\x7E)]{8,}), which results in a large list of potential passwords strings. This could be improved by filtering out known static strings.

Memory regions

When reviewing a process’s memory with Process Hacker, you’ll notice there are dozens of memory regions, several megabytes worth. Many of which will likely contain results matching the above regex pattern, leading to multiple false positives.

To help filter out false positives, I started to focus on a specific memory region where the password is located, and looked for more patterns. I noticed that each time a new process was started, it contained a static string of bytes at the beginning of the region. This can be seen in the screenshot below:

You can see what appears to be a CSS (Cascading Style Sheet) resource, along with other attributes. It is encoded with UTF-16 Big Endian, typically seen on Windows platforms. To decode this value one can use CyberChef with the options From Hex and Decode Text (UTF-16 BE).

Do that and we get the following:

@font-face {
  font-family: "Open Sans";
  font-style: italic;
  font-weight: 300;
  font-display: auto;
  src: url(../popup/fonts/Open_Sans-italic-300.woff) format("woff");
  unicode-range: U+0-10FFFF;
}

I now had a static string which I knew existed in the same memory region as the master password, and a search pattern to use. To make searching more efficient, I applied another filter to only include specific memory regions.

To inspect memory regions types and protections you can use Process Hacker, or the VMMap sysinternals tool by Microsoft, which is shown below.

As shown, the selected memory region (where the master password is found) has a memory type of Private Data with Read/Write Protection. The size (188 KB) varied between 150 KB - 260 KB, which was also used as a search criterion.

Now that I had good understanding of what to search for, I started to think about building a tool to try automate the process.

Developing a tool

I decided to take this opportunity to get more familiar with working with Windows API calls and Golang. My goal was to simulate the same actions performed with Process Hacker, but with a few key benefits.

These are:

• Doesn’t require Admin privileges - uses the permissions of the logged-in user.
• Automatically find the right processes.
• Choose specific memory regions rather than scanning ALL regions.
• Search for patterns in those memory regions.
• Filter out possible matches.

I’m not going to go into any real details on the programming side in this blog - as it’s already too long. But I would like to mention the resources I used for development.

Windows APIs

Microsoft has an extensive list of Windows API functions available for developing Windows programs. The Windows App Development online document is probably the best resource for low-level Windows development. For anything memory related, I recommend reviewing the memoryapi.h section found here.

Offensive Go libraries

There are dozens of offensive security Go libraries available. I opted to use the following because they worked well during initial testing, and were simple for me to understand.

• Getting a list of processes (https://github.com/shirou/gopsutil)
• Reading process memory (https://github.com/0xrawsec/golang-win32)

To help get a better understanding of Windows API bindings for Golang, I used the Black Hat Go book by Tom Steele, Chris Patten, and Dan Kottmann, which is available on No Starch Press. I also used a few examples they provided as a template for my tool.

BW-dump

BW-dump is a Windows based tool that is capable of extracting master vault passwords from a locked Bitwarden vault. The tool is written in Golang, and makes use of Windows API functions. For the tool to work, you need two requirements:

• A supported process running.
• The vault must have been unlocked at least once.

Here is a screenshot of extracting a master password from Bitwarden Desktop (2023.2.0):

You’ll notice results may include the password several times, sometimes with extra characters. This was because it was much harder to figure out the length of the master password to know where the string ends. The earlier version of the tool (which worked on the web browser extension) was much more reliable.

Download

The proof of concept tool can be downloaded from my GitHub, which also includes a release binary. BW-dump version v1.0.2 only works with the Bitwarden desktop client due to a patch released a few months for the web extension.

Note: Since the tool reads the memory of other processes, and I didn’t bother with applying any sort of obfuscation techniques, Windows Defender will rightly flag the compiled binary as malicious.

Conclusion

Reviewing closed GitHub issues (marked as stale) can lead to interesting bits of research. This blog demonstrates a relatively simple approach to identifying patterns inside process memory, which can be used to find unknown master vault passwords. We developed a proof of concept tool to automate the process.

• Security Advisory - Vulnerabilities with Crocus Iris App
• Parental Protections Pose Problems for PlusNet Pixel Patching
• Vulnerability disclosure: how many top-million sites use security.txt?

About Naz Markuta

Naz is an OSCP-certified technical cyber security professional with experience in security and penetration testing, and vulnerability research. Naz has found credited vulnerabilities in hardware devices, mobile and web applications. At Red Maple he helps deliver our cyber security consulting services, as well as research and development.