Array of unicode characters

[Krispin Kilmurray]

I'm working on a project in college and am confused about some MS documentation related to it. I'm trying to recreate an encryption key. The initial vectors for the key are the (password + salt). Ok sounds easy but the saltValue stored within my file is 32 bytes and not the specified 16 bytes in the description. The decoded hex string from base64 is 82a264407ea974a3cbccb4af5f3364fd. I know this is hex and each two characters is an ascii char but they are totally gobbledygook when converted to ascii. The output is ‚¢d@~©t£ËÌ´¯_3dý. Am I missing a step here?

The second conundrum is the password value. A string of unicode characters is specified. So say my password is "abc", this would be "\u0062\u0062\u0063". Would anyone know if this is the correct format to put them in as.

https://msdn.microsoft.com/en-us/library/dd925430(v=office.12).aspx

This is a link to the MS-offcrypto documentation which mentions the unicode characters.

[Stephan van Hulst]

Cryptography is done on binary data, so the salt and password identifiers used in the article refer to byte arrays. That means you first need to convert the password "abc" to a byte array using some sort of character encoding.

Concatenation (The '+' operator) refers to "pasting two byte arrays together" so they form one with the combined size.

The salt looks like gobbledygook when you try to encode it to ASCII, because it's random binary data, and never intended to be read as text.

If the specification says that salts must be 16 bytes, then obviously the file you have does not match the specification.

Honestly, the article is really poor because they specify the password to be an array of Unicode characters, while hashing is done on binary data. They don't specify whether the data is UTF-8, UTF-16 or something else still. They also use the identifier block without specifying what it means. I wouldn't be confident writing an algorithm based on this article alone. You may actually have to dive into the ECMA-376 specification.

[Krispin Kilmurray]

Thanks Stephan. The saltValue is in the XML field next to the salt size so it's definitely correct. Saly size is specified at 16 while the size of the base64 decoded string is 32 bytes. Like I said if this was hex it would make sense. I understand what you're saying about the unicode characters. In the documentation it specifically mentions that the password be broken down to unicode before being hashed. My SHA512 algorithm was getting a byte array from the string already so this threw me off.

[Stephan van Hulst]

What XML? Do you have an example?

Salts are only encoded to store them more easily in text files. That doesn't mean the length of the encoded string is also the length of the original salt value.

[Krispin Kilmurray]

<p:encryptedKey spinCount = "100000" saltSize = "16" blockSize = "16" keyBits = "256" hashSize = "64" cipherAlgorithm = "AES" cipherChaining = "ChainingModeCBC" hashAlgorithm = "SHA512" saltValue = "gqJkQH6pdKPLzLSvXzNk/Q==" encryptedVerifierHashInput = "jjyuf6DaqqL81TLEovnr0w==" encryptedVerifierHashValue = "VZUku9tIkOMd2HyoF+Qq1E9wjmgaPvujUaauptYHxbtVpvvG4tfhbYg2vDJJFxo8V73DlAcWESDQyLlrD8/y3A==" encryptedKeyValue = "u1BGSkPmuWr0gGoVTZiSV6QuwFWzwDAuCTcmu4CGBiw=" />
This is the xml from the end of the encrypted document. The saltSize is 16 and the saltValue which should conform to the saltSize, but when decoded from base64 to hex is 32 chars long.

[Stephan van Hulst]

I formatted your XML a little bit so it's more readable.

There are more inconsistencies with the article. The spin count is 100000 while the article specifies 50000. The cipher mode is CBC while the article specifies ECB (why I'm not sure, ECB is horrible). The hashing algorithm is SHA-512 while the article specifies SHA-1.

This file simply does not seem to conform to what the article prescribes.

[Krispin Kilmurray]

The article, although it says is updated, seems to be based on office 2007(I think). There are a few other things like the SpinCount=50,000 mentioned in the documentation but as per the XML you can see it's 100,000. There are a few other slight changes made for the latest version (2013), like the final iteration of the hash in the key generation previously used SHA(H(max),0) but instead of 0 there is a specific array of bytes (0xfe, 0xa7, 0xd2, 0x76, 0x3b, 0x4b, 0x9e, and 0x79.). Strangely enough these made their way into the article.

I'm trying to put together what sections are up to date and what are not, I'm relying on two references

http://www.cjmorgan.org/tech-blog/2015/1/8/default-encryption-settings-and-behaviors-for-onenote-2013-office-365

and a PDF by Yoshinori Takesako. Unfortunately both are missing slight but important points for password verification. I've also found this

http://source.dussan.org/raw/mirrors/poi.git/trunk/src/java/org/apache/poi/poifs/crypt/

The source code from the crypto library is either helpful or information overload when mixed with all other links.

[Krispin Kilmurray]

I have the section of my project related to this thread finished. It does everything I think it should but I can't seem to get a match in the output of the two values that confirm the correct password. If I were to post my code here would anyone mind having a look to see if they can spot a glaring problem in the code? There's a little bug in there somewhere I reckon but I can't find it.

import java.util.Arrays;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.Base64;
import java.lang.reflect.Field;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;  
import java.io.*;

import java.security.MessageDigest;
import java.security.DigestException;
import java.security.GeneralSecurityException;
import java.security.Key;

import javax.crypto.Cipher;
import javax.crypto.spec.SecretKeySpec;

import org.apache.poi.poifs.crypt.*;
import org.apache.poi.util.StringUtil;
import org.apache.poi.util.LittleEndianConsts;
import org.apache.poi.EncryptedDocumentException;
import org.apache.poi.poifs.crypt.CipherAlgorithm;

import java.util.Base64;

public class KeyGen
{
	public static void main(String args[])
	{
		/**********************************************************************
		*
		*		variables are extracted from docx in base64 format
		*
		*		then manually transformed to hex then byte array as per JtR
		*
		***********************************************************************/
		String salt = "gqJkQH6pdKPLzLSvXzNk/Q==";//initial values retrieved from word doc
		String HashInput ="jjyuf6DaqqL81TLEovnr0w==";
		String HashValue=
"VZUku9tIkOMd2HyoF+Qq1E9wjmgaPvujUaauptYHxbtVpvvG4tfhbYg2vDJJFxo8V73DlAcWESDQyLlrD8/y3A==";
//this was truncated in JtR but will only work at full length here
		
		String hexSalt="82a264407ea974a3cbccb4af5f3364fd";
		String hexHashInput="8e3cae7fa0daaaa2fcd532c4a2f9ebd3";
		String hexHashValue = "559524bbdb4890e31dd87ca817e42ad44f708e681a3efba351a6aea6d607c5bb";
		
		Hex h = new Hex();
		
		//hexSalt = h.hexToAscii(hexSalt);
		//hexHashInput = h.hexToAscii(hexHashInput);
		//hexHashValue = h.hexToAscii(hexHashValue);
		
		int spinCount = 100000;//loop count
		int blockSize = 16;
		int keySize = 32;//size in bytes = 256 bits
		String password = "abc";
		byte[] blockKeyInput = 
{(byte)0xfe,(byte)0xa7,(byte)0xd2,(byte)0x76,(byte)0x3b,(byte)0x4b
,(byte)0x9e,(byte)0x79};//block key for input given by microsoft
		byte[] blockKeyValue = 
{(byte)0xd7,(byte)0xaa,(byte)0x0f,(byte)0x6d,(byte)0x30,(byte)0x61,
(byte)0x34,(byte)0x4e};//block key for value given by microsoft
		
		byte[] decodedValueIn = Base64.getDecoder().decode(HashInput); //decode base64 values to byte array
		byte[] decodedValueVal = Base64.getDecoder().decode(HashValue);
		byte[] decodedSalt = Base64.getDecoder().decode(salt);
		
		//decodedValueVal = hexHashValue.getBytes();
		//decodedValueIn = hexHashInput.getBytes();
		//decodedSalt = hexSalt.getBytes();
		
		SHA512 sha = new SHA512();//create sha object
		
		/**********************************************************************
		*
		*		variables used in cryptofunctions class parameters
		*
		***********************************************************************/
		
		HashAlgorithm ha = HashAlgorithm.sha512;//define hash algorithm
		CipherAlgorithm ca = CipherAlgorithm.aes256;//define encryption algorithm
		ChainingMode cm = ChainingMode.cbc;//define chaining mode
		int cipherMode = 0;//unused as cryptofunctions class code changed
		boolean iteratorFirst = true;//order of i (i+hash)
		
		CryptoFunctions cf = new CryptoFunctions();//call class instead of using library (for troubleshooting)
		
		byte[] hash = cf.hashPassword(password,ha,decodedSalt,spinCount,iteratorFirst);//get hash from loop
		
		/**********************************************************************
		*
		*	generate an IV (initialization vector) for each key generated
		*
		***********************************************************************/

		
		byte[] IVIn = cf.generateIv(ha,decodedSalt,blockKeyInput,blockSize);//get IV from salt+blockKey
		byte[] IVVal = cf.generateIv(ha,decodedSalt,blockKeyValue,blockSize);
	
		/**********************************************************************
		*
		*		generate final byte arrays which are part of the key
		*
		***********************************************************************/	
		
		byte[] valueKey = cf.generateKey(hash,ha,blockKeyValue,keySize);
		byte[] inputKey = cf.generateKey(hash,ha,blockKeyInput,keySize);//as above
		SecretKeySpec inKey = new SecretKeySpec(inputKey,"AES");//generate key for cipher parameter
		SecretKeySpec valKey = new SecretKeySpec(valueKey,"AES");//generate second key
		
		Cipher cipherV = cf.getCipher(valKey,ca,cm,IVVal,cipherMode);//get cipher from combined values
		Cipher cipherI = cf.getCipher(inKey,ca,cm,IVIn,cipherMode);
		
		/**********************************************************************
		*
		*		final step decrypt two values then hash input
		*
		***********************************************************************/
		
		byte[] decryptedIn = {};//initialize byte arrays outside try/catch
		byte[] decryptedVal = {};
		byte[] hashedBytesIn = {};
		
		try
		{
			decryptedIn = cipherI.doFinal(decodedValueIn);//do final decrypts using aes256+key
			decryptedVal = cipherV.doFinal(decodedValueVal);
			hashedBytesIn = sha.hashBytes(decryptedIn);//input must be hashed again
		}
		catch(Exception ex){}
		
		/**********************************************************************
		*
		*		compare arrays to retrieve candidate key result
		*
		***********************************************************************/
		
		boolean Result = Arrays.equals(hashedBytesIn,decryptedVal);
		
		if(Result)//check for matching values
		{
			System.out.println("Success!");
		}
		else
		{
			System.out.println("");
			System.out.println("Fail!");
			System.out.println("");
		}
		System.out.println("decryptedVal"+Arrays.toString(decryptedVal));
		System.out.println("");
		System.out.println("hashedBytesIn"+Arrays.toString(hashedBytesIn));		
	}
	
}
	
	/****************************************POSSIBLE ERRORS********************************************
	*
	*			No padding found in docx so field is empty in CryptoFunctinos.getCipher()
	*
	*			Secret Key is used in cf but in this class its SecretKeySpec, can't instantiate SecretKey
	*
	****************************************************************************************************/

import java.security.MessageDigest;

public class SHA512 
{
            
    public static String convertByteToHex(byte data[])
    {
        StringBuffer hexData = new StringBuffer();
        for (int byteIndex = 0; byteIndex < data.length; byteIndex++)
            hexData.append(Integer.toString((data[byteIndex] & 0xff) + 0x100, 16).substring(1));
        
        return hexData.toString();
    }
    
    public static String hashText(String textToHash) throws Exception
    {
        final MessageDigest sha512 = MessageDigest.getInstance("SHA-512");
        sha512.update(textToHash.getBytes());
        
        return convertByteToHex(sha512.digest());
    }
	
	//returns a byte array from a byte array
	public static byte[] hashBytes(byte[] bytesToHash) throws Exception
    {
        final MessageDigest sha512 = MessageDigest.getInstance("SHA-512");
		sha512.update(bytesToHash);
        
        return sha512.digest();
    }
}

public class Hex
{
	public static String bytesToHex(byte[] bytes) 
	{
		StringBuilder sb = new StringBuilder();
		for (byte b : bytes) 
		{
			sb.append(String.format("%02X ", b));
		}
		String hex = new String(sb);
		hex = hex.replaceAll("\\s+","");
		return hex;
	}
	
	public String hexToAscii(String hexValue)
	{
		StringBuilder output = new StringBuilder("");
		for (int i = 0; i < hexValue.length(); i += 2)
		{
			String str = hexValue.substring(i, i + 2);
			output.append((char) Integer.parseInt(str, 16));
		}
		return output.toString();
	}
	    
    
}

[Stephan van Hulst]

Krispin, I deleted the Apache CryptoFunctions class from your post. Its source and documentation are readily available online.

I don't know the specification and don't know exactly what you're trying to prove in your main class. You have a bunch of magical constants that you pulled from somewhere, and it's not clear what you're doing with them or why.

Furthermore, you're instantiating instances of CryptoFunctions, SHA512 and Hex, while these are utility classes and you can call methods on the classes directly.

You're also not supposed to pass 0 as the cipher mode, but Cipher.DECRYPT_MODE.

[Krispin Kilmurray]

Sorry, I did a poor job of explaining anything. The constants were taken from my encrypted documented that I'm trying to open by verifying the hard coded password. The salt, hashValue and hashInput are decoded from base64 then used to create a key with a function from the crypto class. The decoded salt and blockKey are use to create an IV key. Both these byte arrays (IV and key) are used to create a cipher which in turn is used to decrypt the decoded hashInput and hashValue values. The input is then hashed and they are compared. If the password was correct they should match.

The Hex and SHA512 classes were my own which is why I included them. I instantiated the cryptoClass as I had initially found the code online before discovering apache.poi library whivh they are in. The class asked for an int in the getCipher parameter which corresponds to Cipher.DECRYPT_MODE or Cipher.ENCRYPT_MODE. I didn't know what int corresponds to what so I edited the getCipher method in the cryptoClass to just use Cipher.DECRYPT_MODE regardless of what int was passed in.