Yes, you have to read the bytes and make sure first few bytes are the JPEG SOI format identifier. You then need to make sure the next few bytes is either the JFIF marker or you likely have an Exif. For JPEG, you need to read the first two bytes, which should be
0xFF 0xD8 (this is the 'Start of Image' or SOI marker required by JPEG). For JFIF (the JPEG File Interchange Format - the format defined by the mime type for .jpg files transferred around the web), the next 9 byte should be:
0xFF 0xE0 <two bytes> 0x4A 0x46 0x49 0x46 0x00 (This is the JFIF APP0 header marker, required by JFIF just after the SOI marker. Those last 5 bytes are the
string "JFIF" followed by 0). If the file has the JPEG SOI marker but not the JFIF APP0 marker then it is likely an Exif format, used by Photoshop and some cameras I think. It is more like a TIFF but can have a jpg or jpeg extension, and has the metadata stored differently.