Hash Verification (MD5/SHA)

The Undercode Testing forensics reference describes the foundational role: “In digital forensics, hashing is a fundamental practice to ensure data integrity. When you create forensic images or handle critical files, generating hash values (like MD5, SHA-1, or SHA-256) is essential. These hashes act as digital fingerprints, allowing you to verify that files remain unchanged during transfers or storage.”¹

The fundamental concept

Hash verification rests on the mathematical properties of cryptographic hash functions:

• Variable input, fixed output: the function accepts any size input (1 byte to terabytes) and produces a fixed-size output regardless.
• Deterministic: the same input always produces the same output; there is no randomness.
• Avalanche effect: a single bit change in the input produces a completely different output; small changes do not produce small differences.
• One-way: the hash cannot be reversed to recover the original input.
• Collision-resistant (in theory): finding two different inputs that produce the same hash should be computationally infeasible.
• Practical purpose: if two pieces of data produce the same hash, they are with extremely high probability identical.

The verification process

Hash verification follows a simple but rigorous three-step process:

1. Calculate source hash: compute the hash of the data at its origin.
2. Calculate destination hash: compute the hash of the data after transfer, copy, processing, or storage.
3. Compare: match the two hash values character-by-character.

If the hashes match exactly, the data is verified as identical. If they differ even by one character, the data has been modified somewhere between source and destination. There is no partial match; hashes either match completely or they do not.

The “digital fingerprint” analogy

Hash values function similarly to fingerprints in identification:

• Uniqueness: each unique data input produces a unique hash (with extremely high probability).
• Compactness: the hash is much smaller than the data it represents (32 hex characters for any-size input with MD5).
• Identification capability: given a hash, one can determine if a candidate file matches without revealing the file content.
• Tamper evidence: any modification produces an entirely different fingerprint, making tampering detectable.
• Public verifiability: hashes can be shared publicly without revealing the data they represent.

Why hashing matters in recovery contexts

The MCSI Library forensic reference describes the importance: “During a forensic investigation, it is of utmost importance to ensure that the integrity of the acquired evidence remains the same throughout the investigation. The state of evidence must remain the same from the moment it was acquired till the moment the investigation is complete. The best way to ensure that is by using hashing.”² Specific recovery scenarios requiring hash verification:

• Imaging a damaged drive and confirming the image is byte-equivalent to the source.
• Transferring a forensic image between storage devices without corruption.
• Verifying that recovery tools have not modified the source drive.
• Confirming that a recovered file matches the original (when source hash is available).
• Validating downloaded recovery software (ISO images, tools) against published hashes.
• Detecting bit rot or storage corruption in long-term archives.

The three forensic principles

The Undercode Testing reference identifies three primary forensic functions of hash verification:

• Data Integrity: detect accidental or malicious alterations to evidence files between acquisition and analysis.
• Chain of Custody: prove that files have not been tampered with at any point in the evidence handling process.
• Legal Admissibility: courts rely on hash verification for evidence authenticity; matching hashes establish that analyzed evidence is the same as acquired evidence.

Several cryptographic hash algorithms are used for integrity verification, each with different output sizes, security properties, and performance characteristics. The choice depends on the threat model and compatibility requirements.

Hash algorithm comparison

Algorithm	Year	Output Size	Hex Chars	Status
MD5	1991	128 bits	32	Broken (collisions); used for non-security integrity
SHA-1	1995	160 bits	40	Broken (SHAttered 2017); deprecated for security
SHA-224	2001	224 bits	56	SHA-2 family; secure but rarely used
SHA-256	2001	256 bits	64	Current standard; secure, widely supported
SHA-384	2001	384 bits	96	SHA-2 family; used in TLS, code signing
SHA-512	2001	512 bits	128	SHA-2 family; faster on 64-bit systems
SHA-3 / Keccak	2015	224/256/384/512 bits	56-128	NIST alternative; different internal construction
BLAKE2 / BLAKE3	2012/2020	Variable	Variable	Modern fast hashes; used in some systems

MD5 (1991, Ronald Rivest)

MD5 was designed by Ronald Rivest at MIT and published in 1991 as RFC 1321:

• Output: 128 bits, displayed as 32 hexadecimal characters.
• Speed: fast on all hardware; one of the most-optimized hash algorithms.
• Collision attacks: Wang and Yu demonstrated collisions in 2004; chosen-prefix collisions in 2007.
• Security status: cryptographically broken; not suitable for digital signatures, password hashing, or any adversarial scenario.
• Practical use: still widely used for non-security integrity checks (file copy verification, download corruption detection).
• Forensic role: retained for backward compatibility with legacy evidence; secondary verification alongside SHA-256.

SHA-1 (1995, NSA)

SHA-1 (Secure Hash Algorithm 1) was published by NIST in 1995 as FIPS 180-1:

• Output: 160 bits, displayed as 40 hexadecimal characters.
• Speed: slightly slower than MD5; widely supported in legacy systems.
• Collision attacks: theoretical attacks demonstrated 2005-2015; Google’s SHAttered demonstrated practical collision in February 2017.
• Security status: deprecated by NIST in 2011; no longer accepted for digital signatures.
• Practical use: still found in legacy systems, some TLS certificates, Git commit hashes.
• Forensic role: phasing out; SHA-256 is the current standard replacement.

SHA-2 family (2001, NSA)

The SHA-2 family was published in 2001 as FIPS 180-2 and includes SHA-224, SHA-256, SHA-384, and SHA-512:

• Variants: SHA-224 (224 bits), SHA-256 (256 bits), SHA-384 (384 bits), SHA-512 (512 bits).
• Internal construction: Merkle-Damgård construction with Davies-Meyer compression function (similar to SHA-1 but expanded).
• Word sizes: SHA-256 uses 32-bit words; SHA-512 uses 64-bit words; SHA-512 is faster on 64-bit systems.
• Security status: no known practical collision attacks; current standard for security-relevant integrity.
• Practical use: TLS certificates, blockchain (Bitcoin uses double SHA-256), code signing, digital signatures, forensic integrity.
• Forensic role: SHA-256 is the recommended primary algorithm; SHA-512 for very large files where its speed advantage matters.

SHA-3 / Keccak (2015, NIST competition)

SHA-3 was selected through a NIST competition (2007-2012) and standardized in 2015:

• Origin: Keccak algorithm by Bertoni, Daemen, Peeters, and Van Assche; selected from 64 submissions.
• Internal construction: sponge construction (different from SHA-2’s Merkle-Damgård).
• Variants: SHA3-224, SHA3-256, SHA3-384, SHA3-512 (matching SHA-2 output sizes for drop-in replacement).
• Status: alternative to SHA-2 rather than replacement; both are NIST-approved.
• Adoption: slower than SHA-2 on most current hardware; less widely deployed.
• Forensic role: SHA-2 (specifically SHA-256) remains the practical standard; SHA-3 available for environments requiring NIST-approved alternative.

CRC32 vs cryptographic hashes

CRC32 is sometimes confused with cryptographic hashes but serves a different purpose:

• CRC32: Cyclic Redundancy Check producing 32-bit output; designed for accidental error detection.
• Use case: network packets, file format internal checksums (ZIP CRC, PNG CRC, Ethernet frames).
• Speed: very fast; often hardware-accelerated.
• Security: not cryptographically strong; deliberate collisions are trivial to find.
• When CRC32 is appropriate: error detection where adversarial collision is not a concern.
• When CRC32 is inappropriate: any context requiring tamper detection or evidence integrity.

Fuzzy hashes for similarity detection

Standard cryptographic hashes are binary: data is identical or it is not. Fuzzy hashes detect similarity rather than identity:

• ssdeep: Context Triggered Piecewise Hashing (CTPH); produces hashes that can be compared for similarity score.
• TLSH: Trend Micro Locality Sensitive Hash; designed for malware similarity detection.
• Use case: identifying files that have been slightly modified (malware variants, document edits).
• Forensic role: finding related files across an investigation; clustering similar samples.
• Limitation: not cryptographically secure; similarity can be intentionally manipulated.

Hash verification tools span operating systems, command-line utilities, and dedicated forensic software. The right tool depends on the platform, the data being verified, and the workflow integration requirements.

Linux command-line tools

Linux distributions include hash tools in GNU coreutils:

• md5sum: computes MD5 hashes; available on every Linux distribution.
• sha1sum: computes SHA-1 hashes; same syntax as md5sum.
• sha224sum / sha256sum / sha384sum / sha512sum: SHA-2 family with corresponding output sizes.
• Common syntax: sha256sum filename outputs hash and filename; sha256sum -c hashfile verifies files against hash list.
• Pipe usage: dd if=/dev/sda bs=4M | sha256sum hashes a drive without creating an image first.
• Batch operation: find . -type f -exec sha256sum {} \; > hashes.txt hashes all files in a directory tree.

macOS hash tools

macOS includes BSD-style hash tools with slightly different conventions from GNU:

• md5: BSD-style MD5 utility; output format differs from GNU md5sum (hash on right, filename in parentheses).
• shasum: Perl-based utility supporting -a 1, -a 256, -a 512 for algorithm selection.
• openssl dgst: openssl dgst -sha256 filename works as a cross-platform alternative.
• Homebrew installation: brew install coreutils provides Linux-style md5sum, sha256sum (prefixed with “g”: gmd5sum, gsha256sum).
• Compatibility note: hash values are identical across implementations; only output format differs.

Windows hash tools

Windows provides several hash verification options:

• CertUtil: built-in command line tool; certutil -hashfile filename SHA256 outputs hash.
• Get-FileHash (PowerShell): Get-FileHash filename -Algorithm SHA256 for object output; supports MD5, SHA-1, SHA-256, SHA-384, SHA-512.
• HashCheck Shell Extension: right-click Properties dialog integration showing multiple hashes simultaneously.
• HashTab: third-party Properties tab adding hash display.
• 7-Zip integration: right-click context menu can compute hashes via 7-Zip’s CRC SHA submenu.
• PowerShell scripting: Get-FileHash output can be piped to compare-object for batch verification.

Forensic tools with integrated hashing

Professional forensic tools build hashing into the imaging and analysis workflow:

• FTK Imager (Exterro, free): computes MD5 and SHA-1 hashes during imaging; can verify existing images.
• Guymager (Linux GUI): open-source forensic imager with hash on the fly during acquisition.
• dc3dd: dd variant by DoD Cyber Crime Center; dc3dd if=/dev/sda hash=sha256 hashlog=hashes.txt of=disk.img hashes during imaging.
• dcfldd: dd variant by DoD Computer Forensics Lab; similar to dc3dd with hash and verification features.
• OSForensics: commercial tool with volume/disk/image hash calculation feature.
• EnCase: commercial forensic suite; hashing built into all evidence handling operations.
• Sleuth Kit: command-line forensic tools; integration with hashing utilities for evidence verification.

File transfer verification tools

Specialized tools verify hashes during file copy operations to detect transfer corruption:

• TeraCopy (Windows): replaces Windows Explorer copy; verifies hashes after copy to detect corruption.
• FastCopy (Windows): high-speed file copier with optional hash verification.
• rsync (Linux/macOS): uses block-level checksums during transfer; rsync -c forces full checksum comparison.
• cp -v with separate verification: standard Linux copy followed by md5sum/sha256sum comparison.
• BorgBackup, Restic: backup tools with built-in deduplication and integrity verification via SHA-256.

Hash database lookups

Several public databases enable hash-based identification of known files:

• NSRL (National Software Reference Library): NIST database of hashes for legitimate software; used to filter known-good files from forensic analysis.
• VirusTotal: hash lookup against malware sample database; identifies known threats by hash.
• Hashes.org / cracking communities: reverse hash lookups for password cracking (separate use case from forensic verification).
• Threat intelligence feeds: commercial and open-source feeds of known-bad hashes for IOC matching.
• Internal hash inventories: organizations maintain hash lists of approved software for allowlist verification.

Hash verification is integrated into multiple stages of the recovery workflow, from initial drive imaging through final evidence handling.

The pre-acquisition / post-acquisition workflow

The Quora forensic reference describes the standard procedure: “Hash the original drive (source) prior to acquisition, if possible, using a write-blocker and a trusted hashing tool (e.g., md5sum, sha1sum, SHA-256 via hashlib, or forensic tools like FTK Imager, Guymager, dc3dd). Hash the forensic image (target) immediately after acquisition using the same hashing algorithm and tool.”³ The complete forensic workflow:

1. Connect via write-blocker: hardware or software write-blocker prevents accidental modifications to source.
2. Pre-acquisition hash: hash the source drive with SHA-256 (and optionally MD5).
3. Document source hash: record value, timestamp, examiner ID, case number, drive serial number.
4. Image with hashing imager: use dd/dc3dd/FTK Imager/Guymager with hash integration.
5. Post-acquisition hash: hash the resulting image immediately after creation.
6. Compare: verify image hash matches source hash exactly.
7. Document image hash: record alongside source hash with same metadata.
8. Periodic re-verification: re-hash image during analysis to detect inadvertent modifications.
9. Final verification: hash image one more time at end of analysis as final integrity check.

Why hash both source and image

Hashing both source and image creates a verifiable chain:

• Source hash establishes baseline integrity at the moment of acquisition.
• Image hash demonstrates the imaging process produced an exact copy.
• Matching hashes prove no modifications occurred during acquisition.
• If hashes differ, the imaging is invalid; analysis cannot proceed.
• The matched hash pair becomes part of the chain of custody documentation.
• Subsequent analysis works on the image; the source can be returned to evidence storage.

Hashing during dd or ddrescue imaging

The standard dd command can be combined with hashing in several ways:

• Hash after imaging: sudo dd if=/dev/sda of=disk.img bs=4M; sha256sum /dev/sda > source.hash; sha256sum disk.img > image.hash
• Concurrent hashing via tee: sudo dd if=/dev/sda bs=4M | tee disk.img | sha256sum > image.hash
• dc3dd with integrated hashing: dc3dd if=/dev/sda hash=sha256 hashlog=hashes.txt of=disk.img bs=4M
• dcfldd similar: dcfldd if=/dev/sda hash=sha256 hashlog=hash.log of=disk.img bs=4M
• Verification after restore: sha256sum -c hash.log verifies restored images against logged hashes.

Verifying recovery tool output

After running recovery tools, hashes verify which files were correctly recovered:

• If source hash is known: hash recovered file and compare; matching hashes confirm exact recovery.
• If multiple recovered fragments: hash each fragment and compare to source; identify which are complete recoveries vs partial.
• For batch verification: generate hashes for all recovered files; compare against source hash database where available.
• For forensic context: hashes of recovered files become part of chain of custody documentation.
• NSRL filtering: hash recovered files against NSRL database; remove known-good system files from analysis.

ISO and software distribution verification

Linux distributions, recovery tools, and forensic suites publish hashes for download verification:

• Distribution sites publish SHA256SUMS files alongside ISO downloads.
• After downloading, users compute SHA-256 of the downloaded file.
• Computed hash is compared to the published value.
• Matching hash confirms the download is intact and authentic (assuming the published hash itself is trusted).
• GPG signatures often accompany hash files for additional authentication.
• Tools like sha256sum -c SHA256SUMS automate batch verification of multiple downloads.

Long-term archive integrity

Hash verification protects against bit rot in long-term storage:

• Generate hashes when files are archived; store hashes in separate manifest.
• Periodically re-hash archived files (annually for example).
• Compare current hashes to archived manifest.
• Mismatched hashes indicate bit rot or storage corruption.
• ZFS and Btrfs perform this automatically at the filesystem level using internal block-level hashing.
• Tape archives and cold storage particularly benefit from periodic hash verification.

Hash verification is powerful but not a complete security solution. Understanding its limitations clarifies when it provides protection and when additional measures are required.

What hash verification does not provide

• Confidentiality: hashes do not encrypt data; the original content remains visible to anyone with access to it.
• Authenticity (without trusted reference): if an attacker controls both the file and the published hash, hash verification provides no protection.
• Non-repudiation: hash matching alone does not prove who created or sent the file.
• Recovery from damage: when hashes do not match, hashing reveals the problem but cannot fix it.
• Protection against insider threats: someone with access to both data and hash records can re-hash modified files.

The collision attack threat model

Collision attacks affect security-relevant uses of broken hash algorithms:

• Identical-prefix collision: attacker finds two arbitrary inputs with same hash; works against MD5 and SHA-1.
• Chosen-prefix collision: attacker can control the prefix of both colliding inputs; works against MD5 (much harder against SHA-1, no practical attack on SHA-2).
• Practical impact: attacker can create two documents with same MD5 hash, sign one, and substitute the other.
• Forensic relevance: for evidence integrity (no adversarial signature substitution attempted), MD5 collisions are typically not a real threat; for code signing or authentication, they are critical.
• Mitigation: use SHA-256 or SHA-512 for any security-relevant integrity verification.

HMAC for authenticated integrity

HMAC (Hash-based Message Authentication Code) extends hash functions with a secret key:

• Combines a hash function with a shared secret key.
• Provides both integrity (data hasn’t changed) and authenticity (sender knows the key).
• Prevents collision-substitution attacks because attacker must know the key.
• Common forms: HMAC-MD5, HMAC-SHA1, HMAC-SHA256, HMAC-SHA512.
• HMAC-MD5 remains secure for authentication despite MD5 collision attacks because the key prevents collision exploitation.
• Standard tools: openssl can compute HMAC; programming libraries include HMAC functions.

Best practices for forensic hash verification

• Use SHA-256 as primary: current standard for security-relevant integrity.
• Compute MD5 secondarily: for backward compatibility with legacy evidence.
• Hash before and after every operation: document integrity at each stage.
• Use write-blockers during source hashing: prevent accidental modifications.
• Document everything: hash values, timestamps, examiner ID, case number, drive identifiers.
• Store hashes separately: hash logs in different location from evidence files.
• Verify on every transfer: any time evidence moves between storage, re-verify hashes.
• Use forensically-validated tools: FTK Imager, dc3dd, EnCase have established legal precedent.
• Never modify hash records: any correction must be a new entry, not a modification to existing entry.
• Plan for algorithm transitions: maintain capability to verify against legacy MD5/SHA-1 hashes.

Common hash verification mistakes

• Hashing only the destination: without source hash, there’s nothing to compare to.
• Using different algorithms for source and destination: SHA-256 of source vs MD5 of image cannot be compared.
• Comparing partial hashes: first 8 characters matching means almost nothing; full match required.
• Trusting hash from same source as data: if attacker controls both, hash verification provides no protection.
• Ignoring case: hash comparison is technically case-insensitive in hex but tool output may vary.
• Whitespace in hash files: stray spaces or line endings can cause false mismatches.
• Hashing mounted volumes: filesystem activity changes timestamps; hash unmounted source for stable values.