How deduplication works using File Hashes?

💻 Environment/Context

• Upload
• Collections
• Storage

❓ Issue/Question

• How does merging duplicates work on the ScorePlay platform based on file hash?

👌 Resolution/Answer

• What a file hash is
  • A hash function (like SHA-256, SHA-1 or MD5) reads the file's data and produces a fixed-length string called a hash
  • Two files with the same hash are treated as identical
    • Example: File A and File B both give the hash 9a0364b9e99bb480dd25e1f0284c8555, so they are duplicates
  • Even a tiny change in a file gives a completely different hash
• How ScorePlay uses it
  • When a file is uploaded, ScorePlay computes its hash and checks if it already exists
  • If the hash already exists, the file isn't stored again and the existing file is used
  • If the hash is new, the file is stored and its hash is saved
• Limits in ScorePlay
  • Files are only deduplicated within the same collection, not across collections
  • Hashes don't work properly on files over 20 GB
• Why it helps
  • Identical files are stored only once, which saves storage
  • Comparing hashes is much faster than comparing whole files
  • Hashes can also detect corrupted or changed files

🔖 Notes

• Hash collisions (two different files with the same hash) are very rare, but possible with weaker hashes like MD5
• How does merging duplicates logic work on ScorePlay?