NTFS compression not working

4481 Views 10 Replies 3 Participants Last post by cluberti, Apr 9, 2011

kms2061

Discussion Starter · Apr 8, 2011

I have a very weird problem - NTFS compression is not working, or at least free space is not being reported correctly. Here's the deal:
I create a 128MB file filled with 0x7F's. I compress the file. Size of file on disk goes from 128MB -> 8.00MB. But, free space on disk goes from 97860kB -> 97220kB (500 kB was somehow lost)! The file IS physically compressed, if I check the raw data with a hex editor. Waiting or restarting doesn't help.
Any ideas? This is the first time something like this happened to me... :sigh:

Save

Status: Not open for further replies.

1 - 11 of 11 Posts

jcgriff2

· #2 · Apr 8, 2011

kms2061 said:
free space on disk goes from 97860kB -> 97220kB (500 kB was somehow lost)!

How are you measuring KBs on a 100's GB HDD?

There may be other HDD activity occurring.

Assuming a 250 GB HDD, 500kb = 0.00005%; insignificant amount of space.

Save

kms2061

Discussion Starter · #3 · Apr 8, 2011

In that case I was simply using a 512MB virtual RAM disk for testing. No other disk activity was occurring to that volume and it had no open handles at the time of measurements. The measurements occurred within 5 seconds.
The point is that I compressed a very compressable file (filled with 0x7F's only) and no space was freed (500kB was even taken away). The file properties indicate that the size went down 128MB -> 8MB while the free space count does not indicate this change at all. Decompressing the file reclaims the 500kB. The same effect can be observed on physical media, such as both SATA and USB connected hard disks.
This has never occurred before and I have no idea what might be wrong. Could this be some driver conflict or some service failing to start? Any help or insights are really appreciated.

Save

jcgriff2

· #4 · Apr 9, 2011 (Edited)

Maybe binary/ decimal conversion plays a part depending on where you get your #'s from - https://www.techsupportforum.com/posts/3068497/

General info Windows NTFS compression - http://support.microsoft.com/kb/251186

Save

kms2061

Discussion Starter · #5 · Apr 9, 2011

It's not a conversion error, something really is wrong with NTFS compression. I'll state what I did more explicitly:
Let's say we have a file "R:\test" filled with 0x00's.
In cmd, I type "dir r:" and get

Code:

 Directory of R:\

04/08/2011  19:23 PM    <DIR>          TEMP
04/08/2011  15:57 PM       134,217,728 test
               2 File(s)    134,247,936 bytes
               1 Dir(s)     347,791,360 bytes free

Now I type "compact r:\test /c" and get

Code:

 Compressing files in r:\

test                134217728 :   8388608 = 16.0 to 1 [OK]

Now if I type "dir r:" again, I get

Code:

 Directory of R:\

04/08/2011  19:23 PM    <DIR>          TEMP
04/08/2011  15:57 PM       134,217,728 test
               2 File(s)    134,247,936 bytes
               1 Dir(s)     347,136,000 bytes free

Instead of freeing about about 120MB, some space was used! Previously, compression always worked as it was supposed to, but now something is really wrong. Thank you very much for trying to help, and please let me know if you can think of a reason why this is happening. I'm starting to think I'm going to have to reinstall... :sigh:

See less See more

Save

jcgriff2

· #6 · Apr 9, 2011

Use dir r: /a /s

Take screenshot of Disk Management - be sure to maximize screen
START | type diskmgmt.msc

Save

kms2061

Discussion Starter · #7 · Apr 9, 2011

As I said, R: is a virtual disk, so it is not shown in Disk Management. I took a screenshot anyway, though, please see attached file.

Using dir /a /s again on R: (I deleted all other files since TEMP contained hundreds of temp files and doing recursive dir on it produced too many lines to post here) Here are the results:
>dir r: /a /s

Code:

 Volume in drive R has no label.
 Volume Serial Number is 168E-17A9

 Directory of R:\

04/08/2011  15:57 PM       134,217,728 test
               1 File(s)    134,217,728 bytes

     Total Files Listed:
               1 File(s)    134,217,728 bytes
               0 Dir(s)     370,806,784 bytes free

>compact r:\test /c

Code:

 Compressing files in r:\

test                134217728 :   8388608 = 16.0 to 1 [OK]

1 files within 1 directories were compressed.
134,217,728 total bytes of data are stored in 8,388,608 bytes.
The compression ratio is 16.0 to 1.

>dir r: /a /s

Code:

 Volume in drive R has no label.
 Volume Serial Number is 168E-17A9

 Directory of R:\

04/08/2011  15:57 PM       134,217,728 test
               1 File(s)    134,217,728 bytes

     Total Files Listed:
               1 File(s)    134,217,728 bytes
               0 Dir(s)     370,151,424 bytes free

Just for completeness, here are the results from doing this to test on my physical drive D: I'm not using the /s switch since the subfolders contain thousands of files.

Code:

D:\>dir /a
 Volume in drive D has no label.
 Volume Serial Number is 470F-C032

 Directory of D:\

03/24/2011  20:42 PM    <DIR>          $RECYCLE.BIN
04/08/2011  15:55 PM    <DIR>          KMS
03/29/2011  23:07 PM    <DIR>          share
04/08/2011  15:41 PM    <DIR>          System Volume Information
04/08/2011  15:57 PM       134,217,728 test
               1 File(s)    134,217,728 bytes
               4 Dir(s)  13,340,741,632 bytes free

D:\>compact test /c

 Compressing files in D:\

test                134217728 :   8388608 = 16.0 to 1 [OK]

1 files within 1 directories were compressed.
134,217,728 total bytes of data are stored in 8,388,608 bytes.
The compression ratio is 16.0 to 1.

D:\>dir /a
 Volume in drive D has no label.
 Volume Serial Number is 470F-C032

 Directory of D:\

03/24/2011  20:42 PM    <DIR>          $RECYCLE.BIN
04/08/2011  15:55 PM    <DIR>          KMS
03/29/2011  23:07 PM    <DIR>          share
04/08/2011  15:41 PM    <DIR>          System Volume Information
04/08/2011  15:57 PM       134,217,728 test
               1 File(s)    134,217,728 bytes
               4 Dir(s)  13,340,082,176 bytes free

On this volume, there were open handles to D:\$Extend\$ObjId and D:\System Volume Information\tracking.log by svchost, but still you can observe the same effect. Around 120MB freed by compression were not reclaimed.

UPDATE: chkdsk fixes the free space miscount! Here's the log:

Code:

R:\>chkdsk /F /X
The type of the file system is NTFS.
Cannot lock current drive.
Volume dismounted.  All opened handles to this volume are now invalid.

CHKDSK is verifying files (stage 1 of 3)...
  256 file records processed.
File verification completed.
  23 large file records processed.
  0 bad file records processed.
  0 EA records processed.
  0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
  274 index entries processed.
Index verification completed.
  0 unindexed files scanned.
  0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
  256 file SDs/SIDs processed.
Security descriptor verification completed.
  9 data files processed.
Windows has checked the file system and found no problems.

    524287 KB total disk space.
     29760 KB in 6 files.
        72 KB in 11 indexes.
         0 KB in bad sectors.
      5359 KB in use by the system.
      4672 KB occupied by the log file.
    489096 KB available on disk.

      4096 bytes in each allocation unit.
    131071 total allocation units on disk.
    122274 allocation units available on disk.

And now if I do dir r: /a /s, I get just as I'm supposed to:

Code:

Volume in drive R has no label.
 Volume Serial Number is 168E-17A9

 Directory of R:\

04/08/2011  15:57 PM       134,217,728 test
               1 File(s)    134,217,728 bytes

     Total Files Listed:
               1 File(s)    134,217,728 bytes
               0 Dir(s)     496,640,000 bytes free

But if I uncompress the test file and recompress it, we're back to where we started. It is unacceptable for me to dismount and check the drive during normal operation, as I will have to invalidate any open handles.
Please let me know if you have any idea what might be causing this strange problem and anything I can do to analyze it further. Thank you very much for your help.

See less See more

Save

jcgriff2

· #8 · Apr 9, 2011

What are you using to create the RAMDISK?

Dataram RAMDisk v3.5 publication -

2. Can I format a RAMDisk drive?

Yes. RAMDisk drives can be formatted using Windows Disk Management

http://memory.dataram.com/__documents/file/Dataram_User_Manual_35.pdf

Save

kms2061

Discussion Starter · #9 · Apr 9, 2011

I'm using ImDisk (Tools and utilities for Windows, 64-bit digitally signed driver), but that's not really relevant - the same problem occurs on physical media as well, as I demonstrated in my previous post. Formatting the media does not help, that was one of the first things I tried. There's definitely something wrong with how free space is counted after compressing a file - as I demonstrated, and chkdsk notes and corrects the miscount. The problem is the miscount occurring in the first place.
I work with many sparse files and working NTFS compression is very important to me. Thank you for your help and please let me know if you have any ideas as to how I can try diagnosing this problem further. This is not a common problem - an hour of searching did not yield any results.

Save

kms2061

Discussion Starter · #10 · Apr 9, 2011

UPDATE: Looks like I found the problem. As I suspected, Microsoft Security Essentials was to blame. Somehow it was interfering with free space reclamation after compressing. Stopping the Microsoft Antimalware Service fixes the problem. I can't believe nobody else has observed this problem yet.

How can I report this issue to Microsoft so that it can be fixed?

Save