Weird free space problem with 2TB drive

[jtr1962]

I purchased a Samsung F4 2TB drive about a month ago. Generally, before using a drive, I'll let it burn in a while and also surface scan it a number of times. No problems there, but it seems the drive is reporting its free space incorrectly. The free space is correct after I format the drive. It shows as 2,000,270,127,104 bytes free. However, after a few days, or in some cases only a few hours, it shows about 2.4 GB less. As best I can determine, the problem starts occurring after I delete a file ( from any drive, not just from the Samsung ), or after a reboot. It does the same thing whether or not there are files on the drive. For example, I copied one folder with 7.29 GB to the drive, and yet the drive properties showed 9.74 GB in use after a few days. The weird part is doing a chkdsk shows the correct number of allocation units and bytes free, but XP reports the free space incorrectly. Has anyone else come across this problem? And is there a solution? BTW, I checked for hidden files and such, but there are none.

On another note, is there any way to get XP to report file and disk sizes in the correct base 10 manner instead of the idiotic base 2 convention where 2 TB reports as 1.81 TB? Last I checked, K = 1000, M = 1,000,000, G = 1,000,000,000, and T = 1,000,000,000,000. You should at least have the option to choose between base 2 and base 10.

 

[Bozo]

If you are running XP 32bit, could it be that it is choking on a drive that big???

 

[jtr1962]

If you are running XP 32bit, could it be that it is choking on a drive that big???

From what I've read, XP 32-bit should be able to handle drives up to 2^32 sectors. With 512-byte sectors, that's 2,199,023,255,552 bytes.

I've read of a few cases of motherboards which couldn't handle even 1 TB drives. My A7N8X-E was one of them, but I updated the BIOS. It had no problem reading and formatting every sector. I'm thinking this is some kind of arcane problem at the file system level.

 

[BingBangBop]

check to see if the Recycle Bin has anything (that will show up as less space on the HD). Go into recycle bin properties to see if it automatically reserves space for the Recycle Bin.
Next, check System Restore for it can allocate space on a drive too.
Also, check for Shadow Copy settings (the OS might be reserving space for shadow copies).

 

[LunarMist]

From what I've read, XP 32-bit should be able to handle drives up to 2^32 sectors. With 512-byte sectors, that's 2,199,023,255,552 bytes.

Sure, but the F4 has 4KB sectors. Do you use the alignment tool?

 

[LunarMist]

On another note, is there any way to get XP to report file and disk sizes in the correct base 10 manner instead of the idiotic base 2 convention where 2 TB reports as 1.81 TB? Last I checked, K = 1000, M = 1,000,000, G = 1,000,000,000, and T = 1,000,000,000,000. You should at least have the option to choose between base 2 and base 10.

I don't think so, at least not with Windows itself.

The base 2 convention is not idiotic; it is perfectly natural for digital systems. OTOH, base 10 is an ancient holdover from the 5-fingered x 2-handed primates.

 

[Bozo]

check to see if the Recycle Bin has anything (that will show up as less space on the HD). Go into recycle bin properties to see if it automatically reserves space for the Recycle Bin.
Next, check System Restore for it can allocate space on a drive too.
Also, check for Shadow Copy settings (the OS might be reserving space for shadow copies).

Also check to see if Hibernate is off.

 

[jtr1962]

Yes, hibernate is off, the recycle bin has no reserved space, system restore is turned off on this drive. I don't really know anything about shadow copy, but it looks like it's off.

Any other ideas here?

Lunar, my beef with the whole base2 thing is we already have conventions for it-KiB, MiB, GiB, TiB. If the O/S wants to report space using binary, then at least use the correct prefix. Better yet, let me choose which I want. As far as I'm concerned, 1 TB = 1,000,000,000,000 bytes, NOT 2^40 or 1,099,511,627,776 bytes. Storage isn't inherently in powers of two like RAM because hard disks end up with however many sectors fit on the platter. Even sectors don't inherently need to be a power-of-two number of bytes ( if you include the ECC bytes then they're not ), although I suppose that makes things easier when you're mapping things into RAM.

 

[BingBangBop]

Do you have multiple OS's sharing the same drive? Files from another OS can also allocate space on a HD that are invisible, depending on security.

For that matter, on a single OS, if you try, you can effectively totally hide files, from non-administrators, using security and while you can't totally hide files from administrators that know what they are doing you can make them effectively hidden, if they are not working at finding them. So theoretically there could be an application that is deliberately trying to hide stuff using security settings.

Another, would be using compression on the original folder but not on the destination folder. That would make the original size look smaller than it actually is, and the destination would actually be the bigger correct size.

 

[BingBangBop]

Click on the original folder and choose properties. Do the same on the duplicate destination folder that has changed size. Compare the two windows and their numbers, specifically number of files, # of folders, file size, and file size on disk. While it is not perfect, it may give you a clue as to where to look.

 

[jtr1962]

All I copied so far were two DVDs I ripped. Yes, the original folder had compression while the new folder didn't. However, since these were already video files with built-in compression, they don't compress much. They take about about 0.01 GB more on the uncompressed drive.

Here's the statistics when I highlight all the folders on the drive and go to properties:

142 Files, 16 Folders
Size: 12,515,641,853 bytes
Size on disk: 12,515,905,536 bytes

Now here's what I get when I go to properties for the same drive:

Used space: 15,144,558,592 bytes
Free space: 1,985,254,281,216 bytes
Capacity: 2,000,398,839,808 bytes

Note the difference between the two highlighted numbers which should be close to the same size, at least within a few hundred MB of each other, as they are on my other drives ( the difference is obviously file system overhead ). The file system overhead on this drive after it was freshly formatted was only about 128 MB. My apparently unaccounted for lost space here is around 2.5 GB.

No, there aren't multiple OSes sharing this drive. In fact, my O/S resides on my other drive. This drive is formatted as one partition, and will be solely used for storage.

 

[LunarMist]

I'd zap the MBR and start over. Also consider running the full diagnostic utility. I have an empty F4 here that I could test, for something.

 

[BingBangBop]

I really wanted a comparison between folder and folder rather than folder vs drive (like vs like). When comparing folder or file vs drive you can get into the situation where an individual file/folder could be one bite in size but the minimum size allocated to the drive is your cluster size (4096 bytes). There is also File system overhead like the MFT and the page file that is counted in the drive used space but will not be counted folder/file size.

If you compare folder vs folder, you will get information that can be used to id the problem. For example knowing that they have the same number of files and the same number of bytes will tell you that there are or are not any files being hidden. You'll also find out if compression actually matters. You will also get the total bytes used rather than the amount allocated.

If everything is the same folder vs folder then I'm almost certain that the difference between folder and drive is that you are dealing with actual file size vs allocated space + file system overhead.

 

[jtr1962]

The difference between the used space of the folders between the compressed and non-compressed drives is minuscule. For example, one folder used 15,148,750 bytes more on the uncompressed drive than on the compressed drive. I'm using 4K allocation units on both drives. Also, the page file is disabled. Even if it wasn't, it would be on my other drive, not this one.

Remember that this problem comes up even when the drive is freshly formatted, with absolutely nothing at all on it. The free space mysteriously drops suddenly by about 2.5 GB, usually either after a reboot, or after I delete something on one of the other drives. When it does, I can find absolutely nothing on the drive to account for this. There might be a few KB of files in the system volume information folder, but absolutely nothing else on the drive.

 

[BingBangBop]

I don't know why you are resisting my suggestion the test takes less than a minute and is totally non-destructive.

Try the MFT size ...

 

[jtr1962]

I'd zap the MBR and start over. Also consider running the full diagnostic utility. I have an empty F4 here that I could test, for something.

I've surface tested the drive about a dozen times already. All is well.

I used diskpar to partition the drive. Maybe I should do it over with diskpart and see if the problem reappears? I just have to figure out the parameters to use with diskpart to start the partition at sector 64 instead of the default 63. Using disk management just doesn't allow that kind of control.

Only thing I might ask you to test with your empty F4 is to partition and format it, then reboot the machine once or twice, and also perhaps copy something to the drive, delete it, and empty the recycle bin. In theory your free space should more or less match what it was after you freshly formatted the drive. I might be a few KB smaller on account of the recycling bin folder being added, but it should be very close. If you find it's a few GB less, then you're having the same issues as me.

 

[jtr1962]

I don't know why you are resisting my suggestion the test takes less than a minute and is totally non-destructive.

Try the MFT size ...

Exactly what would you want me to try here? I'm not really resisting your suggestion. I just don't know any other method here to compare folders than the one I used. I you tell me step by step what you want me to do, I'll gladly do it. I only have two data folders on the drive in question. They're only using about 22 MB more combined on this drive compared to what they're using on the other one. That can't possibly account for the problem. Neither can a page file because I disabled paging. Like I said, tell me exactly what to do, and I'll try it. I'm pretty sure that I did do what you asked me to, but maybe I misunderstood.

On the compressed drive, the two folders were allocated the following amount of space ( i.e. this was the number next to "size on disk" ):

4,674,981,888 bytes
7,819,893,042 bytes

On the uncompressed 2TB drive, the corresponding figures were:

4,680,855,552 bytes
7,835,041,792 bytes

The difference between the two is very small. My point is when you add the totals for both folders, and also the other system folders on the drive, and subtract it from the drive's capacity, it gives a number which is about 2.5 GB higher than what windows says is the available free space. If I were to delete both those folders, the drive will still show ~2.5 GB in use even though the system folders are only a few KB.

Here's the chkdsk results:

1953514495 KB total disk space.
12222568 KB in 144 files.
44 KB in 24 indexes.
0 KB in bad sectors.
126295 KB in use by the system.
65536 KB occupied by the log file.
1941165588 KB available on disk.

4096 bytes in each allocation unit.
488378623 total allocation units on disk.
485291397 allocation units available on disk.

Interestingly, chkdsk shows the correct number of allocation units and free space available. This free space is about 2.5 GB higher than what Windows is reporting. Apparently then the disk housekeeping is OK, but for some reason Windows is reading it wrong.

 

[BingBangBop]

The point of comparing folder to folder was to exclude a whole host of potential issues. Things like extra files or data being added by some other activity. If the number of files are identical, the sizes are identical, compression isn't a factor, then we can look at simply file system overhead rather than potential security issues.

At this point my best guess is that the MFT is simply expanding and then contracting as unnecessary portions are cleaned up and removed.

 

[sdbardwick]

Maybe this MS article has some ideas:

In summary, use the following methods to correctly determine how the disk space is being used on a volume:

View the output from the chkdsk command.
Use the Ntbackup GUI or view the backup logs.
View the disk quotas.

On the other hand, Windows Explorer and the dir command have some limitations and drawbacks when you use them to determine how disk space is being used.

 

[sechs]

When comparing folder or file vs drive you can get into the situation where an individual file/folder could be one bite in size but the minimum size allocated to the drive is your cluster size (4096 bytes).

On NTFS, files smaller than the cluster size are stored in the MFT. This saves a lot of otherwise wasted space.

 

[sechs]

On the compressed drive, the two folders were allocated the following amount of space ( i.e. this was the number next to "size on disk" ):

4,674,981,888 bytes
7,819,893,042 bytes

On the uncompressed 2TB drive, the corresponding figures were:

4,680,855,552 bytes
7,835,041,792 bytes

The difference between the two is very small. My point is when you add the totals for both folders, and also the other system folders on the drive, and subtract it from the drive's capacity, it gives a number which is about 2.5 GB higher than what windows says is the available free space. If I were to delete both those folders, the drive will still show ~2.5 GB in use even though the system folders are only a few KB.

Here's the chkdsk results:

1953514495 KB total disk space.
12222568 KB in 144 files.
44 KB in 24 indexes.
0 KB in bad sectors.
126295 KB in use by the system.
65536 KB occupied by the log file.
1941165588 KB available on disk.

4096 bytes in each allocation unit.
488378623 total allocation units on disk.
485291397 allocation units available on disk.

Interestingly, chkdsk shows the correct number of allocation units and free space available. This free space is about 2.5 GB higher than what Windows is reporting. Apparently then the disk housekeeping is OK, but for some reason Windows is reading it wrong.

Are you comparing "available" to "free" space? These aren't the same thing.

 

[LunarMist]

I've surface tested the drive about a dozen times already. All is well.

I used diskpar to partition the drive. Maybe I should do it over with diskpart and see if the problem reappears? I just have to figure out the parameters to use with diskpart to start the partition at sector 64 instead of the default 63. Using disk management just doesn't allow that kind of control.

Only thing I might ask you to test with your empty F4 is to partition and format it, then reboot the machine once or twice, and also perhaps copy something to the drive, delete it, and empty the recycle bin. In theory your free space should more or less match what it was after you freshly formatted the drive. I might be a few KB smaller on account of the recycling bin folder being added, but it should be very close. If you find it's a few GB less, then you're having the same issues as me.

Oh crap. :crap: I forgot if there is a Samsung utility to align the partition like the one WD has.

 

[jtr1962]

Well, I think I partially solved it, or at least found out what's behind the problem. After reformatting the disk, my free space was back to 2,000,270,127,104 bytes. I tried deleting and then restoring a file from my other hard disk. After I restored it, I noticed that the free space on the 2TB drive decreased by 2.44 GB, and I now had a recycler folder on the drive. Same thing if I delete something on the other drive, and then empty the recycle bin instead of restoring it. Apparently Windows reserves some space for the recycle bin on the 2TB drive once the recycle bin is in use on any other drive in the system. Weird.

I then reformatted the 2TB drive, but this time dropped the space for the recycle bin on that drive to zero, and also turned off recycling. This partially fixed the problem. Now no matter what I do as far as deleting files on my other drive, the free space on the 2TB drive is unaffected, and no recycler folder appears. However, if I delete any files on the 2TB drive, then Windows again appropriates 2.44 GB even though recycling is turned off. Emptying the recycle bin doesn't free up the space. Basically then I'm OK so long as I don't delete any files on the drive. Maybe there's another setting somewhere to completely keep Windows from appropriating any space on the drive for recycling, or at least to give all the space back once the recycling bin is empty.

I'm going to guess here that Windows perhaps always keeps something like 0.1% of the disk space in reserve for recycling no matter what. On even a 100 GB drive this isn't that noticeable, but on a 2TB drive you're talking around 2 GB.

 

[ddrueding]

Something tells me that if you weren't being so meticulous in this it wouldn't actually affect your use of the drive. If it is the standard behavior of the drive (which it looks like it is), then it has happened to every drive I've had without me noticing. I'll take a look at the 11TB test array at the office and see what I can come up with.

 

[jtr1962]

Ugh, it seems this behavoir is normal. I just checked one of the partitions on my 200GB drive. Total space is 150,271,983,616 bytes. Windows reports the free space as 102,551,552 bytes. However, chkdsk reports 289,656,832 bytes available on disk. The "missing" space is 187,105,280 bytes, or just about 0.1245% of the total size of the partition. Interestingly, this almost exactly matches the percentage of space I lose on the 2TB drive. It seems then that Windows always reserves roughly 1/8 of a percent of the total space for recycling, no matter what, at least on NTFS partitions. On my FAT partitions the free space reported by both Windows and chkdsk are exactly the same.

 

[LunarMist]

So it is just the MFT?

 

[jtr1962]

So it is just the MFT?

According to both chkdsk and defrag, the MFT isn't taking a huge amount of space. The MFT is only using about 128MB on the 2TB drive. Rather, it appears by default that Windows considers roughly 1/8% of the drive space to be "used" for recycling purposes when it reports the amount of free space on NTFS volumes, even if the recycle bin is empty.

 

[ddrueding]

So what would be the fastest way for me to try and replicate this?

 

[Bozo]

When a file is deleted it is not really removed. It is just removed from the MFT ie:index.
Could it be that windows is counting the 'deleted' file as used space because the clusters still have data in them?

CCleaner has an option to 'clean' the empty areas of the hard drive without removing the MFT. I wonder if that would make a difference in your test.

 

[jtr1962]

When a file is deleted it is not really removed. It is just removed from the MFT ie:index.
Could it be that windows is counting the 'deleted' file as used space because the clusters still have data in them?

Except that the deleted file here was only a few kB, and I suddenly lost 2.44GB of space after I either restored it, or emptied the recycle bin.

 

[jtr1962]

So what would be the fastest way for me to try and replicate this?

Is the 11TB array empty and freshly formatted? If so, then check the free space and used space. Now copy any file into the array, and delete it. After that either restore it, or empty the recycle bin. If your system is behaving like mine, then you'll suddenly see less free space. Since the lost space seems to be about 1/8% on all the drives I've checked, you'll probably find that you're losing close to 14GB.

If the array isn't empty, then compare the free space reported by Windows to the available space reported by chkdsk. Again, if you're experiencing the same problem as me, they will differ by close to 14 GB.

 

[ddrueding]

I copied an 4KB file and lost 8MB.

 

[sechs]

Again, if you're experiencing the same problem as me, they will differ by close to 14 GB.

Are you comparing "available" to "free" space?

The Windows Explorer's "free space" is capacity minus used space. "Used space" does not include files and folders that you can't see, i.e. those with the hidden attribute.

Go ahead and pick in volume with hidden files and folders. Compare the disks's "used space" with the size of everything, including the hidden items, in properties (select everything, right-click, and select properties).

 

[jtr1962]

Are you comparing "available" to "free" space?

The Windows Explorer's "free space" is capacity minus used space. "Used space" does not include files and folders that you can't see, i.e. those with the hidden attribute.

Go ahead and pick in volume with hidden files and folders. Compare the disks's "used space" with the size of everything, including the hidden items, in properties (select everything, right-click, and select properties).

I have all my hidden files visible in Explorer. Besides that, on the drive in question the only system folder is the restore folder, and that's empty.

Anyway, it looks like I more or less figured out what's going on here. It seems when using the NTFS, Windows reserves roughly 1/8% of the drive for something, most likely recycling, and subtracts this from the available space. Hence the discrepancy in free space between what Windows reports, and what chkdsk reports. This discrepancy exists on all my NTFS partitions. I just never noticed it before because it was relatively small, and I already had files on the drive. With the empty 2TB drive, 1/8% works out to around 2.5GB, so it was pretty apparent.

Now to see if it's possible to fix this behavoir, although practically speaking it's not a huge problem, but more like a book keeping issue where things just don't balance out.

 

[Howell]

VSS?

 

[Mercutio]

VSS?

That's probably not turned on.

 

[sechs]

I have all my hidden files visible in Explorer. Besides that, on the drive in question the only system folder is the restore folder, and that's empty.

It also excludes files that you don't have the rights to (and, therefore, can't see). This includes some system files, as well as metadata.

 

[Sol]

Does it actually stop you using the last 1/8% or does it just show the wrong figures?

 

[jtr1962]

Does it actually stop you using the last 1/8% or does it just show the wrong figures?

It actually does stop using that last 1/8%. I just found this out when the 150 GB NTFS partition on my boot drive was full. Chkdsk still said I had about 190MB free but Windows refused to write to the disk.

 

[jtr1962]

Well, no direct solutions to this but I hit upon an indirect one - use exFAT. Windows doesn't mysteriously mark space as in use when it isn't with exFAT. From what I'm gathering the only downsides are no built-in compression ( I wasn't planning to compress any folders anyhow ), and possibly more wasted space due to larger cluster size ( again not much of an issue since this drive will mostly hold video files and backups of my primary disk - both cases mean a few very large files ).

I played around with different cluster sizes. Here is the relative loss/gain in total space on a freshly formatted partition relative to using NTFS with 4K clusters:

4K - ( -1880 MB )
8K - ( -880 MB )
16K - ( -361 MB )
32K - ( -124 MB )
64K - ( + 1 MB )
128K - ( + 64 MB )
256K - ( + 94 MB )
512K - ( + 110 MB )
1M - ( + 116 MB )
2M - ( + 115 MB )

With cluster sizes above 1M you actually lose space compared to smaller clusters. Default cluster size is 128K. I used 32K for the time being although I can always reformat since nothing important is on the drive yet. Note that if I use the same 4K cluster size as NTFS then the slack space on stored files is the same BUT I lose close to 2 GB of drive space right off. Keep in mind however that once NTFS marks that ~2.5 GB of space as being used I'm still ahead by 600 MB using exFAT and 4K clusters.

Any thoughts? Am I crazy using exFAT?