[ic] Performance hit from having many images in one directory?

[John1]

On Tuesday, August 23, 2005 1:39 PM, mike at perusion.com wrote:

> Quoting Jeff Fearn (jefffearn at gmail.com):
>> On 8/23/05, John1 <list_subscriber at yahoo.co.uk> wrote:
>>> At the moment we have all our item images in a single directory
>>> (and all the thumbnails in one directory).  There are currently
>>> only about 1000 items, but this could increase significantly as
>>> time goes on (especially as we tend to leave any images of old
>>> discontinued products there just in case search engines etc still
>>> have them catalogued).
>>>
>>> We run the Linux ext3 filing system.  Is there a performance hit
>>> from storing loads of files in a single directory?  I am sure I
>>> read somewhere once that there *is* a performance hit, but I have
>>> no idea how many image files we would need to store in one
>>> directory before the performance implications became significant,
>>> bearing in mind that each page loads between 10 and 20 images from
>>> the thumbnails directory and 1 image from the main items directory?
>>>
>>> Please can anyone enlighten me on this?  Thanks.
>>
>> Good discussion at
>> http://answers.google.com/answers/threadview?id=122241
>>
Thanks Jeff - good overview.

>
> There are two major options once you hit 10,000 files or so:
>
OK, so 2000 or so files in the same directory on a Pentium 4, 3.0GHz with 
2GB RAM,  Western Digital Raptors SATA mirrored drives should be OK on ext3 
then?

Presumably the 10,000 is very much a ball park figure?  I guess the more 
important factor is how many times the file system has to access that big 
directory to serve up a single web page - yes?  In my set up, there might be 
15 thumbnail images on a typical webpage, so that's 15 searches through the 
1000 files then - does anybody have any subjective experience of when 
performance I am likely to see a noticable performance hit? 2000, 10000, 
50000 files?

>    1. Switch filesystems to XFS or ReiserFS for their tree-based
>       scheme. They don't degrade like ext2.
>
I like this idea.  Are there any good reasons why I *shouldn't* switch to 
one of these filing systems?  If not, then I guess this has to be the way to 
go with any new partitions?

>    2. Use a hashing scheme of some type to store images
>       in subdirectories. It can be as simple as manually
>       by category, or something automated based on say,
>       the first two letters of the image name. We do
>       this for session and temporary files, and we have
>       some routines which support finding the file name
>       automatically.
>
This is an option would not be too difficult for me to implement though not 
as nice to live with - it's so much nicer for maintenance and admin of the 
images to have them all stored together in one directory.

Thanks Jeff & Mike for your informative replies - much appreciated.