How Old is that Data on the Hard Drive?

The vast of amount of data being stored in this day and age, naturally leads to files sitting unused for longer and longer periods of time. A new app, agedu, can quickly tell you what data on your filesystem is lying fallow.

There is a system administrator theorem that is basically written as “Users will find a way to use space faster than new space can be added to systems.” The corollary to this theorem is, “Users will always insist that all of their data is critical and must be retained on-line.” The end result of this theorem and corollary is that the explosion of storage is really a burden that all system administrators must bear. What administrators need is a tool or a set of tools to help them examine file systems to determine what files and/or directories are being used and which ones haven’t been used recently (i.e. the “age” of files and directories).

In this article I want to introduce a new application called agedu, that can be used to get a snapshot of the age of files and directories. From this information you can get a general sense of what directories have older data that hasn’t been accessed (or modified) in a while. It can also be used in scripts to create reports about systems or for simply understanding what’s going on with your storage, even on your desktop or laptop.

Recent Data Age Study

There was a fairly recent study from the University of California, Santa Cruz and Netapp that examined CIFS storage within Netapp the company itself. Part of the storage was deployed in the corporate data center where the hosts were used by over 1,000 marketing, sales, and finance employees. The second part of the storage was a high-end file server deployed in the engineering data center and used by over 500 engineering employees.

Robin Harris at Storage Mojo writes a very good blog that looks at Storage. Robin has written about this study and come up with some very interesting observations. From his blog:

Some significant differences from prior studies:

  • Workloads more write oriented. Read/write byte ratios are now only 2:1 compared to the 4:1 or higher ratios reported earlier.
  • Workloads less read-centric. Read/write workloads are now 30x more common.
  • Most bytes transferred sequentially. These runs are 10x the length found in the old studies.
  • Files 10x bigger.
  • Files live 10x longer. Less than half are deleted within a day of creation.

Cool new findings:

  • Files rarely re-opened. Over 66 percent are re-opened once and 95 percent fewer than 5 times.

  • Over 60 percent of file re-opens are within a minute of the first open.

  • Less than 1 percent of clients account for 50 percent of requests.

  • Infrequent file sharing. Over 76 percent of files are opened by just 1 client.

  • Concurrent file sharing is very rare. As the prior point suggests, only 5 percent of files are opened by multiple clients, and 90 percent of those are read only.

  • Most file types have no common access pattern.

And there’s this observation from his blog:

Over 90 percent of the active storage was untouched during the study. That makes it official: data is getting cooler.

While one cannot take this data and make sweeping conclusions, it is a very interesting data point. In particular, there are two important observations that are very pertinent:

  1. Files are rarely reopened.

  2. Over 90 percent of the active storage was untouched during the study.

This begs the question of whether the data is needed at all since the data was created and never touched again. Wouldn’t it be more appropriate to archive the data? While the results are very enlightening, how could one scan their own systems in a similar fashion?

Data Comes in All Ages

As a background or primer on the “age” of data on *nix systems, it’s good to remind yourself that there are really 3 ages when talking about files:

  • File or Directory’s Change Time (ctime)
  • File or Directory’s Access Time (atime)
  • File or Directory’s Modify Time (mtime)

The first time, ctime is the time when changes were last made to the file or directory’s inode. This can include changes to the actual data, file or directory permission changes, file or directory ownerships, etc. You can view the ctime by using the command, ls -lc. The second time, atime is the time when the file was last accessed. You can view the access times by the command ls -lu. The third time, mtime is the modify time which is time when the actual file contents were changed. You can view the modify time by the command ls -l. To get all of the information in a quasi-readable format, in Linux you can use the stat command.

This means that when the age of data is discussed on *nix systems you need to be very specific about what age you using. One of the most important times to consider is the access time. This will tell you the last time the data was accessed (the last time it was used). This information is very useful when determining if data should be removed and/or archived. However, also tracking the “create” time and the “modify” time are very important as well. So, how does one get all of this information?

Next: Aegdu – A Tool for Displaying Data Usage »

Comments on "How Old is that Data on the Hard Drive?"

It lies within the decrease reaches of the Xijiang, Beijiang and Suijiang rivers and forms a part of the Pearl River Delta financial belt.Feel free to visit my site – Daisy Daze Necklace (Elena)

my web-site … reviews of regal assets (about.me)

Feel free to visit my web page regal assets reviews

Just beneath, are a lot of completely not associated web-sites to ours, nevertheless, they’re certainly worth going over.

It’s the best time to make some plans for the future and it’s time to be happy. I have read this post and if I could I want to suggest you few interesting things or advice. Maybe you can write next articles referring to this article.I want to read even more things about it!Feel free to visit my site … drake views from the 6 mp3 complete album download

Here is my web blog … regal assets review, Jake,

Visit my page :: reviews of regal assets – Jesenia -

Every once inside a whilst we select blogs that we read. Listed beneath would be the most up-to-date sites that we choose.

Feel free to surf to my homepage: reviews of regal assets

The time to read or take a look at the content or internet sites we’ve linked to below.

Check below, are some completely unrelated sites to ours, nevertheless, they are most trustworthy sources that we use.

I just want to say I am just newbie to blogs and absolutely enjoyed you’re web-site. Very likely I’m likely to bookmark your blog . You actually have fabulous articles and reviews. Thanks a lot for sharing your blog site.

Here are a number of the websites we advocate for our visitors.

The details talked about inside the post are a number of the best available.

Visit my web blog … regal assets reviews – http://wingchunsd.com/xe/?document_srl=664401 -

Feel free to surf to my webpage; regal assets reviews (Barry)

Also visit my web-site :: regal assets review (Jan)

I used to be recommended this website by means of my cousin. I’m no longer certain whether this submit is written by means of him as nobody else recognise such specific about my problem.You’re amazing! Thanks!Here is my website: ace public adjusters (Malissa)

Here are a few of the sites we advocate for our visitors.

my web site; regal assets review (Delila)

Feel free to surf to my blog post reviews of regal assets (Beatrice)

Here is my weblog; regal assets review (http://www.spidermangameshub.com)

Review my web-site reviews of regal assets, Phillis,

Here is my web page: reviews of regal assets (http://www.management.su.ac.th)

my web-site; reviews of regal assets (Louise)

Every once in a when we decide on blogs that we study. Listed below would be the most current web pages that we pick out.

Also visit my blog … reviews of regal assets (Oliva)

my website – reviews of regal assets (Rosaria)

That could be the end of this article. Right here you will obtain some sites that we believe you will value, just click the hyperlinks.

My web-site: reviews of regal assets; Leroy,

Also visit my page :: regal assets reviews (Emile)

Feel free to visit my web-site: regal assets reviews

Also visit my blog: reviews of regal assets (Alberta)

Feel free to surf to my webpage :: reviews of regal assets (Charline)

Usually posts some pretty interesting stuff like this. If you are new to this site.

Stop by my blog: regal assets review (http://www.anim8bar.com)

Also visit my weblog reviews of regal assets (Alejandra)

Please take a look at the web sites we follow, like this 1, because it represents our picks in the web.

We prefer to honor several other internet web-sites on the web, even when they aren?t linked to us, by linking to them. Below are some webpages worth checking out.

We like to honor quite a few other net internet sites around the web, even when they aren?t linked to us, by linking to them. Below are some webpages worth checking out.

Leave a Reply