HELP!

[Camy]

I desparately need a recommendation for software to remove duplictate files.

I've got Gb and Gb of data and a lot of it is redundant duplicates. It's especially annoying when it comes to finding the last version of a story that I want to start work on again.

So ... any decent freeware - that you can recommend - out there?

Cheers,

Camy

[Trab]

I can't promise results, but I've asked my computer literate nephew and he says he'll take a quick look with his sources to see what he can find.

[Camy]

I can't promise results, but I've asked my computer literate nephew and he says he'll take a quick look with his sources to see what he can find.

Thanks, Bart. There are so many different programs out there that promise the world ... but files are precious and a recommendation is definitely worth waiting for!

Camy

[The Pecman]

This one seems to be recommended by CNET and PC Magazine, plus it was a ZDNet Editor's Choice (for what that's worth):

http://www.ashisoft.com/

[Camy]

This one seems to be recommended by CNET and PC Magazine, plus it was a ZDNet Editor's Choice (for what that's worth):

http://www.ashisoft.com/

Thanks!

Camy

[The Pecman]

BTW, I could make a smartass Mac vs. Windows argument, but this is a big problem on Mac, too. Weeding out unwanted duplicate files is a royal pain for every operating system.

I recently started a part-time job where we're dealing with 1.5 petabytes (thousands of terabytes) of data, mostly old Hollywood movies being tweaked. It's a nightmare dealing with this much data -- racks and racks and racks of servers, in a room about 40' square, kept to about 50 degrees. And I thought I had a lotta data...

[Camy]

Blimey! THAT is a lot of data, but I'm thinking it's probably not as many files, just much, much bigger ones.

The real problem is keeping track of the most recent version across multiple computers and multiple drives. Especially when it comes to writing. USB drives are a nightmare too. I'd almost like to go back to the time when I had one zip disk with all my work on it.

I use drop box (cloud storage) for some stuff, and that works really well: one set of data accessed by whatever ... but I'm frightened the cloud might vanish. And broadband isn't near fast enough (in the UK) to store large media files up there.

[Mountain Dude]

Blimey! THAT is a lot of data, but I'm thinking it's probably not as many files, just much, much bigger ones.

The real problem is keeping track of the most recent version across multiple computers and multiple drives. Especially when it comes to writing. USB drives are a nightmare too. I'd almost like to go back to the time when I had one zip disk with all my work on it.

I use drop box (cloud storage) for some stuff, and that works really well: one set of data accessed by whatever ... but I'm frightened the cloud might vanish. And broadband isn't near fast enough (in the UK) to store large media files up there.

Camy -- I'm thinking you really do have a major problem. What with files on multi drives and computers I'm not very hopeful that any software will help.

I think the only way to correct this is to get all files onto ONE 'EXTERNAL/ZIP' hard drive. During this process any true duplicate files will be flagged and NOT moved to the external drive (this will require your best judgement ~~ if a file is questionable - rename that file and save it to the external drive. Once they are ALL on the external drive then at least the true duplicates are handled.

Once this is done you will need to go file by file based on title search and date/time search to come up with the latest file time stamp on the external drive.

A lot of work -- YES. But the only way as I see it. In the future NEVER save a 'multi use file' to anything but your 'EXTERNAL HARD DRIVE'. That way the latest file is ALWAYS on the external.

Zip drives are similar to external drives in that they are both movable - as in take your work with you. It's the ONLY way to avoid duplicate files all over the place.

You must also do a backup of the external hard drive on a REGULAR basis. Just in case!!! You will be most unhappy if you don't -- shit happens!!!

[Trab]

Camy, was my email any help to you?

I tend to agree with Mtn Dude, and recommend an external drive, although I'd go with several memory sticks as I've lost entirely too many hard drives in the past.

[Trab]

Camy, maybe you should develop a naming system the incorporates name, date, and time, and always use Save As, rather than Save. For instance, as Story named Blue Shadow might be named "blue_shadow_CH01_2010_7_13_1417"

If you put one somewhere else, you can still tell which is the more recent, as it would end in "_7_12_0932" or whatever

[DesDownunder]

I too, found that it is all too easy to end up with not only duplicate files, but also versions of the same file that differed from each other as I revised them.

My solution was to firstly create a folder for each and every project I work on. Inside that folder I always 'save-as' just as Trab suggests. You do have to make sure when you save -as, that you are in the correct folder.

Now if I have two or more different versions of a file I will call them the same name with a suffix, such as 'Really-good-story-No85-V1'.

The next time I work on the file and then save it, I will make a judgement as whether I should keep the original file or save the new one as V2, because I have changed things so much from what I had previously done.

This means I only save the file and its variations to one place on the hard drive.

My file structure (simplified for clarity) looks like this.

MyDocuments

DesData

-Des-Stories

-CompletedStories

--StoriesWorking

---Story-ideas

---Story-85

And of course other stories also have their own folder.

When it comes to Sound files I do something similar, except there is a separate folder for sounds that might be considered general effects (Sirens, crashes, etc.,) and they are stored under the 'SoundFX folder.' Music has two folders Original, and Modified. Actual sound projects also, each have their own title folder, which may also contain the recording program's generated files in its own subfolder. This will depend on the recording software, but it helps to keep them in one place if possible, and always with the name of the project as the files name, along with its version number.

If I am doing a recording of a reading of poem, then in each poem's folder is another folder called Recording-Poem12, or what ever it happens to be.

In a recording folder there will be several 'takes' of each reading that I have done, IF I thought they were worth saving. These do tend to mount up and clearing them out is relatively easy as they are all in one place.

This is quite more difficult to describe than actually do, once you get the hang of the idea.

So to sum up, each project has it's own folder under the 'Working' folder. They are never saved anywhere else but in their own folder. (Except for backups to the USB or DVD drive)

When a project is completed, The final file is saved in the Completed folder and usually noted as published at AD or CW or wherever.

As Trab points out, the secret is to use a naming system. To that I have added a specified storage folder space for each story or project in a hierarchical format suited for my purposes, and work practices. How you arrange it will depend on how you work, but once it is done and you are happy with it, it saves time and a heap of hassles.

Do be wary of making it too complicated. Think it out on paper first, try it, alter it, work with it, change it and you will soon work out what suits you best so you can concentrate on the creativity rather than wondering where the hell the file has gone, or if you are working on the right one.

One further problem is with Windows 7, which is not friendly towards you wanting to save documents and image and sound files anywhere but where it thinks they should reside on your hard drive. When compounded by the Win7 Library system, things get messy fast. The trick is to ignore what Win 7 (or XP) wants to do. There are ways to bypass what it wants to do, but it is a head f*k that should never have been imposed on those of us who want to do things our way and not M$'s way. Obviously I do not want all my files of a certain type to be lumped together in a mysterious location on the hard drive.

Hope this gives you some ideas.

[Camy]

Thank you all for your help and ideas. I can't express how chuffed I am with both the software suggestions and the filing ideas.

As Eddy said it's going to take a lot of work, but better that than a hair tearing tearful digital disaster. I don't know quite what I expected .... Perhaps a magic word that would sort out all my problems without any serious slaving. Sadly, that didn't materialise. :(

But I do know what I'm going to do which is adapt Des's file structure for my avian brain, and additionally use Bart's end of file date. I think, provided my head doesn't explode, it'll work and I'll be okay. Bart also suggested locate32 (http://www.locate32.net/) which I've installed and is excellent.

Luckily, all the studio stuff is on a Mac which has 'Time Machine' and pretty much takes care of itself.

So thanks, y'all! You guys ROCK!

Camy

[The Pecman]

Watch out for Time Machine -- it's not always 100% safe. I've had at least one of those completely fill up, and you have no choice except to toss out that drive (or reformat it) and start all over again, once it's filled. I use SuperDuper to make real backups of most of stuff, typically to 1TB drives on Firewire (though I'm starting to much prefer eSATA).

I'm a big fan of the idea of putting the operating system and applications on your boot drive, then keeping all your actual data files (stories, music, finances, whatever) on an external drive. External drives are real cheap -- they're down to maybe US$60-$70 these days. This way, if your main computer ever fails, you can still access the files on a laptop or a friend's computer in a pinch.

I keep my correspondence in separate folders (organized by year), plus writing projects in separate folders inside a "Current Projects" folder. Older stories are in a secondary folder, like "Current Projects 2" or something like that. Come up with a logical organized system that makes sense to you in terms of what you need to grab.

BTW, if I'm really on a tear and writing very quickly, I don't trust backups at all. I print out everything I write that day and toss the paper into a file cabinet. My theory is, hard drives crash, computers blow up, but the paper is probably going to outlive me (short of a fire or nuclear holocaust). Only once have I had to retype or scan printed copy for a project, but it was still better than trying to start from scratch.

[JamesSavik]

My system for managing writing works like this:

WritingProjects

-Broken

-Twilight

-Valhalla

Each project has its own subdirectory.

I back the whole thing up to external drives

[Cole Parker]

Hey, I'm going to weigh in on a computer problem!

That's something like a 6-year-old advising Einstein on warp theory.

But, I can say this.

Computers make it very easy to save stuff. I've found, however, that most stuff I save, I never look at again. I could, right now, go and dump tons of files and never miss them. I don't, I have a relucatance to dump things both in real life and computer life, but I could.

I save everything I edit, everything I write, all ideas, all fragments of ideas, and, well, you get the idea.

I have room for all this, but if I get to a point where I don't, I'll start dumping. It's difficult, but if you ask yourself, do I really need this, am I ever going to use this again, you could probably dump a quarter of what you have immdediately and reduce the problem accordingly.

C

[Mountain Dude]

The real problem is keeping track of the most recent version across multiple computers and multiple drives. Especially when it comes to writing. USB drives are a nightmare too. I'd almost like to go back to the time when I had one zip disk with all my work on it.

I think most responders are missing the point that these files span multi computers and drives.

This fact requires the combining of ALL FILES onto ONE DRIVE! Be it flash drive zip drive or another moveable drive. Combined file size will dictate the external drive size required. Comparison of files requires they all be on the same drive -- or at least on the same computer.

The Zip drive you mention IS the answer to your current problem, All files need to be in one place -- only! -- ESPECIALLY when changing files such as your story files.

[The Pecman]

The Zip drive you mention IS the answer to your current problem, All files need to be in one place -- only! -- ESPECIALLY when changing files such as your story files.

Uh... with all due respect, the Zip format died about 7 years ago. In 2006 PC World magazine called the Zip Disk "the 16th worst technology product of all time." Speaking as a guy who owned several Zip drives and a lot of disks... it sucked. Read this link on Zip Disk history, or this link on the dreaded Click of Death for details why.

In 2010, I think a cheap external hard disk or even recordable CD-R or DVD-R are a far better, cheaper alternative. Having said that... any backup is better than no backup.

[DesDownunder]

I agree with Pecman, Zips were in my experience nothing more than a huge drain on my finances in those days, not at all reliable.

I now share my data and backup on 3 x 500gig hard drives and also on several USB flash drives for the most important data.

Even so I too, print out the documents I must not be without, such as my photos of fondly remembered bananas, and my tax records.

[Camy]

I have to agree that Zip (as in Iomega Zip drives and their 100Mb discs) are long gone. They were good when they worked and my beautiful beige G3 had an internal version.

But, replace Zip with a USB flash drive and that's part of the answer ... except USB sticks are really easy to lose, or leave in your jeans when they get washed. For that matter I have a neat 2GB micro SD in my phone that I use to store stuff. It's too much data is the problem. As Cole says dumping stuff is good, but for those who hoard very painful.

Ultimately the only thing that works is what works for you. I don't think there is one overall answer to the problem and storage is sooooo cheap!

Hmm, perhaps a new branch of therapy is in the offing? ;)

[Trab]

You might all want to take a look at this beauty. I'm on a Mac AND also a PC netbook AND a PC tower, and this lovely works with them all. In fact, I use it for me Time Machine backup mostly. For whatever reason, which started before my iMac, I have a history of burning out hard drives, whether they be internal or external. The last about 3 months, then pfizz and they're dead, with little curlicues of smoke often signalling the finality of it all. Only once was I able to recover the information with forensic help, and then only because they did it for me for free, not the $3k they originally requested.

Side story: I had sent it in for determination of the cost, they told me, I said I'd think about it, I paid the estimate fee, decided to have them return it to me, and they LOST IT. They found it 2 years later, and as an apology they recovered almost all my data at no cost.

Anyway, the point here is that you can pop any old or new HD into this unit and I haven't lost one to heat-stroke since.

[Camy]

That's brilliant.

[The Pecman]

Yeah, I agree 100% with Trab -- those little Thermaltake docks are fantastic. I think it's almost a perfect backup machine -- just a bare drive and a box. Once you're done, pop the drive out and put it in a drawer. (I found some little plastic boxes to hold them, just to keep the dust out.) A really terrific, super-cheap backup solution.

[DesDownunder]

That's strange, because I have had least problems with Western Digital drives.

Checking with my techie nerd friend, he tells me that Western Digital had problems some 15 years or so ago, and that Seagate had problems about 12 years ago. He said that most manufacturers of Hard drives have had some reliability problems at one time or another. Of course anything mechanical will develop faults eventually.

My Computer tower case has front of case slots for 4 hard drives to allow easy replacement and they have their own fan.

[Mountain Dude]

I have been using USB external drives -- I've had my problems with them too. These docking stations look really interesting. Have to give them a try.

[The Pecman]

I used to be a big Seagate fan, but I switched to Western Digital's "Caviar Green" line. They run very cool, momentarily sleep when not in use, and are great for non-intense storage (like music, photos, and text). If you do heavy-duty video editing, they're not ideal. For everything else, they're fine.

Both the WD and Seagates (and Hitachis) are all priced about the same -- maybe $60 for a 1TB, $120 for a 2TB -- so there isn't much difference beyond that.