Suggestions for automating some tasks

[mubs]

I'd like to automate something for a client - file copy based on some logic.

Folder "P" keeps accumulating files (it lives on a dedicated 250 GB drive). Periodically, the new files added to folder "P" need to be copied to sub-folders inside folder "Q" (on another drive). The sub-folders are temporary holding places for the files and will be emptied when processing is done, so they are empty at the start of this process. The sub-folders need to be CD-sized, ie., ~640MB or ~690MB. Copy will start with the first sub-folder in "Q" as destination, and when it approaches CD-size, switch to the next sub-folder in "Q". The number of new files generated in a given period is indeterminate. The individual files range in size from ~20MB to ~55MB.

I was thinking that maybe a script could do this. My days of scripting batch files are long gone, but I'm willing to learn Windows scripting. The issues here are destination folder size monitoring and date comparisons (some way of knowing which the newly added files are; either keyboard input for start date, or saved/read from a text file).

The m/c is running WinXP Pro, and performance is not an issue; a "start it and go for a coffee" type solution is fine.

I know this can be done in VB or C++ and similar tools, but I'm looking for simpler tools than that. Anything more expensive (e.g. VB) than a decent shareware-type price is a no-go. I have no idea what tools I could use, and where to find them.

Do I make sense?

Any pointers and advice would be greatly appreciated.

 

[Handruin]

Perl would do the job you ask. I don't know how to write it in Perl, but I suspect it's definitly possible.

Next options is a php CLI script. PHP5 has more support for a CLI based environment to make it function like Perl. I'll look into it and see if I can make it do what you ask.

 

[Handruin]

Reference for me later:
http://www.phpfreaks.com/tutorials/86/1.php

 

[JMP]

You might want to try autoit, it is free and easy to use

http://www.autoitscript.com/autoit3/

Jeff

 

[Howell]

Folder "P" keeps accumulating files (it lives on a dedicated 250 GB drive). Periodically, the new files added to folder "P" need to be copied to sub-folders inside folder "Q" (on another drive).

Do you expect P to be empty except for the files that need to be copied? IE, do the files need to remain on P after they are copied.

 

[mubs]

Thanks, Handruin, I'll look into Perl / PHP.

Jeff, thanks for the suggestion. My understanding was that AutoIt records mouse and keyboard movements for playing back later (like recording a macro in Word or Excel). Maybe I'm wrong, I'll take another deeper look at it.

Howell,

Folder "P" is an archive of critical stuff, so it will keep growing. I believe the stuff has to be around for 7 years for legal reasons, so the contents of P will be a rolling 7-year window. I am doubtful that 250GB will be enough to store 7-years worth, but this thing just got started, so we have enough time to fix things later.

Folder P lives on its own drive, drive #2. Drive #1 is the system drive, on which the application runs, and maintains its own version of the files. Thanks to Handruin, I've installed SecondCopy on this m/c, which maintains an identical copy of "P" on another PC with a similar 250GB drive. So there are 3 copies overall of the data.

To answer your questions, if "P" is empty, there's a major problem! And "P" will always have more files than are to be copied. Files from "P" will always be copied, never moved, so yes, the files will need to stay on "P" after copy.

 

[sechs]

Does no one think that this can be done with a batch file or Windows script?

 

[Handruin]

Just to recap:

"P" will be continuously increasing in size as an archive.

As new files are added to "P", the will also be added to sub folders. Each sub folder has a max capacity of 640 to 690 MB.

(Temporary sub folders)
"Q" will fill up to 640-690 MB
Once the file size limit is reached in "Q", a new folder named "R" is created. As new files fill into "P" they are now copied into "R" instead of "Q".
"R" will fill up to 640-690MB and the process continues into "S"...

The process continues like this, or did you have a max number of temp sub folders?

 

[Handruin]

Does no one think that this can be done with a batch file or Windows script?

I think it can be done, but it might be hard to determine folder sizes without a lot of temp text files with exported directory sizes. I mentioned Perl/PHP because I feel it has more capabilities to do the job. After doing some reading (the link I posted), I think it might be easy to write you a script using PHP 4.3.6 and then utilize the php.exe script engine. Since it is free and would require little to no configuration, it might be easier than trying to figure out how to do this with a batch file.

I have no background with windows scripting, so I can't argue it's advantages/disadvantages compared to these other tools.

 

[Will Rickards WT]

Does the guy have word or excel on the machine?
You could write this in a VBA script and have a button on the toolbar that starts the process.

Batch File? Probably but unless you've got a good reference I think some of the things will be hard to do.
Windows Scripting? This is essentially vbscript and some nice object libraries to make file manipulation easier. The problem is one of security. I wouldn't exactly want this enabled on my box. Your client may feel differently.

It is definitely a job for a scripting language though.
VB would be overkill.

 

[Howell]

Does no one think that this can be done with a batch file or Windows script?

Just relax. I like to get a complete problem definition before I go shooting off solutions.

 

[mubs]

Thank you, everybody, for your time, effort and patience!

Sechs and Will:

If you've read the problem def (as our friend Howell puts it), you'll see there's date logic and file/folder size issues involved. This is not a simple copy from one folder to another; I'd have written a script/batch file/whatever in 5 minutes to do that.

Now on to Mr. Handuin's questions:

"P" will be continuously increasing in size as an archive.

Yes.

As new files are added to "P", they will also be added to sub folders.

No! New files are only written to "P" by the app sw. Copying to sub-folders is done manually now, and is what I'm trying to automate with this post. The issue that is of most concern in manual copying is proper identification of new files by date, and proper copying to the destinations - no missed files, no duplicates.

Each sub folder has a max capacity of 640 to 690 MB.

Correct. For simplicity, assume it is one or the other.

(Temporary sub folders)
"Q" will fill up to 640-690 MB
Once the file size limit is reached in "Q", a new folder named "R" is created. As new files fill into "P" they are now copied into "R" instead of "Q".
"R" will fill up to 640-690MB and the process continues into "S"...

Think of "Q" as a root folder within which the sub-folders reside. For clarity, you may call the sub folders S1, S2, S3. Also assume that S1, S2 and S3 are already created and always exist inside "Q" (there's no point deleting and recreating them every time). When copy happens (manually or through a tool), there will be anywhere from ~600MB to ~1500MB of new files to be copied, depending on what happened inside the app sw. At start, assume S1, S2 and S3 are empty. Fill up S1 upto ~640MB, then start filling up S2 upto 640MB, then move on to S3 when S2 is full. Of course, if all the new files fit in S1, mission is accomplished, and there is no need to touch S2 or S3.

The process continues like this, or did you have a max number of temp sub folders?

We can safely assume that there will be a max of 3 sub folders.

 

[Howell]

The first thing the computer needs to do is to distinguish between old files and new files. Is it permissible to copy the files to a subdirectory of P to indicate that they have already been dealt with after they have been dealt with?

I suspect you want the old files to stay on P so that they are readily accessible for recovery purposes.

PS, Beware I think there is a quirky limit to how many files you can have in a single directory.

 

[mubs]

PS, Beware I think there is a quirky limit to how many files you can have in a single directory.

I've never heard of this one -- tell me it ain't so! There used to be such a limit to files in the root of a logical drive, but AFAIK there is none for for a folder that is in the root of a drive.

I suspect you want the old files to stay on P so that they are readily accessible for recovery purposes.

Yup.

Is it permissible to copy the files to a subdirectory of P to indicate that they have already been dealt with after they have been dealt with?

I honestly don't know; it depends on the app, and the contact for the app left that company. I could find out, but it might take a week to track down the right person and get an answer. I will also try this out when I am there next (this week-end).

The app maintains an index file that contains brief details of each data file, then expects to find the data files in a folder called "Archive" that is in the same place as the index file. So technically the app will not find the data files if they are moved to subfolders within the "Archive" folder.

But eventually things will get rather unweildly, so let's assume it ok to create sub-folders in "Archive" and move the files there. Maybe sub-folders named for the year and month, as in "2004-05". Approximately 120 files will be generated in a month, consuming ~ 3GB of space. When needed, the files can be moved back to the "Archive" folder temporarily.

[offtopic]

BTW, the data file names are generated by the app and are all gobbledy-gook - an ~30 character filename, consisting of seemingly random alphabets, dashes and digits. This is one factor that makes it hard to do things manually. Dumb-asses should have prefixed the yyyy-mm-dd in the file name; things would be a lot easier then. One more complication (a really bad one) is that the files are created and stored on the HDD in the ultrasound m/c, then transferred via LAN to this PC running the app. Ideally, the files are transferred every day, but in practice, it so happens that the transfer may be done a day or two or three later. During transfer, the files that end up on the PC have the date and time of the transfer, not the date and time of creation. So the correlation between say, the paperwork and the time-stamp of the files on the PC is lost. The index file contains the correct date, but there is no filename displayed when viewing the index file with the app, so again the specific data file one wants cannot be identified.

[/offtopic]

 

[Mercutio]

I have a set of perl scripts that move stuff around to my file shares in a fairly automated fashion, but the business rules you're dealing with are way more complicated than my set-up.

 

[blakerwry]

Is the reason for copying files to CD sized individual folders for backup purposes?

If so there is a program, "Simply safe backup 2004". They have a free edition that will backup a folder to CD sized chunks and can easily be told to backup only new/changed files since the last backup.

Even better, it can burn directly to CD for you. Some of the interesting features include encryption, compression, ECC data, as well as verifying the backup after it has completed.

 

[mubs]

Blake, that was very perceptive of you. The sub-folders will indeed be burned to CD from time to time.

Thank you for mentioning Simply Safe Backup 2004. I checked it out, and it indeed has the date criteria, which is great. But it seems to create a single file in the destination, a back-up file.

The CDs are transported to another organization for use. So the files need to be burned as individual files, not as a single back up file. For this reason, I think Simply Safe Backup 2004 will not meet the requirements.

 

[Fushigi]

Could you let SSB create the CD-sized files and then run a 'restore' scriupt to unpack them from the backup to their original file -- but in the new location?

It seems to me the file selection and CD size limits are the two 'hard' pieces of the puzzle and SSB handles those from what you folks are saying.

Note: Be sure to disable any compression schemes or the resulting restored directory will likely be too big to fit on a CD.

 

[Howell]

Could you let SSB create the CD-sized files and then run a 'restore' scriupt to unpack them from the backup to their original file -- but in the new location?

Yeah I was considering advocating using command-line winzip.

The file selection might be done by utilizing the archive bit and "xcopy /m" to a temporary location. I just did not have enough time to see if disk spanning might be used to limit the directory size.

Good find, Blake.

 

[mubs]

Sorry folks, I've been busy.

An abosulute requirement is simplicity. If ever there was a plcae that need the KISS principle like fire needs oxygen, this is it. We're talking major computer phobes here - people who don't know what Windows Explorer is, and for whom Windows for Dummies would be the worst nightmare of their lives.

I've gone there a few tmes to do this for them, and have written up a 4-page set of instructions (checkpointed as Arabic numerals within Roman numerals with aplhabetized steps). It was agony for me to write, since I had to document every click and every mouse movement -- things we do so automatically we don't even know we're doing it.

So having them use SSB would be a cure worse than the disease. I'll have to play with it to try Fushigi's idea. In the comparison matrix, it says one can "Use Batch files to automate (unattended) jobs".

I'll have to play with Winzip, likewise.

Thanks everybody.

 

[blakerwry]

SSB includes an automation wiz, which is another reason I thought it might be useful. set it and forget it.

 

[Will Rickards WT]

I started to work on a VBA solution. I'd figure I'd post it. I hit a road block in that there is no real file search capability that I could find that would give me a date option. I didn't want to code it to loop through all the files every time.

Option Explicit

Private Const cstrFolderP = "C:\Documents And Settings\williamr\My Documents\P"
Private Const cstrFolderQ = "C:\Documents And Settings\williamr\Desktop\Q"
Private Const cstrDateFile = "C:\Documents And Settings\williamr\My Documents\P\datefile.txt"

Public Function MoveNewFilesToCDBackup() As Boolean

Dim fsoObject     As Scripting.FileSystemObject
Dim fldSource     As Scripting.Folder
Dim fldTarget     As Scripting.Folder
Dim dtFrom        As Date
Dim dtThru        As Date
Dim objFileSearch As Office.FileSearch

   ' if source folder does not exist
   If Not fsoObject.FolderExists(cstrFolderP) Then
   
      ' inform user
      MsgBox "Source Folder '" & cstrFolderP & "' Was Not Found", vbExclamation + vbOKOnly, "MoveNewFilesToCDBackup"
   
      ' bail out
      Exit Function
   
   End If
   
   ' if target folder does not exist
   If Not fsoObject.FolderExists(cstrFolderQ) Then
      
      ' inform user
      MsgBox "Target Folder '" & cstrFolderQ & "' Was Not Found", vbExclamation + vbOKOnly, "MoveNewFilesToCDBackup"
   
      ' bail out
      Exit Function
      
   End If
   
   ' if both folders exist
   
   ' get reference to folders
   Set fldSource = fsoSource.GetFolder(cstrFolderP)
   Set fldTarget = fsoTarget.GetFolder(cstrFolderQ)
   
   ' now we figure out which files to take
   dtFrom = GetFileDate()
   
   ' search for files created after date
   
      ' move to target folder

   ' now that we are done write the date thru
   WriteFileDate dtThru

End Function

Public Function GetFileDate() As Date

Dim fsoObject     As Scripting.FileSystemObject
Dim txtDateFile   As Scripting.TextStream
Dim strContents   As String

   ' if date file exists
   If fsoObject.FileExists(cstrDateFile) Then
   
      ' open datefile
      Set txtDateFile = fsoObject.OpenTextFile(cstrDateFile, ForReading, False)
      
      ' read contents
      strContents = txtDateFile.ReadAll
      
      ' if the first 8 characters are a positive number
      If IsNumeric(Left(strContents, 8)) And IsNumeric(Left(strContents, 1)) Then
      
         ' get the file date
         GetFileDate = DateSerial(CLng(Left(strContents, 4)), CLng(Mid(strContents, 5, 2)), CLng(Mid(strContents, 7, 2)))
      
      ' if the first 8 characters are not a positive number
      Else
      
         ' invalid date file
         fsoObject.DeleteFile cstrDateFile, True
      
         ' use jan 1 1980
         GetFileDate = DateSerial(1980, 1, 1)
         
         ' write it to the datefile
         WriteFileDate DateSerial(1980, 1, 1)
      
      End If
      
      
   ' if date file does not exist
   Else
   
      ' use jan 1 1980
      GetFileDate = DateSerial(1980, 1, 1)
      
      ' write it to the datefile
      WriteFileDate DateSerial(1980, 1, 1)

   End If
   
End Function

Public Function WriteFileDate(ByVal dtNewerThan As Date)

   ' write it to the datefile
   Set txtDateFile = fsoObject.CreateTextFile(cstrDateFile, True, False)
   txtDateFile.Write Format(dtNewerThan, "YYYYMMDD")
   txtDateFile.Close

End Function