More *nix gubbins.

[nath]

I'm trying to sort out some software that will convert an entire folders worth of text files (lots and lots of subdirectories) from dos text to unix (line feed?) acceptable text.

I found a nice piece of software called edit pad pro, which does just about exactly what I want to do but it doesn't do batches of these things. Only one at a time, which when you have a few thousand text files, it becomes a pain.

Anyone got any ideas?

Cheers

 

[Testin da Cable]

this uses the *nix command sed. don't know if it can help

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # sed v1.5 only

 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
 sed 's/$//'                          # method 1
 sed -n p                             # method 2

as to a program that will batch it for you....sorry

 

[xane]

You can just pipe through sed from ls.

 

[nath]

You guys have obviously mistaken me for someone who has a clue :|

edit, plus wouldn't that do the files in the directory you run it from, but not the countless sub dirs I have?

 

[Sibanac]

Originally posted by Testin da Cable
this uses the *nix command sed. don't know if it can help

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # sed v1.5 only

 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
 sed 's/$//'                          # method 1
 sed -n p                             # method 2

as to a program that will batch it for you....sorry

depending on the *nix flavour there might actualy be a script on the system already called dos2unix and unix2dos

 

[Testin da Cable]

oh aye, forgot about that can it handle recursive stuff?


edit: well, slowaris has it :/

Reformatting page. Please Wait... done

User Commands dos2unix(1)

NAME
dos2unix - convert text file from DOS format to ISO format

SYNOPSIS
dos2unix [ -ascii ] [ -iso ] [ -7 ] originalfile conver-
tedfile

DESCRIPTION
The dos2unix utility converts characters in the DOS extended
character set to the corresponding ISO standard characters.

This command can be invoked from either DOS or SunOS. How-
ever, the filenames must conform to the conventions of the
environment in which the command is invoked.

If the original file and the converted file are the same,
dos2unix will rewrite the original file after converting it.

OPTIONS
-ascii
Removes extra carriage returns and converts end of
file characters in DOS format text files to conform
to SunOS requirements.

-iso This is the default. It converts characters in the
DOS extended character set to the corresponding ISO
standard characters.

-7 Convert 8 bit DOS graphics characters to 7 bit space
characters so that SunOS can read the file.

 

[nath]

Hmm, nothing seems to do lots of files at once.

I read somewhere someone said "I just let my ftp convert it on the fly" but neither flash FXP or cute FTP do it.. anyone got any idea what might? A windows one would be good if poss

 

[Testin da Cable]

this *may* work:

put your dir with testfiles on the unixbox in a directory called (example) test. cd to one level above test and type the following command: find test/* > files
this creates a file called files containing a list of everything in test

now open up an editor and type the following:

for i in `cat files`
do
if [ -f $i ]; then dos2unix $i $i.new
else echo "skipping $i"
fi
done

save to whatever name you like and chmod it to +x or 775 or whatever. the little script takes the filename, passes it to dos2unix with all flags default and saves the output as filename.new. dunno if it will help but it's worth a shot maybe

 

[nath]

madskillz teed

Problem is, I think I need all the files sepeate, and all of them in their respective directories. I've got one directory with loads of sub dirs and in each sub dir is a list of a few files.

Thousands of files, hundred odd folders. I'm amazed it's so difficult to find a solution for this. *nix blows man!


Perhaps what *would* be handy is if you could do an LS that lists ALL files in ALL subdirectories, then you could pipe that to dos2unix, or something similar.

 

[Testin da Cable]

you're not paying attention f00

in the example...

find test/* > files

will make a recursive list of everything in the directory "test". everything.

that scriptlet will take every item in the list "files", check it to see if it's a file (and not something else like a directory) and then create a new file cleaned of dos shite, next to the original, in place, in it's directory, with the extention "new".

ph34r!

then all you have to do is figure out a way to rename all the ".new" files to their original names. since you've gone and doubted me I shan't tell you how

 

[nath]

God damnit teed, my boss is stressing :|

We've managed to convert the files now, that's all hunky dorey but how do you move then back to their original names, rather than having a bunch of copies of the files. Plz? (oh great one yadda yadda)

edit: I think we may have solved it fnar.

*me* tdc

 

[Testin da Cable]

a way to get back to the original names:

make the files list again:

find test/* > files

grab everything that ends in ".new":

cat files | grep ".new" > newfiles

new scriptlet:

for i in `cat newfiles`
do
newfilen=$i
#chop off that ".new" shite
#use safe way to rename files thanks to Sibanac
oldfilen=`echo $newfilen |  sed -e "s/\.new^//"`
#do the move
#no need to check for directories cos we lost them all making the "newfiles" file
mv $newfilen $oldfilen
done

should work. I shall bill you my consultancy fee


oO had to edit

 

[Testin da Cable]

oh, if you've gotten it already then it's cool, glad to help out
can I be OS forum mod now?

 

[Sibanac]

Originally posted by Testin da Cable

"s/.new//g"`

should work. I shall bill you my consultancy fee


oO had to edit

thats a bit dangerous isnt it ?
wont that replace all ocurances of .new in a file name ?
and . matches averything

report1new.txt.new -> report.txt ?

wouldnt s/\.new^// be better


could be wrong tho, maybe i am screwing up between diff regex engines

 

[Testin da Cable]

Originally posted by Sibanac

wouldnt s/\.new^// be better

oh aye, troo dat. that's what you get for dashing code out off the top of my head :/


/me watches nath's server crash

 

[Sibanac]

Originally posted by Testin da Cable
oh aye, troo dat. that's what you get for dashing code out off the top of my head :/


/me watches nath's server crash

hehe easy mistake to make, i was just fighting a mean regex in perl, probly the only reason i caught it

 

[Testin da Cable]

when they start getting longer than ~20 chars or so is when it really started to hurt

 

[Sibanac]

Originally posted by Testin da Cable
when they start getting longer than ~20 chars or so is when it really started to hurt

its one off them nasty url parsing ones writen by sameone who as it seems never heard of modules , and to make it wurse he used / as a delimiter so all the other slashes are escaped .... my head hurts now

 

[Testin da Cable]

ugh, that sounds nasteh

 

[nath]

Crikey, teed.. 's a bloody good thing we didn't use your renamer thing

 

[Testin da Cable]

funny man good thing indeed werd that: I was actually thinking of chopping off the last four chars of each filename when my fingers just typed that global remove_every_instance_of_new() thing. that's what you get for playing fast&loose hehe
i never do that at work naturally