Databases and Managing Tons of Files

[Guest Rustic Monk]

Hey,

Umm . . . I hate to admit short-comings. But I'm having a heck of a time managing all the poems I have for the poetry corner. It would be a lot easier if I had them in a database with some PHP to generate each poem and biography with a predefined CSS. The only problem is I never learned PHP. Does anyone here want to volunteer to help get the database setup so v.2 of the Poetry Corner can get online? Maybe someone who has poetry there?

--Gabe

.

[blue]

I wish I knew PHP and could help with online database skills. That's a hole in my skills set too... and the same problem is driving me bonkers on projects. Looks like I'm going to have to self-teach or find a college class or two for the spring semester.

[Tanuki Racoon]

Sorry. Out my skill range :(

[Trab]

What's PHP? I guess that explains MY skill level, huh?

[JamesSavik]

Gabe-

I have a similar problem but it involves pdf files. I have a huge library of ebooks. When I mean huge, I'm talking over 300 GB. In order to find anything at all, you have to have some sort of organizational scheme. Even search engines won't do the trick as the larger the textbase, the slower the search and common terms return unwieldy output.

I started off with a hierarchal scheme where major subjects got a directory/subordinate subjects got subdirectories: Astronomy & Astrophysics, Business, Classics, etc. Soon Subdirectories were in order: Astronomy & Astrophysics/reference, Business/resumes, Classics/Kant, Fiction/Tolkien

This scheme grew like weeds. Major subject heads appeared and disappeared absorbed into other subject areas. Subdirectories grew to handle various branches of subject areas.

Top Level---------------------Branch--------------------Branch

...........................................| |

...........................................| ------------------------Branch------------------Branch

...........................................|...................................|----------------------Branch

...........................................|-------------------------Branch

History------------------------Ancient---------------------Mesopotamia

| | |........................................|_________________Egypt

| | |........................................|_________________China

| | |

| | |-------------------------- Classical Period------------Greece

| |..........................................|_________________Rome

|

disregard "....", it is filer

History also organizes rather neatly by country like Russia, US or UK history or regional like History of China, History of the Middle East.

This schema allows for the construction of very complex "tree" relationships by extending the root-branch-leaf model to some depth. As this becomes even more complex with the addition of material, it soon becomes too complex for a human to track.

When the library reached ~200 Mbytes, I found it increasingly difficult to manage all of the various overlapping relationships. For instance:

• Is crystallography subordinate to or equal to mineralogy?
  
• Is mineralogy subordinate to geology?
  
• Are mineralogy and crystallography subordinate to geology?
  
• Is crystallography a more appropriate subordinate to chemistry?
  

In some cases, I used links to bridge these gaps but this was really only a band-aid.

It soon became apparent that I needed was file management software that could index everything and provide search facilities.

There are a couple of public domain programs that do a reasonable job of file management. My favorite is Avignon Concept 2008 but it is best suited for smaller textbases and personal applications. I tried a number of freware products and found that they just weren't industrial strength enough to handle 300GB and over a million files.

I ended up with a cheap commercial product called Advanced File Manager that's easy and powerful enough to do the job.

For the web side of your task, I suggest NVU that can quickly format raw ascii to prettified html.

For web work, I also suggest the Crimson Editor. It will allow you to switch from source code to browswer to see exactly how your code will appear.

Hope some or all of it helps.

[Camy]

Gabe, the Dude was looking at using Joomla for the front page of the site. It's a content management system, and might also be suitable for the poetry site too.

I'll help if I can, but it's not really up my street ;)

James is right about Crimson Editor. It's free and excellent.

Camy

[Mountain Dude]

It is truly amazing the grey matter availabe here -- have to say that I am impressed!!