Information retrieval systems evaluation help pls

[Ezteq]

hay kids,

I'm currently working on a module about information retrieval and for my first assignment I have to evaluate some IR Systems, I'm not sure what the exact systems will be but they will consist of search engines, metasearch engines, subject portals, databases, OPACs, Inet directories and social tagging IR systems.

I was wondering, and hoping tbh, if any of you have had to acquire a new IR system for your work and if you undertook any evaluation tests...and if you did would you mind telling me what the tests were and how to do them?

It would be for a specific user group that I have to choose so am already on to finding out about specific IR systems I just want some ideas about evaluation methods and testing (anything on recall and precision testing would be aces tbh) also any tests anyone may have done that did not work so well would also be useful.

Ok many thanks I'll let you all get back to looking at porn and eating wotsits now,

xxx

 

[Madmaxx]

hay kids,

I'm currently working on a module about information retrieval and for my first assignment I have to evaluate some IR Systems, I'm not sure what the exact systems will be but they will consist of search engines, metasearch engines, subject portals, databases, OPACs, Inet directories and social tagging IR systems.

I was wondering, and hoping tbh, if any of you have had to acquire a new IR system for your work and if you undertook any evaluation tests...and if you did would you mind telling me what the tests were and how to do them?

It would be for a specific user group that I have to choose so am already on to finding out about specific IR systems I just want some ideas about evaluation methods and testing (anything on recall and precision testing would be aces tbh) also any tests anyone may have done that did not work so well would also be useful.

Ok many thanks I'll let you all get back to looking at porn and eating wotsits now,

xxx

i didnt read the whole post as i dunno what your on about... but I saw wotsits and I could really eat some i havent had em in ages

 

[Cerb]

right im not sure about exaclty what thses retrival systems are but im getting the impressionn they are like databases where like lots of journals are held on specific topics? if so ive used alot of them recently for college but im not sure exavtly how i can help? if i can i will!

 

[Ezteq]

lol everyone knows what information retrieval systems are...think about the name:

basically they are things that help you find things, so google and yahoo are IR systems, a phone book is an IR system. Sorry I am so used to everyone on here knowing more about everything than me I thought you'd all know what I was on about (wow the day came when I...me! blinded someone with science!!, ha!! whoda thought?).

Anyway I'm kind of after some evaluation strategies that have actually been implemented because I want to make sure that what I am doing will work so I am hoping to hear from people who have actually evaluated IR systems.

Though any ideas are welcome I mean how many heads do we have on F:hot? thats gotta be better than just my empty melon.

 

[tris-]

I know precisely fuck all about the particulars of an IRS, but I can help more generally onw what exactly is meant by evaluating something if thats any use.

 

[Ezteq]

yep, evaluation strategies are welcome too please.

I need to get some really good figures and stuff to demonstrate exactly why the IRS of my choice is more suitable for my chosen user group than the others I test. So things like testing the range of information held and the ability to access the correct information easily (or would you need specialised searching like boolean etc etc), things like user frendliness, accessibility options anything and any methods of presenting evaluation results would be excellent.

it has all got to go into a formal report when compiled so plenty of time, I am aiming to have it roughed and ready for polishing by say a month

 

[tris-]

Im not sure I know strategies.

If you really want to push the boat out and suprise them, can you find any previous research on this topic? Then you could find out the methods other people used, analyse them and choose the 'best' one to carry out your own research. Or go even further, and find a way that you could modify someone elses strategy to show how other research methods have been lacking. But all this maybe more in depth than you need and could take several thousand words to do propley!

 

[Ezteq]

nope Tris, thats not pushing the boat out that is what we are supposed to do anyway lol and tis what I was lanning to do.

Tbh I want to go as in depth as I can without actually assimilating myself into whatever I'm going in depth into!

 

[tris-]

What is the word limit though?
youre nearly on to a dissertation with that route.

 

[Ezteq]

lol its only like 2.5k but I can use the appendices for graphs and additional data, so I can do all the tests and stuff and just write up the results and include those in the main body, and I mean it is a report so I shant be waffling (like I do here Oo) and using unnecessary wordage.

Anyhow if anyone comes up with ideas of how to efficiently perform an evaluation please let me know.

 

[tris-]

lol its only like 2.5k .

.

Thats what I mean. To do what Im saying above might take 6-8k words.

 

[Bugz]

I'm not sure how fitting it would be for this, but a SWOT analysis or indeed an adapted type of it could be investigated. Even if you don't follow it as a framework but make reference to it. All evaluations need to include the strengths, weaknesses, opportunities and threats BUT the real evaluative skill comes from making judgments based on the information you have at hand, the information you don't have at hand and any possibilities that could arise.

 

[Ezteq]

the report criterion are that the systems must meet the needs of the user group and the areas to be evaluated are efficiency, support, management and cost effectiveness... all I have to do is come up with ways to evaluate these aspects though I shall probably be looking deeper into them but i really don't think the word count is a problem, after all thats one of the ways they mark you is on the ability to actually say what needs to be said in limited space.

Bugz sweetie can you explain SWOT for me please? Remember use Ezeese because I probably won't understand it lol

 

[tris-]

I understand that. What im talking about is carrying out a literature review in relation to the past research, whcih i dont think you can do in a few 100 words, no matter how succinct you are

 

[Ezteq]

oh god no i dont have to do a whole review of it (possibly I could do it in the word count if I resorted to morse ode though...) all I have to do is demonstrate that I have read widely upon the subject by referencing certain relevant bits in the main body and bibliography lol

[edit Morse ode is the poetical version of morse code... i.e. shall I compare thee to a summers dot dot dash dash dotdotdot]

 

[tris-]

AH HA.
Well what Id say there then is still to be critical of other peoples methods and compare them to each other to show where one did something different etc.
If that helps, but you might already know that!

 

[Ezteq]

I might, but what I don't know and where you professional student types can help is if there are any actual methods/mechanisms for doing tests while comparing/contrasting the works of others on the same topics?

So like if you had to do an evaluation how would you fit in the works of others so that it was also linked to an evaluation method (trying to think of ways to kill two birds with one stone so that I don't have to d seperate bits on the methods of others and how the same evaluation worked for my chosen systems).


Oooh also, if anyone can give me any tips or pointers or anything on report writing I'd love it, reports are my kryptonite atm, I've only done about 3 previously so any advice is desperately needed


many taas

 

[tris-]

What format is the report: scientific, something else? I have dozens of reports of different kinds including my dissertation I could email to you. Depending on the type of report, you might need to get technical with how it flows, with the linking of paragraphs and keeping a theme running throughout etc.

Im not sure i understand your first quetion. You would usually use other peoples methods to inform your own method of a test. You find out what other people did, compare their methods and make an argument about how you will do it is better or more suitable than how it has been done before. Or even why Smiths* method is better than Jones method for your particular situation. Then your discussion will be talking about how sucesfull your method was etc.

unless of course you can make one up completley from scratch that is better than other people who have published work, then your talking clever stuff

*names i made up.

 

[Bugz]

A SWOT analysis basically incorporates four different areas: strengths, weaknesses, opportunities, threats and sub-divides them further into 'external' and 'internal' - although a lot of models do away with internal & external unless they are based on business management (which is the primary use of the model).

Upon looking at what you are trying to evaluate, a SWOT analysis would seem the perfect idea. It will also make cross comparison between systems so much easier; i.e. the strengths of project 1 are far more valid than those of project 2 but the weakness of so and so in project 1 nullifies vastly any advantage it may present overall against project 2. This is the beauty of a SWOT analysis: you are stating each and every possible strength weakness etc and then you can make as many judgment-calls as you want.

I would also suggest using more than one evaluative method but with a word count of 4.5k, as Tris said, the area of evaluation is so massive that really, I can't see you fitting it all in.

However, I would contact your tutor to ask if such an approach is allowed as my knowledge of using SWOT analysis at dissertation level is nill.

 

[Bugz]

The one thing that bugs me the most about all this Ez is your use of 'evaluative methods'

What exactly do you mean by that?

 

[Cerb]

Well in th last week or so Ez i had to do a literature review for my dissertation proposal and it was 3-4k words. The thing is in the end i had to just decide for myself that i really was going to limit myself to the amount of sources i used to 5 because no matter how concise i had to be about each they still ended up needing atleast 600-800 words each and that was before id done an intro/methods. not sure why im telling you this really but i guess what im saying is maybe id advise not tring to include absoloutly everything you find on your study into it even though its tempting as you think it will get you a better mark, but it could end u going against you.

As for the compare and contrast thing I always find that after you've read a few chapters/online articles or journals that you begin to see similarities and ways authors have done things differently yourself, and usually know which is better.

hope this has helped, if not sorry i waffle

 

[Access Denied]

I'm thinking utilising an .xml graph to provide a visual representation of your written report. Y'know, give each product a rating from 1-5 for each subject addressed. Something like that.

 

[Ezteq]

Tris: Module spec just says formal report, info studies is considered a social science so maybe a scientific report would be good? My tutor has expressed his liking for headings and numbering systems so I have to try and break it all down to stuff like 1.2.1 blahblah 1.2.2. blah blah etc. Any examples you would be able to email would be brilliant, many thanks.

Bugz: swot does sound like the way to go, I like the idea of breaking everything down then breaking it down some more, nice one. By evaluation methods I mean...um well...evaulation, lol I really don't know how to explain it, where you evaluate aaargh um...aw jeeze, you know when you examine something and measure its qualities to do the task in hand as opposed to the other items you are comparing it against, so if I was evaluating a chip (for example) I might use the factors of taste, heat retention, crispyness and cost.

Cerb: Thanks for that buddy, I know it is soo tempting to 'write all you know about...' but I have the brutal editing skills of a library ninja, and quite enjoy hacking out all the superflous waffle hehe.

Access Denied: Nice touch, I like your style! That'd be a great addition to the end of the report tbh.

Thx for all your help guys, if you think of anything else (anything at all) just let me know

 

[tris-]

Coudl you send me your email ad again.
I will send you my dissertation and a couple of other smaller reports.

 

[Ezteq]

i'll send you a PM with my uni email addy because that is on all day when I'm studying, thx man

 

[tris-]

ive sent 2 docs, theyre in .docx so if they dont work tell me and ill change them.

 

[Ezteq]

got em, but am on my netbook atm so wouldn't open I'll try on the PC later on and let you know if no joy, cheers for that