Source Credibility Rating System - Rough Draft

[cosine]

Hello, in http://www.scienceforums.net/forums/showpost.php?p=231931&postcount=18 a post about congenial debate, I said I would start a rough draft of a Source Credibility Rating System for us to use. Sources referring to anything used to support an arguement.

 

It is certainly a rough draft because I expect and encourage feedback. The goal is for when enough people post that they are satisfied with the current state of the rating system, all the edits will be gathered into a "Second Draft thread" which hopefully can be sticky-ed and used as a reference for source credibility. (Of course it is labeled Second Draft so future SFN members can look at it and add to it over time until it is deemed neccessary to create a "Third Draft thread" and so on.)

 

A - Non-controversial science journals, independently verified government reports, and first hand accounts from those involved in the discussed events rate as the best sources one can use. Perhaps a little lower but still in this rating would be widely agreed upon premium news sources such as the New York Times, or local newssources that are able to get in-depth information relevent to a mainly local topic.

 

 

B - Encyclopedias, dictionaries, Wikipedia, and books without an agenda. All of these are sturdy (with the possible exception of Wikipedia) references usually written by an accountable authority, but can be slow to update. Books may be biased, depending on who's writing them, although publishers usually require a degree of fact checking before using resources to publish. And Wikipedia, while it can be updated quikcly, is open to anyone to edit, so it can possibly be tainted, though is usually self corrected.

 

 

C - Government reports, perhaps more controversial though still somewhat sound scientific articles. Though these can easily be biased resources, these sorts of things still maintain a credibility since they did use some sort of method to arrive at their report.

 

 

D - Member hypothesi and slanted sources. While these may be used for a lead or starting point to suggest a further investigation, they do not actually constitute support for any given point. Using these to support an arguement against someone who uses more credible sources is a nuisance, and even using them as a starting point for inventigation should be used for chronicle purposes only and should be accomponied by further support you were led to.

 

 

F - Refuted sources and generalizations. Refuted sources should obviously be left out of debates, and generalizations, which are the basis of bigotry, even if based on an event, should be left out as well.

 

 

 

My own suggestions is that these could be the larger catagories, and subjectively people could add a + or - suffix to the rating to show an expression of how they feel it compares to other sources in the same catagory, while it needn't be neccessary for us to lay out formally what a + or - means, as long as the catagories are agreeable.

 

Edit: I would like to be able to edit this document directly to reflect suggestions, but I don't think I will be able to after 4 hours of the original post's posting time. So all suggestions are in subsequent posts, this post is just the original post.

[Cap'n Refsmmat]

How do you propose we would use these? Would the original poster rate their sources or the people replying to them?

[ecoli]

How do you propose we would use these? Would the original poster rate their sources or the people replying to them?

 

I think it would be the poster, to ultimately let the community know how reliable the source is.

[cosine]

How do you propose we would use these? Would the original poster rate their sources or the people replying to them?

The ultimate goal is to encourage the use of better sources. As to this question specifically it would be more likely that a reader would analyze a writer's sources this way. For instance, I cite michaelmoore's website that says bush is an alien, you may request a better source, tell me I really shouldn't be bothering you with a source, or say "yeah I've seen this on this C- source (source), but I still don't think its credible. anyone have a B or A source?" Basically in political discourse there should be a heavy emphisas on the use of sources when trying to put foward a point, especially to avoid side-tracks into source credibility (although this system may lead to questions if it is ambiguous as to which catagory a certain source lies in) and to avoid circular debates where two people drone on argueing one theory against the other using only D sources and never really getting anywhere. For example, can creationism even offer a B source? a C even?

 

I think it would be the poster' date=' to ultimately let the community know how reliable the source is.[/Quote']

 

This too, as it may be said outright in order to lend some credibility to sources. It may also be stated in response to a request. "Here's an A source demonstrating such and such." Seeing an A source should lead one to believe that there is some merit or worth to the point being put forth.

[bascule]

Well first let me say this is an awesome idea.

 

What I don't see on the list would be the sole recognized international authority on a particular subject. For example, in a recent "debate" regarding Peak Oil, I presented a press release from the International Platinum Association regarding the remaining worldwide platinum reserves. (they did cite a peer reviewed study, but for the sake of argument let's just say this is "their word")

 

This is an international consortium, not a government body, but I think it's safe to say if there's one group in the world that understands platinum it would be this one.

 

I really do get tired of "Well, this source has been wrong five times in a row so I'm just going to assume from here on out that everything on the site is wrong because I'm tired of debunking it" "That's an ad hominem argument!" type debates.

[ecoli]

Well first let me say this is an awesome idea.

 

What I don't see on the list would be the sole recognized international authority on a particular subject. For example' date=' in a recent "debate" regarding Peak Oil, I presented a press release from the International Platinum Association regarding the remaining worldwide platinum reserves. (they did cite a peer reviewed study, but for the sake of argument let's just say this is "their word")

 

This is an international consortium, not a government body, but I think it's safe to say if there's one group in the world that understands platinum it would be this one.

[/quote']

 

 

I think this would be covered under C. Because it's interpretation of data, it might still be slanted. And even if their report is still based off of peer reviewed papers, you're still going through a "middle" man to get the data. You'd have to find what papers the consortium used and get those to present then as A or B sources.

[cosine]

Well first let me say this is an awesome idea.

 

Thankyou!

 

What I don't see on the list would be the sole recognized international authority on a particular subject. For example' date=' in a recent "debate" regarding Peak Oil, I presented a press release from the International Platinum Association regarding the remaining worldwide platinum reserves. (they did cite a peer reviewed study, but for the sake of argument let's just say this is "their word")

 

This is an international consortium, not a government body, but I think it's safe to say if there's one group in the world that understands platinum it would be this one. [/quote']

 

Yes, good idea. It seems to me that something like this should be in the A catagory, like it is sort of a mix between an independently verified government report (though more like an "independent respected institution report") and like a premium news source (due to its respect and authority). But yeah, IMHO it should definitely be put on the list, in the A catagory.

 

I really do get tired of "Well, this source has been wrong five times in a row so I'm just going to assume from here on out that everything on the site is wrong because I'm tired of debunking it" "That's an ad hominem argument!" type debates.

[cosine]

I think this would be covered under C. Because it's interpretation of data, it might still be slanted. And even if their report is still based off of peer reviewed papers, you're still going through a "middle" man to get the data. You'd have to find what papers the consortium used and get those to present then as A or B sources.

This is true too, you would not want an institute with a flashy name to be used in an effort to sneak C evidence into an A credential. Such as using information from say the Heritage Foundation. As ecoli points out, the best solution to this problem is to cite the sources that the Institutes research itself uses. However, to add onto that point, it doesn't mean that the International Platinum Association should not be cited, but perhaps also suffixed with "in an study that uses A sources." If it is research that the institute itself carries out, it will be much like a premium or slanted news source, but can definitively fall into A or C depending on the existance of slant and methodology and peer-review of the institution.

[Pangloss]

I'm following this with great interest, and I also appreciate the submission/suggestion.

 

I've got a couple of thoughts that might supplement this from a logistical point of view, but I'm going to hold off commenting for a bit.

[In My Memory]

What would you rate the following sources:

 

- The Daily Show with John Stewart

 

- The O'Reilly Factor with some fat boob

 

- Blogs

[Cap'n Refsmmat]

D, all of them, unless they provide a source.

[ecoli]

D, all of them, unless they provide a source.

 

Some blogs I might move up to C-ish... depending how many sources and credentials of the author. I've seen some very intellegent blogs out there.

[Pangloss]

I'd like us to give some more thought to when and where these labels would be applied, and more importantly when (specifically) they would be expected.

[cosine]

Some blogs I might move up to C-ish... depending how many sources and credentials of the author. I've seen some very intellegent blogs out there.

Right, but the validity of their blogs usually rests on the research on which they are based. So like you and Cap'n Refsmmat said, it really depends on sources they provide. And perhaps credentials wise if the person is an authority on or involved in the matter, though this may be basis for a slant, depending on the context of the arguement.

 

For example, President Bush's blog would probably be considered slanted if used as an arguement to justify a given policy. However it would be a credible source as to, say, what color his eyes are, or for example, what the name of his dog is.

[cosine]

I'd like us to give some more thought to when and where these labels would be applied, and more importantly when (specifically) they would be expected.

When they can be brought up is at the disgresson of those discussing. However, they should be providable at any time, being that it should be a property of the source. This thread and future draft threads should be the place to work out where types of sources should lay in the SCRS (Source Credibility Rating System) if it is currently ambiguous. Though, I emphasise types since hopefully discussers of a particular topic will be able to realize amongst themselves in that thread where a source such as President Bush's blog as in the example above would lie.

 

Um, just to sum up the short answer, these ratings should be able to be provided at any time. However, it may not always be to challenge a source, it may be a request for a better source, or a comparison of sources.

 

Also, though I noted before that certain situations may have an ambiguity (no one expects SCRS to be perfect, which is why this is the rough draft of hopefully many more to come), in most cases it should be obvious as to a sources rating. Nature is a pretty A source, Encarta is a pretty B source, President Bush is a pretty C source (come on, honestly, most politicians are), Al Franken is a pretty D source, and race generalizations are a pretty F source.

[Pangloss]

When they can be brought up is at the disgresson of those discussing.

 

This is what I'd like to expand on.

[JohnB]

Sorry bascule, I missed that debate but it does bring up a point.

 

At a guess they said that there would be a shortage in the future? Such comments serve to drive up the price of a commodity and they are a consortium that exists to sell platinum aren't they? There is still the almost certainty that an "Independent" group does indeed have a motive for slanting a Press Release. Cane growers know sugar, but that mean you should believe what they say about the Sugar industry? See what I'm getting at?

 

I like the rating idea, but I would put industry releases at a C, grading down to a D. The sources used for their release would be another matter altogether.

 

I for one would like to get away from the definition "A reliable source is one that agrees with what I believe."

[ecoli]

This is what I'd like to expand on.

 

I think a source should be provided with any argument made, especially if the argument is trying to make a point about the intereptation of statistics. The peak oil debates are a good example.

 

I suppose, if the writer feels that information is common knowledge (Co2 gasses are responsible for the greenouse effect, for example) then he can leave out the source, but if anybody distrusts this information, the poster would be obligated to post a source for it.

 

 

Also, where would educational web pages be? They aren't peer reviewed, but there usually made by proffesionals in the field, so they're ususally right, IFAIK. Such sources wouldn't necessarily be A material, but I also don't think that B would apply either.

 

And also what about educational pages but not from Proffesionals? (Bob's Cell Respiration Webpage). The information in this pages is not necesarily certified, but can be verified by another source.

[bascule]

Sorry bascule' date=' I missed that debate but it does bring up a point.

 

At a guess they said that there would be a shortage in the future?[/quote']

 

No, they said "world reserves are more than adequate for expected uses"

 

http://www.platinuminfo.net/press_releases/platinum_availability_pr_030303.pdf

[cosine]

I think a source should be provided with any argument made' date=' especially if the argument is trying to make a point about the intereptation of statistics. The peak oil debates are a good example.

 

I suppose, if the writer feels that information is common knowledge (Co2 gasses are responsible for the greenouse effect, for example) then he can leave out the source, but if anybody distrusts this information, the poster would be obligated to post a source for it. [/quote']

 

I agree. If it is common knowledge, there should be some source for it.

 

Also' date=' where would educational web pages be? They aren't peer reviewed, but there usually made by proffesionals in the field, so they're ususally right, IFAIK. Such sources wouldn't necessarily be A material, but I also don't think that B would apply either.[/quote']

 

Yeah, personally it sounds like they would have the same weight as a book or encyclopedia, though it depends if they cite sources or not. I think it also depends on the nature of the institution that constructs it (are they a research university, or a Kaplan SAT review course?). For this I would think it falls into a similar catagory as bascule's question in post #5.

 

And also what about educational pages but not from Proffesionals? (Bob's Cell Respiration Webpage). The information in this pages is not necesarily certified' date=' but can be verified by another source.[/quote']

 

Well it would really depend on Bob's source cited, but if he has none, I would give him a D for possible hypothesis. If it can be verified by another source, its best to use that source instead of Bob's website.

[ecoli]

Well it would really depend on Bob's source cited, but if he has none, I would give him a D for possible hypothesis. If it can be verified by another source, its best to use that source instead of Bob's website.

 

Yeah, I just thought of the obviousness of that statement after I posted it... woops.

[JohnB]

Thanks Bascule. I assumed the opposite because nearly always when there is a press release it's along the lines of "Things are bad and going to get worse. Please give us more protection/power/money.":-) I guess I'm just a cynic.