Ranking Service Historical Success

[terp]

Hundreds of players have been ranked by each of these sources.  Has anyone ever attempted to objectively evaluate the evaluators?  Is there any reason why I would be more inclined to rely on the evaluation of one versus another?  No doubt, there could be a fair amount of “play” in any attempt to evaluate the evaluators.  But has anyone tried?

[mojo1917]

 

In the age of I'm always right and everyone else is stupid this is a great question.

 

I think it would be difficult to find metrics that make the case "these guys know what they're looking at" ironclad.

 

It would need to be a very analytics heavy study for certain though.  I don't think I'd even want to read the report beyond the summary. 

 

 

[terp]

2 hours ago, mojo1917 said:

 

In the age of I'm always right and everyone else is stupid this is a great question.

 

I think it would be difficult to find metrics that make the case "these guys know what they're looking at" ironclad.

 

It would need to be a very analytics heavy study for certain though.  I don't think I'd even want to read the report beyond the summary. 

 

 

You mean….I have to do it myself?  

It might not be that bad if it is done in baby steps.  Maybe look five years of first round picks for two sources, starting 10 years ago, NHL games played, GPG, PPG, something like that. After the first round, it’s like roulette anyway and would probably look that way statistically.

[mojo1917]

@terp

If "we're" doing it, it would definitely have to be in small manageable chunks.

And I had a similar thought, even starting to look at the task that narrowly- the results would most likely appear scattershot.

 

It would be an intimidating project to take on.

 

The discrepancies between services really shone brightly this year. 

Matthew Savioe was rated really high by some services and not in the top 15 in others.

One guy looked at his skating and wrote it's not explosive, yet every other report listed skating as strength. It's tough to know who to believe. The guy with the question had specific reasons for writing what he did and explained it.

Your example of Cutter Gauthier is another perplexing kid.

Is he top 5 or not even top 20?

Such a crap shoot.

[terp]

3 hours ago, mojo1917 said:

@terp

If "we're" doing it, it would definitely have to be in small manageable chunks.

And I had a similar thought, even starting to look at the task that narrowly- the results would most likely appear scattershot.

 

It would be an intimidating project to take on.

 

The discrepancies between services really shone brightly this year. 

Matthew Savioe was rated really high by some services and not in the top 15 in others.

One guy looked at his skating and wrote it's not explosive, yet every other report listed skating as strength. It's tough to know who to believe. The guy with the question had specific reasons for writing what he did and explained it.

Your example of Cutter Gauthier is another perplexing kid.

Is he top 5 or not even top 20?

Such a crap shoot.

This is what drew my attention: a guy who is somewhere between the 3rd and 20th best prospect begs for someone to look back and see who does and doesn’t know what he’s talking about.  I get that these are 18 year old guys and by nature it is hard to predict.  But, even so, 17 places?  It means there are some significant disagreements among these people, whoever they are.