Plan to close out as many pairs from the last contest as possible, pave the way for new contests
Thread poster: Jared Tabor
Jared Tabor
Jared Tabor
Local time: 04:36
SITE STAFF
Sep 22, 2017

Hello all,

First of all, another and much-belated apology on my part for the failure to adequately close out the last translation contest in a reasonable amount of time. I know a lot of people put time into it, to varying degrees of satisfaction. I put a lot of time into it as well, had a lot of fun in the five contests I was able to oversee, and I would like for there to be more contests.

Quite a number of language pairs were stuck in the Qualification phase, without f
... See more
Hello all,

First of all, another and much-belated apology on my part for the failure to adequately close out the last translation contest in a reasonable amount of time. I know a lot of people put time into it, to varying degrees of satisfaction. I put a lot of time into it as well, had a lot of fun in the five contests I was able to oversee, and I would like for there to be more contests.

Quite a number of language pairs were stuck in the Qualification phase, without finalists being defined, and another large number were stuck in Finals, where no clear winning entry had been determined.

Here is the plan, if you're with me:

  1. The contest Celebrations has been reopened in pairs where no finalists or winners were determined.
  2. A new promotional "push" will be made, on and off ProZ.com, to attempt to get as many new raters and voters in those pairs, so that finalists can be determined where needed, and winners determined from finalists.
  3. New contests will be held.


Tatiana and Enrique, two of the newer additions to the ProZ.com team, will be helping me promote rating and voting to close as many language pairs as possible, and they will head up the new contests (I'll still be poking around too, because it's fun). Next week they will post more details about the next steps.


One thing to keep in mind moving forward, with this and with future contests: ProZ.com contests are peer-based. This means that competition in a language pair can only go from Qualifications to Finals, and from Finals to a winning entry, if there is sufficient participation from contest participants. It is possible that, in pairs where this level of participation cannot be reached, competition remains open forever, or until peers come along to rate, comment, vote. So if you have made an entry in a pair and have not rated or voted, please consider doing so.

Thanks to everyone who has participated so far, for the patience many of you have had to have over this last contest, and thanks in advance to those who rate, comment and vote to help close out the remaining language pairs.

Jared
Collapse


 
Gerard de Noord
Gerard de Noord  Identity Verified
France
Local time: 08:36
Member (2003)
English to Dutch
+ ...
Just a tip Sep 22, 2017

Hello Jared,

Moving forward, please bear in mind that competitors often don't want to judge other entries to keep the competition fair. Urge other members to vote but not the competitors.

Cheers,
Gerard


Verónica Girbau (She/Her)
Thomas T. Frost
 
Samuel Murray
Samuel Murray  Identity Verified
Netherlands
Local time: 08:36
Member (2006)
English to Afrikaans
+ ...
@Jared Sep 22, 2017

Jared Tabor wrote:
One thing to keep in mind moving forward, with this and with future contests: ProZ.com contests are peer-based. This means that competition in a language pair can only go from Qualifications to Finals, and from Finals to a winning entry, if there is sufficient participation from contest participants.


Yes, but the "sufficient participation" thresholds are arbitrary and are set by ProZ.com staff.

Clearly the thresholds (which are decided upon by ProZ.com itself) are too high for many language combinations. You should tell participants of future competitions what the thresholds are so that they can decide whether or not to bother participating if they suspect that their language pair will unlikely reach the threshold.

It is possible that, in pairs where this level of participation cannot be reached, competition remains open forever...


If that is so, then the competition is poorly designed. The competition should be able to reach its end regardless of the number of voters or entrants.


 
Maria Teresa Borges de Almeida
Maria Teresa Borges de Almeida  Identity Verified
Portugal
Local time: 07:36
Member (2007)
English to Portuguese
+ ...
@Gerard Sep 22, 2017

Totally agree! As someone directly involved in the contest I do ask myself: Am I impartial enough to be fair to other people’s entries?

Thomas T. Frost
 
Claudia Cherici
Claudia Cherici  Identity Verified
Italy
Local time: 08:36
Member (2010)
English to Italian
+ ...
streamline voting procedure Sep 23, 2017

Hi Jared
Thanks for your information. My advice for the next contests is please streamline the voting process. This is my old forum comment in which I explained what happened in my language pair (other translators agreed):

"Congratulations to the winner in my language pair, English to Italian, a wonderfully competent translator. It is, however, a bit baffling to see that my translation was 4th, despite having the highest overall rating: clearly, voters who have already express
... See more
Hi Jared
Thanks for your information. My advice for the next contests is please streamline the voting process. This is my old forum comment in which I explained what happened in my language pair (other translators agreed):

"Congratulations to the winner in my language pair, English to Italian, a wonderfully competent translator. It is, however, a bit baffling to see that my translation was 4th, despite having the highest overall rating: clearly, voters who have already expressed their preference by rating the texts don't feel, or don't know, that they then should go back to actually vote for an entry. And why should they, life's too short, can't blame them. I think this is a big flaw in the system: expecting peers to rate as many texts are possible, in quite a detailed manner, and then go back and vote again for finalists makes the procedure unnecessarily complicated and should be corrected, IMO"
Collapse


Mina Chen
Thomas T. Frost
 
Riccardo Schiaffino
Riccardo Schiaffino  Identity Verified
United States
Local time: 01:36
Member (2003)
English to Italian
+ ...
Very poorly designed Sep 23, 2017

Samuel Murray wrote:

It is possible that, in pairs where this level of participation cannot be reached, competition remains open forever...


If that is so, then the competition is poorly designed. The competition should be able to reach its end regardless of the number of voters or entrants.


I feel that these competitions have been designed very poorly. I was rather keen on them at the beginning, ages ago: participated a couple of times, devoted the time necessary to judge them several other times. Then nothing seemed to happen. Forever. Life is too short to waste any more time on them.

Just declare all of them closed, apologize to the participants, and move on. Don't start any new contests until you have a good plan in place to declare a winner within a specific amount of time, no matter how many participants or judges you have or don't have.


Thomas T. Frost
 
Mirko Mainardi
Mirko Mainardi  Identity Verified
Italy
Local time: 08:36
Member
English to Italian
Polls Sep 24, 2017

Jared Tabor wrote:

This means that competition in a language pair can only go from Qualifications to Finals, and from Finals to a winning entry, if there is sufficient participation from contest participants. It is possible that, in pairs where this level of participation cannot be reached, competition remains open forever, or until peers come along to rate, comment, vote.


Perhaps, when organizing a contest, you could use a poll (e.g. SurveyMonkey, etc.) to gauge potential interest in the various language pairs. E.g. 'Would you be interested in submitting an entry in...', 'Would you be interested in evaluating submissions in...', etc. This way you would at least have an overall picture of the situation and be able to better decide what pairs to include in the competition, based on the chance of actually having a winner...


Verónica Girbau (She/Her)
 
Katalin Horváth McClure
Katalin Horváth McClure  Identity Verified
United States
Local time: 03:36
Member (2002)
English to Hungarian
+ ...
Agree Sep 25, 2017

Samuel Murray wrote:
If that is so, then the competition is poorly designed. The competition should be able to reach its end regardless of the number of voters or entrants.


Riccardo Schiaffino wrote:
Just declare all of them closed, apologize to the participants, and move on. Don't start any new contests until you have a good plan in place to declare a winner within a specific amount of time, no matter how many participants or judges you have or don't have.


 
Jared Tabor
Jared Tabor
Local time: 04:36
SITE STAFF
TOPIC STARTER
I understand the hesitation of some participants to also rate and vote, but Sep 25, 2017

Hi Gerard,

Gerard de Noord wrote:

Hello Jared,

Moving forward, please bear in mind that competitors often don't want to judge other entries to keep the competition fair. Urge other members to vote but not the competitors.

Cheers,
Gerard


This has been an issue. I don't think it is the main obstacle to closing out a contest, though. I understand the hesitation or refusal to participate in rating/voting if one has submitted an entry. However, I feel it is important to point out (and perhaps better communicate moving forward) that, while every vote counts, a competitor's rating or voting does not negatively affect the fairness of the contest. A single person's vote will not swing the contest for or against their own entry unless it is backed by other people's votes, and over five contests what I have seen is that entry-makers who also rate and vote tend to do so responsibly, and are often able to evaluate and contribute thoughtfully and on a different level, having also studied the text and struggled with translating the same turns of phrase as the other competitors have done.


 
Jared Tabor
Jared Tabor
Local time: 04:36
SITE STAFF
TOPIC STARTER
Thanks Samuel Sep 25, 2017

Hi Samuel,

Samuel Murray wrote:

Jared Tabor wrote:
One thing to keep in mind moving forward, with this and with future contests: ProZ.com contests are peer-based. This means that competition in a language pair can only go from Qualifications to Finals, and from Finals to a winning entry, if there is sufficient participation from contest participants.


Yes, but the "sufficient participation" thresholds are arbitrary and are set by ProZ.com staff.

Clearly the thresholds (which are decided upon by ProZ.com itself) are too high for many language combinations. You should tell participants of future competitions what the thresholds are so that they can decide whether or not to bother participating if they suspect that their language pair will unlikely reach the threshold.

It is possible that, in pairs where this level of participation cannot be reached, competition remains open forever...


If that is so, then the competition is poorly designed. The competition should be able to reach its end regardless of the number of voters or entrants.


My take on the Celebrations contest is that two main things happened.

1) We may have reached a different threshold, where the number of participants was greater than the number of people willing to rate and vote, aggravated by the tendency for entry-makers to not want to rate/vote. The contest had just over 2,000 participants, which was more than double the number of participants in any of the previous contests. 1,922 people put in ratings, and just under 900 people cast votes.

2) I feel like I did an excellent job of promoting for the submissions phase, but failed to promote participation in Qualifications and Finals to a similar degree, which may have made the difference for at least some pairs, if not all of those that remained open.

Note that currently, at the submissions level, the platform will inform an entry-maker if the language pair they are creating/entering did not reach Qualifications or Finals in a previous contest.


 
Jared Tabor
Jared Tabor
Local time: 04:36
SITE STAFF
TOPIC STARTER
Rating vs Voting phases Sep 25, 2017

Hi Claudia,

Claudia Cherici wrote:

Hi Jared
Thanks for your information. My advice for the next contests is please streamline the voting process. This is my old forum comment in which I explained what happened in my language pair (other translators agreed):

"Congratulations to the winner in my language pair, English to Italian, a wonderfully competent translator. It is, however, a bit baffling to see that my translation was 4th, despite having the highest overall rating: clearly, voters who have already expressed their preference by rating the texts don't feel, or don't know, that they then should go back to actually vote for an entry. And why should they, life's too short, can't blame them. I think this is a big flaw in the system: expecting peers to rate as many texts are possible, in quite a detailed manner, and then go back and vote again for finalists makes the procedure unnecessarily complicated and should be corrected, IMO"


Interesting. There may be improvements to make there, though I imagine there are other factors at work as well. There were 65 entries in English to Italian, so during Qualification the focus was on determining a smaller pool of entries that would go on to the finals. Seven entries made it to the finals. From what I can see, a good number of raters did return to vote in that pair, though this doesn't mean that a rater returning to vote followed up every 4 or 5 rating with a vote for first place.


 
Samuel Murray
Samuel Murray  Identity Verified
Netherlands
Local time: 08:36
Member (2006)
English to Afrikaans
+ ...
@Jared Sep 25, 2017

Jared Tabor wrote:
Samuel Murray wrote:
Jared Tabor wrote:
It is possible that, in pairs where this level of participation cannot be reached, competition remains open forever...

If that is so, then the competition is poorly designed. The competition should be able to reach its end regardless of the number of voters or entrants.

I feel like I did an excellent job of promoting for the submissions phase, but failed to promote participation in Qualifications and Finals to a similar degree, which may have made the difference...


Or not. Don't beat yourself about it. It is the design that is flawed, not the execution.

The contest had just over 2,000 participants ... [but only] 1,922 people put in ratings, and just under 900 people cast votes.


Thanks for sharing these shocking statistics. And I see that there were over 2800 entries (i.e. over 600 people participated in more than one language combination).

These figures are really counter-intuitive. One would think that it takes far more effort to do a translation than to give even a basic evaluation. One would think that participants would be eager to help the process along by providing an evaluation.

Still, it's an odd kind of punishment for participation if a language combination is stuck in a preliminary phase simply because there is insufficient participation of the right kind. My language combination has 12 entries, and as far as I can tell, only about 6 people added ratings/comments.

The question becomes: should you be allowed to change the rules without notice, if it turns out that the rules don't work? Can you (for languages that lack ratings/comments participation) change the rules so that only entries by translators who also provided ratings/comments/agrees/disagrees to at least 2 or 3 peer texts are considered for the voting round?

We may have reached a different threshold, where the number of participants was greater than the number of people willing to rate and vote, aggravated by the tendency for entry-makers to not want to rate/vote.


The forums are full of suggestions for the contests, so allow me to add some more.

Make it compulsory for entrants to perform a level 1 evaluation on at least 5 other entries. By "level 1" I mean the type of evaluation that simply separates wheat from chaff, in which things should only be marked as errors if the evaluator is fairly certain that most other translators would also consider it an error (i.e. things like (a) spelling and punctuation, (b) basic grammar and syntax, (c) omission and addition, and (d) gross or very obvious meaning errors). Entries with too many level 1 errors that more than a certain number of reviewers agree upon, are then removed from the pool.

In my own language combination I was surprised at the number of spelling errors in the entries.


 
Maria Teresa Borges de Almeida
Maria Teresa Borges de Almeida  Identity Verified
Portugal
Local time: 07:36
Member (2007)
English to Portuguese
+ ...
@Jared Sep 27, 2017

Maybe, as you say, a single person's vote will not swing the contest, but if we analyze the finalists’ entries for the pair EN-PT one just can’t help noticing a few things (please note that I’m not a finalist). For instance, how come all the entries used the same adjective “fatídico” to translate “fateful” and this adjective was marked with red in three cases and with green in one? How come all entries used the substantive “instigadores” to translate “instigators” but only... See more
Maybe, as you say, a single person's vote will not swing the contest, but if we analyze the finalists’ entries for the pair EN-PT one just can’t help noticing a few things (please note that I’m not a finalist). For instance, how come all the entries used the same adjective “fatídico” to translate “fateful” and this adjective was marked with red in three cases and with green in one? How come all entries used the substantive “instigadores” to translate “instigators” but only one is marked in green? I could go on and on…

Regards,

Teresa
Collapse


 


To report site rules violations or get help, contact a site moderator:

Moderator(s) of this forum
Lucia Leszinsky[Call to this topic]

You can also contact site staff by submitting a support request »

Plan to close out as many pairs from the last contest as possible, pave the way for new contests






Anycount & Translation Office 3000
Translation Office 3000

Translation Office 3000 is an advanced accounting tool for freelance translators and small agencies. TO3000 easily and seamlessly integrates with the business life of professional freelance translators.

More info »
CafeTran Espresso
You've never met a CAT tool this clever!

Translate faster & easier, using a sophisticated CAT tool built by a translator / developer. Accept jobs from clients who use Trados, MemoQ, Wordfast & major CAT tools. Download and start using CafeTran Espresso -- for free

Buy now! »