Thursday 6 April 2017

Summary of an Analysis of Scoring Systems for UKAC Contests

Summary of an Analysis of Scoring Systems for UKAC Contests. Issue 1.

Introduction.
A team of volunteers from both inside and outside the RSGB VHF Contest Committee have been working to analyse scoring systems for the Tuesday evening UK Activity Contests. This report gives the output from that study and is background information to feed into the RSGB VHF Contest Committee Rules Consultation for 2017. That consultation, which will follow in the next week, will provide a formal means for providing feedback.

The report is broken into two pieces - this shorter summary of the analysis and conclusions, and a much more detailed report for those people who are interested in looking at the subject in more depth.

Details of Study.
The objective we set ourselves were to come up with a scoring system which,
- Will be recognised as fair by the vast majority of entrants across a wide geography
- Is straightforward enough to be easily understood and explained (and coded)
- Works for all bands from 6m through SHF
- Works for all sections of the contest (Open, Restricted and Low Power)

The primary rule in taking part in this analysis was to put any pre-conceptions behind us, and to use the real data from the newly published open logs of 2016 to tell us what was actually happening, and what would happen if we tried different schemes.

As a result of this work, we believe that we have come up with a proposal for a fairer scoring system for these contests, but in the process we have been surprised by much of what we uncovered, and by how inaccurate some of the speculation on the behaviour of different scoring systems was.

What constitutes fairness has been a critical element of the work. We have adopted the concept of a Potential Score for a square. From the data in real logs showing what stations were active in the contest, the relative score that a station in each of the UK1 squares could achieve if they had equivalent skills, equipment and take-off is calculated. A correction is applied for different types of scoring systems (bonuses or multipliers), and then the closer that the resulting scores are across the whole UK for a particular scoring system, the fairer it is.

We haven’t tried to compensate for differences in station’s equipment, operating skill, or take-off. We don’t think that we should compensate for equipment and skill – these are hard won factors which contesters develop through experience. It would be nice to compensate for take-off, but we haven’t been able to determine a sufficiently reliable way of doing this with public domain data.

What we have tried to compensate for is the differences in the numbers of QSO partners which are accessible to a particular station to work in the contest based on their location and the actual way that activity is spread around the country.

To make sure that the theoretical analysis of this new scoring system doesn’t create unexpected results, we also rescored the real contest logs against each scoring system.
This is to make sure that, [throughout this document, where UK is used – read UK & Crown Dependencies] we’re not changing the basic structure of these highly successful and popular contests, but at the same time we are helping the most disadvantaged stations.

We repeated this analysis for each band from 6m to 23cm, and for Open, Restricted and Low Power, and in some cases for multiple months of logs.

The detailed analysis is described in detail in the main report, but a brief summary is provided here.
- The amount of activity outside the UK, and the impact on scores is much less than commonly assumed, even from the East Coast. UK activity represents 96% of the QSOs made across the UK as a whole on 2m.
- M5 is broadly more favourable to stations in the North of the country, while M7 is more favourable to stations in the South, but both schemes exhibit equivalent levels of bias when measured across the country as a whole.
- Multiplier systems generally make scoring less fair than 1 pt/km or bonus based scoring because they exaggerate the differences between high and low activity areas.
- The people who are most disadvantaged by both M5 and M7 (relative to 1 pt/km) in these contests are those in Central and Northern Scotland. By the time that you venture that far North, activity is very thin indeed.
- Bonus based scoring systems provide a means to reward the most disadvantaged stations better than multiplier based schemes.

A New Proposal – B2
Based on this analysis, we are proposing a new bonus based scheme (B2) with the first contact in each square adding a bonus of 500, 1000, or 2000 points as shown in Figure 1 below.



- We believe that this scheme results in a significantly fairer scoring system relative to either M5 or M7 across the whole of the UK.
- It significantly helps the most disadvantaged stations uplifting them by a few places in the results table
- It provides some incentive for stations to take the necessary time to complete aircraft scatter or meteor scatter contacts with the most remote stations. This won’t work for everyone, but could be a useful technique for mid-table stations to enhance their score, at the same time making the contest more interesting and rewarding for the stations on the far edges of activity.
- It does not result in disruptive changes to results across the majority of stations and so keeps the popular character of the contest
- It generates scores which are closer together across the table. We have received feedback that some entrants (particularly newcomers) are put-off by the huge numerical gap between their score and that of the leaders. This scoring system will help that.

An Alternative Proposal – M8
We have also received some feedback that entrants would like to retain a multiplier based scheme, but not M5 or M7. We have spent some time looking for a fairer system than M5 or M7 across the whole country, and propose a new option called M8 as shown below in Figure 2.



GGG
















This scheme is significantly fairer across the UK as a whole than M5 and M7, and, compared to B2, achieves similar but smaller impacts to positions of the most disadvantaged stations.

Computer Logging
One of the challenges of variable multiplier or bonus scoring systems like this is that they are not currently scored accurately during the contest by any logging packages except for MINOS. Not having the score calculated correctly during the contest doesn't stop you taking part, it just means you won't know your multiplier or bonus score accurately during the contest. After the contest, when you submit your log to the contest entry web pages, our entry system will re-calculate those multipliers or bonuses correctly. You can then use this score to populate the Claimed Score system if you like.

However, knowing your correct score during the contest could help you optimise your strategy as you go along, although this can be achieved quite effectively just by making paper notes.

The results of a recent mini-survey suggest that this is not an issue for the majority of our entrants.
We will discuss how to get wider software support with the authors.

We think these scoring systems work well to at least 23cm, but not at SHF with very low QSO totals and short ranges. We are recommending 1 point/km scoring for the SHF UKACs.

Conclusions
We believe that the B2 proposal outlined here is a very strong candidate for a new UKAC scoring system from 6m through to 23cm. It appears to provide a fairer scoring system across the whole UK than either the existing M5 or M7 mechanisms, without changing the fundamental character of a set of remarkably successful contests. It should also make contesting more enjoyable for those in remote areas.

This should result in maintaining and encouraging higher VHF activity levels in these areas.
We have also offered an alternative multiplier based scheme called M8. This is less effective at achieving the goals of uplifting the positions of the most disadvantaged stations although it achieves this to some degree, but is significantly more even handed across the UK as a whole than M5 or M7.

We don’t believe that the lack of support for real-time scoring for B2 or M8 in all logging software should be a block to adoption of this scoring system and will be approaching software authors to provide support for these scoring systems.

From http://www.rsgbcc.org/vhf/

Tim M0BEW.