The process for selecting abstracts submitted for presentation at annual scientific meetings should ensure both the quality of these meetings and fairness to prospective presenters. The aim of the present study was to review the assessment of radiation oncology abstracts submitted for oral presentation to the 2004 Royal Australian and New Zealand College of Radiologists annual scientific meeting. Selection criteria were developed that were primarily focused on the subjective aspects of abstract quality. All research abstracts were reviewed blindly by five individual reviewers (four radiation oncologists and a statistician), scoring each abstract in five categories. The scores of three reviewers were used to select the top 30 general and top eight trainee entries. For comparison, cluster analysis using the scores of all five reviewers was used to group papers into two ranks. There was a strong correlation in total scores for each paper, between all reviewers. Similarly, the study design subscale was strongly correlated between all reviewers. Abstracts belonging to the first-rank cluster were generally selected. Most trainee entries would have been successful in being accepted into the general programme. The selection process described appears feasible and fair and may improve the quality of meetings.