Welcome image

ICSME/EMSE Registered Reports Track - Call for Registrations


EMSE, in conjunction with the International Conference on Software Maintenance and Evolution (ICSME), is conducting a pilot RR track. This is the second such pilot after the very successful effort at MSR.

See the associated Author's Guide. Please email the ICSME track chairs - Neil Ernst or Tim Menzies - for any questions, clarifications, or comments.

Please note: small changes to the CFP and guide may occur once the MSR pilot is concluded.

What are Registered Reports

  • Methods and proposed analyses are pre-registered and reviewed prior to research being conducted.
  • Reduce/eliminate under-powered, selectively reported, researcher-biased studies.

Two Phase Review

  • (ICSME 2020) Phase 1: Introduction, Methods (including proposed analyses), and Pilot Data (where applicable). In Principle Acceptance.
  • (EMSE) Phase 2: full study, after data collection and analysis. Results may be negative!

Final publication is straightforward if the original protocol is adhered to, regardless of positive or negative results. Additional exploratory analyses may be conducted, if they are justified. Any deviation from the protocol must be justified and logged in detail to ensure replicability. EMSE J. Editors reserve the right to tighten eligibility criteria if necessary.

Phase 1 Criteria

  • Importance of the research question(s).
  • Logic, rationale, and plausibility of the proposed hypotheses.
  • Soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
  • Clarity and degree of methodological detail for replication.
  • Will results obtained test the stated hypotheses?

Phase 2 Criteria (via https://osf.io/pukzy/)

  • Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls)
  • Whether the Introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required)
  • Whether the authors adhered precisely to the registered experimental procedures
  • Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative
  • Whether the authors’ conclusions are justified given the data

Qualitative Research and RR

  • No reason to assume pre-registration cannot be for qualitative methods such as card-sorting, grounded theory, coding, member checking etc.
  • E.g. phase 1 may include details on survey respondents, survey instrument design, data collection techniques.
  • OSF Qualitative Pre-Registration

Organizers

Program Committee

(PC members are guiding the formulation of the CFP, and review Phase 1 and 2 studies)

  1. TBD

Timeline

Date Milestone
1 June 2020 study protocols and plans due
21 June 2020 initial protocol reviews
7 July 2020 rebuttals/clarifications due
21 July 2020 In Principle Acceptance (IPA) decision notifications
7 Aug 2020 1 page summary plan / camera-ready
21 Aug 2020 Reports registered with OSF registry

Submissions

Submit via this EasyChair link. EasyChair will be used to handle Phase 1 reviews and feedback/rebuttal. EMSE's EditorialManager system will be used for the Phase 2 submissions, with OSF managing the registration. Reviews from Phase 1 will be shared with the Phase 2 reviewers.

Submission Details

Papers must strictly adhere to the two-column IEEE conference proceedings format. Please use the templates available here. LaTeX users should use the following configuration: \documentclass[conference]{IEEEtran}. Microsoft Word users should use the US Letter format template.

Follow the template requested in the author's guide to ICSME RR submissions.

Page limit is 4 pages recommended, max 6 pages including references.

Review will be unblinded or single blind. There will be a light-weight rebuttal phase, in which authors have the opportunity to clarify unclear parts of the report. However, the rebuttal is not there to make changes to the experimental design.

FAQ

Q. How will self-plagiarism be handled?
A. Self-plagiarism is where an author includes verbatim text from other, already published work. We expect this to be managed using the existing workshop/extension model; there will be sufficient new content in Phase 2 to clearly indicate this is a new piece of work.

Q. What if I publish my Phase 1 proposal, and then someone scoops me by following the protocol?
A. In practice, this seems quite uncommon. However, OSF has mechanisms to manage embargo periods, so this might be something we also consider in the future. Currently the MSR/EMSE model makes embargos impractical. However, tracks such as "new ideas" already pose this potential risk, so we don't anticipate extensive problems.

Q. How does this process deal with exploratory studies, where there is no well-defined hypothesis?
A. For now, we strongly suggest such studies target the New Ideas and Emerging Results Track of MSR: . We will focus on studies that have a clear, well-formulated hypothesis.

Q. What if my study changes as I gather data?
A. RR have flexibility to deviate from the analysis plan. However, authors will need to provide solid reasons as to why they deviated from the plan.

Other FAQs on RR in general are at the bottom of the OSF page.

Links

  1. https://cos.io/prereg/
  2. See these links