The Journal of Development Economics(JDE) now offers authors the opportunity to have their prospective empirical projects reviewed and approved for publication before the empirical results are known. This pre-results review track for submissions is based on a two-stage peer review process.
In Stage 1, authors submit a proposal that includes an
introduction, methods, and data analysis plan for a prospective study (see the optional Stage 1 submission template to learn more). High-quality studies
are accepted based on pre-results review, which constitutes a commitment by the JDE to
publish the upcoming paper, regardless of the results. Authors can then collect and analyze
the data and submit a full paper for final review and publication (Stage 2). Stage 2 review
makes sure that the final paper is aligned with the research design accepted in Stage
Pre-specifying the details of a research design before data collection and analysis is central to the pre-results review (registered reports) format, pre-analysis plans (PAPs), and study (pre-)registration. What distinguishes pre-results review from the other two is that it also incorporates peer-review of the pre-specified research design at Stage 1. Following data collection and analysis, Stage 2 of peer review then ensures that research projects were implemented in alignment with their pre-specified research design, and the results of all pre-specified analyses are clearly reported.
Note that the Stage 1 proposal or the Stage 2 full paper >must not be submitted to another journal while the respective manuscript is under review at the JDE. The corresponding author(s) need to notify the JDE if they decide to submit the full paper to another journal. Should the paper be rejected by the other journal, the JDE will observe the pre-results acceptance for up to five years after it was granted.
If the full paper is accepted at another journal, the corresponding author should notify the JDE, and a brief note on the outcome will be published alongside the proposal’s summary in the JDE Catalog of Prospective Articles accepted based on pre-results review (e.g., “The proposal <title> by <authors> was accepted by the Journal of Development Economics based on pre-results review basis. The full paper based on this proposal was eventually accepted and published as <title> <citation>,<DOI>). The JDE Editorial Board also asks that working papers and published papers acknowledge that the article underwent peer review and was accepted based on pre-results basis at the JDE, citing the proposal summary in the JDE Catalog of articles accepted based on pre-results review.
The JDE does not require, but strongly encourages authors to pre-register their Stage 1 Proposals before the start of data collection. Authors can use the information from their Stage 1 Proposal submission to easily pre-register their study.
For more information, please consult the “Pre-registration Resources” section in the JDE Author Guidelines for Pre-Results Review.
Pre-results review does not restrict authors’ ability to conduct post hoc, exploratory analyses and report them in the full-length article submitted at Stage 2. However, such analyses must be clearly labeled as exploratory and distinguished from the confirmatory analyses that were specified prior to data collection and analyses.
Stage 1 Proposals will not be published as stand-alone items in the JDE following acceptance based on pre-results review. Instead, the JDE will post the title and abstract of Stage 1 Proposals accepted based on pre-results review in the Catalog of Prospective Articles.
Note that authors are required to submit the accepted Stage 1 Proposal as supporting material to their Stage 2 submissions. Stage 1 Proposals will be included in the appendix of the published article.
Yes. Authors should communicate such concerns to the Editors, and submit a version of the title and abstract where the necessary details have been masked.
During Stage 1 peer review and before data is collected, reviewers will assess whether authors have pre-specified sufficient data quality checks for accuracy, consistency, bias, and completeness. At Stage 2, reviewers will verify that pre-specified data quality checks have been met, and that data is of sufficient quality to test the pre-specified hypotheses.
In addition to pre-specified data quality checks, as part of the Stage 1 Proposal submission, authors are advised to consider potential challenges that may arise during implementation (e.g., attrition, non-compliance with treatment) and offer strategies to address them.
No, this format is not limited to any particular study design, but may be particularly well suited for RCTs. Any study design may be eligible if the data are yet to be collected/accessed and analyzed at the time of submission. This also includes quasi-experimental and observational studies (e.g., Neumark (2001), Burlig (2018), and the 2016 Election Research Pre-acceptance Competition).
Submissions where follow-up data have been collected (outside of pilot and baseline data) are not eligible for this track.
Work should be submitted at a minimum of three months, but ideally six months to a year, before data collection of the key outcome data. All participants, authors, referees and editors, must be blind to the key outcome variables at each stage of revision of the Stage 1 submission. This is useful for two reasons: (a) to maximize the possibility that feedback can be used in shaping final data collection, and (b) to ensure enough time for a revise and resubmit round before the final data come in.
(Note: it is typically quite useful to have baseline data in hand when submitting the Stage 1 Proposal for pre-results review: baseline data provides useful information on the nature of measures, including their means and variances, which can help to understand the context and carry out meaningful statistical power calculations.)
No, however authors may include pilot data in their Stage 1 submissions to establish proof of concept, effect size estimations, and/or feasibility of proposed methods.
In this approach, researchers use a training sample to run preliminary analyses that inform the Stage 1 Proposal and then refrain from analyzing the rest of the sample. After the Proposal is accepted, researchers use the remainder of the sample (testing sample) to test the pre-specified hypotheses, which constitute the main results of the published study.
Yes, this type of study design is eligible. However, at the time of the Stage 1 submission, authors will need to verifiably demonstrate they have not seen the testing part of the sample before the submission date (e.g., a download receipt plus a signed statement affirming this was the first download of the data).
PAPs and/or pre-registrations can serve as a helpful base for your Stage 1 submission. Stage 1 Proposals go a step further and require that authors also outline the importance the research question in terms of its contribution or value added to the development economics literature, as in a regular full-length submission (see criterion 1 of the Stage 1 review criteria here). Stage 1 Proposals are expected to mirror full-length conventional articles, with the exception of the “Results” and “Discussion”/”Conclusion” sections. Authors may refer to the optional Stage 1 Submission Template for more guidance.
Stage 1 Proposal submissions should follow the same structure and formatting rules as standard JDE article submissions, however need to provide enough details to allow for the project to be evaluated in the light of Stage 1 assessment criteria (learn more in the JDE Author Guidelines for Pre-Results Review). For items of the Stage 1 Proposal that are not commonly found in JDE papers (e.g. ex-ante plans for data processing, variations from intended sample size, pilot data, etc.), authors can include brief summaries in the body of the paper, and include the rest as an appendix to the Stage 1 Proposal submission.
In studies where parts of an analysis depend on the results of another, authors may consider constructing a decision tree as part of their Stage 1 Proposal (e.g., “If A is observed, then we will adopt analysis A1 but if B is observed then we will adopt analysis B1”). Alternatively, a single primary outcome (or a narrow set of outcomes) can be pre-specified at Stage 1, and the remainder of the analyses may be conducted as exploratory.
Yes, however exceptions may be granted on a case-to-case basis. The JDE will not publish the resulting full-length article unless it has successfully obtained IRB approval.
Publications in this track will look like any other article published in the JDE, with two exceptions:
Here are some additional tips to consider:
Once a Stage 1 Proposal is accepted based on a pre-results review, authors will receive an official notice from the Editors.
Note that due to functionality limitations of our manuscript handling platform, this will be paired with an auto-generated “revise and resubmit” notification, though this is not the actual decision. To submit a full paper for Stage 2 review, authors should submit it by responding to the “revise and resubmit” notification, and choosing the article type from Registered Report Stage 1: Proposal to Registered Report Stage 2: Full Length Article.
No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the ‘Results’ section.
Yes, as long as such analyses are clearly caveatted (e.g. reported in a separate section) from the analyses based on the Proposal that was accepted by the JDE at Stage 1.
This may be relevant in instances where the author(s) submit a Stage 1 Proposal based on a pre-analysis plan (PAP) for a project in collaboration with an external partner (e.g., government, funding agency) that asks for particular empirical analyses (e.g., estimates over sub-sample A, given primary outcome(s), etc.), however underwent revision during Stage 1 of peer review. If partners require the original PAP as is, then those analyses can be presented in the final paper, but in addition to the analyses accepted by the JDE at Stage 1.
During Stage 1, submissions will be either 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Pre-results acceptance constitutes a commitment by the JDE to later publish the resulting full paper regardless of the nature of its empirical findings, subject to a successful Stage 2 review.
Information about the duration of peer review for full-length papers in the standard peer review track at the JDE is available here. Given the novelty of the pre-results review track, review times vary from 60 days (accounting for both rejected and accepted papers) to 115 days (accounting for accepted papers, including time to revise and resubmit) as of October, 2019. We therefore ask that authors submit a minimum of three months, but ideally six months to a year, before data collection of the key outcome data.
Once offered acceptance based on pre-results review, authors can include the article as an upcoming publication in their research portfolios with the label: “Accepted at the Journal of Development Economics based on pre-results review”, and the URL of Catalog of Prospective Articles
In March 2018, the Journal of Development Economics (JDE) began piloting Pre-results Review track (also referred to as “registered reports” in other disciplines) in collaboration with the Berkeley Initiative for Transparency in the Social Sciences (BITSS). Through this track, the JDE reviewed and accepted detailed proposals for prospective empirical projects before results were available, offering a commitment to publish the resulting papers regardless of their findings.
The motivation behind this exercise was simple: both science and policy should reward projects that ask important questions and employ sound methodology, irrespective of what the results happen to be. By shifting the bulk of the review process (including an editorial decision to accept a paper) to before any results are known, pre-results review has the potential to advance scholarly rigor and transparency by empowering authors to conduct research and report their results transparently, free from fear that certain results could render their papers “unpublishable.” Compared to other sub-disciplines within economics, development economics has long been an area for methodological innovation and rigor, from the widespread adoption of randomized controlled trials (RCTs) starting over two decades ago, to the promotion of study registration and pre-analysis plans in the past decade. Pre-results review is a natural addition to this arsenal of open science tools.
Since this was (to our knowledge) the first effort to implement pre-results review in an economics journal, we decided to introduce it as a pilot in the hope of learning about its compatibility with existing scholarly conventions in the discipline and its feasibility at the journal. A little over one year in, we reflect on the experience so far and offer our perspectives on the future of pre-results review at the JDE and in the discipline as a whole. Given the largely positive experiences so far, the JDE has decided to make pre-results review a permanent track for article submission at the journal.
This paper was submitted in the Pre-Results Review track at the Journal of Development Economics. This submissions track involves a two-stage review process where prospective research projects are reviewed and accepted before the results are known. Throughout the publishing process, we refer to these articles as “Registered Reports”, in line with the terminology adopted by other academic journals and the existing Elsevier publishing infrastructure.
At Stage 1, authors submit a Proposal, which typically contains a literature review, research question(s), hypotheses, and detailed methodological framework. Following Stage 1 peer review, high quality studies are accepted based on pre-results review, which constitutes a commitment by the JDE to later publish the full paper regardless of the nature of its results. After Stage 1, authors collect and analyze data, and then submit a full paper (including results and discussion sections) for Stage 2 review. This paper is published as long as the data collection and analysis maintain standards of quality and the study was implemented in alignment with the research design accepted at Stage 1.
Learn more in the JDE Author Guidelines for Pre-Results Review.
Assessment of Stage 1 Proposals should be based on the following criteria:
Proposals need to satisfy all of the criteria above. Note that Stage 1 Proposals should be evaluated based on the ability of the research design to effectively test the proposed hypotheses, and arguments that the study may end up with a null result will not be taken into consideration.
Based on the referee recommendations, Stage 1 Proposals will be 1) rejected; 2) returned to authors for the opportunity to revise and resubmit; or 3) accepted based on pre-results review. Learn more about Stage 1 of the review process in the JDE Author Guidelines for Pre-Results Review.
You are welcome to make recommendations that may help Proposals satisfy one or more of the Stage 1 review criteria and improve the quality of the resulting papers.
However, in some cases Stage 1 Proposals are based around work that is well underway. If for example, an author has a sample and baseline data and/or the analysis is pre-registered, it may be problematic or quite costly for the author to construct a new randomization strategy, or to significantly deviate from his/her proposed analysis. Recommendations should be in the direction of ensuring that the proposed study meets the quality standards of the JDE, however may be disregarded if the proposed study already is of satisfactory quality. The Editorial Board will decide which changes are essential for the study to be accepted based on pre-results review, and which ones are discretionary for the author(s).
Yes. Successful Proposals should be positively evaluated based on all of the Stage 1 criteria. If you find that the Stage 1 Proposal is not relevant to the field, or has a limited potential to make a contribution to the development economics literature, you are free to recommend changes or a rejection.
Stage 2 of peer review ensures that data collection and analysis maintain high standards of quality and the study was implemented in alignment with the research design accepted at Stage 1.
Assessment of Stage 2 Articles should be based on the following criteria:
Referee reports should provide an evaluation of the submission strictly based on the Stage 2 criteria above. Editorial decisions will not be based on the statistical significance, perceived importance, novelty or conclusiveness of the results of the study. This is a key feature of pre-results review. Reviewers at Stage 2 may suggest that authors report additional tests that were not pre-specified; however, authors are not obliged to complete these tests unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.
Learn more about Stage 2 of the review process in the JDE Author Guidelines for Pre-Results Review
Referees can assess the soundness and feasibility of the research design and make relevant suggestions only at Stage 1 of the peer review process. At Stage 2, the focus of the peer review process should switch to assessing the quality of implementation of the pre-specified research design based on the Stage 2 criteria.
Whereas you are free to make comments regarding the pre-specified research design in your Stage 2 referee report, these will be considered irrelevant at Stage 2, and will not influence editorial decisions.
Authors are expected to implement the study based on the research design accepted at Stage 1. However, in the case of a deviation from the pre-specified design, the Editors will strive to err in favor of the authors, as long as the deviation(s) are reported transparently and justified based on methodological and/or theoretical considerations. Referees will evaluate whether the deviations compromise the integrity of the pre-specified research design and the credibility of the findings, and ultimately whether they find the author’s justification acceptable.
Please reference the “Deviations from the Pre-specified Research Design” section in the JDE Author Guidelines for Pre-Results Review for a non-exhaustive list of deviations that may render a study ineligible to be published as part of this process. Referees are encouraged to make recommendations in this regard in their Stage 2 referee reports.
No. The results of all hypotheses pre-specified at Stage 1 must be included in the full manuscript submitted at Stage 2. In instances where a pre-specified hypothesis is subsequently shown to be logically flawed or unfounded, authors may include it in an Appendix (if particularly lengthy) or as a footnote, rather than in the “Results” section.
The following Stage 1 Proposals have been accepted based on pre-results review at the Journal of Development Economics.
The following papers were accepted after a pre-results review at the Journal of Development Economics and have subsequently been published with full results in either JDE or elsewhere.
Andrew Foster, George and Nancy Parker Professor of Economics, Professor of Health Services, Policy and Practice, and Director of the Population Studies and Training Center received his Ph.D. in Economics from the University of California at Berkeley in 1988. He is an empirical microeconomist with interests in the areas of population, environment, development, and health. Recent work has examined economic growth in rural India, exploring such issues as growth in the non-farm economy, the effects of local democratization, groundwater usage, forest cover, household structure, inequality, and schooling. He also is exploring the effects of recent changes in air quality in Delhi. Foster also has a series of projects with colleagues in the Center for Gerentology examining the market for nursing home care.
Dean Karlan is a Professor of Economics and Finance at Northwestern University. Karlan received a Ph.D. in Economics from M.I.T., an M.B.A. and an M.P.P. from the University of Chicago, and a B.A. in International Affairs from the University of Virginia. His research focuses on microeconomic issues of poverty, typically employing experimental methodologies and behavioral economics insights to examine what works, what does not, and why in interventions in sustainable income generation for those in poverty, household and entrepreneurial finance, health behavior, and charitable giving. He works on issues for low-income households in both developing countries and the United States. Karlan is Founder of Innovations for Poverty Action and is on the Executive Committee of the Board of Directors of the M.I.T. Jameel Poverty Action Lab.