Transitions between self-compatibility and self-incompatibility and the evolution of reproductive isolation in the large and diverse tropical genus Dendrobium (Orchidaceae)

Background and Aims The evolution of interspecific reproductive barriers is crucial to understanding species evolution. This study examines the contribution of transitions between self-compatibility (SC) and self-incompatibility (SI) and genetic divergence in the evolution of reproductive barriers in Dendrobium, one of the largest orchid genera. Specifically, it investigates the evolution of pre- and postzygotic isolation and the effects of transitions between compatibility states on interspecific reproductive isolation within the genus. Methods The role of SC and SI changes in reproductive compatibility among species was examined using fruit set and seed viability data available in the literature from 86 species and ∼2500 hand pollinations. The evolution of SC and SI in Dendrobium species was investigated within a phylogenetic framework using internal transcribed spacer sequences available in GenBank. Key Results Based on data from crossing experiments, estimations of genetic distance and the results of a literature survey, it was found that changes in SC and SI significantly influenced the compatibility between species in interspecific crosses. The number of fruits produced was significantly higher in crosses in which self-incompatible species acted as pollen donor for self-compatible species, following the SI × SC rule. Maximum likelihood and Bayesian tests did not reject transitions from SI to SC and from SC to SI across the Dendrobium phylogeny. In addition, postzygotic isolation (embryo mortality) was found to evolve gradually with genetic divergence, in agreement with previous results observed for other plant species, including orchids. Conclusions Transitions between SC and SI and the gradual accumulation of genetic incompatibilities affecting postzygotic isolation are important mechanisms preventing gene flow among Dendrobium species, and may constitute important evolutionary processes contributing to the high levels of species diversity in this tropical orchid group.