It is well concluded that microbial composition and diversity of coral species can be affected under temperature alterations. However, the interaction of environmental accumulation of corals and temperature stress on symbiotic Symbiodiniaceae and bacterial communities are rarely studied. In this study, two groups of soft coral Sarcophyton trocheliophorum were cultured under constant (26 °C) and inconstant (22 °C to 26 °C) temperature conditions for 30 days as control treatments. After that, water was cooled rapidly to decrease to 20 °C in 24 h. The results of diversity analysis showed that symbiotic Symbiodiniaceae and bacterial communities had a significant difference between the two accumulated groups. The principal coordinate analyses confirmed that symbiotic Symbiodiniaceae and bacterial communities of both control treatments were clustered into two groups. Our results evidenced that rapid cooling stress could not change symbiotic Symbiodiniaceae and bacterial communities’ composition. On the other hand, cooling stress could alter only bacterial communities in constant group. In conclusion, our study represents a clear relationship between environmental accumulation and the impact of short-term cooling stress in which microbial composition structure can be affected by early adaptation conditions.