One thing that has helped me lately with regard to the second point is acknowledging and getting more comfortable with the fact that software development is a creative activity. The quality of the product is highly dependent on many factors, including the knowledge, skills, and creative capacity of the people producing it. I've managed teams of translators and teams of UX/UI designers and understand how hard it is to drive their work through formulas and rigid process structures.
I had always thought of software development as more like electrical or mechanical engineering. If I could give the developers a spec, even in the form of a well-written user story, the implementation should be relatively straightforward. Over time my understanding of the software development process has deepened and along with it my appreciation for the craft and those who practice it.
As for the first point, my fear and lack of trust are mitigated by my own effort to reduce them along with my experience working with a team and seeing them consistently provide excellent outcomes without my interference.
I'll use a sports team analogy. I enjoy watching our local MLS club, Real Salt Lake. I watch videos of their practice sessions and listen to interviews with their coach. Then I watch them play matches. Soccer is unique in professional sports in that once the official begins the match the players and coaches have no opportunity to call time out. The coach doesn't get the chance to assemble his team during a time out or between plays to provide guidance and direction. He can only stand on the side and shout encouragement and directions. The team works out how it's going to organize itself, counter attacks from the other team and execute it's own attacks on the opponent's goal. These rules create the dynamic of constant communication among players on the field. On the best teams, players display an ability to intuit what their teammates are going to do in a given set of circumstances.
The best coaches in soccer, like some of the best managers, product owners, and scrum masters in software development, understand that excellent outcomes are the result of drills, practice, clear vision and goals, and ultimately trusting those whose responsibility it is to perform to put their best efforts into the team's success.