For an all-in-one reference, I just recently found a textbook that I’ve found to be very satisfactory, but it does gloss over some of the finer aspects of Causality that aren’t necessarily related to ML. It’s called “Elements of Causal Inference: Foundations and Learning Algorithms” by Jonas Peters.

For Causality and Causal Inference introduction, application to stats and a comprehensive reference to the mathematics, I recommend reading “The Book of Why” by Judea Pearl, then “Causal Inference in Statistics: A Primer”, and “Causality: Models, Reasoning and Inference (2nd ed)” by the same author.

One of the main books the ML one draws its work from is “Probabilistic Reasoning in Intelligent Systems” by Judea Pearl.

Here are the links to the books:

https://www.amazon.com/Elements-Causal-Inference-Foundations-Computation-ebook/dp/B078X5GRXD/ref=mp_s_a_1_4?dchild=1&keywords=causal+inference+machine+learning&qid=1585529360&sr=8-4

https://www.amazon.com/gp/aw/d/B075CR9QBJ/ref=tmm_kin_title_0?ie=UTF8&qid=1585529211&sr=8-3

https://www.amazon.com/Causal-Inference-Statistics-Judea-Pearl-ebook/dp/B01B3P6NJM/ref=mp_s_a_1_2?dchild=1&keywords=causal+inference+in+statistics+a+primer&qid=1585529591&sprefix=causal+in&sr=8-2

https://www.amazon.com/Causality-Judea-Pearl-ebook/dp/B00AKE1VYK/ref=mp_s_a_1_2?dchild=1&keywords=causality+models%2C+reasoning%2C+and+inference&qid=1585529622&sprefix=causality&sr=8-2

https://www.amazon.com/Probabilistic-Reasoning-Intelligent-Systems-Representation-ebook/dp/B016DABXUG/ref=mp_s_a_1_1?dchild=1&keywords=probabilistic+reasoning+in+intelligent+systems&qid=1585529664&sprefix=probabilistic+reasoning+in+intelligent+systems&sr=8-1

The text by Daphne Koller I hear is also good, but haven’t looked at it myself.

Based on what you said, you’ve got more than enough mathematics under your belt to get through all of these texts. CI math isn’t really that hard actually. It’s basically just Structural Equation Models and Bayesian Networks erected on top of the very basics of Graph Theory.

I hope this helps!