Part 1 Inclusion Criteria:

* Age 18 years and older.
* Patients must have a prior diagnosis of one of the following:

* (Note: Mutational and cytogenetic studies performed at sites other than Dana-Farber Cancer Institute or Brigham and Women's Hospital will require review of the outside cytogenetic and/or molecular pathology reports by the overall PI.)

* High-risk MDS, which is defined as one of the following subsets:

* IPSS Intermediate-2 or higher
* Presence of a mutation in TP53
* Prescence of mutation in the RAS pathway including NRAS, KRAS, PTPN11, CBL, NF1, RIT1, FLT3, and KIT
* Therapy-related MDS
* High-risk AML, which is defined as one of the following subsets:

* AML with adverse risk disease according to ELN guidelines including one of the following features:

* a history of mutation in TP53, RUNX1, or ASXL1
* t(6;9)(p23;q34.1); DEK-NUP214
* t(v;11q23.3); KMT2A rearranged
* t(9;22)(q34.1;q11.2); BCR-ABL1
* inv(3)(q21.3q26.2) or t(3;3)(q21.3;26.2); GATA2,MECOM(EVI1)
* -5 or del(5q)
* -7
* -17/abn(17p)
* Complex karyotype
* Monosomal karyotype
* Wild-type NPM1 and FLT3-ITDhigh
* Secondary AML, which is defined as a history of antecedent hematologic disorder (an MPN or MDS), a diagnosis of therapy-related myeloid neoplasm including t-MDS and t-AML, or AML with myelodysplasia-related changes, OR
* "Secondary-type" AML, which is defined by the presence of a mutation in any of the following eight genes with high specificity for the presence of antecedent myelodysplastic syndrome including SRSF2, SF3B1, U2AF1, ZRSR2, ASXL1, EZH2, BCOR, or STAG2
* Patients with AML with evidence of measurable residual disease by multiparameter flow cytometry (≥ 0.1%) despite morphologic remission on the pre-transplant/screening bone marrow biopsy. Review required by overall PI.
* High-risk chronic myelomonocytic leukemia (CMML) or MDS/MPN unclassifiable (MDS/MPN-U), which is defined by the presence of trisomy 8, chromosome 7 abnormalities or complex karyotype (3 or more abnormalities); or by the presence of a mutation in ASXL1
* Measurable disease is not required for eligibility.
* Patient is determined to be a suitable candidate for an allo-HCT using a reduced intensity conditioning (RIC) regimen using peripheral blood stem cell as stem cell source.
* Patient must have a matched related or an 8/8 unrelated donor option for his/her allo-HCT.
* Patient must have an ECOG performance status ≤ 2
* There are no limitations or minimum on the amount of prior therapy for patient's advanced myeloid malignancy. Prior exposure to venetoclax is allowed.
* Patient must have normal organ function as defined below:

* Total bilirubin ≤ 2.0 x institutional upper limit of normal (In patients with Gilbert's Syndrome, Total Bilirubin ≥ 2.0 is permitted.)
* AST(SGOT)/ALT(SGPT) ≤ 3 × institutional upper limit of normal
* Creatinine clearance ≥ 30 mL/min using Cockcroft Gault formula
* The effects of venetoclax on the developing human fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of venetoclax administration. For women who are of child-bearing potential, they must have a negative serum beta-HCG (human chorionic gonadotropin) test to be eligible.
* Ability to understand and the willingness to sign a written informed consent document

Part 1 Exclusion Criteria:

* Patients who have had chemotherapy (with the exception of hydroxyurea and/or dexamethasone) or radiotherapy or investigational therapy within 14 days prior to starting treatment on study. Exceptions: Patients already on venetoclax therapy prior to transplant need a three day wash out prior to first treatment dose on study. Patients on BCR-ABL, IDH and FLT3 small molecule inhibitors can stay on this treatment up until 5 days prior to first treatment dose on study.
* Patients with \> 10% morphologic blasts on bone marrow biopsy if they have a diagnosis of MDS or MDS/MPN. Patients \> 5% morphologic blasts on bone marrow biopsy if they have a diagnosis of AML.
* Patients recommended to receive a myeloablative conditioning regimen prior to transplantation (since there is a known survival advantage for AML using higher intensity).
* Patients who have a history of prior allogeneic stem cell transplantation.
* Symptomatic or untreated known CNS involvement of disease
* Patients with active heart disease (New York Heart Association class 3-4 as assessed by history and physical exam, or a critical event including unstable angina/stroke/myocardial infarction within the last 6 months prior to first dose on study).
* Patients who have consumed grapefruit, grapefruit products, Seville oranges or starfruit within 3 days prior to study treatment. Patients who a strong or moderate inducer within 7 days prior to the first dose of study drug.
* Malabsorption syndrome or other clinically significant condition that would preclude enteral administration.
* Other severe acute or chronic medical or psychiatric condition or laboratory abnormality that in the opinion of the investigator may increase the risk associated with study participation or investigational product administration or may interfere with the interpretation of study results and/or would make the patient inappropriate for enrollment into this study.
* Patients with known active hepatitis B virus (HBV) infection should be excluded because of potential effects on immune function and/or drug interactions. However, if a patient has HBV history with an undetectable HBV load by polymerase chain reaction (PCR), no liver-related complications, and is on definitive HBV therapy that is not contraindicated on this study, then he/she would be eligible for study.
* Patients with known active hepatitis C virus (HCV) infection. However, if a patient with a history of HCV infection has received definitive therapy (and is now HCV viral load negative), or if a patient has a reactive HCV antibody test but has an undetectable viral load by PCR, then he/she would be eligible.
* Patients with known active HIV infection out of concern for the drug-drug interaction with venetoclax and HAART therapy.
* Pregnant or breastfeeding women or those intending to become pregnant during the study or within 3 months after the final dose of study treatment are excluded from this study. Women of childbearing potential must have a negative serum pregnancy test resulted during screening and repeated within 7 days prior to study drug (local labs are allowed).
* Vaccination with live, attenuated vaccines within 4 weeks prior to initiation of study treatment or anticipation of need for such a vaccine during the study.
* Patients with uncontrolled infection at time of first dose of treatment on study. Patients receiving anti-microbial agents including antibiotics, antiviral and antifungal therapies are allowed if hemodynamically stable.
* Part 2 and Part 3 only: Patients recommended to receive FLT3 inhibitor therapy or any other antileukemic therapy for AML as maintenance post allo-HCT.

Part 2 and Part 3 Eligibility Criteria:

* No DLT event (including delay in neutrophil engraftment) due to the addition of venetoclax to conditioning chemotherapy prior to start of maintenance therapy.
* No morphologic evidence of relapse (defined as 5% or more morphologic blasts) prior to start of maintenance therapy confirmed by bone marrow biopsy after day +28.
* ANC ≥ 1.0 K/uL without growth factor support and platelet level ≥ 50 K/uL without platelet transfusion within 7 days of starting maintenance therapy.

--Exception: Patients without morphologic evidence of disease relapse but with evidence of persistent molecular or cytogenetic residual disease at the time of assessment can start maintenance therapy as long as ANC ≥ 0.75 K/uL without growth factor support and platelet level ≥ 25 K/uL.
* Absence of overall grade II-IV acute GVHD per investigator. Upon acute GVHD resolution, patients are eligible. Patients that are on prednisone 0.5 mg/kg daily dose or lower are allowed to initiate maintenance.
* Total bilirubin less than or equal to 2 x institutional ULN (unless has known Gilbert's disease).
* AST and ALT less than or equal to 3 x ULN.
* Cr Cl ≥ 30 mL/min or higher (Cockgroft Gault formula).
* No concurrent illnesses that would prevent taking oral therapy and interfere with safety assessment.