Inclusion Criteria:

* In Cohorts I and II, patients must have R/R AML and MDS (venetoclax naïve or venetoclax exposed)

* Relapsed AML is defined as the appearance of 5% or greater myeloblasts in the bone marrow or peripheral blood after achieving a complete remission (CR), CR with partial hematologic recovery (CRh), or CR with incomplete hematologic recovery (CRi). Patients with mutations in FLT3, IDH1 or IDH2 must have failed or been intolerant of a Food and Drug Administration (FDA) approved FLT3, IDH1 or IDH2 inhibitor before enrolling on study
* Refractory AML is defined as failure to achieve a CR, CRh, or CRi after one of the following regimens: (i) Two cycles of intensive induction chemotherapy with a cytarabine containing regimen (e.g., 7+3, mitoxantrone, etoposide, cytarabine \[MEC\], high-dose cytarabine \[HIDAC\], etc.) or, (ii) Two cycles of hypomethylating agent (HMA)/venetoclax or low-dose cytarabine (LDAC)/glasdegib or, (iii) Four cycles of HMA monotherapy
* Relapsed MDS is defined as: (i) Intermediate, high, or very high-risk disease by International Prognostic Scoring System-Revised (IPSS-R) and, (ii) Any relapse after achieving any 2023 IWG MDS defined response
* Refractory MDS is defined as: (i) Intermediate, high, or very high-risk disease by IPSS-R and \> 5% blasts in the bone marrow or peripheral blood, (ii) Failure to achieve a response (as per IWG 2006 criteria) after four cycles of HMA monotherapy, or (iii) Two cycles of HMA + venetoclax
* In Cohort III, patients must have prior untreated high-risk MDS

* MDS with \> 5% blasts in the bone marrow or peripheral blood AND
* IPSS-R high or very high-risk disease OR
* Molecular International Prognostic Scoring System (IPSS-M) high or very high-risk disease
* No prior use of deoxyribonucleic acid methyltransferase inhibitor (DNMTi) therapy with the exception of one single cycle of DNMTi. Prior use of erythropoiesis stimulating agents (ESA), thrombopoietin agonists, lenalidomide, and luspatercept are allowed
* Age ≥ 18 years. Because no dosing or adverse event data are currently available on the use of SM08502 (cirtuvivint) in combination with ASTX727 in patients \< 18 years of age, children are excluded from this study
* Eastern Cooperative Oncology Group (ECOG) performance status ≤ 3
* Total bilirubin ≤ 2 x institutional upper limit of normal (ULN) (unless bilirubin rise is due to Gilbert's syndrome or of non-hepatic origin; in that case a cut off of ≤ 4 × institutional ULN will be used)
* Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamic pyruvic transaminase \[SGPT\]) ≤ 3 x institutional ULN (unless considered due to organ involvement by the patient's myeloid malignancy; in that case a cut off of ≤ 5 x institutional ULN will be used)
* Glomerular filtration rate (GFR) ≥ 45 mL/min/1.73m\^2
* If female, patient must be either:

* Postmenopausal (surgically sterile or age \> 55 years with no menses for 12 or more months without an alternative medical cause or age equal to 55 or less with no menses for 12 or more months without an alternative medical cause and a follicle stimulating hormone \[FSH\] level \> 40 IU/L); or
* Of children bearing potential. These patients must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Patient must agree to have a negative urine or serum beta-human chorionic gonadotropin (HCG) test result during screening and repeated within 7 days prior to study drug (local labs are allowed) to be eligible
* The effects of SM08502 (cirtuvivint) and ASTX727 on the developing human fetus are unknown. For this reason, and because these agents are known to be teratogenic, women of child-bearing potential must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and during the treatment therapy. Women of childbearing age should agree to use adequate contraception for 7 months after completion of SM08502 (cirtuvivint) administration. For ASTX727, adequate contraception must continue for at least 6 months after last dose of ASTX727. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study and for the duration of the study and at least 4 months after the last dose of SM08502 (cirtuvivint) and 3 months after last dose of ASTX727. Women who are lactating must refrain from breastfeed during the study and at least for two weeks after last dose of ASTX727
* Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial
* HIV-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial
* Ability to understand and the willingness to sign a written informed consent document. Legally authorized representatives may sign and give informed consent on behalf of study participants

Exclusion Criteria:

* Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) with the exception of alopecia or abnormal blood counts
* Patients who are receiving any other investigational agents
* Systemic anti-leukemic therapy within 14 days of first day of study treatment. If on venetoclax, then a wash-out period of at least five times the half-life of the treatment. Exceptions: No wash-out required for intrathecal chemotherapy, hydroxyurea, cytarabine (Ara-C), or palliative radiation therapy to painful sites of leukemic disease
* Patient is receiving known inhibitors or activators of flavin-containing monooxygenases (FMO1 or FMO3), and these cannot be stopped at least 5 days prior to SM08502 (cirtuvivint) treatment start. Known inhibitors of FOMO are chlorpromazine and imipramine
* Patient is receiving strong inhibitors or strong inducers of CYP3A4/5 and these cannot be stopped at least 5 days prior to SM08502 (cirtuvivint) treatment start

* Strong inhibitors include grapefruit juice or grapefruit/grapefruit related citrus fruits (e.g., Seville oranges, pomelos), ketoconazole, miconazole, itraconazole, voriconazole, posaconazole, clarithromycin, telithromycin, indinavir, saquinavir, ritonavir, nelfinavir, amprenavir, fosamprenavir, nefazodone, lopinavir, troleandomycin, mibefradil, and conivaptan. Examples may be found at the FDA website. Refer to the prescribing information of concomitant medications if in doubt or consult the sponsor for guidance
* Strong inducers include phenobarbital, rifampin, phenytoin, carbamazepine, rifabutin, rifapentin, clevidipine, and St. John's Wort. Examples may be found at the FDA website. Refer to the prescribing information of concomitant medications if in doubt or consult the sponsor for guidance.
* While moderate inhibitors or moderate inducers of CYP3A4/5 are not an exclusion criterium for the trial, it is preferred that moderate inhibitors or moderate inducers of CYP3A4/5 be replaced prior to the first dose of SM08502 (cirtuvivint) and during study conduct where this is possible.

* Moderate inhibitors include erythromycin, ciprofloxacin, verapamil, diltiazem, atazanavir, fluconazole, darunavir, delavirdine, amprenavir, fosamprenavir, aprepitant, imatinib, tofisopam, and cimetidine. Examples may be found at the FDA website. Refer to the prescribing information of concomitant medications if in doubt or consult the sponsor for guidance
* Moderate inducers include bosentan, efavirenz, etravirine, modafinil, and nafcillin. Examples may be found at the FDA website. Refer to the prescribing information of concomitant medications if in doubt or consult the sponsor for guidance
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to SM08502 (cirtuvivint) or ASTX727
* Chronic, active hepatitis B virus (HBV) or hepatitis C (HCV) requiring treatment. Note: patients with serologic evidence of prior vaccination to HBV (i.e. hepatitis B surface \[HBs\] antigen negative, anti-HBs antibody positive and anti-hepatitis B core \[HBc\] antibody negative) or positive anti-HBc antibody from intravenous immunoglobulins (IVIG) may participate. Patients who had prior HCV that has been definitively treated with negative HCV viral load prior to study initiation and no evidence of cirrhosis, are allowed to participate. If there is no known history of HBV infection no HBV studies need to be obtained. If there is no known history of HCV infection, no HCV studies need to be obtained
* Patients with uncontrolled intercurrent illness or any other significant condition(s) that would make participation in this protocol unreasonably hazardous
* Pregnant and lactating women are excluded from this study because SM08502 (cirtuvivint) is a small molecule inhibitor of CLK DYRK with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with SM08502 (cirtuvivint), breastfeeding should be discontinued if the mother is treated with SM08502 (cirtuvivint). These potential risks may also apply to other agents used in this study
* Patients with acute promyelocytic leukemia
* Subject has symptomatic central nervous system (CNS) involvement with AML
* Patient has immediate life-threatening, severe complications of their myeloid malignancy such as uncontrolled bleeding and/or uncontrolled infection
* Patient has significant active cardiac disease within 6 months prior to the start of study treatment, including uncontrolled New York Heart Association (NYHA) class III or IV congestive heart failure; acute coronary syndrome (ACS); and/or stroke
* Left ventricular ejection fraction (LVEF) \< 40% by echocardiogram (ECHO) or multi-gated acquisition (MUGA) scan obtained within 30 days prior to the start of study treatment
* Patient is known to have dysphagia, short-gut syndrome, gastroparesis, or other conditions that limit the ingestion or gastrointestinal absorption of drugs administered orally. Patient needs to be able to swallow pills
* Patient has active uncontrolled systemic fungal, bacterial, or viral infection (defined as ongoing signs/symptoms related to the infection without improvement despite appropriate antibiotics, antiviral therapy, and/or other treatment)
* Subject has corrected QC (QTc) interval (i.e., Fridericia's correction \[QTcF\]) ≥ 480 ms or other factors that increase the risk of QT prolongation or arrhythmic events (e.g., heart failure, hypokalemia, family history of long QT interval syndrome) at screening