Inclusion Criteria:

* Patients 18 years of age or older with evidence of relapsed or refractory disease as defined by International Myeloma Working Group (IMWG) criteria and measurable disease as defined by any of the following:

* Serum M-protein \>= 1.0 g/dl
* Urine monoclonal protein \>= 200 mg/24h
* Involved free light chain (FLC) level \>= 10mg/dl (\>= 100mg/l) and an abnormal serum free light chain ratio (\< 0.26, or \> 1.65)
* Patients must have had at least 2 prior lines of therapy including lenalidomide, proteasome inhibitor (PI), anti-CD38 directed antibody, and BCMA-targeted therapy

* Prior elotuzumab is permitted but patients with progressive disease (PD) on elotuzumab are excluded; at least 6 months must have lapsed from prior elotuzumab exposure
* Patients must have hemoglobin \>= 7g/dL
* Absolute neutrophil count (ANC) \>= 1000/uL
* Platelets \>= 70,000/uL

* If plasma cell percentage on bone marrow biopsy aspirate or core is \> 30%, platelet requirement will be adjusted to 50,000/ul
* Total bilirubin =\< 1.5 x the upper limit of normal (ULN)
* Aspartate aminotransferase (AST)/alanine aminotransferase (ALT)/alkaline phosphatase \< 2.5 x the ULN
* Calculated creatinine clearance of \>= 45ml/min using Modification of Diet in Renal Disease (MDRD) formula
* Left ventricular ejection fraction \>= 30%; baseline echocardiography (ECHO) is not required if ECHO was done within the preceding one year and patients do not have new signs/symptoms suggestive of heart failure
* No uncontrolled arrhythmias
* No New York Heart Association class III-IV heart failure
* 12-lead electrocardiogram (ECG) with QT interval calculated by Fridericia formula (QTcF) interval of =\< 470 msec
* Patient must be able to swallow capsule or tablet
* Patients must provide informed consent
* Patients must have an Eastern Cooperative Oncology Group (ECOG) performance status (PS) score of \< 2
* Women of child bearing potential (WOCBP) must commit to either abstain continuously from heterosexual sexual intercourse or to use 2 methods of reliable birth control simultaneously. This includes one highly effective form of contraception (tubal ligation, intrauterine device \[IUD\], hormonal \[birth control pills, injections, hormonal patches, vaginal rings or implants\] or partner's vasectomy) and one additional effective contraceptive method (male latex or synthetic condom, diaphragm, or cervical cap). Contraception must begin 4 weeks prior to dosing and continue to 6 months after study treatment ending. Reliable contraception is indicated even where there has been a history of infertility, unless due to hysterectomy
* Investigators shall counsel WOCBP and male participants who are sexually active with WOCBP on the importance of pregnancy prevention and the implications of an unexpected pregnancy
* A negative pregnancy test will be required for all WOCBP within 24 hours before starting treatment drugs
* Breast feeding is not permitted
* Male patients must agree to use an adequate method of contraception (latex or synthetic condom) for the duration of the study and up to 6 months after study treatment ending
* Criteria also applies to azoospermic males
* Males should refrain from sperm donation during this time and continue for 6 months after study treatment ending

Exclusion Criteria:

* Patients with Waldenstrom macroglobulinemia, primary amyloid light chain (AL) amyloidosis, primary plasma cell leukemia, or polyneuropathy, organomegaly, endocrinopathy, monoclonal plasma cell disorder, skin changes (POEMS) syndrome

* Patients with secondary plasma cell leukemia are permitted
* Patients with peripheral neuropathy \> National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade 2, or grade 2 peripheral neuropathy with pain
* Patients receiving concurrent corticosteroids at the time protocol therapy is initiated other than for physiologic maintenance treatment
* Concurrent use of complementary or alternative medicines that would confound the interpretation of toxicities and antitumor activity of the study drugs
* Patients with history of anaphylaxis or hypersensitivity to elotuzumab, lenalidomide, or pomalidomide
* Concurrent use of strong CYP3A modulators; concurrent use of proton-pump inhibitors =\< 2 weeks prior to started CC-92480
* Unacceptable respiratory risk factors defined by any one of the following criteria:

* Chronic obstructive pulmonary disease (COPD) with a forced expiratory volume in 1 second (FEV1) less than 50% of predicted normal
* Moderate or severe persistent asthma within the past 2 years, or currently has uncontrolled asthma of any classification
* Unacceptable cardiac risk factors defined by any of the following criteria:

* Left ventricular ejection fraction \< 30%
* Complete left bundle branch, bifascicular block or clinically significant abnormal electrocardiogram (EKG) finding at screening
* A prolongation of QT interval on screening ECG as defined by repeated demonstration of a QTc interval \> 470 msec using Fridericia's QT correction formula; a family history of long QT syndrome
* Myocardial infarction within 6 months
* Unstable angina
* Patients who have received targeted or investigational agents within 2 weeks or within 5 half-lives of the agent and active metabolites (whichever is shorter) and who have not recovered from side effects of those therapies
* Patients who have undergone major surgery =\< 2 weeks prior to starting study drug or who have not recovered from the side-effects of surgery
* Patients with known positivity for human immunodeficiency virus (HIV) or hepatitis C; baseline testing for HIV and hepatitis C is not required
* Patients with active hepatitis B (defined as hepatitis B surface antigen \[HBsAg\]+); hepatitis b virus (HBV) screening is required prior to beginning therapy

* Patients with prior hepatitis B vaccine are permitted (defined as HBsAg-, hepatitis B virus surface antibody \[anti-HBs\]+, hepatitis B virus core antibody \[anti-HBc\]-)
* Non-active hepatitis B (HBsAg-, anti-HBs+, anti-HBc+) may only be enrolled following approval by the sponsor after consideration of risk of reactivation (additional screening and monitoring for hepatitis B and consultation with a liver disease specialist may be required)
* Patients with a history of another primary malignancy that is currently clinically significant or currently requires active intervention, other than non-melanoma skin cancer and carcinoma in situ of the cervix or breast, should not be enrolled
* Patients with a history of gastrointestinal surgery or other procedure that might, in the opinion of the investigator(s), interfere with the absorption or swallowing of the study drugs
* Patients with any significant history of non-compliance to medical regimens or unwilling or unable to comply with the instructions given to them by the study staff
* Any other medical condition, including mental illness or substance abuse, deemed by the investigator(s) to likely interfere with the patient's ability to sign informed consent, cooperate and participate in the study, or interfere with the interpretation of the results