This phase II trial tests whether adding nivolumab to the usual treatment (encorafenib and cetuximab) works better than the usual treatment alone to shrink tumors in patients with colorectal cancer that has spread to other places in the body (metastatic) or that cannot be removed by surgery (unresectable) and whose tumor has a mutation in a gene called BRAF. Encorafenib is in a class of medications called kinase inhibitors. It is used in patients whose cancer has a certain mutation (change) in the BRAF gene. It works by blocking the action of mutated BRAF that signals cancer cells to multiply. This helps to stop or slow the spread of cancer cells. Cetuximab is in a class of medications called monoclonal antibodies. It binds to a protein called EGFR, which is found on some types of cancer cells. This may help keep cancer cells from growing. Immunotherapy with monoclonal antibodies, such as nivolumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Giving nivolumab in combination with encorafenib and cetuximab may be more effective than encorafenib and cetuximab alone at stopping tumor growth and spreading in patients with metastatic or unresectable BRAF-mutant colorectal cancer.