I recently saw a video where Adele talked about her English teacher and it turned out while she was speaking about her, her teacher was ALSO there and she broke down crying to see the one teacher who changed her life even though she only taught Adele for a year when she was 12 years old. You may not know it, but seeing how dedicated and passionate you are teaching kids, maybe you did make a difference and time will tell. Then one day, you'll look back and say, it was worthed.
Now you are doing what's best for you as well, so don't beat yourself too hard. You've done all you can as well. Cheers and Merry Christmas!!