Hi,
Rather than just list a series of references, I'd like to answer your question by explaining a little of my history with math or: How I Got Addicted to Mathematics.
When I dropped out of college for the second time (no money), I eventually went to work in the public school system as an instructional aid. At that time, my math education went no further than some high school algebra—which I didn't complete—and some college algebra—which I also didn't complete. I had my own computer and had been learning to program and, of course, wanted to try making my own games. As I began reading about various graphics techniques, more and more math came up and, before long, I was at a stand-still, spending hours agonizing over single sentences, trying to understand the math. One technique involving camera positioning mentioned something called "quaternions". I actually went to the university library and checked out the original book on the subject. I didn't make it past the first few pages.
Meanwhile, at work, it quickly became apparent (even to the students) that my teaching ability far outshone the teacher's and I was not asked to help with the instruction anymore. I became quite efficient at making copies, so I found myself with lots of free time at work. Then I made one of the most important realizations of my life. I still had the text from my college algebra course and I realized: this is a book. I realized that books are made to be read—there's text in there that explains the subject. I've discovered that my school experiences differ wildly from others', so I should explain that we never read, or were assigned reading from, our math books in public school. They were simply the source of the exercises. I thought, "I don't know how far I'll be able to get on my own, but I bet I could at least get past the first chapter or two." So I opened to page 1 and started reading. I finished that book, snagged a free geometry book from a cart on its way out the door to the dumpster, worked through that, borrowed a book on trigonometry from one of the only math teachers at that school that was worth a hoot, and worked through that (I was really fast at making the copies now, so I had lots of time). This took about a year and a half, after which I joined the Navy. On a side note, towards the end, as I was studying and would try to ask for help from the teachers (it was a school after all), there came a point where one of the teachers said outright, "I don't know how to do the math you're doing. I can't help you."
Not to worry though, that was in the 1990's. I'm sure the standards have gone up since then.
My math progressed a little bit slower in the Navy. I also had a great deal of free time while out to sea, but I spent a lot of it learning to program. I taught myself calculus. I got as far as the equivalent of first semester calculus, which I CLEP'd out of and dabbled a little in other areas—probability, counting theory, number theory, linear algebra, etc. I left active duty and immediately rolled right into full-time school. First semester I took calc II—and hated it. Remember, this was the first math class that I actually completed after junior high school. I did not enjoy any of the math classes that I took while a full-time student. The pace of the courses precludes any attempt at reflection or digestion—there's no time to play with any of it. They just shove it down your throat and want you to regurgitate it at the exam. I'm so glad that I'm done with that. Ironically, I think that I'll go back for a Master's in math at some point, but first I want to spend a few years studying on my own again, if for no other reason than to remember what I like about math.
So, how would I recommend learning math? Don't take it too seriously. Do it for fun. Enjoy it (and maybe find a job where you can get paid to sit in the back of the room and study). If you go the solo route, then find resources to ask questions of, but you must also learn to self-assess. It's easy to think the words "I understand this", but it's much more difficult to actually determine whether you do or not.
Figure out what you're wanting to accomplish and why. If you want a degree in math or a job that requires such a degree, you'll have to go to school. If you simply want to understand more about math—stay as far away from school as possible (I'm only partly kidding). Here are a couple of excellent articles on math education:
Academia.edu
Either way (school or no school), you'll want a book. Mostly, the textbooks are homogenized these days, so I don't know how much it matters which one you get. Personally, what I look for in a textbook is generality. If I'm just starting a subject, I don't want something very specific or niche. I want a broad treatment. A good rule of thumb is the fewer words it takes to describe it the better. College Algebra with Applications to Social Sciences for Art Majors should probably go back on the shelf. The book that I have is below, but there is nothing special about this book aside from it being the book required for the course that I never completed (I do like the book though). Heh, I just realized that I never did take a college algebra course.
Finally, if the goal is only to be able to understand the Mandelbrot set, that is a mountain at the end of a road that is much longer than you might think. As I said, for all of my study, I can only glimpse some of the pieces; I can not claim mastery. A book you should probably check out is:
This book was my introduction to the Mandelbrot set and what inspired the first version of my plotting program.
I've taken the machine gun approach to answering your question; I've thrown lots of stuff out there. Take what you find helpful and throw the rest away.