The short answer is no, or at least, it shouldn’t be. Sure, the presence of tones in the phonetic system can be confusing and the tens of thousands of characters (hanzi) in existence is certainly daunting, but it is a mistake to overly fixate on these pain points.
In truth, tones and hanzi are not really problems for learning Chinese. Firstly, tones do not actually do as much work as the textbook examples suggest they do. Certainly, it would be embarrassing to create confusion between “horse” and “mother” (ma3 and ma1, respectively), but a surprising number of Chinese syllable segments have a restricted number of tones to choose from. For example, “te” has to be “te4”, so even if your pronunciation is off, the listener will apply their implicit knowledge of Chinese phonological system and automatically correct. Once context is introduced, the inclination to listen over “mispronunciations” is even stronger. Finally, the majority of native Chinese speakers were not brought up in perfect standard Mandarin, meaning that their version of Chinese is likely to have some variations in tonal pronunciation. There is even a common saying to this point: “天不怕，地不怕，只怕广东人讲普通话:” (Tian1 bu4 pa4, di4 bu4 pa4, zhi3 pa4 guang3dong1ren2 jiang3 pu3tong1hua4). “There is nothing to fear in heaven or earth except a Cantonese speaking Mandarin.”
Regarding hanzi, though they are many and have complex internal structures, the threshold for competency has been vastly lowered in the technological age. We don’t need to overly concern ourselves with stroke order and proper calligraphy techniques when writing hanzi is a matter of hitting a few keys and selecting one from the IME provided list. Of course, one still needs to put in the effort to recognize hanzi, but to a certain degree, learning hanzi is the critical element of learning Chinese. One’s vocabulary is directly proportional to the number of hanzi in one’s tool bag. Furthermore, given the fact that Chinese does not have the inflections, tenses, and cases of other languages and that the main grammatical points can be boiled down to special cases of a number of hanzi, it is not overly reductive to equate the task of learning Chinese to learning hanzi.
So, all told, as Chinese is “simpler” than some languages and the so-called most difficult part is the essence of learning, it really isn’t a hard language to learn.
Note: Obviously the thesis “learn characters –> learn Chinese” has some logical fallacies. To get going, one needs to learn the basic patterns of sentence construction, clearly differentiate vocabulary from characters (and build up a sense of morphology), and practice speaking and listening as a separate set of skills. Nonetheless, the main effort of progressing from beginner to intermediate and to advanced is in building up a large knowledge base of characters.