As a promising paradigm to collaboratively train models with decentralized data, Federated Learning (FL) can be exploited to fine-tune Large Language Models (LLMs). While LLMs correspond to huge size, the scale of the train- ing data significantly increases, which leads to tremendous amounts of computation and com- munication costs. The training data is generally non-Independent and Identically Distributed (non-IID), which requires adaptive data pro- cessing within each device. Although Low- Rank Adaptation (LoRA) can significantly re- duce the scale of parameters to update in the fine-tuning process, it still takes unaffordable time to transfer the low-rank parameters of all the layers in LLMs. In this paper, we propose a Fisher Information-based Efficient Curriculum Federated Learning framework (FibecFed) with two novel methods, i.e., adaptive federated cur- riculum learning and efficient sparse parameter update. First, we propose a fisher information- based method to adaptively sample data within each device to improve the effectiveness of the FL fine-tuning process. Second, we dynami- cally select the proper layers for global aggre- gation and sparse parameters for local update with LoRA so as to improve the efficiency of the FL fine-tuning process. Extensive experi- mental results based on 10 datasets demonstrate that FibecFed yields excellent performance (up to 45.35% in terms of accuracy) and superb fine-tuning speed (up to 98.61% faster) com- pared with 17 baseline approaches). Our code will be publicly available.