Herbify: an ensemble deep learning framework integrating convolutional neural networks and vision transformers for precise herb identification

Farhan Sheth, Ishika Chatter, Manvendra Jasra, Gireesh Kumar, Richa Sharma

Abstract: Herbs have historically been central to medicinal practices, representing one of the earliest forms of therapeutic intervention. While synthetic drugs are often highly effective in treating acute conditions, their use is frequently accompanied by adverse side effects. In addition, the growing dependence on synthetic pharmaceuticals has raised concerns regarding affordability, thereby fostering a renewed interest in herbal medicine as a cost-effective and holistic alternative. In response to this need, the current study introduces a computer vision framework for accurate herb identification. A novel dataset, Herbify, was compiled from two different herb datasets and refined through rigorous cleaning, preprocessing, and quality control procedures. The resulting dataset underwent standardization via the Preprocessing Algorithm for Herb Detection (PAHD), producing a refined dataset of 6104 images, representing 91 distinct herb species, with an average of about 67 images per species. Utilizing transfer learning, the research harnessed pre-trained Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), then integrated these models into an ensemble framework that leverages the unique strengths of each architecture. Experimental results indicate that EfficientNet v2-Large achieved a noteworthy F₁-score of 99.13%, while the ensemble of EfficientNet v2-Large and ViT-Large/16, termed EfficientL-ViTL, attained an even higher F₁-score of 99.56%. Additionally, the research also introduces ‘Herbify’ application, an AI-driven framework designed to identify herbs using the developed model. By directly tackling the principal obstacles in herb identification, the proposed system achieves a highly accurate and operationally viable classification mechanism. The experimental outcomes showcase top-tier performance in herb identification and emphasize the transformative potential of AI-based solutions in supporting botanical applications.

External IDs:doi:10.1186/s13007-025-01421-5