BankNote-Net: Open Dataset for Assistive Currency Recognition

Millions of people around the world have low or no vision. Assistive software applications have been developed for a variety of day-to-day tasks, including currency recognition. To aid with this task, we present BankNote-Net, an open dataset for assistive currency recognition. The dataset consists of a total of 24,816 embeddings of banknote images captured in a variety of assistive scenarios, spanning 17 currencies and 112 denominations. These compliant embeddings were learned using supervised contrastive learning and a MobileNetV2 architecture, and they can be used to train and test specialized downstream models for any currency, including those not covered by our dataset or for which only a few real images per denomination are available (few-shot learning). We deploy a variation of this model for public use in the last version of the Seeing AI app developed by Microsoft, which has over a 100 thousand monthly active users.

If you make use of this dataset or pre-trained model in your own project, please consider referencing this GitHub repository and citing our paper:

@article{oviedoBankNote-Net2022,
  title   = {BankNote-Net: Open Dataset for Assistive Currency Recognition},
  author  = {Felipe Oviedo, Srinivas Vinnakota, Eugene Seleznev, Hemant Malhotra, Saqib Shaikh & Juan Lavista Ferres},
  journal = {https://arxiv.org/pdf/2204.03738.pdf},
  year    = {2022},
}

Data Structure

The dataset data structure consists of 256-dimensional vector embeddings with additional columns for currency, denomination and face labels, as explained in the data exploration notebook. The dataset is saved as 24,826 x 258 flat table in feather and csv file formats. Figure 1 presents some of these learned embeddings.

Figure 1: t-SNE representations of the BankNote-Net embeddings for a few selected currencies.

Setup and Dataset Usage

1. Install requirements.

   Please, use the conda environment file env.yaml to install the right dependencies.

   # Create conda environment
   conda create env -f env.yaml
   
   # Activate environment to run examples
   conda activate banknote_net
   

2. Example 1: Train a shallow classifier directly from the dataset embeddings for a currency available in the dataset. For inference, images should be encoded first using the keras MobileNet V2 pre-trained encoder model.

   Run the following file from root: train_from_embedding.py

   python src/train_from_embedding.py --currency AUD --bsize 128 --epochs 25 --dpath ./data/banknote_net.feather
   

     usage: train_from_embedding.py [-h] --currency
                                 {AUD,BRL,CAD,EUR,GBP,INR,JPY,MXN,PKR,SGD,TRY,USD,NZD,NNR,MYR,IDR,PHP}
                                 [--bsize BSIZE] [--epochs EPOCHS]
                                 [--dpath DPATH]
   
     Train model from embeddings.
   
     optional arguments:
     -h, --help            show this help message and exit
     --currency {AUD,BRL,CAD,EUR,GBP,INR,JPY,MXN,PKR,SGD,TRY,USD,NZD,NNR,MYR,IDR,PHP}, --c {AUD,BRL,CAD,EUR,GBP,INR,JPY,MXN,PKR,SGD,TRY,USD,NZD,NNR,MYR,IDR,PHP}
                             String of currency for which to train shallow
                             classifier
     --bsize BSIZE, --b BSIZE
                             Batch size for shallow classifier
     --epochs EPOCHS, --e EPOCHS
                             Number of epochs for training shallow top classifier
     --dpath DPATH, --d DPATH
                             Path to .feather BankNote Net embeddings
                         
   

3. Example 2: Train a classifier on top of the BankNote-Net pre-trained encoder model using images in a custom directory. Input images must be of size 224 x 224 pixels and have square aspect ratio. For this example, we use a couple dozen images spanning 8 classes for Swedish Krona, structured as in the example_images/SEK directory, that contains both training and validation images.

   Run the following file from root: train_custom.py

   python src/train_custom.py --bsize 4 --epochs 25 --data_path ./data/example_images/SEK/ --enc_path ./models/banknote_net_encoder.h5
   

   usage: train_custom.py [-h] [--bsize BSIZE] [--epochs EPOCHS]
                     [--data_path DATA_PATH] [--enc_path ENC_PATH]
   
   Train model from custom image folder using pre-trained BankNote-Net encoder.
   
   optional arguments:
   -h, --help            show this help message and exit
   --bsize BSIZE, --b BSIZE
                         Batch size
   --epochs EPOCHS, --e EPOCHS
                         Number of epochs for training shallow top classifier.
   --data_path DATA_PATH, --data DATA_PATH
                         Path to folder with images.
   --enc_path ENC_PATH, --enc ENC_PATH
                         Path to .h5 file of pre-trained encoder model.                       
   

4. Example 3: Perform inference using the SEK few-shot classifier of Example 2, and the validation images on example_images/SEK/val

   Run the following file from root: predict_custom.py, returns encoded predictions.

     python src/predict_custom.py --bsize 1 --data_path ./data/example_images/SEK/val/ --model_path ./src/trained_models/custom_classifier.h5
   

     usage: predict_custom.py [-h] [--bsize BSIZE] [--data_path DATA_PATH]
                             [--model_path MODEL_PATH]
   
     Perform inference using trained custom classifier.
   
     optional arguments:
     -h, --help            show this help message and exit
     --bsize BSIZE, --b BSIZE
                             Batch size
     --data_path DATA_PATH, --data DATA_PATH
                             Path to custom folder with validation images.
     --model_path MODEL_PATH, --enc MODEL_PATH
                             Path to .h5 file of trained classification model.                           
   

License for Dataset and Model

Copyright (c) Microsoft Corporation. All rights reserved.

The dataset is open for anyone to use under the CDLA-Permissive-2.0 license. The embeddings should not be used to reconstruct high resolution banknote images.

Model Loading and Trust Model

predict_custom.py and train_custom.py load Keras models from local .h5 (HDF5) files. The legacy HDF5 Keras format can embed Lambda layers, whose Python code is serialized as marshalled bytecode and executed the moment the file is deserialized by tensorflow.keras.models.load_model. Loading an arbitrary or untrusted .h5 file is therefore equivalent to running arbitrary code (CWE-502), and this holds true for the TensorFlow/h5py versions pinned in env.yaml.

To protect against this, both scripts verify the provenance of any .h5 file before it is deserialized, using src/model_security.py:

1. Exact path + digest allow-list. src/trusted_models.json maps repository-relative model paths to their SHA-256 digest. A model is only loaded if its fully-resolved path matches an entry and its SHA-256 digest matches exactly. A file is never trusted merely because it shares a filename with a trusted model.
2. No symlinks. Model paths that are symlinks are rejected outright.
3. Verify-then-load on the same bytes. The file is read into memory, hashed, and then handed to Keras as an in-memory h5py.File built from those exact verified bytes — the same bytes that were hashed are the bytes that get deserialized, removing the window for a file-swap (TOCTOU) attack between verification and loading.
4. Structural inspection. Before deserialization, the model's embedded configuration is parsed and rejected if it contains a Lambda/TFOpLambda layer, or any layer/object type outside of a small, explicit allow-list of standard Keras layers used by this repository's models. This is defense in depth, since TensorFlow 2.4.1 (pinned in env.yaml) predates any Keras safe_mode protections.
5. compile=False. Models are loaded without restoring the training configuration/optimizer state, since inference and feature-extraction don't require it.

If validation fails for any reason (unregistered path, digest mismatch, malformed manifest, symlink, disallowed layer type, etc.), the script prints a clear error and exits without ever calling load_model. There is no unsafe fallback and no opt-out flag.

The models tracked in this repository (src/trained_models/custom_classifier.h5, src/trained_models/shallow_classifier.h5, and models/banknote_net_encoder.h5) are already registered in src/trusted_models.json, so the documented usage above works out of the box.

Registering your own trained model

If you train your own model with train_custom.py and want to load it with predict_custom.py (or use it as an encoder), you must register it as trusted first:

sha256sum ./src/trained_models/<your_trained_model>.h5

Add (or update) an entry for the model's path (relative to the repository root) in src/trusted_models.json with the resulting digest, for example:

{
  "src/trained_models/<your_trained_model>.h5": {
    "sha256": "<digest from sha256sum>"
  }
}

Re-run sha256sum and update the manifest any time the model file changes; the manifest is intentionally strict and will refuse to load a model whose digest no longer matches.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.