AI Receipt Scanner & Expense Tracker Android App | Vietnam

AI-Powered Receipt Scanning, Multi-Currency Expense Management & Budget Analytics — Vietnam

Project Overview

We developed an Automated Smart Receipt & Expense Tracker for a Vietnam-based business that wanted to simplify how individuals and small businesses capture receipts, organize expenses, track spending, and prepare financial information for reporting.

The application transforms physical paper receipts into structured digital expense records using Android camera technology, OCR, image processing, and machine-learning-assisted document parsing.

Instead of manually entering every expense, users can photograph a receipt and allow the application to extract important information such as the merchant, date, total amount, currency, and relevant expense details.

The platform also supports multi-currency expense tracking, automated tax-reporting tags, monthly budget analytics, digital receipt storage, and local/cloud data synchronization.

The Android application was developed using Kotlin, Android CameraX, and Material Design 3. The receipt-processing pipeline uses Python, OpenCV, Tesseract OCR / ML Kit, while Firebase Functions provide server-side processing and integration. Cloud Firestore manages cloud data and SQLite through Room provides local persistence.

The core transformation was:

Paper receipt → Camera scan → OCR → AI-assisted parsing → Structured expense → Tax classification → Budget analytics.

 

---

The Business Challenge

Expense tracking is still highly manual for many individuals and businesses.

A customer may receive a paper receipt containing:

• Merchant information

• Purchase date

• Product information

• Subtotal

• Tax

• Discount

• Total

• Currency

• Payment information

Manually transferring this information into an expense tracker is repetitive and time-consuming.

The client wanted to create an application that could turn a physical receipt into a structured digital expense record with minimal user input.

The desired experience was:

Paper Receipt
      ↓
Open App
      ↓
Scan Receipt
      ↓
Image Processing
      ↓
OCR / ML
      ↓
Extract Receipt Data
      ↓
Review
      ↓
Save Expense
      ↓
Categorize
      ↓
Tax Tag
      ↓
Budget Analytics

The application also needed to work with expenses in different currencies, making it more suitable for users who travel, purchase internationally, or operate across multiple markets.

---

Project Objectives

The primary objectives were to:

1. Build a native Android expense management application.

2. Allow users to scan paper receipts using the phone camera.

3. Automatically extract receipt information.

4. Use OCR to recognize printed receipt text.

5. Apply image preprocessing to improve OCR quality.

6. Use ML-assisted parsing to identify relevant receipt fields.

7. Support multiple currencies.

8. Convert foreign expenses into a selected base currency.

9. Automatically assign expense categories.

10. Add tax-reporting tags to expenses.

11. Provide monthly spending analytics.

12. Allow users to create and monitor budgets.

13. Provide local data storage through Room.

14. Synchronize important information through Cloud Firestore.

15. Provide a fast and mobile-friendly Android experience.

16. Minimize manual expense entry.

---

The Solution

The application was designed as an intelligent expense-management system rather than a simple receipt scanner.

The overall architecture was:

                    USER
                     │
                     ▼
              ANDROID APPLICATION
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       Camera      Expenses   Analytics
          │          │          │
          ▼          ▼          ▼
       CameraX      Room       Charts
          │
          ▼
    Image Processing
          │
          ▼
   Python / OpenCV
          │
          ▼
 Tesseract OCR / ML Kit
          │
          ▼
 Receipt Data Extraction
          │
          ▼
   Firebase Functions
          │
          ▼
     Cloud Firestore

The architecture combines on-device Android functionality with cloud-based processing and persistent storage.

---

Smart Receipt Scanning

The central feature of the application is automated receipt scanning.

Instead of manually creating an expense, the user can:

Open App
   ↓
Tap Scan Receipt
   ↓
Point Camera
   ↓
Capture Receipt
   ↓
AI Processing
   ↓
Review Extracted Data
   ↓
Save

This significantly reduces the number of fields the user needs to enter manually.

---

Android CameraX Integration

Android CameraX was used to provide the receipt-scanning experience.

CameraX provides a modern Android camera framework suitable for:

• Camera preview

• Image capture

• Image analysis

• Lifecycle management

• Device compatibility

• Orientation handling

The application can guide the user to position the receipt correctly before taking the picture.

---

Receipt Capture Experience

A dedicated scanning screen can provide:

┌─────────────────────────────┐
│       Scan Receipt          │
│                             │
│   ┌─────────────────────┐   │
│   │                     │   │
│   │   PLACE RECEIPT     │   │
│   │      HERE           │   │
│   │                     │   │
│   └─────────────────────┘   │
│                             │
│       [ Capture ]           │
└─────────────────────────────┘

The objective is to make receipt capture straightforward for users who may scan receipts frequently.

---

Image Preprocessing with OpenCV

Raw receipt photographs are not always ideal for OCR.

Common problems include:

• Shadows

• Uneven lighting

• Perspective distortion

• Low contrast

• Small text

• Background noise

• Folded receipts

• Angled photographs

OpenCV was used as part of the image-processing pipeline to improve receipt images before OCR.

A simplified workflow is:

Captured Image
      ↓
Resize
      ↓
Crop
      ↓
Perspective Correction
      ↓
Noise Reduction
      ↓
Contrast Enhancement
      ↓
Thresholding
      ↓
OCR

---

Perspective Correction

Receipts are often photographed at an angle.

For example:

       /──────────/
      / Receipt  /
     /──────────/

Perspective transformation can help convert the photographed receipt into a more document-like image.

This can improve downstream text recognition.

---

Image Enhancement

The preprocessing pipeline can apply techniques such as:

• Grayscale conversion

• Noise reduction

• Thresholding

• Contrast adjustment

• Sharpening

• Cropping

• Perspective correction

The objective is not to make the image visually attractive but to create a cleaner input for machine-readable text extraction.

---

OCR Receipt Processing

After preprocessing, the receipt image is passed through an OCR pipeline.

Tesseract OCR and ML Kit can be used to recognize text from the receipt.

The general process is:

Receipt Image
      ↓
OpenCV Processing
      ↓
OCR Engine
      ↓
Raw Text
      ↓
Parser
      ↓
Structured Receipt

For example, raw OCR output might resemble:

ABC MARKET
DATE 12/08/2026
MILK       4.50
BREAD      2.30
TAX        0.68
TOTAL      7.48

The parsing layer converts this into structured data.

---

Automated Receipt Parsing

OCR alone only produces text.

The application therefore needs a parsing layer that identifies which pieces of text correspond to which receipt fields.

The parser can identify:

• Merchant

• Date

• Time

• Currency

• Subtotal

• Tax

• Discount

• Total

• Receipt number

The structured result might become:

Merchant: ABC Market
Date: 12/08/2026
Currency: USD
Subtotal: $6.80
Tax: $0.68
Total: $7.48

---

ML-Assisted Receipt Understanding

The receipt-processing system can use machine-learning assistance to improve the interpretation of OCR output.

For example, OCR might return:

TOTAI 7.48

instead of:

TOTAL 7.48

The parsing layer can use contextual information to understand that the value likely represents the receipt total.

This creates a pipeline that goes beyond basic OCR.

---

Tesseract OCR

Tesseract provides an open-source OCR engine capable of extracting text from receipt images.

It can be integrated into the Python processing layer and combined with preprocessing.

Its role is primarily:

Image → Text

while the parsing layer handles:

Text → Structured Expense

---

ML Kit

ML Kit can support on-device or mobile-friendly machine-learning capabilities within the Android ecosystem.

It can assist with text recognition and other machine-learning-related processing depending on the final implementation.

The combination provides flexibility between:

• Android-side recognition

• Python processing

• Cloud processing

---

Python Receipt Processing Engine

Python was used for receipt-processing and image-analysis functionality.

A conceptual backend pipeline is:

Receipt Upload
      ↓
Python Service
      ↓
OpenCV
      ↓
OCR
      ↓
Text Normalization
      ↓
Field Extraction
      ↓
Expense Object
      ↓
Firebase

Python provides a flexible environment for image processing, OCR pipelines, and data transformation.

---

Receipt Data Extraction

The application can extract structured fields such as:

The user can review these values before saving the expense.

---

Human Verification

Automated extraction should not be treated as infallible.

The application therefore provides a review step:

AI Extracted Data
       ↓
User Review
       ↓
Edit if Necessary
       ↓
Confirm
       ↓
Save

This prevents a small OCR mistake from automatically becoming an incorrect financial record.

---

Expense Categories

The system can automatically categorize expenses.

Example categories include:

• Groceries

• Dining

• Transportation

• Travel

• Utilities

• Office

• Software

• Healthcare

• Shopping

• Entertainment

• Education

• Other

The category system can be customized according to the user's requirements.

---

Automated Expense Categorization

The system can use merchant names, receipt text, and transaction information to suggest a category.

For example:

Merchant:
Uber

Suggested Category:
Transportation

Or:

Merchant:
Whole Foods

Suggested Category:
Groceries

The user can override the suggestion when necessary.

---

Multi-Currency Expense Tracking

A key feature of the application is support for multiple currencies.

A user may record:

USD
EUR
GBP
VND
JPY
AUD
CAD

The system can maintain the original transaction currency while also calculating an equivalent amount in the user's selected base currency.

For example:

Original:
€50 EUR

Base Currency:
USD

Converted:
≈ $XX.XX USD

The exact conversion rate should come from the configured exchange-rate source at transaction time.

---

Original Currency Preservation

It is important not to overwrite the original transaction value.

A robust expense record should preserve:

Original Amount
Original Currency
Exchange Rate
Converted Amount
Base Currency
Conversion Timestamp

For example:

Amount: 50
Currency: EUR

Rate:
EUR → USD

Converted:
USD equivalent

This allows users to understand how the converted value was calculated.

---

Currency Conversion Architecture

A typical workflow is:

Receipt
  ↓
Currency Detection
  ↓
Original Amount
  ↓
Exchange Rate Service
  ↓
Base Currency Conversion
  ↓
Expense Record

The system can cache exchange rates for a defined period while preserving the rate associated with the transaction.

---

Automated Tax Reporting Tags

Another major feature is automated tax classification.

Expenses can receive tags such as:

• Tax deductible

• Business expense

• Office expense

• Travel expense

• Meal expense

• VAT/GST-related

• Personal expense

• Review required

The exact tax treatment should be configurable because tax rules differ between jurisdictions.

The application is designed to organize tax-related information rather than independently determine legal tax liability.

---

Tax Data Extraction

Where receipts expose tax information, the system can attempt to identify:

Subtotal
Tax
Tax Rate
Total

For example:

Subtotal: 100.00
Tax: 10.00
Total: 110.00

These values can then be associated with appropriate reporting tags.

---

Expense Review

Before an expense becomes part of the permanent record, the user can review:

Merchant
Date
Amount
Currency
Category
Tax Tag
Notes

This creates a verification layer between AI extraction and financial reporting.

---

Monthly Budget Analytics

The application provides monthly spending analytics.

Users can see:

• Total spending

• Spending by category

• Budget used

• Remaining budget

• Largest expense categories

• Monthly comparison

• Currency-adjusted totals

A dashboard can present:

MONTHLY EXPENSES

Total Spent
$2,480

Budget
$3,000

Remaining
$520

Budget Used
82.6%

---

Category Analytics

The application can visualize where money is being spent.

Example:

Groceries       $650
Transport       $320
Dining          $280
Shopping        $450
Utilities       $380
Other           $400

This allows users to identify spending patterns.

---

Monthly Comparison

Historical analytics can compare different months.

For example:

June
$2,100

July
$2,480

August
$2,250

The application can calculate trends such as:

• Spending increase

• Spending decrease

• Category changes

• Budget performance

---

Budget Management

Users can define monthly budgets.

For example:

Monthly Budget
$3,000

Groceries
$600

Transport
$400

Entertainment
$200

The system can compare recorded expenses against those limits.

---

Budget Alerts

The application can provide alerts when spending approaches a configured threshold.

Example:

Groceries Budget

$560 / $600

93% Used

Warning:
You are approaching your monthly grocery budget.

This provides proactive financial awareness.

---

Expense Dashboard

The main dashboard can combine the most important financial information.

┌──────────────────────────────┐
│       EXPENSE TRACKER        │
├──────────────────────────────┤
│ Monthly Spending             │
│ $2,480                       │
│                              │
│ Budget Used       82.6%      │
│ ████████████████░░░          │
│                              │
│ Top Categories               │
│ Groceries       $650         │
│ Shopping        $450         │
│ Transport       $320         │
│                              │
│ [Scan Receipt]               │
└──────────────────────────────┘

---

Expense History

Users can browse previous expenses.

Each record can show:

• Merchant

• Amount

• Currency

• Date

• Category

• Tax tag

• Receipt image

Example:

ABC Market
$42.80
Groceries
12 Aug

City Restaurant
€36.00
Dining
11 Aug

---

Digital Receipt Storage

The original receipt image can be associated with the expense record.

This allows users to return to the source document when reviewing an expense.

A record can therefore contain:

Expense
 ├── Merchant
 ├── Date
 ├── Amount
 ├── Category
 ├── Tax Tag
 └── Receipt Image

---

Room Database

Room, built on SQLite, was used for local persistence.

It provides local storage for:

• Expense records

• Cached receipt data

• Categories

• Budget settings

• Recent analytics

• User preferences

This allows the application to remain responsive without requiring every screen to query the cloud.

---

Local-First Expense Experience

A local-first architecture can follow:

User Creates Expense
       ↓
Room Database
       ↓
UI Updates
       ↓
Cloud Synchronization
       ↓
Cloud Firestore

This provides a fast local experience while maintaining cloud synchronization.

---

Cloud Firestore

Cloud Firestore provides cloud-based storage and synchronization.

It can store:

• User profiles

• Expense records

• Receipt metadata

• Categories

• Budgets

• Tax tags

• Analytics data

• Application preferences

---

Firebase Functions

Firebase Functions can handle server-side operations such as:

• Receipt-processing triggers

• Expense-processing workflows

• Data validation

• Cloud synchronization

• Notification logic

• Backend automation

• External API integrations

A simplified architecture is:

Android App
     ↓
Firebase
     ↓
Cloud Function
     ↓
Python / Processing Service
     ↓
Structured Data
     ↓
Firestore

---

Backend Receipt Workflow

The complete receipt workflow can be represented as:

1. Capture Receipt
        ↓
2. Upload / Process Image
        ↓
3. OpenCV Enhancement
        ↓
4. OCR
        ↓
5. Text Normalization
        ↓
6. Field Extraction
        ↓
7. Currency Detection
        ↓
8. Expense Categorization
        ↓
9. Tax Tag Suggestion
        ↓
10. User Review
        ↓
11. Save Expense
        ↓
12. Update Analytics

---

Material Design 3

The Android interface was designed using Material Design 3 principles.

This provides a consistent Android user experience across:

• Buttons

• Forms

• Navigation

• Cards

• Dialogs

• Inputs

• Bottom sheets

• Lists

• Color and typography systems

The application was designed around clarity because financial information needs to be quickly understandable.

---

Android Application Navigation

A logical navigation structure is:

Home
│
├── Scan Receipt
├── Expenses
├── Analytics
├── Budgets
└── Profile

Users can access the core functions without navigating through unnecessary screens.

---

Receipt Scanner Screen

The scanner is one of the application's primary actions.

The application can place the scan action prominently on the home screen:

Scan Receipt

The goal is to make capturing a new expense a one-step action from the dashboard.

---

Expense Details Screen

After processing, users receive a structured expense form.

Expense Details

Merchant: ABC Market
Date: 12 Aug 2026
Amount: $46.20
Category: Groceries
Tax: $4.20
Tax Tag: Business Expense

[Save Expense]

Users can correct any field before saving.

---

Analytics Screen

The analytics section can provide:

• Monthly spending

• Category distribution

• Budget progress

• Spending trends

• Currency-adjusted totals

This transforms individual receipt records into useful financial insights.

---

Budget Screen

The budget section allows users to define:

• Monthly spending limits

• Category limits

• Budget periods

• Warning thresholds

The application can compare actual spending against these limits.

---

Search & Filtering

As the number of expenses grows, users need to find specific records quickly.

Filtering can include:

• Date

• Category

• Merchant

• Currency

• Tax tag

• Amount range

For example:

Category: Travel
Currency: USD
Date: August

---

Data Synchronization

The application synchronizes relevant information between Room and Firestore.

A conceptual synchronization process is:

Android
  │
  ├── Room
  │     ↓
  │  Local State
  │
  └──── Sync ────► Firestore
                       ↓
                  Cloud State

Conflict handling and synchronization rules can be implemented according to the application's requirements.

---

Security Architecture

Expense data can contain sensitive financial information, so security is an important part of the application architecture.

Security considerations include:

• HTTPS

• Firebase Authentication

• Firestore security rules

• Protected Firebase Functions

• Server-side validation

• Secure local storage

• Access control

• Input validation

• Restricted cloud permissions

Users should only be able to access their own expense records.

---

Receipt Image Privacy

Receipt images can contain sensitive information such as:

• Merchant details

• Payment information

• Addresses

• Transaction identifiers

The application should therefore implement appropriate access controls and retention policies for stored receipt images.

Where possible, only the data required for the application should be retained.

---

Performance Optimization

The application was optimized across multiple layers.

Android

• Efficient CameraX processing

• Lightweight UI rendering

• Room caching

• Background synchronization

• Lazy loading

Image Processing

• Image resizing

• Compression

• Efficient preprocessing

• Controlled OCR resolution

Backend

• Firebase Functions

• Efficient Firestore queries

• Cached data

• Reduced unnecessary processing

---

OCR Accuracy Strategy

Receipt OCR accuracy depends heavily on image quality.

The application therefore uses a combination of:

Camera guidance + OpenCV preprocessing + OCR + parsing + user verification.

This layered approach is more reliable than simply sending a raw camera image directly to an OCR engine.

---

Error Handling

Receipt processing can fail for several reasons:

• Blurry image

• Unsupported receipt format

• Poor lighting

• Missing total

• OCR error

• Network failure

The application can handle these situations through:

Processing Failed
       ↓
Explain Problem
       ↓
Retry Scan
       ↓
Manual Entry

A manual fallback is important because users should never be blocked from recording an expense because automated recognition failed.

---

Development Process

Phase 1 — Requirements & Expense Workflow

We mapped the complete expense lifecycle:

Capture → Process → Review → Categorize → Save → Analyze.

---

Phase 2 — Android Application Architecture

The Android application was structured around Kotlin, CameraX, Material Design 3, and Room.

---

Phase 3 — Camera & Receipt Capture

CameraX was integrated to provide the document-scanning experience.

---

Phase 4 — Image Processing

OpenCV was integrated into the processing pipeline to enhance receipt images before OCR.

---

Phase 5 — OCR

Tesseract OCR and ML Kit were incorporated to recognize receipt text.

---

Phase 6 — Intelligent Parsing

The raw OCR output was converted into structured expense fields.

---

Phase 7 — Firebase Backend

Firebase Functions and Firestore were integrated for cloud processing, synchronization, and data storage.

---

Phase 8 — Multi-Currency Support

Currency identification and conversion logic were incorporated while preserving the original transaction currency.

---

Phase 9 — Tax Classification

Automated tax-reporting tags were added to help users organize expenses for financial reporting workflows.

---

Phase 10 — Budget Analytics

Monthly expense calculations, category analytics, and budget tracking were implemented.

---

Phase 11 — Local Persistence

Room was integrated to provide fast local storage and an offline-friendly foundation.

---

Phase 12 — Security & Optimization

The application was optimized for performance and protected with appropriate authentication and database access controls.

---

Phase 13 — Testing

The application was tested across:

• Receipt capture

• OCR

• Image processing

• Expense parsing

• Currency conversion

• Tax tags

• Budget calculations

• Analytics

• Room database

• Firestore synchronization

• Network failures

• Authentication

• Android UI

---

Key Technical Challenges

Challenge 1 — Poor Receipt Image Quality

Receipts are frequently photographed in difficult lighting or at an angle.

OpenCV preprocessing was therefore used to improve the input before OCR.

Challenge 2 — OCR Is Not Structured Data

OCR produces text, not financial records.

A parsing layer was required to transform:

Raw text → Merchant + Date + Amount + Tax + Currency + Category.

Challenge 3 — Multiple Currencies

International expenses require preservation of the original transaction currency while also providing a normalized base-currency value.

Challenge 4 — Tax Classification

Tax-related information differs by jurisdiction, so the application was designed around configurable reporting tags rather than treating automated tagging as professional tax advice.

Challenge 5 — Local and Cloud Data

Expense applications need fast access to records while also maintaining cloud synchronization.

Room and Firestore were therefore used together.

---

Business Impact

The Automated Smart Receipt & Expense Tracker provides the Vietnam-based client with a digital alternative to manual receipt and expense management.

The application can help users:

• Capture receipts faster

• Reduce manual data entry

• Digitize paper expenses

• Organize transactions automatically

• Track multiple currencies

• Categorize spending

• Apply tax-reporting tags

• Monitor monthly budgets

• Analyze spending patterns

• Maintain searchable digital expense records

The core transformation was:

Paper receipts → structured digital expenses → automated categorization → financial analytics.

---

Why Kotlin Was Used

Kotlin was selected for native Android development because it provides a modern Android development environment with strong support for application architecture and Android libraries.

It is particularly suitable for an application that combines:

• Camera functionality

• Local databases

• Cloud APIs

• Background processing

• Material Design

• Mobile-first financial workflows

---

Why CameraX Was Used

CameraX provides a modern camera framework for Android.

For this application it supports:

• Receipt preview

• Image capture

• Image analysis

• Camera lifecycle

• Device compatibility

It creates the foundation for the automated receipt-scanning experience.

---

Why OpenCV Was Used

OpenCV provides powerful image-processing capabilities.

For receipt scanning, it can help with:

• Perspective correction

• Noise reduction

• Grayscale conversion

• Thresholding

• Contrast enhancement

• Image normalization

Better image input generally improves the downstream OCR process.

---

Why Tesseract & ML Kit Were Used

OCR is the foundation of automated receipt extraction.

Tesseract provides a flexible OCR engine while ML Kit offers Android-oriented machine-learning and text-recognition capabilities.

Using these technologies allows the system to support different processing strategies depending on the application's requirements.

---

Why Room & Firestore Were Used Together

Room provides fast local storage while Firestore provides cloud synchronization.

This creates a hybrid architecture:

Room = local performance

Firestore = cloud persistence and synchronization

This is particularly useful for expense applications where users expect their financial records to remain accessible and responsive.

---

Future Expansion

The platform can later be expanded with:

• AI expense assistant

• Automatic recurring expense detection

• Bank transaction integration

• Credit card integration

• Invoice scanning

• PDF invoice processing

• Expense report generation

• Business expense approval workflows

• Team expense management

• Employee reimbursement

• Receipt fraud detection

• Duplicate receipt detection

• Advanced financial forecasting

• AI spending recommendations

• Voice-based expense entry

• Automated accounting integrations

• Export to CSV / Excel / PDF

• Business dashboards

• Multi-user company accounts

A future AI financial assistant could work like:

User:
"How much did I spend on restaurants this month?"

        ↓

AI Expense Assistant

        ↓

Analyze Expenses

        ↓

Restaurant Category

        ↓

Monthly Total

        ↓

Response
"You spent X in restaurants this month."

---

Technology Stack

Android Frontend

• Kotlin

• Android CameraX

• Material Design 3

• Android SDK

AI & Document Processing

• Python

• OpenCV

• Tesseract OCR

• ML Kit

• Automated Receipt Parsing

• Machine-Learning-Assisted Classification

Backend

• Firebase Functions

• Cloud APIs

• Server-side Processing

Database

• Cloud Firestore

• SQLite

• Room Database

Core Features

• Smart Receipt Scanner

• OCR Receipt Processing

• Automated Expense Extraction

• Expense Categorization

• Multi-Currency Tracking

• Currency Conversion

• Tax Reporting Tags

• Monthly Budget Analytics

• Expense History

• Digital Receipt Storage

• Budget Management

• Financial Analytics

• Local Data Persistence

• Cloud Synchronization